Graceful restart (RFC 4724, RFC 8538)

A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/content/docs/getting-started/introduction.md

@@ -65,8 +65,8 @@ Ferrix 0.1.0 does not support the following features:
   ADD-PATH for advertised routes.
 - TCP Authentication Option (TCP-AO), TCP MD5 signatures, and the
   Generalized TTL Security Mechanism (GTSM).
-- Graceful restart, Bidirectional Forwarding Detection (BFD), and
-  configuration changes without a restart.
+- Long-Lived Graceful Restart, Bidirectional Forwarding Detection (BFD),
+  and configuration changes without a restart.
 - Recursive resolution of next hops that are not directly connected.
 
 For the implementation status of each RFC, see

docs/content/docs/guides/examples.md

@@ -1,7 +1,7 @@
 +++
 title = "Configuration Examples"
 description = "Provides complete configurations for a single-homed edge network, a network with an Internet exchange connection, and a transit network."
-weight = 7
+weight = 8
 +++
 
 This topic provides complete configurations for three common deployments.

docs/content/docs/guides/graceful-restart.md

@@ -0,0 +1,168 @@
++++
+title = "Graceful Restart"
+description = "Describes BGP graceful restart (RFC 4724, RFC 8538), which retains routes while a BGP speaker restarts, and how to restart Ferrix without withdrawing its routes."
+weight = 7
++++
+
+Graceful restart is a BGP mechanism, specified in RFC 4724, that retains
+the routes of a BGP speaker while the speaker restarts. The neighbors of
+the restarting speaker continue to forward traffic by its routes, and do
+not withdraw them from their own neighbors. Ferrix supports graceful
+restart in both roles: as the restarting speaker, for example during a
+software upgrade, and as the neighbor of a restarting speaker.
+
+## Understanding Graceful Restart
+
+Graceful restart is in use on a BGP session if both neighbors advertise the
+Graceful Restart capability in their OPEN messages. The capability
+contains the following information:
+
+- The restart time: the time within which the speaker expects to
+  reestablish the session after a restart.
+- The address families for which the speaker retains routes.
+- For each address family, whether the speaker preserved its forwarding
+  state during its last restart (the F bit).
+- Whether the speaker has just restarted (the R bit).
+- Whether the speaker applies graceful restart to sessions that end with a
+  NOTIFICATION message (the N bit, RFC 8538).
+
+When a session with a restarting neighbor ends, the router marks the
+routes of the neighbor as stale. Stale routes remain in the routing table
+and are selected and advertised in the same way as other routes. When the
+neighbor reestablishes the session, it advertises its routes again, and
+then sends an End-of-RIB marker for each address family. When the router
+receives the End-of-RIB marker, it removes the stale routes of the address
+family that the neighbor did not advertise again.
+
+RFC 8538 extends graceful restart to sessions that end with a NOTIFICATION
+message. A Cease NOTIFICATION message with the subcode Hard Reset ends the
+session without graceful restart: the receiving router removes the routes
+immediately.
+
+## Restarting Ferrix
+
+Ferrix leaves for a restart when it receives the SIGUSR1 signal. Ferrix
+then takes the following actions:
+
+1. It stops updating the kernel routing table, and leaves its routes in
+   the kernel. The router continues to forward traffic by these routes.
+2. It closes the TCP connection of each session on which graceful restart
+   is in use, without a NOTIFICATION message. The neighbor retains the
+   routes of Ferrix for the restart time.
+3. It sends a Cease NOTIFICATION message with the subcode Administrative
+   Shutdown to each other neighbor.
+4. It exits with status 75.
+
+The systemd unit restarts the daemon when it exits with status 75. See
+[Upgrading Ferrix](@/docs/operations/deployment.md#upgrading-ferrix).
+
+When the daemon starts again, it takes the following actions:
+
+1. It finds the routes of the previous run in the kernel and retains them.
+   See [Kernel Route Installation](@/docs/guides/kernel.md#startup-and-shutdown-behavior).
+2. It advertises the Graceful Restart capability with the R bit set, for
+   the restart time after startup. It sets the F bit if it found routes of
+   the previous run in the kernel, or if kernel route installation is not
+   configured.
+3. It advertises its originated routes, and sends an End-of-RIB marker for
+   each address family to each neighbor on which graceful restart is in use.
+   To a neighbor that receives learned routes (`learned_routes`), Ferrix
+   sends the End-of-RIB markers after every configured neighbor has sent
+   its own End-of-RIB markers, or after the restart time.
+4. When every configured neighbor has sent its End-of-RIB markers, or after
+   `stale_seconds`, it removes the routes of the previous run that no
+   neighbor advertised again.
+
+When the daemon stops in response to SIGTERM, it ends each session for
+good. On a session on which the N bit is in use, it sends a Cease
+NOTIFICATION message with the subcode Hard Reset.
+
+## When a Neighbor Restarts
+
+Ferrix retains the routes of a neighbor whose session ends in one of the
+following ways, if graceful restart is in use on the session:
+
+| Session ends because | Routes retained |
+|---|---|
+| The TCP connection fails or is closed | Yes |
+| The hold timer expires | If the N bit is in use |
+| The neighbor sends a NOTIFICATION message other than Hard Reset | If the N bit is in use |
+| The neighbor sends a Cease NOTIFICATION message with the subcode Hard Reset | No |
+| Ferrix detects an error, such as a malformed message or an exceeded prefix limit | No |
+
+Ferrix retains the routes of the address families that the neighbor listed
+in its Graceful Restart capability. It removes the routes of other address
+families immediately.
+
+Ferrix retains the routes for the restart time of the neighbor, or for
+`stale_routes_seconds`, whichever is shorter. If the neighbor does not
+reestablish the session within that time, Ferrix removes the routes.
+
+When the neighbor reestablishes the session, Ferrix takes the following
+actions:
+
+- If the neighbor no longer advertises graceful restart, or does not set
+  the F bit for an address family, Ferrix removes the stale routes of the
+  address family immediately.
+- Otherwise, Ferrix removes the stale routes of an address family when the
+  neighbor sends its End-of-RIB marker for the address family, or after
+  `stale_routes_seconds`.
+
+## Configuring Graceful Restart
+
+By default, graceful restart is enabled for every neighbor, with a restart
+time of 120 seconds. The following example sets a shorter restart time for
+all neighbors, and disables graceful restart for one neighbor:
+
+```textproto
+graceful_restart {
+  restart_time_seconds: 60
+  stale_routes_seconds: 60
+}
+
+peers {
+  name: "transit-a"
+  endpoint { address: "192.0.2.1" }
+  remote_asn: 64496
+  graceful_restart { disabled: true }
+}
+```
+
+The restart time determines for how long neighbors forward traffic to the
+router after its session with them ends. Choose a restart time that covers
+a restart of the daemon, including the time for systemd to start it again.
+
+For the statements, see
+[GracefulRestart](@/docs/reference/configuration.md#gracefulrestart).
+
+## Verifying Graceful Restart
+
+The status page and the HTTP API display the following information:
+
+- For each neighbor on which graceful restart is in use, the restart time
+  of the neighbor and whether the N bit is in use. The `/api/peers`
+  endpoint returns this information in the `graceful_restart` field.
+- For a neighbor that is restarting, the number of retained routes and the
+  time at which they are removed (`stale_routes` and `stale_until_unix`).
+- For a neighbor that has reestablished its session, the number of routes
+  that are still stale (`stale_routes`).
+- For each received route, whether it is stale (`stale` in
+  `/api/routes/received`).
+
+Ferrix logs the following events at the `info` level: a session that ends
+with retained routes, a neighbor that reestablishes its session, and the
+removal of stale routes.
+
+## Limitations
+
+- Ferrix does not support Long-Lived Graceful Restart (RFC 9494).
+- Ferrix retains the routes of a neighbor whose session fails for any
+  reason, including a failure of the neighbor or of the link, for the
+  restart time of the neighbor. During that time, traffic is forwarded to
+  the neighbor. Ferrix does not support Bidirectional Forwarding Detection
+  (BFD), which would detect such failures. To remove the routes of a
+  neighbor immediately when its session fails, disable graceful restart
+  for the neighbor.
+- Ferrix does not preserve its forwarding state when the router reboots,
+  or when the daemon is stopped with SIGTERM. In both cases the routes are
+  removed from the kernel.

docs/content/docs/guides/kernel.md

@@ -113,10 +113,15 @@ Ferrix handles its kernel routes as follows:
 
 - **At startup**, Ferrix retains the routes installed by a previous run, so
   that the router continues to forward traffic while the neighbors
-  readvertise their routes. After `stale_seconds`, Ferrix removes the routes
-  that no neighbor has readvertised.
+  readvertise their routes. When every configured neighbor has sent an
+  End-of-RIB marker for each address family, or after `stale_seconds`,
+  whichever is first, Ferrix removes the routes that no neighbor has
+  readvertised.
 - **On SIGINT or SIGTERM**, Ferrix closes its BGP sessions and removes its
   routes from the kernel.
+- **On SIGUSR1**, Ferrix leaves its routes in the kernel and exits, to be
+  started again (graceful restart). See
+  [Graceful Restart](@/docs/guides/graceful-restart.md).
 - **After a crash**, the routes remain in the kernel until the next start.
 
 ## Interface Changes

docs/content/docs/operations/deployment.md

@@ -1,6 +1,6 @@
 +++
 title = "Deploying Ferrix"
-description = "Describes how to run Ferrix as a systemd service, secure the hosts and interfaces around it, change its configuration, and restart it."
+description = "Describes how to run Ferrix as a systemd service, secure the hosts and interfaces around it, change its configuration, and restart or upgrade it without withdrawing its routes."
 weight = 1
 
 [extra]
@@ -48,6 +48,10 @@ The `ExecStartPre` command validates the configuration before the daemon
 starts. If the configuration is invalid, the service does not start, and
 the journal contains the error.
 
+When the daemon exits with status 75, after a graceful restart, systemd
+starts it again (`RestartForceExitStatus=75`). See
+[Upgrading Ferrix](#upgrading-ferrix).
+
 The sandboxing options in the unit have been tested with kernel route
 installation enabled. The command `systemd-analyze security ferrix` rates
 the unit's exposure level as 2.1 (OK).
@@ -134,31 +138,82 @@ configuration:
    sudo mv /etc/ferrix/server.textpb.new /etc/ferrix/server.textpb
    ```
 
-3. Restart the service.
+3. Restart the daemon with a graceful restart.
 
    ```bash
-   sudo systemctl restart ferrix
+   sudo systemctl kill --signal=SIGUSR1 ferrix
    ```
 
+## Upgrading Ferrix
+
+A graceful restart (RFC 4724) restarts the daemon without withdrawing its
+routes. The router continues to forward traffic during the restart, and
+neighbors on which graceful restart is in use retain the routes of Ferrix.
+See [Graceful Restart](@/docs/guides/graceful-restart.md).
+
+To upgrade Ferrix:
+
+1. Install the new executable.
+
+   ```bash
+   sudo install -m 0755 bgp /usr/local/bin/bgp.new
+   sudo mv /usr/local/bin/bgp.new /usr/local/bin/bgp
+   ```
+
+   The `mv` command replaces the executable in one step, while the running
+   daemon continues to use the previous one.
+
+2. Validate the configuration with the new executable.
+
+   ```bash
+   sudo bgp --check /etc/ferrix/server.textpb
+   ```
+
+3. Send the SIGUSR1 signal to the daemon.
+
+   ```bash
+   sudo systemctl kill --signal=SIGUSR1 ferrix
+   ```
+
+   The daemon leaves its routes in the kernel, closes its sessions, and
+   exits with status 75. systemd starts the new executable after
+   `RestartSec` (5 seconds).
+
+4. Verify that the sessions are reestablished.
+
+   ```bash
+   journalctl -u ferrix -f
+   ```
+
+   On the status page, the neighbor table shows the restarted sessions,
+   and the kernel section shows the routes kept from the previous run
+   until every neighbor has sent its routes.
+
+The restart must complete within the restart time that Ferrix advertises
+(`restart_time_seconds`, default: 120 seconds). Otherwise, the neighbors
+remove the routes of Ferrix.
+
 ## Restart Behavior
 
-When the daemon stops in response to SIGTERM, it sends a Cease
-NOTIFICATION message with the subcode Administrative Shutdown to every
-neighbor, and removes its routes from the kernel. The neighbors withdraw
-the routes that they received from Ferrix. Traffic uses alternative paths
-until the sessions are reestablished and the routing tables are exchanged
-again. For a full routing table, this takes several seconds.
+Table 2 lists how the daemon stops, and the effect on its neighbors and on
+forwarding.
 
-If the daemon terminates unexpectedly, its routes remain in the kernel, and
-the router continues to forward traffic. At the next start, Ferrix retains
-these routes for `stale_seconds` (default: 120 seconds) while the neighbors
-readvertise their routes, and then removes the routes that were not
-readvertised.
+| Event | Neighbors | Kernel routes |
+|---|---|---|
+| SIGUSR1 | Neighbors on which graceful restart is in use retain the routes of Ferrix for the restart time. Other neighbors receive a Cease NOTIFICATION message with the subcode Administrative Shutdown. | Retained |
+| SIGTERM (`systemctl stop`, `systemctl restart`) | Every neighbor receives a Cease NOTIFICATION message, with the subcode Hard Reset where the neighbor supports RFC 8538, and withdraws the routes of Ferrix. | Removed |
+| Unexpected termination | Neighbors on which graceful restart is in use retain the routes of Ferrix for the restart time. | Retained |
+
+At startup, Ferrix retains the routes that a previous run left in the
+kernel until every neighbor has sent its routes, or for `stale_seconds`
+(default: 120 seconds), and then removes the routes that no neighbor
+advertised again.
 
 > [!WARNING]
-> A restart interrupts forwarding through the router for the time that the
-> sessions take to converge. Restart Ferrix in a maintenance window, or
-> after traffic has been drained to another router.
+> A stop with SIGTERM interrupts forwarding through the router until the
+> sessions are reestablished and the routing tables are exchanged again.
+> For a full routing table, this takes several seconds. To restart the
+> daemon, use SIGUSR1.
 
 ## Running Ferrix in a Container
 

docs/content/docs/reference/configuration.md

@@ -55,6 +55,7 @@ Top-level statements.
 | `kernel` <span class="type">[Kernel](#kernel)</span> | Disabled | Enables kernel route installation (Linux only). |
 | `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Originated routes only | Specifies the export policy for neighbors that have no `export` statement. |
 | `control_communities` <span class="type">bool</span> | `false` | Enables control communities, which restrict the neighbors to which a route is advertised. See [Control Communities](@/docs/guides/passing-routes-on.md#control-communities). |
+| `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Enabled | Specifies graceful restart for neighbors that have no `graceful_restart` statement. Its `restart_time_seconds` also limits how long Ferrix waits at startup for neighbors to send their routes. |
 
 ## Peer
 
@@ -73,6 +74,7 @@ Statements for a BGP neighbor.
 | `role` <span class="type">[Role](#role)</span> | None | Specifies the BGP role of the neighbor (RFC 9234). Not permitted for IBGP neighbors. |
 | `require_role` <span class="type">bool</span> | `false` | Rejects the session if the neighbor does not advertise a BGP role. Requires `role`. |
 | `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Top-level `export` | Specifies the export policy for the neighbor. Replaces the top-level `export` statement. |
+| `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Top-level `graceful_restart` | Specifies graceful restart for the neighbor. Each statement that is set replaces the corresponding top-level statement. |
 
 ## TcpEndpoint
 
@@ -167,6 +169,17 @@ Statements for kernel route installation. See
 | `without_nexthop_objects` <span class="type">bool</span> | `false` | Installs each route with its own gateway instead of a nexthop object. |
 | `stale_seconds` <span class="type">uint32, optional</span> | 120 | Specifies the time, in seconds, for which routes installed by a previous run are retained at startup. A value of 0 removes them immediately. |
 
+## GracefulRestart
+
+Statements for graceful restart (RFC 4724, RFC 8538). See
+[Graceful Restart](@/docs/guides/graceful-restart.md).
+
+| Statement | Default | Description |
+|---|---|---|
+| `disabled` <span class="type">bool, optional</span> | `false` | Disables graceful restart. Ferrix does not advertise the capability, and removes the routes of a neighbor as soon as its session ends. |
+| `restart_time_seconds` <span class="type">uint32, optional</span> | 120 | Specifies the restart time advertised to neighbors, in seconds: the time for which a neighbor retains the routes of Ferrix after the session ends. Range: 0 through 4095. |
+| `stale_routes_seconds` <span class="type">uint32, optional</span> | 120 | Specifies the maximum time, in seconds, for which Ferrix retains the routes of a restarting neighbor: while the neighbor is down, regardless of the restart time that it advertises, and again after it reconnects, until it has sent all its routes. A value of 0 retains no routes. |
+
 ## Role
 
 | Value | Description |

docs/content/docs/reference/http-api.md

@@ -58,7 +58,15 @@ neighbor:
   "role": null,
   "role_confirmed": null,
   "last_error": null,
-  "retry_at_unix": null
+  "retry_at_unix": null,
+  "graceful_restart": {
+    "restart_time_secs": 120,
+    "notification": true,
+    "address_families": ["IPv4 unicast", "IPv6 unicast"],
+    "forwarding_preserved": ["IPv4 unicast", "IPv6 unicast"]
+  },
+  "stale_routes": null,
+  "stale_until_unix": null
 }
 ```
 
@@ -71,6 +79,9 @@ neighbor:
 | `held_routes` | Number of held routes: routes rejected for a reason that can change without a new UPDATE, such as RPKI validity or next hop resolution. |
 | `role_confirmed` | Whether the neighbor advertised a BGP role that matches the configured role. |
 | `retry_at_unix` | For a neighbor without a session, the time of the next connection attempt. |
+| `graceful_restart` | Graceful restart, if both neighbors advertised it: the restart time of the neighbor, whether the N bit is in use (RFC 8538), the address families whose routes are retained, and those for which the neighbor preserved its forwarding state. |
+| `stale_routes` | Number of the neighbor's routes retained from its previous session and not yet advertised again, or `null` if there are none. |
+| `stale_until_unix` | For a restarting neighbor without a session, the time at which its retained routes are removed. |
 
 ## Received Routes
 
@@ -101,6 +112,7 @@ Prefixes are sorted by address, with IPv4 prefixes first.
     "prefix": "203.0.113.0/24",
     "path_id": 0,
     "best": true,
+    "stale": false,
     "rpki": null,
     "aspa": null,
     "summary": {
@@ -123,6 +135,7 @@ Prefixes are sorted by address, with IPv4 prefixes first.
 |---|---|
 | `path_id` | ADD-PATH path identifier, or 0. |
 | `best` | Whether the route is the best path for the prefix. |
+| `stale` | Whether the route is retained from a previous session of the neighbor and has not been advertised again (graceful restart). |
 | `rpki`, `aspa` | Route origin validation and ASPA verification state, if RPKI validation is configured. |
 | `attributes` | All path attributes, as received. |
 

docs/content/docs/reference/rfcs.md

@@ -101,8 +101,20 @@ The base protocol specification.
 
 | Feature | Status |
 |---------|--------|
-| `GracefulRestart` capability wire format | 🟡 |
-| Restart state and forwarding preserved across sessions | ❌ |
+| `GracefulRestart` capability (parse, serialise) | ✅ |
+| Receiving speaker: routes of a restarting neighbor retained as stale, and removed at its End-of-RIB or after the stale timer | ✅ |
+| Restarting speaker: forwarding state preserved in the kernel over a restart (SIGUSR1); R and F bits | ✅ |
+| End-of-RIB marker sent and received | ✅ |
+| Route selection deferred until neighbors' End-of-RIB (for the removal of kept kernel routes and End-of-RIB markers for learned routes) | ✅ |
+
+### RFC 8538 — Notification Message Support for BGP Graceful Restart
+<https://datatracker.ietf.org/doc/html/rfc8538>
+
+| Feature | Status |
+|---|---|
+| N bit advertised and negotiated | ✅ |
+| Routes retained after a NOTIFICATION other than Hard Reset, and after hold timer expiry | ✅ |
+| Hard Reset sent for errors and for a stop, with the original NOTIFICATION as its data | ✅ |
 
 ### RFC 9494 — Long-Lived Graceful Restart for BGP
 <https://datatracker.ietf.org/doc/html/rfc9494>

docs/examples/ferrix.service

@@ -19,6 +19,10 @@ ExecStartPre=/usr/local/bin/bgp --check /etc/ferrix/server.textpb
 ExecStart=/usr/local/bin/bgp /etc/ferrix/server.textpb --level info
 Restart=on-failure
 RestartSec=5s
+# On SIGUSR1 the daemon leaves its routes in place for a graceful restart
+# and exits with status 75: start it again at once.
+SuccessExitStatus=75
+RestartForceExitStatus=75
 # Time to notify all neighbors and remove the kernel routes.
 TimeoutStopSec=30s