ferrix/docs/content/docs/guides/graceful-restart.md

+++ title = "Graceful Restart" description = "Describes BGP graceful restart (RFC 4724, RFC 8538), which retains routes while a BGP speaker restarts, and how to restart Ferrix without withdrawing its routes." weight = 7 +++

Graceful restart is a BGP mechanism, specified in RFC 4724, that retains the routes of a BGP speaker while the speaker restarts. The neighbors of the restarting speaker continue to forward traffic by its routes, and do not withdraw them from their own neighbors. Ferrix supports graceful restart in both roles: as the restarting speaker, for example during a software upgrade, and as the neighbor of a restarting speaker.

Understanding Graceful Restart

Graceful restart is in use on a BGP session if both neighbors advertise the Graceful Restart capability in their OPEN messages. The capability contains the following information:

• The restart time: the time within which the speaker expects to reestablish the session after a restart.
• The address families for which the speaker retains routes.
• For each address family, whether the speaker preserved its forwarding state during its last restart (the F bit).
• Whether the speaker has just restarted (the R bit).
• Whether the speaker applies graceful restart to sessions that end with a NOTIFICATION message (the N bit, RFC 8538).

When a session with a restarting neighbor ends, the router marks the routes of the neighbor as stale. Stale routes remain in the routing table and are selected and advertised in the same way as other routes. When the neighbor reestablishes the session, it advertises its routes again, and then sends an End-of-RIB marker for each address family. When the router receives the End-of-RIB marker, it removes the stale routes of the address family that the neighbor did not advertise again.

RFC 8538 extends graceful restart to sessions that end with a NOTIFICATION message. A Cease NOTIFICATION message with the subcode Hard Reset ends the session without graceful restart: the receiving router removes the routes immediately.

Restarting Ferrix

Ferrix leaves for a restart when it receives the SIGUSR1 signal. Ferrix then takes the following actions:

1. It stops updating the kernel routing table, and leaves its routes in the kernel. The router continues to forward traffic by these routes.
2. It closes the TCP connection of each session on which graceful restart is in use, without a NOTIFICATION message. The neighbor retains the routes of Ferrix for the restart time.
3. It sends a Cease NOTIFICATION message with the subcode Administrative Shutdown to each other neighbor.
4. It exits with status 75.

The systemd unit restarts the daemon when it exits with status 75. See Upgrading Ferrix.

When the daemon starts again, it takes the following actions:

1. It finds the routes of the previous run in the kernel and retains them. See Kernel Route Installation.
2. It advertises the Graceful Restart capability with the R bit set, for the restart time after startup. It sets the F bit if it found routes of the previous run in the kernel, or if kernel route installation is not configured.
3. It advertises its originated routes, and sends an End-of-RIB marker for each address family to each neighbor on which graceful restart is in use. To a neighbor that receives learned routes (learned_routes), Ferrix sends the End-of-RIB markers after every configured neighbor has sent its own End-of-RIB markers, or after the restart time.
4. When every configured neighbor has sent its End-of-RIB markers, or after stale_seconds, it removes the routes of the previous run that no neighbor advertised again.

When the daemon stops in response to SIGTERM, it ends each session for good. On a session on which the N bit is in use, it sends a Cease NOTIFICATION message with the subcode Hard Reset.

When a Neighbor Restarts

Ferrix retains the routes of a neighbor whose session ends in one of the following ways, if graceful restart is in use on the session:

Session ends because	Routes retained
The TCP connection fails or is closed	Yes
The hold timer expires	If the N bit is in use
The neighbor sends a NOTIFICATION message other than Hard Reset	If the N bit is in use
The neighbor sends a Cease NOTIFICATION message with the subcode Hard Reset	No
Ferrix detects an error, such as a malformed message or an exceeded prefix limit	No

Ferrix retains the routes of the address families that the neighbor listed in its Graceful Restart capability. It removes the routes of other address families immediately.

Ferrix retains the routes for the restart time of the neighbor, or for stale_routes_seconds, whichever is shorter. If the neighbor does not reestablish the session within that time, Ferrix removes the routes.

When the neighbor reestablishes the session, Ferrix takes the following actions:

• If the neighbor no longer advertises graceful restart, or does not set the F bit for an address family, Ferrix removes the stale routes of the address family immediately.
• Otherwise, Ferrix removes the stale routes of an address family when the neighbor sends its End-of-RIB marker for the address family, or after stale_routes_seconds.

Configuring Graceful Restart

By default, graceful restart is enabled for every neighbor, with a restart time of 120 seconds. The following example sets a shorter restart time for all neighbors, and disables graceful restart for one neighbor:

graceful_restart {
  restart_time_seconds: 60
  stale_routes_seconds: 60
}

peers {
  name: "transit-a"
  endpoint { address: "192.0.2.1" }
  remote_asn: 64496
  graceful_restart { disabled: true }
}

The restart time determines for how long neighbors forward traffic to the router after its session with them ends. Choose a restart time that covers a restart of the daemon, including the time for systemd to start it again.

For the statements, see GracefulRestart.

Verifying Graceful Restart

The status page and the HTTP API display the following information:

• For each neighbor on which graceful restart is in use, the restart time of the neighbor and whether the N bit is in use. The /api/peers endpoint returns this information in the graceful_restart field.
• For a neighbor that is restarting, the number of retained routes and the time at which they are removed (stale_routes and stale_until_unix).
• For a neighbor that has reestablished its session, the number of routes that are still stale (stale_routes).
• For each received route, whether it is stale (stale in /api/routes/received).

Ferrix logs the following events at the info level: a session that ends with retained routes, a neighbor that reestablishes its session, and the removal of stale routes.

Limitations

• Ferrix does not support Long-Lived Graceful Restart (RFC 9494).
• Ferrix retains the routes of a neighbor whose session fails for any reason, including a failure of the neighbor or of the link, for the restart time of the neighbor. During that time, traffic is forwarded to the neighbor. Ferrix does not support Bidirectional Forwarding Detection (BFD), which would detect such failures. To remove the routes of a neighbor immediately when its session fails, disable graceful restart for the neighbor.
• Ferrix does not preserve its forwarding state when the router reboots, or when the daemon is stopped with SIGTERM. In both cases the routes are removed from the kernel.