Graceful restart (RFC 4724, RFC 8538)

A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

bin/bgp/src/main.rs

@@ -1,9 +1,10 @@
 use std::path::PathBuf;
+use std::process::ExitCode;
 use std::sync::Arc;
 
 use bgp_server::config::Config;
 use bgp_server::proto::config::BgpConfig;
-use bgp_server::server::{BgpServer, shut_down};
+use bgp_server::server::{BgpServer, leave_for_restart, shut_down};
 use bgp_web::{AppState, RouterInfo};
 use clap::Parser;
 use eyre::{Context, Result};
@@ -20,6 +21,11 @@ mod logging;
 #[global_allocator]
 static ALLOCATOR: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
 
+/// The exit status after leaving to be started again (SIGUSR1):
+/// EX_TEMPFAIL, on which the systemd unit starts the daemon again
+/// (`RestartForceExitStatus=75`).
+const RESTART_EXIT_STATUS: u8 = 75;
+
 /// A BGP speaker.
 #[derive(Parser, Debug)]
 #[command(version)]
@@ -37,7 +43,7 @@ struct ServerArgs {
 }
 
 #[tokio::main]
-async fn main() -> Result<()> {
+async fn main() -> Result<ExitCode> {
     let args = ServerArgs::parse();
     args.logging.install()?;
     log_panics();
@@ -55,7 +61,7 @@ async fn main() -> Result<()> {
             config.peers.len(),
             config.originate.len()
         );
-        return Ok(());
+        return Ok(ExitCode::SUCCESS);
     }
 
     let server = BgpServer::bind(config).await?;
@@ -81,6 +87,7 @@ async fn main() -> Result<()> {
             refusals: server.refusals.clone(),
             rpki: server.rpki.clone(),
             kernel: server.kernel.clone(),
+            restarts: Arc::clone(&server.restarts),
         };
         tokio::spawn(async move {
             if let Err(e) = bgp_web::serve_on(state, listener).await {
@@ -94,13 +101,28 @@ async fn main() -> Result<()> {
     // makes them slower to take us back.
     // And take our routes out of the kernel: with us gone nothing would
     // keep them true.
+    // Unless we are to be back at once (SIGUSR1, for an upgrade): then the
+    // routes stay, and peers that agreed to graceful restart keep ours.
     let (registry, kernel) = (server.registry.clone(), server.kernel.clone());
     tokio::select! {
-        result = server.run() => Ok(result?),
-        signal = shutdown_signal() => {
-            info!(signal, "Shutting down, closing BGP sessions");
-            shut_down(&registry, kernel.as_ref()).await;
-            Ok(())
+        result = server.run() => {
+            result?;
+            Ok(ExitCode::SUCCESS)
+        }
+        signal = signalled() => match signal {
+            Signal::Stop(name) => {
+                info!(signal = name, "Shutting down, closing BGP sessions");
+                shut_down(&registry, kernel.as_ref()).await;
+                Ok(ExitCode::SUCCESS)
+            }
+            Signal::Restart => {
+                info!(
+                    signal = "SIGUSR1",
+                    "Leaving to be started again: routes stay in the kernel, and peers keep ours"
+                );
+                leave_for_restart(&registry, kernel.as_ref()).await;
+                Ok(ExitCode::from(RESTART_EXIT_STATUS))
+            }
         }
     }
 }
@@ -127,20 +149,30 @@ fn log_panics() {
     }));
 }
 
-/// Resolves with the name of the first termination signal received.
-async fn shutdown_signal() -> &'static str {
+/// What a signal asks of the daemon.
+enum Signal {
+    /// To stop: SIGINT or SIGTERM, by name.
+    Stop(&'static str),
+    /// To leave, to be started again at once: SIGUSR1.
+    Restart,
+}
+
+/// Resolves with the first signal received that ends the daemon.
+async fn signalled() -> Signal {
     #[cfg(unix)]
     {
         use tokio::signal::unix::{SignalKind, signal};
         let mut term = signal(SignalKind::terminate()).expect("install SIGTERM handler");
+        let mut restart = signal(SignalKind::user_defined1()).expect("install SIGUSR1 handler");
         tokio::select! {
-            _ = tokio::signal::ctrl_c() => "SIGINT",
-            _ = term.recv() => "SIGTERM",
+            _ = tokio::signal::ctrl_c() => Signal::Stop("SIGINT"),
+            _ = term.recv() => Signal::Stop("SIGTERM"),
+            _ = restart.recv() => Signal::Restart,
         }
     }
     #[cfg(not(unix))]
     {
         let _ = tokio::signal::ctrl_c().await;
-        "ctrl-c"
+        Signal::Stop("ctrl-c")
     }
 }