Graceful restart (RFC 4724, RFC 8538)

A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/fib/src/kernel.rs

@@ -31,7 +31,7 @@ use crate::netlink::{
 /// "Input/output error": for what should not happen.
 const EIO: i32 = 5;
 use crate::plan::{Neighbor, Op, Plan, Target, Unresolved, resolve};
-use crate::{Fib, FibConfig, FibError, InterfaceStatus};
+use crate::{Fib, FibConfig, FibError, InterfaceStatus, Stop};
 
 /// How many best routes are taken from the routing table at a time.
 const BATCH: usize = 1024;
@@ -932,6 +932,18 @@ impl Task {
         Ok(())
     }
 
+    /// Ends with our routes left in the kernel, for the next run to find:
+    /// what was on its way there is written, and nothing is taken out.
+    async fn leave(&mut self) -> Result<(), FibError> {
+        info!(
+            routes = self.plan.installed(),
+            "Leaving our routes in the kernel for the next run"
+        );
+        self.queue = None;
+        (&mut self.writer).await.map_err(io::Error::other)??;
+        Ok(())
+    }
+
     /// Takes everything of ours out of the kernel.
     async fn withdraw(
         &mut self,
@@ -1022,7 +1034,7 @@ pub(crate) async fn run(
     rib: Arc<RibTable>,
     fib: Fib,
     sockets: Sockets,
-    mut stopping: watch::Receiver<bool>,
+    mut stopping: watch::Receiver<Option<Stop>>,
 ) -> Result<(), FibError> {
     // There from the first scrape, not from the first time they count.
     LazyLock::force(&ROUTES);
@@ -1075,6 +1087,11 @@ pub(crate) async fn run(
         release_at = keep.then(|| Instant::now() + config.stale_time);
         task.keeping = keep;
     }
+    task.fib
+        .shared
+        .adopted
+        .send_replace(Some(task.plan.installed()));
+    let mut release = task.fib.shared.release.subscribe();
     task.publish(true);
     info!(
         table = config.table,
@@ -1114,7 +1131,13 @@ pub(crate) async fn run(
                 task.relearn(&mut heard).await?;
                 reader = task.rib.reader();
             }
-            () = at(release_at), if release_at.is_some() => {
+            _ = async {
+                tokio::select! {
+                    () = at(release_at) => {}
+                    // Sooner, if the peers are known to have sent everything.
+                    _ = release.wait_for(|now| *now) => {}
+                }
+            }, if release_at.is_some() => {
                 release_at = None;
                 task.keeping = false;
                 let stale = task.plan.release();
@@ -1133,7 +1156,11 @@ pub(crate) async fn run(
                 });
             }
             _ = stopping.changed() => {
-                task.withdraw(&mut heard).await?;
+                if *stopping.borrow() == Some(Stop::Keep) {
+                    task.leave().await?;
+                } else {
+                    task.withdraw(&mut heard).await?;
+                }
                 task.publish(false);
                 return Ok(());
             }

crates/fib/src/lib.rs

@@ -139,11 +139,25 @@ pub struct Status {
 /// until it has. Whoever holds one is told when they change.
 pub type InterfacesWatch = watch::Receiver<Option<Arc<interfaces::Interfaces>>>;
 
+/// How the task that keeps the kernel's routes is to end.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Stop {
+    /// Taking our routes out of the kernel.
+    Withdraw,
+    /// Leaving them there, for the next run to find.
+    Keep,
+}
+
 struct Shared {
     config: FibConfig,
     status: Mutex<Status>,
     interfaces: watch::Sender<Option<Arc<interfaces::Interfaces>>>,
-    stop: watch::Sender<bool>,
+    stop: watch::Sender<Option<Stop>>,
+    /// How many routes of an earlier run were found in the kernel at the
+    /// start, once that has been looked at.
+    adopted: watch::Sender<Option<usize>>,
+    /// Set to have what is kept of an earlier run go now.
+    release: watch::Sender<bool>,
     /// Whether the task has ended.
     ended: watch::Sender<bool>,
 }
@@ -165,13 +179,15 @@ impl Fib {
         rib: Arc<RibTable>,
         config: FibConfig,
     ) -> Result<(Self, JoinHandle<Result<(), FibError>>), FibError> {
-        let (stop, stopping) = watch::channel(false);
+        let (stop, stopping) = watch::channel(None);
         let fib = Self {
             shared: Arc::new(Shared {
                 config,
                 status: Mutex::default(),
                 interfaces: watch::channel(None).0,
                 stop,
+                adopted: watch::channel(None).0,
+                release: watch::channel(false).0,
                 ended: watch::channel(false).0,
             }),
         };
@@ -208,13 +224,52 @@ impl Fib {
     /// until it has. Returns at once if it has ended already, for whatever
     /// reason; the task's own result says how.
     pub async fn stop(&self) {
+        self.end(Stop::Withdraw).await;
+    }
+
+    /// Has the task stop following the routing table and end, leaving our
+    /// routes in the kernel for the next run to find and keep for a while
+    /// (see [`FibConfig::stale_time`]), and waits until it has. For a
+    /// restart that is to go unnoticed by the packets being forwarded.
+    pub async fn leave(&self) {
+        self.end(Stop::Keep).await;
+    }
+
+    async fn end(&self, how: Stop) {
         let mut ended = self.shared.ended.subscribe();
-        self.shared.stop.send_replace(true);
+        self.shared.stop.send_if_modified(|stop| {
+            let first = stop.is_none();
+            if first {
+                *stop = Some(how);
+            }
+            first
+        });
         // Fails only if the handle that holds the sender is gone, and this
         // is one.
         let _ = ended.wait_for(|ended| *ended).await;
     }
 
+    /// How many routes of an earlier run the kernel had at the start, once
+    /// the task has looked; zero if it ended before it did.
+    pub async fn adopted(&self) -> usize {
+        let mut adopted = self.shared.adopted.subscribe();
+        let mut ended = self.shared.ended.subscribe();
+        tokio::select! {
+            found = adopted.wait_for(Option::is_some) => {
+                found.ok().and_then(|found| *found).unwrap_or(0)
+            }
+            _ = ended.wait_for(|ended| *ended) => 0,
+        }
+    }
+
+    /// Has the routes of an earlier run that the routing table does not
+    /// have go now, rather than when [`FibConfig::stale_time`] is up: the
+    /// peers have all sent their routes, and what is not among them is no
+    /// longer anybody's.
+    pub fn release_kept(&self) {
+        self.shared.release.send_replace(true);
+    }
+
     #[cfg_attr(not(target_os = "linux"), allow(dead_code))]
     fn update(&self, change: impl FnOnce(&mut Status)) {
         change(
@@ -231,7 +286,7 @@ impl Fib {
 fn start(
     rib: Arc<RibTable>,
     fib: Fib,
-    stopping: watch::Receiver<bool>,
+    stopping: watch::Receiver<Option<Stop>>,
 ) -> Result<JoinHandle<Result<(), FibError>>, FibError> {
     // Opened here, so that a machine that cannot is known of at once.
     let sockets = kernel::Sockets::open()?;
@@ -246,7 +301,7 @@ fn start(
 fn start(
     _rib: Arc<RibTable>,
     _fib: Fib,
-    _stopping: watch::Receiver<bool>,
+    _stopping: watch::Receiver<Option<Stop>>,
 ) -> Result<JoinHandle<Result<(), FibError>>, FibError> {
     Err(FibError::Unsupported)
 }

crates/fib/tests/kernel.rs

@@ -649,6 +649,55 @@ async fn routes_of_an_earlier_run_are_kept_while_peers_come_back() {
     task.await.unwrap().unwrap();
 }
 
+/// A restart that is to go unnoticed (graceful restart): the routes are
+/// left in the kernel, by way of their nexthop objects; the next run
+/// finds them, says how many, and lets those nobody announced again go
+/// as soon as it is told the peers have all sent theirs.
+#[tokio::test(flavor = "multi_thread")]
+#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
+async fn routes_stay_in_the_kernel_over_a_restart() {
+    let mut machine = Machine::new(12);
+    machine.stale_time = Duration::from_secs(60);
+    let (fib, task) = machine.start();
+    assert_eq!(fib.adopted().await, 0);
+    let peer = machine.peer(2);
+    let via = machine.via(machine.neighbor(2).into(), 0);
+    machine.rib.announce(
+        peer,
+        &via,
+        &[prefix("203.0.113.0/24"), prefix("198.18.0.0/15")],
+    );
+    let both = [
+        machine.shown("198.18.0.0/15", machine.neighbor(2)),
+        machine.shown("203.0.113.0/24", machine.neighbor(2)),
+    ];
+    machine.kernel_becomes(&both);
+
+    fib.leave().await;
+    task.await.unwrap().unwrap();
+    assert_eq!(machine.kernel_routes(), both.iter().cloned().collect());
+
+    // The next run, with a table that is empty until the peers are back.
+    machine.rib = Arc::new(RibTable::new(4));
+    let (fib, task) = machine.start();
+    assert_eq!(fib.adopted().await, 2);
+    status_becomes(&fib, "keeping both", |status| status.kept == 2);
+    let peer = machine.peer(2);
+    let via = machine.via(machine.neighbor(2).into(), 0);
+    machine
+        .rib
+        .announce(peer, &via, &[prefix("203.0.113.0/24")]);
+    status_becomes(&fib, "one confirmed", |status| status.kept == 1);
+
+    // Every peer has sent its routes: what was not among them goes now,
+    // and not a minute later.
+    fib.release_kept();
+    machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
+
+    fib.stop().await;
+    task.await.unwrap().unwrap();
+}
+
 /// An IPv4 route whose next hop is an IPv6 neighbor: routes over a link
 /// that has only IPv6 addresses (RFC 8950). The kernel takes such a route,
 /// by way of a nexthop object or with its gateway spelled out.