Graceful restart (RFC 4724, RFC 8538)

A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/rib/src/table/mod.rs

@@ -36,6 +36,7 @@ use std::net::{IpAddr, Ipv4Addr};
 use std::sync::atomic::{AtomicU64, AtomicUsize};
 use std::sync::{Arc, Mutex, PoisonError, RwLock, RwLockReadGuard, RwLockWriteGuard};
 
+use bgp_primitives::constants::IpVersion;
 use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
 use foldhash::fast::RandomState;
 use hashbrown::HashTable;
@@ -85,6 +86,9 @@ pub struct Outcome {
     pub removed: usize,
     /// Prefixes whose best route is now another one, or none.
     pub best_changed: usize,
+    /// Stale routes that are no longer: replaced or withdrawn by their
+    /// peer (graceful restart; see [`RibTable::mark_stale`]).
+    pub stale_cleared: usize,
 }
 
 impl std::ops::AddAssign for Outcome {
@@ -93,20 +97,31 @@ impl std::ops::AddAssign for Outcome {
         self.replaced += other.replaced;
         self.removed += other.removed;
         self.best_changed += other.best_changed;
+        self.stale_cleared += other.stale_cleared;
     }
 }
 
 /// One route of a prefix: sixteen bytes.
 struct Path {
-    /// Two things in one word. The high 48 bits are the first steps of
+    /// Three things in one word. The high 48 bits are the first steps of
     /// route selection as one number, the larger the better; see
     /// [`select::preference`]. The low 16 are the peer the route is from.
+    /// And one bit that the number never uses marks the route stale; see
+    /// [`STALE`].
     word: u64,
     attrs: Arc<AttrSet>,
 }
 
+/// The bit of a [`Path`]'s word that marks a route stale: kept from a
+/// session that ended, until the peer announces it again or it is swept
+/// away (graceful restart, RFC 4724 §4.2). It is the lowest bit of the
+/// preference, which is always zero ([`select::preference`]), and is left
+/// out of it: a stale route is chosen like any other, as the RFC has it.
+const STALE: u64 = 1 << 16;
+
 impl Path {
     fn new(preference: u64, peer: PeerIndex, attrs: &Arc<AttrSet>) -> Self {
+        debug_assert_eq!(preference & 1, 0, "the bit for stale routes is used");
         Self {
             word: (preference << 16) | u64::from(peer.0),
             attrs: Arc::clone(attrs),
@@ -114,7 +129,11 @@ impl Path {
     }
 
     fn preference(&self) -> u64 {
-        self.word >> 16
+        (self.word & !STALE) >> 16
+    }
+
+    fn is_stale(&self) -> bool {
+        self.word & STALE != 0
     }
 
     fn peer(&self) -> PeerIndex {
@@ -170,11 +189,6 @@ impl Row {
             .position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id)
     }
 
-    /// Where a route from `peer` is, whatever its identifier.
-    fn position_of_peer(&self, peer: PeerIndex) -> Option<usize> {
-        self.paths.iter().position(|path| path.peer() == peer)
-    }
-
     fn push(&mut self, path: Path, path_id: u32) {
         if path_id != 0 && self.path_ids.is_empty() {
             self.path_ids = vec![0; self.paths.len()];
@@ -211,6 +225,7 @@ impl Row {
             path_id: self.path_id(index),
             attrs: &path.attrs,
             best: index == self.best(),
+            stale: path.is_stale(),
         }
     }
 
@@ -248,6 +263,9 @@ pub struct RouteRef<'a> {
     pub attrs: &'a Arc<AttrSet>,
     /// Whether it is the route chosen for its prefix.
     pub best: bool,
+    /// Whether it is kept from a session of the peer's that ended, and has
+    /// not been announced again since (graceful restart).
+    pub stale: bool,
 }
 
 impl RouteRef<'_> {
@@ -259,6 +277,7 @@ impl RouteRef<'_> {
             path_id: self.path_id,
             attrs: Arc::clone(self.attrs),
             best: self.best,
+            stale: self.stale,
         }
     }
 }
@@ -271,6 +290,7 @@ pub struct Route {
     pub path_id: u32,
     pub attrs: Arc<AttrSet>,
     pub best: bool,
+    pub stale: bool,
 }
 
 struct Shard {
@@ -430,6 +450,100 @@ impl RibTable {
         outcome
     }
 
+    /// Gives a peer number that is in use to a new session of the same
+    /// peer, whose identifier or address may have changed: what route
+    /// selection knows of the peer is replaced. Returns whether the number
+    /// was in use.
+    ///
+    /// This is how a peer that restarts finds the routes kept from its last
+    /// session (graceful restart): they are still under its number.
+    pub fn rebind_peer(&self, peer: PeerIndex, info: PeerInfo) -> bool {
+        let mut peers = self.peers.lock().unwrap_or_else(PoisonError::into_inner);
+        match peers.get_mut(peer.as_usize()) {
+            Some(slot @ Some(_)) => *slot = Some(info),
+            _ => return false,
+        }
+        for shard in &self.shards {
+            if let Some(slot) = write(shard).peers.get_mut(peer.as_usize()) {
+                *slot = Some(info);
+            }
+        }
+        true
+    }
+
+    /// Marks every route of `peer` for prefixes of the IP versions in
+    /// `versions` stale, and returns how many there are. They stay in the
+    /// table and are chosen as before; announcing one again makes it fresh,
+    /// and [`RibTable::sweep_stale`] takes out those that are still stale.
+    ///
+    /// For graceful restart (RFC 4724 §4.2): the routes of a peer that is
+    /// restarting are kept while it does. Like [`RibTable::remove_peer`]
+    /// this looks at every prefix in the table. It changes no route, so
+    /// the readers are not told of anything.
+    pub fn mark_stale(&self, peer: PeerIndex, versions: &[IpVersion]) -> usize {
+        let mut marked = 0;
+        for shard in &self.shards {
+            let mut shard = write(shard);
+            for row in shard.rows.iter_mut() {
+                if !versions.contains(&row.prefix.address_family()) {
+                    continue;
+                }
+                for path in &mut row.paths {
+                    if path.peer() == peer {
+                        path.word |= STALE;
+                        marked += 1;
+                    }
+                }
+            }
+        }
+        marked
+    }
+
+    /// Takes out the routes of `peer` for prefixes of the IP versions in
+    /// `versions` that are still stale; see [`RibTable::mark_stale`]. The
+    /// peer keeps its number.
+    ///
+    /// Safe to run while the peer announces routes: one that is announced
+    /// again, before or during this, is fresh and stays.
+    pub fn sweep_stale(&self, peer: PeerIndex, versions: &[IpVersion]) -> Outcome {
+        let mut outcome = Outcome::default();
+        let mut news = false;
+        let change = self.begin_change();
+        for shard in &self.shards {
+            let mut shard = write(shard);
+            shard.journals.set_stamp(change.stamp);
+            let before = shard.journals.mark();
+            outcome += shard.remove_where(|row, path| {
+                path.peer() == peer
+                    && path.is_stale()
+                    && versions.contains(&row.prefix.address_family())
+            });
+            shard.tidy_journals();
+            news |= shard.journals.has_news_since(before);
+        }
+        drop(change);
+        if news {
+            self.wake_readers();
+        }
+        outcome
+    }
+
+    /// How many routes of `peer` are stale; see [`RibTable::mark_stale`].
+    /// Looks at every prefix in the table.
+    pub fn count_stale(&self, peer: PeerIndex) -> usize {
+        self.shards
+            .iter()
+            .map(|shard| {
+                read(shard)
+                    .rows
+                    .iter()
+                    .flat_map(|row| &row.paths)
+                    .filter(|path| path.peer() == peer && path.is_stale())
+                    .count()
+            })
+            .sum()
+    }
+
     /// Announces `routes` from `peer`, all with the attributes `attrs`:
     /// what one UPDATE does. A route is known by its prefix and path
     /// identifier together ([`Nlri`]; a plain [`IpPrefix`] is the route
@@ -757,6 +871,7 @@ impl Shard {
                 let before = std::mem::replace(&mut row.paths[at], path);
                 let outcome = Outcome {
                     replaced: 1,
+                    stale_cleared: usize::from(before.is_stale()),
                     ..Outcome::default()
                 };
                 if Arc::ptr_eq(&before.attrs, attrs) && before.preference() == preference {
@@ -790,12 +905,14 @@ impl Shard {
             return Outcome::default();
         };
         self.routes -= 1;
+        let stale_cleared = usize::from(row.paths[at].is_stale());
         if row.paths.len() == 1 {
             entry.remove();
             self.journals.disappeared(prefix);
             return Outcome {
                 removed: 1,
                 best_changed: 1,
+                stale_cleared,
                 ..Outcome::default()
             };
         }
@@ -804,11 +921,17 @@ impl Shard {
         Outcome {
             removed: 1,
             best_changed: best_changed.into(),
+            stale_cleared,
             ..Outcome::default()
         }
     }
 
     fn remove_peer(&mut self, peer: PeerIndex) -> Outcome {
+        self.remove_where(|_, path| path.peer() == peer)
+    }
+
+    /// Takes out every route that `doomed` picks, given its row.
+    fn remove_where(&mut self, doomed: impl Fn(&Row, &Path) -> bool) -> Outcome {
         let mut outcome = Outcome::default();
         let peers = &self.peers;
         let scratch = &mut self.scratch;
@@ -818,7 +941,7 @@ impl Shard {
             let mut best_changed = false;
             // A peer that sends several routes for a prefix has several
             // here.
-            while let Some(at) = row.position_of_peer(peer) {
+            while let Some(at) = row.paths.iter().position(|path| doomed(row, path)) {
                 removed += 1;
                 if row.paths.len() == 1 {
                     outcome.removed += removed;

crates/rib/src/table/tests.rs

@@ -445,6 +445,129 @@ fn removing_a_peer_takes_its_routes_and_frees_its_number() {
     assert_eq!(fixture.table.add_peer(external(3)), Ok(fixture.peers[0]));
 }
 
+/// Graceful restart (RFC 4724 §4.2): a restarting peer's routes are kept,
+/// marked stale, and used as before; what it announces again is fresh, and
+/// what it does not is swept away at the end.
+#[test]
+fn stale_routes_are_chosen_as_before_and_swept_unless_announced_again() {
+    let fixture = Fixture::new(&[external(1), external(2)]);
+    let v4 = [
+        prefix("10.1.0.0/16"),
+        prefix("10.2.0.0/16"),
+        prefix("10.3.0.0/16"),
+    ];
+    let v6 = prefix("2001:db8::/32");
+    for p in v4.iter().chain([&v6]) {
+        fixture.announce(0, p, &Route::via(&[1]));
+    }
+    // Peer 1 has a worse route for the first prefix, and loses.
+    fixture.announce(1, &v4[0], &Route::via(&[2, 3]));
+    let mut reader = fixture.table.reader_from_now();
+
+    let peer = fixture.peers[0];
+    let both = [IpVersion::Ipv4, IpVersion::Ipv6];
+    assert_eq!(fixture.table.mark_stale(peer, &both), 4);
+    assert_eq!(fixture.table.count_stale(peer), 4);
+    // Nothing changed for anyone who follows the table, and the stale
+    // route is still the best.
+    let mut events = 0;
+    reader.poll(usize::MAX, |_| events += 1);
+    assert_eq!(events, 0);
+    assert_eq!(fixture.best(&v4[0]), Some(0));
+    assert!(fixture.table.routes(&v4[0]).iter().any(|r| r.stale));
+
+    // The peer comes back and announces two of them again, one with the
+    // attributes it had: they are fresh.
+    assert_eq!(
+        fixture.announce(0, &v4[0], &Route::via(&[1])).stale_cleared,
+        1
+    );
+    assert_eq!(
+        fixture
+            .announce(0, &v4[1], &Route::via(&[1, 4]))
+            .stale_cleared,
+        1
+    );
+    // Again, and it was not stale any more.
+    assert_eq!(
+        fixture
+            .announce(0, &v4[1], &Route::via(&[1, 4]))
+            .stale_cleared,
+        0
+    );
+    assert_eq!(fixture.table.count_stale(peer), 2);
+
+    // Its End-of-RIB for IPv4: the IPv4 route it did not announce goes,
+    // the IPv6 one waits for the IPv6 marker.
+    let outcome = fixture.table.sweep_stale(peer, &[IpVersion::Ipv4]);
+    assert_eq!((outcome.removed, outcome.best_changed), (1, 1));
+    assert_eq!(fixture.best(&v4[2]), None);
+    assert_eq!(fixture.best(&v4[0]), Some(0));
+    assert_eq!(fixture.table.count_stale(peer), 1);
+    let outcome = fixture.table.sweep_stale(peer, &[IpVersion::Ipv6]);
+    assert_eq!(outcome.removed, 1);
+    assert_eq!(fixture.table.count_stale(peer), 0);
+    assert_eq!(fixture.table.len(), 3);
+    // Sweeping again finds nothing.
+    assert_eq!(fixture.table.sweep_stale(peer, &both), Outcome::default());
+}
+
+/// Stale routes are swept from one peer only, and only for the versions
+/// asked: the other peer's routes, stale or not, stay.
+#[test]
+fn sweeping_takes_only_the_stale_routes_of_one_peer() {
+    let fixture = Fixture::new(&[external(1), external(2)]);
+    let p = prefix("10.0.0.0/8");
+    fixture.announce(0, &p, &Route::via(&[1]));
+    fixture.announce(1, &p, &Route::via(&[2, 3]));
+    fixture
+        .table
+        .mark_stale(fixture.peers[1], &[IpVersion::Ipv4]);
+    assert_eq!(
+        fixture
+            .table
+            .sweep_stale(fixture.peers[0], &[IpVersion::Ipv4]),
+        Outcome::default()
+    );
+    assert_eq!(
+        fixture
+            .table
+            .sweep_stale(fixture.peers[1], &[IpVersion::Ipv6]),
+        Outcome::default()
+    );
+    assert_eq!(fixture.held(&p), vec![(0, 0), (1, 0)]);
+    let outcome = fixture
+        .table
+        .sweep_stale(fixture.peers[1], &[IpVersion::Ipv4]);
+    assert_eq!((outcome.removed, outcome.best_changed), (1, 0));
+    assert_eq!(fixture.held(&p), vec![(0, 0)]);
+}
+
+/// A new session of a restarting peer takes over its number, with what
+/// selection knows of it brought up to date: here a new identifier, which
+/// is what breaks the tie.
+#[test]
+fn a_peer_number_can_be_given_to_the_next_session_of_the_peer() {
+    let fixture = Fixture::new(&[external(1), external(2)]);
+    let p = prefix("10.0.0.0/8");
+    fixture.announce(0, &p, &Route::via(&[1]));
+    fixture.announce(1, &p, &Route::via(&[2]));
+    assert_eq!(fixture.best(&p), Some(0));
+    let renumbered = PeerInfo {
+        router_id: Ipv4Addr::new(10, 0, 0, 9),
+        ..external(1)
+    };
+    assert!(fixture.table.rebind_peer(fixture.peers[0], renumbered));
+    assert_eq!(fixture.table.peer(fixture.peers[0]), Some(renumbered));
+    // Chosen again the next time the row changes.
+    fixture.announce(1, &p, &Route::via(&[5]));
+    assert_eq!(fixture.best(&p), Some(1));
+
+    // A number nobody has cannot be taken over.
+    fixture.table.remove_peer(fixture.peers[1]);
+    assert!(!fixture.table.rebind_peer(fixture.peers[1], external(3)));
+}
+
 #[test]
 fn readers_see_every_route_and_which_is_best() {
     let fixture = Fixture::new(&[external(1), external(2)]);