Install best routes in the Linux kernel over netlink

New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.

- netlink: the rtnetlink messages used (routes, addresses, links, errors),
  as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
  kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
  routes of ours found at startup are kept if still right and removed if
  not; a change of interfaces has every route worked out again; a clean
  stop removes our routes.

Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.

Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.

/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.

Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

crates/fib/src/plan.rs

@@ -0,0 +1,619 @@
+//! What should be in the kernel's table, what is, and what to do about the
+//! difference. No I/O here: this is told what the routing table's best
+//! routes are and what the kernel said, and answers with the changes to
+//! ask the kernel for.
+
+use std::collections::{HashMap, HashSet};
+use std::net::{IpAddr, Ipv6Addr};
+
+use bgp_packet::message::NlriNextHop;
+use bgp_primitives::constants::IpVersion;
+use bgp_primitives::ip_prefix::IpPrefix;
+
+use crate::interfaces::Interfaces;
+
+/// Where the kernel is to send packets for a prefix.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub struct Target {
+    pub gateway: IpAddr,
+    /// The index of the interface the gateway is on.
+    pub interface: u32,
+}
+
+/// Why a route's next hop is not something the kernel can be given.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum Unresolved {
+    /// The route has no next hop.
+    Missing,
+    /// The next hop is of the other IP version than the prefix.
+    OtherFamily,
+    /// The next hop is one of this machine's own addresses.
+    Ourselves,
+    /// The next hop is not in the network of any interface that is up.
+    /// Finding the way to such a next hop through other routes is not
+    /// done.
+    NotConnected,
+}
+
+impl Unresolved {
+    pub const ALL: [Self; 4] = [
+        Self::Missing,
+        Self::OtherFamily,
+        Self::Ourselves,
+        Self::NotConnected,
+    ];
+
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Missing => "no next hop",
+            Self::OtherFamily => "next hop of the other IP version",
+            Self::Ourselves => "next hop is this machine",
+            Self::NotConnected => "next hop not directly connected",
+        }
+    }
+}
+
+fn is_link_local(address: &Ipv6Addr) -> bool {
+    address.segments()[0] & 0xffc0 == 0xfe80
+}
+
+/// Works out where the kernel is to send packets for a route to `prefix`
+/// with the next hop `next_hop`, learned from the peer at `peer`.
+///
+/// Only a next hop that is a neighbor on one of our interfaces is
+/// resolved. A link-local next hop means something only on one link, and
+/// that is taken to be the link the peer itself is on.
+pub fn resolve(
+    interfaces: &Interfaces,
+    prefix: &IpPrefix,
+    next_hop: Option<&NlriNextHop>,
+    peer: Option<IpAddr>,
+) -> Result<Target, Unresolved> {
+    let next_hop = next_hop.ok_or(Unresolved::Missing)?;
+    let version = match next_hop {
+        NlriNextHop::Ipv4(_) => IpVersion::Ipv4,
+        NlriNextHop::Ipv6(_) | NlriNextHop::Ipv6WithLl { .. } => IpVersion::Ipv6,
+    };
+    if version != prefix.address_family() {
+        return Err(Unresolved::OtherFamily);
+    }
+    let direct = |gateway: IpAddr| {
+        if interfaces.is_local(gateway) {
+            return Err(Unresolved::Ourselves);
+        }
+        interfaces
+            .connected(gateway)
+            .map(|connected| Target {
+                gateway,
+                interface: connected.interface,
+            })
+            .ok_or(Unresolved::NotConnected)
+    };
+    // The interface the peer is a neighbor on, for a link-local gateway.
+    let on_peers_link = |gateway: Ipv6Addr| {
+        peer.and_then(|peer| interfaces.connected(peer))
+            .map(|connected| Target {
+                gateway: gateway.into(),
+                interface: connected.interface,
+            })
+            .ok_or(Unresolved::NotConnected)
+    };
+    match *next_hop {
+        NlriNextHop::Ipv4(gateway) => direct(gateway.into()),
+        NlriNextHop::Ipv6(gateway) if is_link_local(&gateway) => on_peers_link(gateway),
+        NlriNextHop::Ipv6(gateway) => direct(gateway.into()),
+        // The global address if it is a neighbor; else the link-local one,
+        // which is there for exactly that case (RFC 2545 §3).
+        NlriNextHop::Ipv6WithLl { global, link_local } => match direct(global.into()) {
+            Err(Unresolved::NotConnected) => on_peers_link(link_local),
+            resolved => resolved,
+        },
+    }
+}
+
+/// A change to ask the kernel for.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum Op {
+    /// Put this route there, in place of ours for the prefix if there is
+    /// one.
+    Install {
+        prefix: IpPrefix,
+        target: Target,
+    },
+    Remove {
+        prefix: IpPrefix,
+    },
+}
+
+impl Op {
+    pub fn prefix(&self) -> &IpPrefix {
+        match self {
+            Self::Install { prefix, .. } | Self::Remove { prefix } => prefix,
+        }
+    }
+}
+
+struct Entry {
+    /// What the kernel has; `None` for a route found there at startup that
+    /// is not of a shape we make.
+    target: Option<Target>,
+    /// Whether the prefix was heard of since the last reset.
+    confirmed: bool,
+}
+
+/// The routes of ours in the kernel, kept in step with the best routes.
+///
+/// It follows what a reader of the routing table is given: the whole
+/// table between a reset and its completion, and changes after. Whatever
+/// is in the kernel and was not confirmed by the end of a whole table is
+/// removed: that is how routes left by an earlier run, or made stale by a
+/// change of interfaces, are cleared out.
+#[derive(Default)]
+pub struct Plan {
+    /// What the kernel has been asked to have, and has not refused.
+    installed: HashMap<IpPrefix, Entry>,
+    /// Prefixes whose best route is not installed, and why.
+    unresolved: HashMap<IpPrefix, Unresolved>,
+    /// How many of them for each reason, in the order of
+    /// [`Unresolved::ALL`].
+    unresolved_counts: [usize; 4],
+    /// Prefixes whose route the kernel refused.
+    refused: HashSet<IpPrefix>,
+}
+
+impl Plan {
+    /// How many routes of ours the kernel has.
+    pub fn installed(&self) -> usize {
+        self.installed.len()
+    }
+
+    /// How many prefixes have a best route that could not be given to the
+    /// kernel for want of a usable next hop.
+    pub fn unresolved(&self) -> usize {
+        self.unresolved.len()
+    }
+
+    /// The same, by reason, leaving out the reasons there are none of.
+    pub fn unresolved_by_reason(&self) -> impl Iterator<Item = (Unresolved, usize)> {
+        Unresolved::ALL
+            .into_iter()
+            .zip(self.unresolved_counts)
+            .filter(|(_, count)| *count > 0)
+    }
+
+    /// Notes why the best route of `prefix` is not installed, or with
+    /// `None` that this is no longer so.
+    fn set_unresolved(&mut self, prefix: IpPrefix, reason: Option<Unresolved>) {
+        let before = match reason {
+            Some(reason) => self.unresolved.insert(prefix, reason),
+            None => self.unresolved.remove(&prefix),
+        };
+        for (reason, by) in [(before, -1isize), (reason, 1)] {
+            if let Some(reason) = reason {
+                let count = &mut self.unresolved_counts[reason as usize];
+                *count = count.wrapping_add_signed(by);
+            }
+        }
+    }
+
+    /// How many prefixes have a best route that the kernel refused.
+    pub fn refused(&self) -> usize {
+        self.refused.len()
+    }
+
+    /// The prefixes of the routes the kernel has.
+    pub fn installed_prefixes(&self) -> impl Iterator<Item = &IpPrefix> {
+        self.installed.keys()
+    }
+
+    /// Notes a route of ours that the kernel already has, from before this
+    /// run. It stays only if the routing table turns out to have the same.
+    pub fn adopt(&mut self, prefix: IpPrefix, target: Option<Target>) {
+        self.installed.insert(
+            prefix,
+            Entry {
+                target,
+                confirmed: false,
+            },
+        );
+    }
+
+    /// The whole table follows: nothing is known to be wanted until it is
+    /// heard of again.
+    pub fn reset(&mut self) {
+        for entry in self.installed.values_mut() {
+            entry.confirmed = false;
+        }
+        self.unresolved.clear();
+        self.unresolved_counts = [0; 4];
+        self.refused.clear();
+    }
+
+    /// The best route for `prefix` is now one that resolves as `target`.
+    pub fn best(&mut self, prefix: IpPrefix, target: Result<Target, Unresolved>) -> Option<Op> {
+        self.refused.remove(&prefix);
+        match target {
+            Ok(target) => {
+                self.set_unresolved(prefix, None);
+                let entry = self.installed.entry(prefix).or_insert(Entry {
+                    target: None,
+                    confirmed: true,
+                });
+                entry.confirmed = true;
+                if entry.target == Some(target) {
+                    return None;
+                }
+                entry.target = Some(target);
+                Some(Op::Install { prefix, target })
+            }
+            Err(reason) => {
+                self.set_unresolved(prefix, Some(reason));
+                // Packets must not go on following a route that is no
+                // longer the best.
+                self.installed
+                    .remove(&prefix)
+                    .map(|_| Op::Remove { prefix })
+            }
+        }
+    }
+
+    /// `prefix` has no route any more.
+    pub fn withdrawn(&mut self, prefix: IpPrefix) -> Option<Op> {
+        self.set_unresolved(prefix, None);
+        self.refused.remove(&prefix);
+        self.installed
+            .remove(&prefix)
+            .map(|_| Op::Remove { prefix })
+    }
+
+    /// The whole table has been given: what was not in it goes.
+    pub fn synced(&mut self) -> Vec<Op> {
+        let stale: Vec<IpPrefix> = self
+            .installed
+            .iter()
+            .filter(|(_, entry)| !entry.confirmed)
+            .map(|(prefix, _)| *prefix)
+            .collect();
+        stale
+            .into_iter()
+            .map(|prefix| {
+                self.installed.remove(&prefix);
+                Op::Remove { prefix }
+            })
+            .collect()
+    }
+
+    /// The kernel refused `op`. Returns what to ask of it instead, if
+    /// anything.
+    pub fn refused_by_kernel(&mut self, op: &Op) -> Option<Op> {
+        match op {
+            Op::Install { prefix, target } => {
+                // If the plan has moved on since, this is old news.
+                if self.installed.get(prefix)?.target != Some(*target) {
+                    return None;
+                }
+                self.installed.remove(prefix);
+                self.refused.insert(*prefix);
+                // What the kernel refused to replace, it still has.
+                Some(Op::Remove { prefix: *prefix })
+            }
+            // It was not there to remove: as wanted.
+            Op::Remove { .. } => None,
+        }
+    }
+
+    /// Everything goes: the changes that empty the kernel of our routes.
+    pub fn clear(&mut self) -> Vec<Op> {
+        self.unresolved.clear();
+        self.unresolved_counts = [0; 4];
+        self.refused.clear();
+        self.installed
+            .drain()
+            .map(|(prefix, _)| Op::Remove { prefix })
+            .collect()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use proptest::prelude::*;
+
+    use super::*;
+    use crate::netlink::{Address, Link, Message};
+
+    fn prefix(s: &str) -> IpPrefix {
+        IpPrefix::try_from(s).unwrap()
+    }
+
+    fn target(gateway: &str, interface: u32) -> Target {
+        Target {
+            gateway: gateway.parse().unwrap(),
+            interface,
+        }
+    }
+
+    fn interfaces() -> Interfaces {
+        let mut interfaces = Interfaces::default();
+        for (index, local, network) in [
+            (2, "192.0.2.1", "192.0.2.0/24"),
+            (2, "2001:db8::1", "2001:db8::/64"),
+            (3, "198.51.100.1", "198.51.100.0/24"),
+        ] {
+            interfaces.apply(&Message::Link {
+                present: true,
+                link: Link {
+                    index,
+                    name: format!("eth{index}"),
+                    up: true,
+                },
+            });
+            interfaces.apply(&Message::Address {
+                present: true,
+                address: Address {
+                    interface: index,
+                    local: local.parse().unwrap(),
+                    network: prefix(network),
+                },
+            });
+        }
+        interfaces
+    }
+
+    #[test]
+    fn a_next_hop_resolves_if_it_is_a_neighbor() {
+        let interfaces = interfaces();
+        let v4 = prefix("203.0.113.0/24");
+        let v6 = prefix("2001:db8:100::/48");
+        let peer = Some("192.0.2.2".parse().unwrap());
+        let v4_hop = |s: &str| NlriNextHop::Ipv4(s.parse().unwrap());
+        let v6_hop = |s: &str| NlriNextHop::Ipv6(s.parse().unwrap());
+
+        assert_eq!(
+            resolve(&interfaces, &v4, Some(&v4_hop("192.0.2.2")), peer),
+            Ok(target("192.0.2.2", 2))
+        );
+        // A neighbor other than the peer: a route server's routes.
+        assert_eq!(
+            resolve(&interfaces, &v4, Some(&v4_hop("198.51.100.9")), peer),
+            Ok(target("198.51.100.9", 3))
+        );
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&v6_hop("2001:db8::2")), peer),
+            Ok(target("2001:db8::2", 2))
+        );
+        for (hop, why) in [
+            (Some(v4_hop("10.9.9.9")), Unresolved::NotConnected),
+            (Some(v4_hop("192.0.2.1")), Unresolved::Ourselves),
+            (Some(v6_hop("2001:db8::2")), Unresolved::OtherFamily),
+            (None, Unresolved::Missing),
+        ] {
+            assert_eq!(resolve(&interfaces, &v4, hop.as_ref(), peer), Err(why));
+        }
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&v4_hop("192.0.2.2")), peer),
+            Err(Unresolved::OtherFamily)
+        );
+    }
+
+    #[test]
+    fn a_link_local_next_hop_is_on_the_peers_link() {
+        let interfaces = interfaces();
+        let v6 = prefix("2001:db8:100::/48");
+        let peer = Some("2001:db8::2".parse().unwrap());
+        let link_local: Ipv6Addr = "fe80::2".parse().unwrap();
+        let both = |global: &str| NlriNextHop::Ipv6WithLl {
+            global: global.parse().unwrap(),
+            link_local,
+        };
+        // The global address is a neighbor: it is used.
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&both("2001:db8::2")), peer),
+            Ok(target("2001:db8::2", 2))
+        );
+        // It is not: the link-local one, on the interface the peer is on.
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&both("2001:db8:ffff::2")), peer),
+            Ok(target("fe80::2", 2))
+        );
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&NlriNextHop::Ipv6(link_local)), peer),
+            Ok(target("fe80::2", 2))
+        );
+        // A peer that is not a neighbor says nothing about the link.
+        let far = Some("2001:db8:ffff::9".parse().unwrap());
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&NlriNextHop::Ipv6(link_local)), far),
+            Err(Unresolved::NotConnected)
+        );
+        assert_eq!(
+            resolve(&interfaces, &v6, Some(&both("2001:db8::1")), peer),
+            Err(Unresolved::Ourselves)
+        );
+    }
+
+    #[test]
+    fn the_kernel_is_asked_only_for_what_changes() {
+        let mut plan = Plan::default();
+        let p = prefix("203.0.113.0/24");
+        let (a, b) = (target("192.0.2.2", 2), target("192.0.2.3", 2));
+
+        assert_eq!(
+            plan.best(p, Ok(a)),
+            Some(Op::Install {
+                prefix: p,
+                target: a
+            })
+        );
+        // The best route changes to one with the same next hop: nothing
+        // for the kernel.
+        assert_eq!(plan.best(p, Ok(a)), None);
+        assert_eq!(
+            plan.best(p, Ok(b)),
+            Some(Op::Install {
+                prefix: p,
+                target: b
+            })
+        );
+        assert_eq!(plan.installed(), 1);
+        // A best route that cannot be installed takes the old one out.
+        assert_eq!(
+            plan.best(p, Err(Unresolved::NotConnected)),
+            Some(Op::Remove { prefix: p })
+        );
+        assert_eq!((plan.installed(), plan.unresolved()), (0, 1));
+        assert_eq!(
+            plan.unresolved_by_reason().collect::<Vec<_>>(),
+            [(Unresolved::NotConnected, 1)]
+        );
+        assert_eq!(plan.withdrawn(p), None);
+        assert_eq!(plan.unresolved_by_reason().count(), 0);
+        assert_eq!(plan.unresolved(), 0);
+
+        plan.best(p, Ok(a));
+        assert_eq!(plan.withdrawn(p), Some(Op::Remove { prefix: p }));
+        assert_eq!(plan.withdrawn(p), None);
+    }
+
+    #[test]
+    fn what_a_whole_table_does_not_confirm_is_removed() {
+        let mut plan = Plan::default();
+        let (kept, changed, stale, odd) = (
+            prefix("203.0.113.0/24"),
+            prefix("198.18.0.0/15"),
+            prefix("192.88.99.0/24"),
+            prefix("2001:db8:5::/48"),
+        );
+        let a = target("192.0.2.2", 2);
+        // Left in the kernel by an earlier run.
+        plan.adopt(kept, Some(a));
+        plan.adopt(changed, Some(a));
+        plan.adopt(stale, Some(a));
+        plan.adopt(odd, None);
+        assert_eq!(plan.installed(), 4);
+
+        plan.reset();
+        // The same as the kernel has: nothing to do.
+        assert_eq!(plan.best(kept, Ok(a)), None);
+        let b = target("192.0.2.3", 2);
+        assert_eq!(
+            plan.best(changed, Ok(b)),
+            Some(Op::Install {
+                prefix: changed,
+                target: b
+            })
+        );
+        let mut removed = plan.synced();
+        removed.sort_by_key(|op| *op.prefix());
+        let mut expected = vec![Op::Remove { prefix: stale }, Op::Remove { prefix: odd }];
+        expected.sort_by_key(|op| *op.prefix());
+        assert_eq!(removed, expected);
+        assert_eq!(plan.installed(), 2);
+        // A second completion finds nothing more.
+        assert_eq!(plan.synced(), []);
+    }
+
+    #[test]
+    fn a_route_the_kernel_refuses_is_not_counted_as_there() {
+        let mut plan = Plan::default();
+        let p = prefix("203.0.113.0/24");
+        let (a, b) = (target("192.0.2.2", 2), target("192.0.2.3", 2));
+        let install = plan.best(p, Ok(a)).unwrap();
+        // What it refused to put there may have left an older one of ours.
+        assert_eq!(
+            plan.refused_by_kernel(&install),
+            Some(Op::Remove { prefix: p })
+        );
+        assert_eq!((plan.installed(), plan.refused()), (0, 1));
+        // Refusing to remove what is not there is as wanted.
+        assert_eq!(plan.refused_by_kernel(&Op::Remove { prefix: p }), None);
+        // The next best route is tried afresh.
+        assert!(plan.best(p, Ok(b)).is_some());
+        assert_eq!((plan.installed(), plan.refused()), (1, 0));
+        // News of a refusal of what has since been replaced is ignored.
+        assert_eq!(plan.refused_by_kernel(&install), None);
+        assert_eq!(plan.installed(), 1);
+
+        let mut cleared = plan.clear();
+        assert_eq!(cleared.pop(), Some(Op::Remove { prefix: p }));
+        assert_eq!(plan.installed(), 0);
+    }
+
+    /// What happens to a table of best routes.
+    #[derive(Debug, Clone)]
+    enum Step {
+        Best(u8, Option<u8>),
+        Withdrawn(u8),
+        /// A reader that starts over: the whole table again.
+        Resync,
+    }
+
+    fn any_step() -> impl Strategy<Value = Step> {
+        prop_oneof![
+            4 => (0u8..12, proptest::option::weighted(0.8, 0u8..3)).prop_map(|(p, t)| Step::Best(p, t)),
+            2 => (0u8..12).prop_map(Step::Withdrawn),
+            1 => Just(Step::Resync),
+        ]
+    }
+
+    proptest! {
+        /// Whatever happens to the best routes, and whatever an earlier
+        /// run left behind, a kernel that does as it is asked ends up with
+        /// exactly the best routes that resolve.
+        #[test]
+        fn the_kernel_ends_up_with_the_best_routes_that_resolve(
+            left_behind in proptest::collection::vec((0u8..16, 0u8..3), 0..6),
+            steps in proptest::collection::vec(any_step(), 0..60),
+        ) {
+            let p = |n: u8| IpPrefix::new(IpVersion::Ipv4, [10, n, 0, 0], 16).unwrap();
+            let t = |n: u8| target(&format!("192.0.2.{}", n + 2), 2);
+            let mut kernel: HashMap<IpPrefix, Target> = HashMap::new();
+            let mut plan = Plan::default();
+            for (prefix, gateway) in left_behind {
+                kernel.insert(p(prefix), t(gateway));
+                plan.adopt(p(prefix), Some(t(gateway)));
+            }
+            // The best routes: the prefix's next hop, if it resolves.
+            let mut table: HashMap<u8, Option<u8>> = HashMap::new();
+            let apply = |kernel: &mut HashMap<IpPrefix, Target>, op: Option<Op>| match op {
+                Some(Op::Install { prefix, target }) => { kernel.insert(prefix, target); }
+                Some(Op::Remove { prefix }) => { kernel.remove(&prefix); }
+                None => {}
+            };
+            let resolved = |gateway: Option<u8>| gateway.map(t).ok_or(Unresolved::NotConnected);
+            // A new reader is first given the whole table.
+            let steps = std::iter::once(Step::Resync).chain(steps);
+            for step in steps {
+                match step {
+                    Step::Best(prefix, gateway) => {
+                        table.insert(prefix, gateway);
+                        apply(&mut kernel, plan.best(p(prefix), resolved(gateway)));
+                    }
+                    Step::Withdrawn(prefix) => {
+                        table.remove(&prefix);
+                        apply(&mut kernel, plan.withdrawn(p(prefix)));
+                    }
+                    Step::Resync => {
+                        plan.reset();
+                        for (prefix, gateway) in &table {
+                            apply(&mut kernel, plan.best(p(*prefix), resolved(*gateway)));
+                        }
+                        for op in plan.synced() {
+                            apply(&mut kernel, Some(op));
+                        }
+                    }
+                }
+                let wanted: HashMap<IpPrefix, Target> = table
+                    .iter()
+                    .filter_map(|(prefix, gateway)| Some((p(*prefix), t((*gateway)?))))
+                    .collect();
+                prop_assert_eq!(&kernel, &wanted);
+                prop_assert_eq!(plan.installed(), wanted.len());
+                prop_assert_eq!(plan.unresolved(), table.len() - wanted.len());
+                prop_assert_eq!(
+                    plan.unresolved_by_reason().map(|(_, count)| count).sum::<usize>(),
+                    plan.unresolved()
+                );
+            }
+        }
+    }
+}