Install best routes in the Linux kernel over netlink

New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.

- netlink: the rtnetlink messages used (routes, addresses, links, errors),
  as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
  kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
  routes of ours found at startup are kept if still right and removed if
  not; a change of interfaces has every route worked out again; a clean
  stop removes our routes.

Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.

Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.

/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.

Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

bin/bgp/README.md

@@ -24,6 +24,7 @@ When `http_endpoint` is set the binary also serves:
 | `GET /api/routes/received` | Routes learned from peers; see "Finding a route" |
 | `GET /api/routes/rejected` | Routes peers announced that were refused, by peer |
 | `GET /api/rpki` | Validation with the RPKI: the cache, the data and its age |
+| `GET /api/kernel` | The routes in the kernel: how many, and why the others are not; the machine's interfaces |
 | `GET /api/events` | Server-sent events for session and route changes |
 | `GET /metricz` | Prometheus metrics |
 
@@ -34,9 +35,11 @@ put it behind a reverse proxy.
 
 It establishes sessions (both dialling configured peers and accepting
 connections, with RFC 4271 collision detection), announces its originated
-routes to every peer with itself as next hop, and records what peers send in
-an Adj-RIB-In. It does not select best paths, re-advertise received routes or
-program a forwarding table, and it has no import or export policy.
+routes to every peer with itself as next hop, judges what peers send by each
+peer's import policy, keeps what passes, chooses a best route for each
+prefix, and on Linux can put those best routes in the kernel so that the
+machine forwards by them. It does not yet pass routes learned from one peer
+on to another, and has no export policy.
 
 ## Who may connect
 
@@ -165,6 +168,67 @@ peers {
 }
 ```
 
+## Forwarding: routes in the kernel (Linux)
+
+With `kernel {}` in the configuration the daemon puts the best route of every
+prefix in the kernel's routing table, over netlink, and keeps it there as the
+best routes change. Without it the daemon only speaks BGP and the machine's
+forwarding is not touched.
+
+```
+kernel {
+    table: 254     # the main table; the default
+    metric: 20     # the default
+}
+```
+
+- **What is installed.** A best route whose next hop is a neighbor: an
+  address in the network of one of the machine's interfaces that is up. For
+  an IPv6 route with a link-local next hop, the interface is the one the
+  peer is a neighbor on. A route whose next hop is further away (a route
+  from a peer in our own AS that did not set itself as next hop, say) is not
+  installed, and neither is whatever it replaced: looking up the way to such
+  a next hop through other routes is not done yet. The status page counts
+  these, by reason.
+- **What is ours.** Routes are marked `proto bgp`. Every route with that
+  mark in the configured table with the configured metric is taken to be
+  this daemon's, and no other route is ever changed or removed. So do not
+  run two daemons that install BGP routes into the same table with the same
+  metric.
+- **Starting and stopping.** At startup, routes of ours left by an earlier
+  run are kept if they are still right and removed if not. On SIGINT or
+  SIGTERM the daemon closes its sessions and then removes its routes. After
+  a crash they stay until the next start.
+- **Interfaces.** The daemon learns the machine's interfaces and addresses
+  from the kernel and is told when they change; a second later it works all
+  routes out again, since any next hop may resolve differently.
+- **Privileges.** Changing routes needs the `CAP_NET_ADMIN` capability, and
+  listening on port 179 needs `CAP_NET_BIND_SERVICE`; neither needs root.
+  Without the first the daemon stops as soon as the kernel refuses it.
+
+  ```
+  setcap cap_net_admin,cap_net_bind_service+ep /usr/local/bin/bgp
+  ```
+
+`ip route show proto bgp` shows what is there. `/api/kernel` and the status
+page show how many routes are installed, how many are not and why, the last
+refusal by the kernel, and the interfaces; `/metricz` has `bgp_kernel_routes`,
+`bgp_kernel_routes_unresolved`, `bgp_kernel_routes_refused`,
+`bgp_kernel_route_changes_total{change}` and `bgp_kernel_errors_total`.
+
+To forward packets for others the machine also needs
+`net.ipv4.ip_forward=1` and `net.ipv6.conf.all.forwarding=1`; the daemon does
+not set them.
+
+### Testing against a real kernel
+
+The tests that change a kernel's routes are not run by `cargo test`.
+`scripts/kernel-tests.sh` runs them on Linux, each in a network namespace
+made for it: the route installer against `ip route`, the daemon with a peer,
+and two daemons that route for each other with pings across. From macOS,
+`scripts/linux-vm.sh` runs it (or any command, such as `cargo test`) in a
+Lima virtual machine that it creates on first use.
+
 ## Finding a route
 
 The status page has a look-up box over the received routes, and

bin/bgp/src/main.rs

@@ -2,7 +2,7 @@ use std::sync::Arc;
 
 use bgp_server::config::Config;
 use bgp_server::proto::config::BgpConfig;
-use bgp_server::server::{BgpServer, close_sessions};
+use bgp_server::server::{BgpServer, shut_down};
 use bgp_server::tracing::TracingArgs;
 use bgp_web::{AppState, RouterInfo};
 use clap::Parser;
@@ -52,6 +52,7 @@ async fn main() -> Result<()> {
             rib: Arc::clone(&server.rib),
             refusals: server.refusals.clone(),
             rpki: server.rpki.clone(),
+            kernel: server.kernel.clone(),
         };
         tokio::spawn(async move {
             if let Err(e) = bgp_web::serve_on(state, listener).await {
@@ -63,12 +64,14 @@ async fn main() -> Result<()> {
     // Leave politely: tell every peer we are going (Cease / Administrative
     // Shutdown) rather than letting them find a dead TCP connection, which
     // makes them slower to take us back.
-    let registry = server.registry.clone();
+    // And take our routes out of the kernel: with us gone nothing would
+    // keep them true.
+    let (registry, kernel) = (server.registry.clone(), server.kernel.clone());
     tokio::select! {
         result = server.run() => Ok(result?),
         signal = shutdown_signal() => {
             info!(signal, "Shutting down, closing BGP sessions");
-            close_sessions(&registry).await;
+            shut_down(&registry, kernel.as_ref()).await;
             Ok(())
         }
     }