fib, bench: profile the daemon taking in a full table; count blackholes instead of scanning

Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.

This change:

* Keeps counts of blackhole routes and of routes kept unconfirmed from
  an earlier run in `Plan`, in step with the map through `Plan::put`
  and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
  cost nothing. `kept()` had the same scan; it ran only while routes of
  an earlier run were being kept, which is exactly during a full-table
  load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
  new `profiling` Cargo profile (release with line tables) and frame
  pointers, runs it in a network namespace joined by a veth pair to a
  second namespace, feeds it the synthetic table from there as one or
  more neighbors, samples it with `perf`, and writes a flame graph
  (inferno), flat profiles by self and inclusive time, the time until
  the routes are in the table and in the kernel, and the daemon's
  resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
  and `OUT` control it. From macOS it runs in the Linux VM through
  scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
  from the peers' addresses, completes the handshakes, writes each
  neighbor's table, drains what the daemon sends back and holds the
  sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
  connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
  "Profiling" section, and the daemon's numbers in the VM.

Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.

Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

crates/bench/examples/feed.rs

@@ -0,0 +1,158 @@
+//! Feeds the synthetic Internet routing table (see the crate documentation)
+//! to a running daemon over TCP, as one or more peers would, and holds the
+//! sessions open afterwards. This is the load for profiling the daemon as
+//! it takes in a full table; `scripts/profile/full-table.sh` runs it under
+//! `perf`.
+//!
+//! Settings, from the environment:
+//!
+//! - `FEED_TO`: where the daemon listens (default `127.0.0.1:1790`).
+//! - `FEED_PEERS`: how many peers send the table (default 1).
+//! - `FEED_SCALE_DOWN`: as `BENCH_SCALE_DOWN`; 10 makes the table ten
+//!   times smaller (default 1).
+//! - `FEED_FROM`: the first three octets of an IPv4 /24, "192.0.2". Peer N
+//!   connects from .N, which is also the next hop of its IPv4 routes.
+//!   Without it, peers connect from wherever the system puts them.
+//! - `FEED_ASN`: the first peer's AS number (default 64513); the others
+//!   follow on from it.
+//! - `FEED_HOLD_SECS`: how long the sessions stay open after the table is
+//!   sent (default 30).
+//!
+//! What is printed: when the sessions are up and how long the writes took.
+//! The writes end when the kernel's socket buffers have taken the last
+//! bytes, which is before the daemon has read them: when the table is in
+//! is for the daemon's API to say.
+
+use std::net::{Ipv4Addr, SocketAddr};
+use std::time::{Duration, Instant};
+
+use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
+use bgp_integration_tests::test_peer::TestPeer;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
+use tokio::net::TcpSocket;
+use tokio::net::tcp::OwnedWriteHalf;
+
+/// A KEEPALIVE message: the marker, the length (19) and the type (4).
+const KEEPALIVE: [u8; 19] = {
+    let mut bytes = [0xff; 19];
+    bytes[16] = 0;
+    bytes[17] = 19;
+    bytes[18] = 4;
+    bytes
+};
+
+/// How often KEEPALIVEs go out while the sessions are held open: well
+/// inside the 90 s hold time.
+const KEEPALIVE_EVERY: Duration = Duration::from_secs(20);
+
+fn mib(bytes: usize) -> f64 {
+    bytes as f64 / (1024.0 * 1024.0)
+}
+
+#[tokio::main]
+async fn main() {
+    let to: SocketAddr = std::env::var("FEED_TO")
+        .unwrap_or_else(|_| "127.0.0.1:1790".into())
+        .parse()
+        .expect("FEED_TO is an address and port");
+    let peers = env("FEED_PEERS", 1) as u32;
+    let shape = Shape::INTERNET.scaled_down(env("FEED_SCALE_DOWN", 1));
+    let from = std::env::var("FEED_FROM").ok();
+    let first_asn = env("FEED_ASN", 64513) as u32;
+    let hold = Duration::from_secs(env("FEED_HOLD_SECS", 30) as u64);
+
+    let started = Instant::now();
+    let prefixes = Prefixes::generate(shape);
+    let tables: Vec<Vec<u8>> = (1..=peers)
+        .map(|peer| {
+            encoded(&announcements(&prefixes, shape, peer))
+                .iter()
+                .flat_map(|message| message.iter().copied())
+                .collect()
+        })
+        .collect();
+    let routes = shape.prefixes() * peers as usize;
+    let bytes: usize = tables.iter().map(Vec::len).sum();
+    println!(
+        "table: {} IPv4 + {} IPv6 prefixes from each of {peers} peer(s): {routes} routes, {:.1} MiB on the wire, made in {:.2} s",
+        shape.ipv4_prefixes,
+        shape.ipv6_prefixes,
+        mib(bytes),
+        started.elapsed().as_secs_f64()
+    );
+
+    let mut sessions = Vec::new();
+    for peer in 1..=peers {
+        let socket = TcpSocket::new_v4().expect("a socket");
+        if let Some(from) = &from {
+            let address: Ipv4Addr = format!("{from}.{peer}")
+                .parse()
+                .expect("FEED_FROM is the first three octets of an IPv4 address");
+            socket
+                .bind(SocketAddr::new(address.into(), 0))
+                .expect("the peer's address is ours to connect from");
+        }
+        let stream = socket.connect(to).await.expect("the daemon listens");
+        let mut session = TestPeer::from_stream(stream);
+        session
+            .handshake(
+                first_asn + peer - 1,
+                Ipv4Addr::new(10, 0, 0, peer as u8),
+                90,
+            )
+            .await
+            .expect("the daemon takes the session");
+        sessions.push(session.into_stream());
+    }
+    println!("{peers} session(s) established; sending");
+
+    let started = Instant::now();
+    let senders: Vec<_> = sessions
+        .into_iter()
+        .zip(tables)
+        .map(|(stream, table)| {
+            tokio::spawn(async move {
+                let (mut reader, mut writer) = stream.into_split();
+                // Whatever the daemon sends back, KEEPALIVEs as a rule, is
+                // read and dropped, so that its writer is never held up.
+                tokio::spawn(async move {
+                    let mut sink = [0u8; 65536];
+                    while let Ok(n) = reader.read(&mut sink).await {
+                        if n == 0 {
+                            break;
+                        }
+                    }
+                });
+                let started = Instant::now();
+                writer.write_all(&table).await.expect("the daemon reads");
+                (writer, started.elapsed())
+            })
+        })
+        .collect();
+    let mut writers: Vec<OwnedWriteHalf> = Vec::new();
+    for (index, sender) in senders.into_iter().enumerate() {
+        let (writer, took) = sender.await.expect("the sender runs");
+        println!(
+            "peer {}: table written in {:.3} s",
+            index + 1,
+            took.as_secs_f64()
+        );
+        writers.push(writer);
+    }
+    let elapsed = started.elapsed();
+    println!(
+        "all {routes} routes written in {:.3} s: {:.2} M routes/s into the sockets",
+        elapsed.as_secs_f64(),
+        routes as f64 / elapsed.as_secs_f64() / 1e6
+    );
+
+    let deadline = Instant::now() + hold;
+    while Instant::now() < deadline {
+        tokio::time::sleep(KEEPALIVE_EVERY.min(deadline - Instant::now())).await;
+        for writer in &mut writers {
+            // A session the daemon closed is left alone.
+            let _ = writer.write_all(&KEEPALIVE).await;
+        }
+    }
+    println!("sessions closed after {} s", hold.as_secs());
+}

crates/fib/src/plan.rs

@@ -317,6 +317,14 @@ pub struct Plan {
     unresolved_counts: [usize; Unresolved::ALL.len()],
     /// Prefixes whose route the kernel refused.
     refused: HashSet<IpPrefix>,
+    /// How many entries of `installed` are blackhole routes, and how many
+    /// are kept from an earlier run without having been confirmed. Kept in
+    /// step with the map by [`Plan::put`] and [`Plan::take`], so that
+    /// asking costs nothing however large the table: the kernel task asks
+    /// after every batch of changes, and a scan of a million entries each
+    /// time was a third of its work.
+    blackholes: usize,
+    kept_unconfirmed: usize,
 }
 
 impl Plan {
@@ -327,10 +335,38 @@ impl Plan {
 
     /// How many of the routes the kernel has drop their packets.
     pub fn blackholes(&self) -> usize {
-        self.installed
-            .values()
-            .filter(|entry| entry.target == Some(Target::Discard))
-            .count()
+        self.blackholes
+    }
+
+    /// What `entry` counts for: one if it is a blackhole route, and one if
+    /// it is kept from an earlier run without having been confirmed.
+    fn weigh(entry: &Entry) -> (usize, usize) {
+        (
+            usize::from(entry.target == Some(Target::Discard)),
+            usize::from(entry.kept && !entry.confirmed),
+        )
+    }
+
+    /// Puts `entry` in as the route of `prefix`, in place of what was
+    /// there, keeping the counts in step.
+    fn put(&mut self, prefix: IpPrefix, entry: Entry) {
+        let (blackhole, kept) = Self::weigh(&entry);
+        self.blackholes += blackhole;
+        self.kept_unconfirmed += kept;
+        if let Some(old) = self.installed.insert(prefix, entry) {
+            let (blackhole, kept) = Self::weigh(&old);
+            self.blackholes -= blackhole;
+            self.kept_unconfirmed -= kept;
+        }
+    }
+
+    /// Takes the route of `prefix` out, keeping the counts in step.
+    fn take(&mut self, prefix: &IpPrefix) -> Option<Entry> {
+        let old = self.installed.remove(prefix)?;
+        let (blackhole, kept) = Self::weigh(&old);
+        self.blackholes -= blackhole;
+        self.kept_unconfirmed -= kept;
+        Some(old)
     }
 
     /// How many prefixes have a best route that could not be given to the
@@ -383,7 +419,7 @@ impl Plan {
     /// have sent theirs again, and to remove everything meanwhile would
     /// stop the machine forwarding for as long as that takes.
     pub fn adopt(&mut self, prefix: IpPrefix, target: Option<Target>, keep: bool) {
-        self.installed.insert(
+        self.put(
             prefix,
             Entry {
                 target,
@@ -396,32 +432,36 @@ impl Plan {
     /// How many routes are being kept from an earlier run without the
     /// routing table having confirmed them.
     pub fn kept(&self) -> usize {
-        self.installed
-            .values()
-            .filter(|entry| entry.kept && !entry.confirmed)
-            .count()
+        self.kept_unconfirmed
     }
 
     /// The time for keeping unconfirmed routes of an earlier run is over:
     /// those the routing table still does not have go.
     pub fn release(&mut self) -> Vec<Op> {
         let mut stale = Vec::new();
+        let mut blackholes_gone = 0;
         self.installed.retain(|prefix, entry| {
             let goes = entry.kept && !entry.confirmed;
             entry.kept = false;
             if goes {
                 stale.push(Op::Remove { prefix: *prefix });
+                blackholes_gone += usize::from(entry.target == Some(Target::Discard));
             }
             !goes
         });
+        // What stays was confirmed, and is now not kept either.
+        self.kept_unconfirmed = 0;
+        self.blackholes -= blackholes_gone;
         stale
     }
 
     /// The whole table follows: nothing is known to be wanted until it is
     /// heard of again.
     pub fn reset(&mut self) {
+        self.kept_unconfirmed = 0;
         for entry in self.installed.values_mut() {
             entry.confirmed = false;
+            self.kept_unconfirmed += usize::from(entry.kept);
         }
         self.forget_unresolved_and_refused();
     }
@@ -442,20 +482,26 @@ impl Plan {
                     confirmed: true,
                     kept: false,
                 });
+                let before = Self::weigh(entry);
                 entry.confirmed = true;
-                if entry.target == Some(target) {
-                    return None;
-                }
-                entry.target = Some(target);
-                Some(Op::Install { prefix, target })
+                let op = if entry.target == Some(target) {
+                    None
+                } else {
+                    entry.target = Some(target);
+                    Some(Op::Install { prefix, target })
+                };
+                let after = Self::weigh(entry);
+                self.blackholes += after.0;
+                self.blackholes -= before.0;
+                self.kept_unconfirmed += after.1;
+                self.kept_unconfirmed -= before.1;
+                op
             }
             Err(reason) => {
                 self.set_unresolved(prefix, Some(reason));
                 // Packets must not go on following a route that is no
                 // longer the best.
-                self.installed
-                    .remove(&prefix)
-                    .map(|_| Op::Remove { prefix })
+                self.take(&prefix).map(|_| Op::Remove { prefix })
             }
         }
     }
@@ -464,9 +510,7 @@ impl Plan {
     pub fn withdrawn(&mut self, prefix: IpPrefix) -> Option<Op> {
         self.set_unresolved(prefix, None);
         self.refused.remove(&prefix);
-        self.installed
-            .remove(&prefix)
-            .map(|_| Op::Remove { prefix })
+        self.take(&prefix).map(|_| Op::Remove { prefix })
     }
 
     /// The whole table has been given: what was not in it goes.
@@ -480,7 +524,7 @@ impl Plan {
         stale
             .into_iter()
             .map(|prefix| {
-                self.installed.remove(&prefix);
+                self.take(&prefix);
                 Op::Remove { prefix }
             })
             .collect()
@@ -495,7 +539,7 @@ impl Plan {
                 if self.installed.get(prefix)?.target != Some(*target) {
                     return None;
                 }
-                self.installed.remove(prefix);
+                self.take(prefix);
                 self.refused.insert(*prefix);
                 // What the kernel refused to replace, it still has.
                 Some(Op::Remove { prefix: *prefix })
@@ -508,6 +552,8 @@ impl Plan {
     /// Everything goes: the changes that empty the kernel of our routes.
     pub fn clear(&mut self) -> Vec<Op> {
         self.forget_unresolved_and_refused();
+        self.blackholes = 0;
+        self.kept_unconfirmed = 0;
         self.installed
             .drain()
             .map(|(prefix, _)| Op::Remove { prefix })
@@ -1163,6 +1209,60 @@ mod tests {
         ]
     }
 
+    /// What the counts would be if the entries were scanned.
+    fn scanned(plan: &Plan) -> (usize, usize) {
+        plan.installed
+            .values()
+            .map(Plan::weigh)
+            .fold((0, 0), |(b, k), (eb, ek)| (b + eb, k + ek))
+    }
+
+    #[test]
+    fn the_counts_keep_step_with_the_routes() {
+        let mut plan = Plan::default();
+        let p = |n: u8| prefix(&format!("203.0.113.{n}/32"));
+        let via = target("192.0.2.2", 2);
+        let check = |plan: &Plan| assert_eq!((plan.blackholes(), plan.kept()), scanned(plan));
+
+        // Adopted from an earlier run: two kept, one a blackhole.
+        plan.adopt(p(1), Some(Target::Discard), true);
+        plan.adopt(p(2), Some(via), true);
+        plan.adopt(p(3), Some(via), false);
+        check(&plan);
+        // Heard of again, as a blackhole and as a route by way of a
+        // gateway; one changes kind.
+        plan.best(p(1), Ok(Target::Discard));
+        plan.best(p(2), Ok(Target::Discard));
+        plan.best(p(4), Ok(Target::Discard));
+        check(&plan);
+        plan.best(p(4), Ok(via));
+        check(&plan);
+        // The whole table again: everything unconfirmed, then confirmed.
+        plan.reset();
+        check(&plan);
+        plan.best(p(1), Ok(Target::Discard));
+        plan.synced();
+        check(&plan);
+        plan.reset();
+        plan.release();
+        check(&plan);
+        // Gone for want of a next hop, withdrawn, refused, adopted over.
+        plan.best(p(1), Err(Unresolved::Missing));
+        plan.best(p(5), Ok(Target::Discard));
+        plan.withdrawn(p(5));
+        plan.best(p(6), Ok(Target::Discard));
+        plan.refused_by_kernel(&Op::Install {
+            prefix: p(6),
+            target: Target::Discard,
+        });
+        plan.adopt(p(7), Some(Target::Discard), true);
+        plan.adopt(p(7), Some(via), true);
+        check(&plan);
+        let had = plan.installed();
+        assert_eq!(plan.clear().len(), had);
+        assert_eq!((plan.blackholes(), plan.kept()), (0, 0));
+    }
+
     proptest! {
         /// Whatever happens to the best routes, and whatever an earlier
         /// run left behind, a kernel that does as it is asked ends up with

crates/integration-tests/src/test_peer.rs

@@ -51,6 +51,12 @@ impl TestPeer {
         }
     }
 
+    /// The connection itself, for a test or a tool that goes on without
+    /// the framing, once the handshake is done.
+    pub fn into_stream(self) -> TcpStream {
+        self.stream
+    }
+
     /// Our end of the connection: what the server sees the peer as.
     pub fn local_addr(&self) -> Result<SocketAddr> {
         Ok(self.stream.local_addr()?)