fib, bench: profile the daemon taking in a full table; count blackholes instead of scanning

Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.

This change:

* Keeps counts of blackhole routes and of routes kept unconfirmed from
  an earlier run in `Plan`, in step with the map through `Plan::put`
  and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
  cost nothing. `kept()` had the same scan; it ran only while routes of
  an earlier run were being kept, which is exactly during a full-table
  load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
  new `profiling` Cargo profile (release with line tables) and frame
  pointers, runs it in a network namespace joined by a veth pair to a
  second namespace, feeds it the synthetic table from there as one or
  more neighbors, samples it with `perf`, and writes a flame graph
  (inferno), flat profiles by self and inclusive time, the time until
  the routes are in the table and in the kernel, and the daemon's
  resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
  and `OUT` control it. From macOS it runs in the Linux VM through
  scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
  from the peers' addresses, completes the handshakes, writes each
  neighbor's table, drains what the daemon sends back and holds the
  sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
  connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
  "Profiling" section, and the daemon's numbers in the VM.

Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.

Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs/content/docs/development/testing.md

@@ -172,11 +172,83 @@ performance: approximately 1 million IPv4 prefixes, mostly /24, and
 in UPDATE messages; AS paths of several hops; and communities on most
 routes.
 
-Table 5 lists results on an Apple M-series computer with a release build.
+Table 5 lists `full_table` results on an Apple M-series computer with a
+release build: 1.23 million prefixes from each of four neighbors, 4.92
+million routes.
 
 | Measurement | Result |
 |---|---|
-| 1.23 million routes from one neighbor, accepted into the routing table | 0.47 seconds |
-| The same routes received by a second neighbor (`transit`) | 2.1 seconds, approximately 4 routes per UPDATE message |
-| 1 million IPv4 routes installed in the kernel (Linux virtual machine) | Slightly more than 1 second |
-| The same routes removed from the kernel | 0.5 seconds |
+| Decoding the UPDATE messages, one thread | 0.43 seconds, 11 million routes per second |
+| Decoding, interning attributes, inserting and selecting, one thread | 1.9 seconds, 2.5 million routes per second |
+| The same, 12 threads | 0.7 seconds, 6.9 million routes per second |
+| Four neighbors send the table over TCP to the in-process server | 0.9 seconds, 5.2 million routes per second |
+| Memory per route, attribute sets included (`--features count-memory`) | 77 bytes |
+
+## Profiling
+
+The `scripts/profile/full-table.sh` script measures the daemon itself, as
+built for production, while one or more neighbors send it a full routing
+table, and records where its CPU time goes. It requires Linux with `perf`,
+`inferno` (`cargo install inferno`), `curl`, `jq`, and `sudo`. On another
+operating system, run it in the Linux virtual machine:
+
+```bash
+scripts/linux-vm.sh scripts/profile/full-table.sh
+```
+
+The script builds the daemon with the `profiling` Cargo profile, which is
+the release profile with line tables, and with frame pointers, so that
+`perf` can attribute samples to functions. It starts the daemon in a
+network namespace of its own, connected by a virtual Ethernet pair to a
+second namespace in which the load generator
+(`crates/bench/examples/feed.rs`) connects as the neighbors and sends the
+synthetic table. Routes that the daemon installs go into the first
+namespace's routing table and do not affect the host. Table 6 lists the
+environment variables that control the run.
+
+| Variable | Effect | Default |
+|---|---|---|
+| `PEERS` | The number of neighbors that send the table | 1 |
+| `KERNEL` | `1` installs the best routes in the kernel; `0` keeps them in the routing table only | 1 |
+| `SCALE_DOWN` | Divides the size of the table | 1 |
+| `HOLD_SECS` | How long the sessions stay open after the table is sent | 30 |
+| `FREQ` | The sampling frequency in hertz | 1999 |
+| `OUT` | The directory for the results | `~/profile` |
+
+When run through `scripts/linux-vm.sh`, pass the variables with `env`,
+because the virtual machine does not receive the host's environment:
+
+```bash
+scripts/linux-vm.sh env PEERS=4 KERNEL=0 scripts/profile/full-table.sh
+```
+
+The script prints how long the routes take to arrive in the routing table
+and in the kernel, and the daemon's resident memory. It writes the
+following files to the results directory:
+
+| File | Contents |
+|---|---|
+| `flame.svg` | An interactive flame graph of the daemon's threads |
+| `top-self.txt` | Functions by the time spent in their own code |
+| `top-children.txt` | Functions by the time spent in them and everything they call |
+| `stacks.folded` | The collapsed stacks the flame graph is drawn from |
+| `perf.data`, `daemon.log`, `feed.log` | The raw samples and the logs |
+
+Table 7 lists results in the Linux virtual machine (8 virtual CPUs on an
+Apple M-series computer) with the synthetic table of 1.23 million
+prefixes.
+
+| Measurement | Result |
+|---|---|
+| One neighbor; routes in the routing table | 0.7 seconds, 1.7 million routes per second |
+| One neighbor; the best routes installed in the kernel | 2.0 seconds after the first route |
+| One neighbor, without the kernel; routes in the routing table | 0.6 seconds, 2.0 million routes per second |
+| Four neighbors; 4.92 million routes in the routing table | 1.1 seconds, 4.5 million routes per second |
+| Four neighbors; the best routes installed in the kernel | 3.0 seconds after the first route |
+| Resident memory with one neighbor's table installed | 520 MB |
+| Resident memory with four neighbors' tables installed | 680 MB |
+
+A session is processed on one thread, so one neighbor's table arrives at
+the rate of one CPU core, and several neighbors' tables arrive in
+parallel. Installing routes in the kernel takes longer than receiving
+them, and continues after the last route has arrived.