rib: give a row room for every peer's route at its first growth

A row's routes are in a vector that grew by one route at a time, so with
four peers sending full tables every row's vector was reallocated three
times as their routes arrived, and the reallocations were 7% of the
daemon's CPU time in a profile of a four-peer load.

This change has the first growth of a row take it to as many routes as
the table has peers, at most 8 (`FIRST_ROOM`), since a prefix that one
full table has, the others nearly always have too; after that it grows
by a quarter as before. A table with one peer never grows a row and is
unchanged. A route server's clients do not all announce every prefix,
hence the cap: a row of a prefix that fewer peers announce has at most
seven spare slots of 16 bytes.

Measured in the Linux VM with four peers sending the synthetic table of
1.23 million prefixes: the kernel is in step 1.9 s after the first route
instead of 2.2 s, the daemon's CPU time over the load is 5% lower, and
it is 604 MiB resident instead of about 630, since there are no
reallocation leftovers. With one peer nothing changes. Memory per route
in the benchmarks, allocations counted: 77 bytes before and after with
the synthetic table from four peers; 64 before and 65 after with the
RIPE RIS dump's eight largest peers, whose load went from 2.8 s to
1.8 s on one thread, the allocation counting making every reallocation
dearer than it is in the daemon.

Also tried and not adopted, with the new `JEMALLOC_CONF` setting of the
profiling script: transparent huge pages for the allocator
(`thp:always,metadata_thp:always`). Over three pairs of runs with one
peer the table arrived in 0.66, 1.17 and 0.70 s with them against 0.87,
0.75 and 0.87 s without, the CPU time was within the noise, and the
daemon was 60 to 100 MiB larger. An arena for the routes would not help
either: jemalloc's size classes already give rows exact 16, 32, 48 and
64 byte allocations, and the cost was the number of reallocations, not
the allocator.

Testing: the table tests, including the one that grows a row past
65,535 routes, pass; the kernel tests and the interoperability lab pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

crates/rib/src/table/mod.rs

@@ -193,13 +193,16 @@ impl Row {
             .position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id)
     }
 
-    fn push(&mut self, path: Path, path_id: u32) {
+    /// Adds a route. `peers` is how many peers the table has: a prefix
+    /// that one of them announces, the others mostly announce too, so a
+    /// row is given room for that many at its first growth.
+    fn push(&mut self, path: Path, path_id: u32, peers: usize) {
         if path_id != 0 && self.path_ids.is_empty() {
             self.path_ids = vec![0; self.paths.len()];
         }
-        push_tight(&mut self.paths, path);
+        push_tight(&mut self.paths, path, peers);
         if !self.path_ids.is_empty() {
-            push_tight(&mut self.path_ids, path_id);
+            push_tight(&mut self.path_ids, path_id, peers);
         }
     }
 
@@ -855,12 +858,25 @@ fn spread(hash: u64, count: usize) -> usize {
     ((stirred * count as u64) >> 32) as usize
 }
 
+/// The most room a row is given at its first growth, however many peers
+/// there are: a route server's clients do not all announce every prefix.
+const FIRST_ROOM: usize = 8;
+
 /// Adds to a vector without the doubling that `push` does: a row of a few
-/// hundred routes should not hold room for as many again. Grows by a
+/// hundred routes should not hold room for as many again. The first
-/// quarter, which keeps adding cheap on average.
+/// growth is to `expected` (at most [`FIRST_ROOM`]), since with several
-fn push_tight<T>(vec: &mut Vec<T>, value: T) {
+/// full tables nearly every row gets that many routes, and growing a row
+/// one route at a time was a twelfth of what taking in four tables cost.
+/// After that it grows by a quarter, which keeps adding cheap on average.
+fn push_tight<T>(vec: &mut Vec<T>, value: T, expected: usize) {
     if vec.len() == vec.capacity() {
-        vec.reserve_exact((vec.len() / 4).max(1));
+        let first = expected.min(FIRST_ROOM);
+        let more = if vec.capacity() < first {
+            first - vec.len()
+        } else {
+            (vec.len() / 4).max(1)
+        };
+        vec.reserve_exact(more);
     }
     vec.push(value);
 }
@@ -902,7 +918,7 @@ impl Shard {
 
         let (outcome, best_changed) = match row.position(peer, path_id) {
             None => {
-                row.push(path, path_id);
+                row.push(path, path_id, self.peers.len());
                 self.routes += 1;
                 let outcome = Outcome {
                     added: 1,

crates/rib/src/table/tests.rs

@@ -957,13 +957,13 @@ fn a_row_can_hold_more_routes_than_a_sixteen_bit_index_counts() {
     let mut scratch = select::Scratch::default();
     let count = usize::from(u16::MAX) + 10;
     for path_id in 2..=count as u32 {
-        row.push(path(&worse), path_id);
+        row.push(path(&worse), path_id, peers.len());
     }
     // Equal routes from one peer: the lowest identifier is the best.
     assert!(!select::after_add(&mut row, &peers, &mut scratch));
     assert_eq!(row.best(), 0);
 
-    row.push(path(&better), count as u32 + 1);
+    row.push(path(&better), count as u32 + 1, peers.len());
     assert!(select::after_add(&mut row, &peers, &mut scratch));
     assert_eq!(row.best(), count);
     assert_eq!(row.route(row.best()).path_id, count as u32 + 1);

docs/content/docs/development/testing.md

@@ -214,6 +214,7 @@ environment variables that control the run.
 | `HOLD_SECS` | How long the sessions stay open after the table is sent | 30 |
 | `FREQ` | The sampling frequency in hertz | 1999 |
 | `OUT` | The directory for the results | `~/profile` |
+| `JEMALLOC_CONF` | jemalloc options for the daemon (its `MALLOC_CONF`), for example `thp:always` | none |
 
 When run through `scripts/linux-vm.sh`, pass the variables with `env`,
 because the virtual machine does not receive the host's environment:

scripts/profile/full-table.sh

@@ -104,7 +104,9 @@ if [ "$kernel" = 1 ]; then
     echo 'kernel { stale_seconds: 0 }' >> "$out/daemon.textpb"
 fi
 
-run "$daemon_bin" "$out/daemon.textpb" --level warn > "$out/daemon.log" 2>&1 &
+# `JEMALLOC_CONF`, if set, is jemalloc's MALLOC_CONF for the daemon:
+# "thp:always", say. The symbols are prefixed, hence the variable's name.
+run env _RJEM_MALLOC_CONF="${JEMALLOC_CONF:-}" "$daemon_bin" "$out/daemon.textpb" --level warn > "$out/daemon.log" 2>&1 &
 for _ in $(seq 100); do
     run curl -sf http://127.0.0.1:8080/api/status > /dev/null 2>&1 && break
     sleep 0.1