A router with many peers has a reader of the table's journal for each
session that is sent learned routes, seventy on ddln, and every change
to the table wakes them all. Each pass of a reader went round every
shard, and for each shard took a count on the table (an Arc clone) to
hold it while the shard was read, and gave it back after: two atomic
adds on one cache line, by seventy readers, 256 shards a pass. A
profile of ddln taking a table from the other router had the reader's
poll as the top function at two to three cores, and in the Linux VM
with 64 such sessions half of all CPU time was in the two atomic add
instructions. The sessions writing to the table waited behind it: the
table came in at 58,000 routes a second.
The count is now taken once a poll. And each journal keeps its latest
sequence number in an atomic the table holds the other end of, so that
a reader can tell a shard with nothing new for it from one with
something without the shard's lock, and pass over it: on a quiet
table, which is what the readers see most of the time, a pass is a
load a shard and no lock. An exporter that is sent little of a table,
as one to a peer at an exchange is, no longer makes a verdict of
nothing for a prefix the peer does not hold, which cost a map insert
and a lookup on the way out, and tells a route that came down or
across (RFC 9234) from its attributes' bytes before parsing them.
Testing: the mesh benchmark can have any number of collector sessions
of the first router (BENCH_COLLECTORS): peers in the sense of RFC 9234,
sent learned routes, with the feeders' routes marked as having come
across from a peer, so that each has a reader following the table
while it fills and passes nearly all of it over, as a router with many
peers at exchanges has. In the Linux VM with 64 of them, three rounds
alternating with the previous commit, the routers' exchange of their
tables takes 11 s instead of 20 s, at 86 s of CPU time instead of
155 s; without them it is unchanged. The table's tests, the server
tests, the integration tests, the kernel tests and every lab phase
pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The last change held a session's leave to gather a table, of which
there are four, until its table had gone out. A peer that reads at the
pace of its receive window takes hours over a table, and two such
peers held two of the four for that long, keeping other sessions from
gathering theirs: they waited for a leave before their first UPDATE.
The leave is for the memory a gather takes; what is still held once
the table is gathered is let go of as it goes out, as it was before
the gather was reworked.
The mesh benchmark's new peer can also come once the first router has
its own peers' routes and the other router's are still arriving
(BENCH_RECEIVER_EARLY=2), as a router's sessions come up after a
restart while its peers' tables arrive; the walk of a full table while
the table fills costs the arriving routes nothing measurable in the
VM.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A table given out anew to a session, at its start or when the journal
had let changes go, was gathered as a verdict for each prefix, each
with the route dressed for the session, and sent once the table was
all there. Dressing a route parses its attributes and encodes them for
the session, so a table of five million routes took seconds before
the first UPDATE went, on one worker thread without a yield. And the
routes that share attributes went out in separate UPDATEs after all:
they were grouped by the dressed form, of which only so many are kept,
and the routes of a set are spread over the whole table, so by the
time the next was reached the form had been let go of and made anew.
The mesh benchmark's new peer got four million routes in 3.8 million
UPDATEs; on a RIPE RIS table, grouped whole, a full table is 160,000
UPDATEs and 21 MB, and grouped in slices of 4,096 routes it is 845,000
UPDATEs and 72 MB, so slices are not the answer either.
The table is now gathered by what dresses its routes: the attribute
set as the table has it, who the route is from, and its IP version.
Nothing is dressed while gathering, which is a map insert a route.
Once the table is all there, each set is dressed once, with the thread
lent to the other tasks every few thousand, and then the sets go out
one after the other, each as few UPDATEs as its routes fit in, with
the peer reading the first while the rest are sent. A prefix heard of
again while the table is read supersedes what was heard before: each
word on a prefix is numbered, and only the latest goes. Path
identifiers are given at gathering, in the order the table has a
prefix's routes, so that they are as before. What was announced before
and not heard of again is withdrawn after the table, as before. The
exporter never blocks the thread on a table: the steady state's batch
is small, and the gathered table is gone through with yields.
Testing: a new test gives a session a table of 300 routes sharing
three sets of attributes, spread over the shards, and checks they go
in three UPDATEs. The mesh benchmark has a new phase in which a peer
takes the first router's table once the mesh is complete, timing the
first UPDATE and the whole; its feeders now negotiate no hold timer,
since they say nothing after their tables, and `BENCH_LOG` prints the
routers' logs. A new example, packing, counts the UPDATEs and octets
a RIS table takes grouped whole and in slices. The server tests, the
integration tests, the kernel tests and every lab phase pass. In the
Linux VM, three rounds alternating with the previous commit, the new
peer's first UPDATE comes 2.9 s after its session is up instead of
5.3 s, and it has all four million routes in 4.0 s instead of 8.0 s,
in 667,000 UPDATEs instead of 3.76 million; the whole run takes
22.4 s of CPU time instead of 26.2 s. The routers' exchange of their
tables, which does not go through this path once their sessions are
up, is unchanged within its noise.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The full-table benchmark measures one router taking in tables from its
peers. What two routers of one AS do with each other, each sending the
other every path it has (ADD-PATH both ways with all_paths, as a mesh
of routers that each hold the network's whole view is configured), was
measured nowhere: the exporter following the table's journal while the
sessions fill the table, at the size of the Internet. In production
that exchange took 70 seconds for a table that arrives from the
Internet in 20.
A new benchmark, mesh, starts two servers in one AS with distinct
router IDs, peers them over IPv6 loopback with ADD-PATH both ways and
all_paths, and feeds the first from four synthetic peers and the second
from two over IPv4 loopback (BENCH_PEERS, BENCH_PEERS_B,
BENCH_SCALE_DOWN). It prints how long each router takes to hold its
own peers' routes and then the other's, with the rate of the latter.
BENCH_DEADLINE prints the sessions' state if a router is not there in
time.
The benchmarks now allocate with jemalloc, as the daemon does, except
when memory is being counted: the C library's allocator put its own
locks in the profiles. Two routers on one host have each other's
addresses as next hops, loopback addresses, which the import check
refuses as it refuses a next hop that is this machine; the permissive
import policy, which only tests use and no configuration can ask for,
now takes them.
Testing: the next-hop test covers both policies; the benchmark runs on
macOS and in the Linux VM under perf.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
