rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: rayhaan/ferrix:6d8f89e5092d4e19820703c1ce047679315a2a54
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
...
compare: rayhaan/ferrix:main
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
28 Commits
6d8f89e509 ... main
Author SHA1 Message Date
Rayhaan JaufeerallyandClaude Fable 5.1 db8edaae8e policy, config, web: the TCP gauges named once, one revalidator, one peer-naming rule
ci / interop (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A code-quality pass over policy, configuration and the status API.
The twenty TCP gauges were spelled out in the struct, the
constructor, the removal, the sampler in the web crate and the status
view; one macro names each once with its help text, `PeerTcp::set`
fills them from a connection, and the view is a `From`. Every metric
and the list that forces them at start come from one declaration,
with a test that every declared family is registered. The two
revalidators, for next hops and for RPKI, share the task body that
scans the table off the sessions' threads and fans the findings out
to every session; the RPKI scan judges a route with the same `Checks`
the import path uses. Which peer a query names is one rule for the
received, rejected and TCP endpoints, so `/api/tcp` takes a name or an
address like the others. The config parser says proto3's "zero is
unset" in three helpers instead of a dozen shapes, and `parse_peer`
is three named parts shorter. The policy's test-only next-hop flag is
named for what it does, `accept_own_host_next_hop`; the "listed, or
else ours" prefix rule and the mark-then-prefer step each exist once.
The status page derives its spanning rows from the table's header,
shares one key-value grid and one detail-row rule, and defines its
helpers before their use. The web crate no longer re-exports views
nothing uses, and the TCP endpoint sits above the test module.

Testing: the TCP view's arithmetic (a share of no busy time is zero,
not NaN), the peer-naming rule, the RPKI scan finding only what a
change can have made invalid, and `/api/tcp` by address, by name and
for an unknown peer. The server and web tests, the HTTP, roles, RPKI
and import-policy integration tests, the docs build and every lab
phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 4cddd6fb04 export: one rule each for admission, identifiers and what the peer had 
A code-quality pass over the exporter. The resync path (gather then
stream) had grown beside the steady-state path (judge then flush)
with its own copies of the admission predicate, the path identifier
rule, the reconciliation of a route the peer already had, the parser
contexts, the withdrawal loop and the advertised gauge. Each is one
function now: `prefix_may_go`, `lowest_free_id`, `Sent::renewed`,
`parser_contexts`, `send_withdrawals`, `within_limit` and
`note_held`. The 3,072-octet ceiling on a route's attributes, which
is what keeps an UPDATE within its size, is a named constant where it
is relied on. The UPDATEs are framed with the packet crate's helpers
instead of a bare type code and a length written by hand. The
positional tuples for a prefix's paths are structs with names. A
mismatched kind of bookkeeping, which nothing should produce, is
asserted in tests and logged in release instead of dropped silently.
Types used only inside the crate are `pub(crate)`. Five doc comments
that had drifted onto neighbours or named former designs are put
right; the exporter's reference encoder no longer breaks the CI doc
build with a link to an item that exists only under test.

Testing: ADD-PATH export has unit tests at last: two routes for a
prefix go with their own identifiers and keep them, and the
withdrawal of one carries its identifier; what is not heard of again
after a reset is withdrawn after the table. The byte-for-byte test of
the encoded UPDATEs covers IPv4 routes with an IPv6 next hop (RFC
8950) too. The test fixtures are shared helpers. The server tests and
the ADD-PATH, transit, limits, roles and deadlock integration tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 bef875e50b session: the way out to the peer as a type of its own 
A code-quality pass over the per-peer session. The outgoing side of a
session, eight fields served from six places in a two-thousand-line
file, is `Outgoing`: the write buffer, the batches waiting for the
writer task, the task and its word on why it ended. One real race
went with it: when the writer task failed with the queue full, the
session took whichever of two ready branches the select chose, and
half the time ended with an invented send hold timer expiry instead
of the connection's error, which decides whether the peer's routes
are kept over a restart; the writer's own word is taken now. The run
loop's handling of what the rest of the server has to send is a
method that goes by the state machine's state, so that pieces after a
NOTIFICATION in the same batch do not follow it out; `execute_all`
reads one line an action with the sending, delivering and
after-failure steps named. A peer's routes have one way onto the
table at a session's end, whether removed or kept for a restart. An
UPDATE whose attributes cannot be kept ends the session under an
error of its own, `Unstorable`, where a message of ours that cannot
be encoded is dropped. What is kept of a restarting peer is a value,
not seven arguments. The dialer no longer turns a session's end into
a Result and back. The server's screening of a connection is called
that, apart from the collision check's `admit`.

Testing: the back-pressure machinery has unit tests at last: batches
wait while the writer has no room, room goes to the oldest batch, a
piece is counted as the messages in it, and a lost connection is
reported as one while batches wait. The routes module gains tests of
holding at its bound, forgetting and revalidation. The server tests
and the whole integration suite pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 25597ab3ef packet, fib, bench: framing helpers, one netlink reader, a mesh with named phases 
A code-quality pass over the periphery. The packet crate frames a
message with `begin_message` and `finish_message`, which the
exporter's hand-written UPDATE header can use instead of a bare type
code, and keeps the NLRI reader and writer together as a pair; the
rule that the body's prefixes are IPv4 unicast lives once; a session
with two-octet AS numbers no longer reserves the attributes' wire
copy it never keeps. The TCP diagnostics read netlink with the
netlink module's own attribute walk and helpers instead of copies of
them. The kernel follower learns what the kernel has in one named
step used at start and on relearning, keeps its batch buffers between
wakes, and builds its task in a constructor. The mesh benchmark,
which had grown six phases in one main, is a `Mesh` with named phases
and a `Receiver`, with one report of the sessions for both deadlines;
the collector capabilities, the KEEPALIVE bytes, the draining of a
session and the printing of a table row are helpers shared by the
benches, the examples and the integration tests. The MRT reader uses
the packet crate's attribute splitter.

Testing: a message framed by hand is the one `to_wire` writes; a
short tcp_info from an old kernel leaves the later fields zero; the
synthetic table is the same every run and every announcement decodes
to itself. The packing example gives the same figures on the RIS dump
before and after. The crates' tests, the integration tests, the
kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 bcfd6e013f rib: the lock-free number in a slot of its own, and the reader in named steps 
A code-quality pass over the table. The latest sequence number of
each journal, which readers compare their cursors with without the
shard's lock, was an atomic inside the journal under the lock with
the table holding the other end of an Arc to it, stored on every
route; it is now in a slot beside each shard's lock, `ShardSlot`,
stored once a shard a change while the write guard is still held and
before the readers are woken. "One of each kind of journal" is one
type, `ByKind<T>`, where three hand-written dispatchers and an array
index were. `Reader::poll` is the loop skeleton over `resync_step`
and `follow_pass`; the journal's `tidy` is three named stages;
`live`'s nested Option is an enum that says what an entry says; the
place of a cursor in a journal is found by one function for both the
time chosen for a pass and the pass itself. The attribute set's four
byte-readers share one walk over the raw attributes, with the next
hop decoded once for both of its readers. `best` returns a `Route`
like the other lookups. Docs that described an earlier design (the
reader's `seen`, "every method takes &self", a `Routes::is_empty`
that could not be true) are put right.

Testing: new tests pin the lock-free skip, which nothing checked
directly before: the number equals the journal's after every kind of
change, with and without readers of each kind; a shard passed over is
read again once it has news, from a reader of the whole table and
from one that starts now. The OTC flag and the received-attributes
interning gain tests. The crate's tests, both proptests included, the
server tests, the integration tests, the kernel tests and every lab
phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c70a223d0d rib: a reader counts the table once a poll, not once a shard
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A router with many peers has a reader of the table's journal for each
session that is sent learned routes, seventy on ddln, and every change
to the table wakes them all. Each pass of a reader went round every
shard, and for each shard took a count on the table (an Arc clone) to
hold it while the shard was read, and gave it back after: two atomic
adds on one cache line, by seventy readers, 256 shards a pass. A
profile of ddln taking a table from the other router had the reader's
poll as the top function at two to three cores, and in the Linux VM
with 64 such sessions half of all CPU time was in the two atomic add
instructions. The sessions writing to the table waited behind it: the
table came in at 58,000 routes a second.

The count is now taken once a poll. And each journal keeps its latest
sequence number in an atomic the table holds the other end of, so that
a reader can tell a shard with nothing new for it from one with
something without the shard's lock, and pass over it: on a quiet
table, which is what the readers see most of the time, a pass is a
load a shard and no lock. An exporter that is sent little of a table,
as one to a peer at an exchange is, no longer makes a verdict of
nothing for a prefix the peer does not hold, which cost a map insert
and a lookup on the way out, and tells a route that came down or
across (RFC 9234) from its attributes' bytes before parsing them.

Testing: the mesh benchmark can have any number of collector sessions
of the first router (BENCH_COLLECTORS): peers in the sense of RFC 9234,
sent learned routes, with the feeders' routes marked as having come
across from a peer, so that each has a reader following the table
while it fills and passes nearly all of it over, as a router with many
peers at exchanges has. In the Linux VM with 64 of them, three rounds
alternating with the previous commit, the routers' exchange of their
tables takes 11 s instead of 20 s, at 86 s of CPU time instead of
155 s; without them it is unchanged. The table's tests, the server
tests, the integration tests, the kernel tests and every lab phase
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 17:12:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a15efaa26c export: give the leave to gather back once the table is gathered 
The last change held a session's leave to gather a table, of which
there are four, until its table had gone out. A peer that reads at the
pace of its receive window takes hours over a table, and two such
peers held two of the four for that long, keeping other sessions from
gathering theirs: they waited for a leave before their first UPDATE.
The leave is for the memory a gather takes; what is still held once
the table is gathered is let go of as it goes out, as it was before
the gather was reworked.

The mesh benchmark's new peer can also come once the first router has
its own peers' routes and the other router's are still arriving
(BENCH_RECEIVER_EARLY=2), as a router's sessions come up after a
restart while its peers' tables arrive; the walk of a full table while
the table fills costs the arriving routes nothing measurable in the
VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:43:48 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 624d20f965 Release 0.2.6 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:30:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b5ca42244d export: a lighter gather, and UPDATEs to the session by the piece 
Two things kept an exporter from its peer. A table being gathered to
go out anew kept a map of every prefix heard, to let a later word on a
prefix supersede an earlier one, and dressed every set of attributes
before the first UPDATE went: a map insert and a parse-and-encode a
route that the peer waited on. And the session's queue took UPDATEs
one by one, a thousand at most, so the exporter stood at the queue
once a hundred kilobytes were in it, behind a writer that takes a
megabyte: the two took turns instead of running together.

The journal gives a table out shard by shard, each prefix once, and a
change meanwhile comes after the table as news of its own, so there is
no second word to keep track of while gathering: the map is gone, and
a prefix the peer is not to have is simply not gathered, as what it
held of it is withdrawn once the table is through. Each set is dressed
as its turn to go comes, so the first UPDATE goes once the table is
gathered; only when the gathered routes alone would be over the
peer's limit are the sets dressed first, to know which cannot be and
take their routes with them, so that none goes if too many would. The
UPDATEs for a set now go to the session in pieces of 64 KiB or so,
each of whole messages, and the queue takes 64 of those: a few
megabytes ahead of the writer, not a hundred kilobytes. The session
counts the messages in a piece by their headers.

Testing: the byte-for-byte test of the encoded UPDATEs now encodes
thirty thousand routes and checks the pieces are of whole messages
and of the size meant. The server tests, the integration tests, the
kernel tests and every lab phase pass. In the Linux VM, three rounds
alternating with the previous commit, a new peer's first UPDATE comes
1.8 s after its session is up instead of 3.1 s, and it has its four
million routes in 3.8 s instead of 4.1 s; the smaller of the two
routers' tables crosses in 3.0 s instead of 4.2 s and the larger in
5.0 s instead of 5.2 s, at the same CPU time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:28:40 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 7425b101ed export: gather a table by attributes, then stream it 
A table given out anew to a session, at its start or when the journal
had let changes go, was gathered as a verdict for each prefix, each
with the route dressed for the session, and sent once the table was
all there. Dressing a route parses its attributes and encodes them for
the session, so a table of five million routes took seconds before
the first UPDATE went, on one worker thread without a yield. And the
routes that share attributes went out in separate UPDATEs after all:
they were grouped by the dressed form, of which only so many are kept,
and the routes of a set are spread over the whole table, so by the
time the next was reached the form had been let go of and made anew.
The mesh benchmark's new peer got four million routes in 3.8 million
UPDATEs; on a RIPE RIS table, grouped whole, a full table is 160,000
UPDATEs and 21 MB, and grouped in slices of 4,096 routes it is 845,000
UPDATEs and 72 MB, so slices are not the answer either.

The table is now gathered by what dresses its routes: the attribute
set as the table has it, who the route is from, and its IP version.
Nothing is dressed while gathering, which is a map insert a route.
Once the table is all there, each set is dressed once, with the thread
lent to the other tasks every few thousand, and then the sets go out
one after the other, each as few UPDATEs as its routes fit in, with
the peer reading the first while the rest are sent. A prefix heard of
again while the table is read supersedes what was heard before: each
word on a prefix is numbered, and only the latest goes. Path
identifiers are given at gathering, in the order the table has a
prefix's routes, so that they are as before. What was announced before
and not heard of again is withdrawn after the table, as before. The
exporter never blocks the thread on a table: the steady state's batch
is small, and the gathered table is gone through with yields.

Testing: a new test gives a session a table of 300 routes sharing
three sets of attributes, spread over the shards, and checks they go
in three UPDATEs. The mesh benchmark has a new phase in which a peer
takes the first router's table once the mesh is complete, timing the
first UPDATE and the whole; its feeders now negotiate no hold timer,
since they say nothing after their tables, and `BENCH_LOG` prints the
routers' logs. A new example, packing, counts the UPDATEs and octets
a RIS table takes grouped whole and in slices. The server tests, the
integration tests, the kernel tests and every lab phase pass. In the
Linux VM, three rounds alternating with the previous commit, the new
peer's first UPDATE comes 2.9 s after its session is up instead of
5.3 s, and it has all four million routes in 4.0 s instead of 8.0 s,
in 667,000 UPDATEs instead of 3.76 million; the whole run takes
22.4 s of CPU time instead of 26.2 s. The routers' exchange of their
tables, which does not go through this path once their sessions are
up, is unchanged within its noise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:16:18 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 92b9bdd3c5 export: keep the peer reading while the next batch is dressed 
An exporter following the table took up to 65,536 changes at a time,
dressed them all, and only then sent them; and before each batch it
slept the 50 ms that lets a run of changes be gathered, whether or not
the table was already far ahead of it. The session's queue holds about
a thousand UPDATEs, so once the batch went out the exporter waited on
the peer reading it, and while the next batch was dressed the peer had
nothing to read. A profile of two routers exchanging their tables,
sampled in 25 ms windows, showed the two stages taking turns: the
exporters at two cores while the sessions were at none, then the other
way round, with a gap of 50 to 75 ms at the start of each cycle for the
sleep. Each task ran at about half a core.

A batch is now 4,096 changes, so that what one batch sends is being
read by the peer while the next is being dressed, and the gathering
sleep is skipped while the reader is behind the table: a run is in hand
already. A reader says whether its last poll caught up.

Testing: the server tests, the integration tests, the kernel tests and
every lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the larger table
(4.9 million routes) is across in 5.6 s instead of 6.7 s and the
smaller (2.5 million) in 4.4 s instead of 6.4 s; the whole exchange
takes 7.3 s of wall-clock time instead of 8.6 s, and 18.9 s of CPU time
instead of 20.5 s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 14:19:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 381eabd886 export: encode a route's attributes once a session 
The exporter built each UPDATE as a message: the attributes of the
route's set, parsed from the table, were cloned, had the next hop
added, were sorted, and were handed to the session, which encoded them
to the wire with the rest of the message. For a table whose attribute
sets each cover a few routes that is an encoding of the same octets for
nearly every route, and a copy of the message's bytes on the way to the
writer besides. A profile of a production router taking the other
router's table with ADD-PATH had the exporters at 55% of the CPU time,
their dressing of routes at 23%, and memcpy at 12%.

A route's attributes are now encoded once for a session, when it is
first dressed for it, and kept with where in the octets a NEXT_HOP or
an MP_REACH_NLRI belongs so that the attributes stay in order of type
code. An UPDATE is then its header, the encoded octets with the next
hop or the MP_REACH_NLRI spliced in, and the prefixes: written straight
into a buffer that goes to the session as it is, and from the session
to the writer without a copy. The session's queue takes either a
message or an encoded UPDATE; an encoded one for a session that is no
longer established is dropped, as the exporter made it for the session
as it was. What is kept for a route, now a few hundred octets instead
of a kilobyte, lets twice as many be kept. The exporter's maps hash
with foldhash instead of SipHash, and the parser context for an UPDATE
is made once a flush instead of once an UPDATE.

Testing: a new test encodes a thousand routes with each of the forms
an UPDATE takes (IPv4 with the next hop in its own attribute, IPv6 in
MP_REACH_NLRI, with and without path identifiers, to two-octet and
four-octet AS speakers, over several messages) both ways and checks the
octets are the same byte for byte; the message-building form is kept
for the test. The exporter tests decode what the session is handed.
The server tests, the integration tests, the kernel tests and every
lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the two routers'
exchange of 7.4 million routes takes 20.6 s of CPU time instead of
25.4 s (-19%) over 9.0 s of wall-clock time instead of 11.1 s, and the
larger of the two tables is across in 7.1 s instead of 9.3 s. The
smaller table's time varies too much from run to run (5-8 s) for a
difference to be read from it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:47:47 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 f1aca37ebe rib: a short row grows by doubling 
A prefix's row of paths is given room for as many paths as there are
peers, and from there grows by a quarter at a time, one path at a time
while it is short. A peer sending every path it has (ADD-PATH) gives a
prefix several paths from one peer, so a row goes past the peers' count
from the first few routes and was then reallocated for each path.

A row shorter than the first allocation's room doubles instead, and
grows by a quarter only from there. Measured on the mesh benchmark, the
reallocations in the table's insert path fall by half; the memory held
is as before, as the rows that reach the quarter-growth are the ones
that were already allocated for it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:45:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 cbc1dc7e44 bench: two routers exchanging their tables with ADD-PATH 
The full-table benchmark measures one router taking in tables from its
peers. What two routers of one AS do with each other, each sending the
other every path it has (ADD-PATH both ways with all_paths, as a mesh
of routers that each hold the network's whole view is configured), was
measured nowhere: the exporter following the table's journal while the
sessions fill the table, at the size of the Internet. In production
that exchange took 70 seconds for a table that arrives from the
Internet in 20.

A new benchmark, mesh, starts two servers in one AS with distinct
router IDs, peers them over IPv6 loopback with ADD-PATH both ways and
all_paths, and feeds the first from four synthetic peers and the second
from two over IPv4 loopback (BENCH_PEERS, BENCH_PEERS_B,
BENCH_SCALE_DOWN). It prints how long each router takes to hold its
own peers' routes and then the other's, with the rate of the latter.
BENCH_DEADLINE prints the sessions' state if a router is not there in
time.

The benchmarks now allocate with jemalloc, as the daemon does, except
when memory is being counted: the C library's allocator put its own
locks in the profiles. Two routers on one host have each other's
addresses as next hops, loopback addresses, which the import check
refuses as it refuses a next hop that is this machine; the permissive
import policy, which only tests use and no configuration can ask for,
now takes them.

Testing: the next-hop test covers both policies; the benchmark runs on
macOS and in the Linux VM under perf.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:45:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d0f092ff06 session: write to the peer from a task of its own 
A session read from and wrote to its socket in one loop: while it was
writing, it was not reading. Writing waits when the peer's receive
window is closed, and the peer's window closes when the peer is not
reading, which it is not while it is writing to us and waiting on our
window in turn. Two routers sending each other every path at once, as
they do with ADD-PATH both ways after a restart, each stopped reading
within seconds and waited on the other until the send hold timer ended
both sessions eight minutes later. The status page's TCP row showed it:
both windows zero, the send queue full, the receive queue unread. The
batching of writes in the previous change made a session spend far
longer writing and so made this certain; it was possible before.

Now each session has a writer task. The session encodes its messages
into batches and hands them to the task through a bounded queue without
ever waiting: a batch the queue has no room for is kept, and while any
is kept the session takes no more messages from the rest of the server
(which then waits, as before, in its own task) but goes on reading the
peer. The writer task's failure, or a write that makes no progress for
the send hold time, comes back to the session as a local failure and
ends it. When the session ends, what is still to go out, a NOTIFICATION
among it, is handed over and the writer shuts our side once it has
written it.

Testing: a new integration test has one peer's table in the server and
a second peer send a table of its own while reading nothing of what the
server sends it, and checks that the server takes the second table all
the same; against the previous commit the test hangs until its timeout.
The send hold timer test now reads the writer's failure. The session
tests, the integration tests, the kernel tests and every lab phase
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 12:22:06 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ce1b68ca59 bgp: jemalloc's background thread on Linux only 
The last change built jemalloc with its background thread on, so that
freed pages go back to the kernel on time. jemalloc has that thread on
Linux only: on macOS every start of the daemon, and every test that
starts one, printed "option background_thread currently supports
pthread only".

The feature now comes from a Linux-only dependency entry of the daemon
crate, so the macOS build asks for nothing jemalloc cannot do. The
Linux build is as before: measured in the VM with four peers and a
collector, resident memory falls after the load without any setting
in the environment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 12:06:39 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9276bc9f7a session: write messages in batches, with TCP_NODELAY 
A session wrote every message to its socket as it came: one write and,
with Nagle's algorithm, mostly one segment for each UPDATE, a hundred or
two octets. On the IPv6 cross-connect between the two routers, whose
round trip is 50 microseconds, a million routes took 42 seconds to go
out: the status page's new TCP row showed the connection busy for that
long with neither window nor buffer limiting it, a congestion window
stuck at ten segments, and a smoothed round-trip time of 18 ms. That is
the receiver's delayed acknowledgements, 40 ms at a time, on a stream
of small segments, holding the sender to a few segments a round trip.

Now a session gathers what is waiting for it, up to the outbound queue's
length, in one wake, encodes the messages into one buffer, and writes
the buffer when it reaches 64 KiB or when the actions of the wake are
done; the socket has TCP_NODELAY, so a batch's last, partial segment is
not held back for the acknowledgement of the one before. The
NOTIFICATION that ends a session, including one sent after a panic, is
flushed on its own.

Measured in the Linux VM with four peers' tables going to a collector
over a veth pair, 151 MiB: 121,850 segments instead of 313,773, the
connection busy 0.8 s instead of 2.0 s, and the congestion window grown
to 47 segments instead of 13. Over loopback the transit benchmark is
unchanged at 2.5 s for the second neighbor to have the table: there the
receiver's parsing sets the pace, not the segments.

Testing: the session tests, including the one that sends a NOTIFICATION
after a panic; the integration tests; the kernel tests and every lab
phase.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 11:14:08 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b0dd357986 export: a kilobyte less a route advertised with all_paths; memory accounted for
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A router feeding a collector every path (all_paths) was 4.8 GB resident
with a table that should take a third of that. A heap profile of the
daemon in the Linux VM, with four full tables and such a collector,
showed where it went: the exporter kept, for every route it had
advertised, the route's outgoing form, the parsed attribute list it was
dressed with, through an `Arc<Dressed>` in the per-route record. One
such form lived for every distinct attribute set advertised, about a
kilobyte each, for the life of the session, on top of the exporter's
cache of 262,144 of them. The record needs only to tell whether a route
changed, and the table's own attribute set, which is in memory anyway,
tells it that: the outgoing form is a function of the set, the peer the
route came from and the policy.

This change:

* `Sent` holds an `Arc<AttrSet>` instead of an `Arc<Dressed>`, and a
  route is sent again when its set differs. `Verdict::All` carries both
  while a flush is on.
* `DRESSED_KEPT` goes from 262,144 to 65,536: a kilobyte a set, so 64 MB
  rather than 256 MB a session at most; what a flush is sending holds
  its own.
* The status page's header shows the daemon's memory as jemalloc
  accounts for it (allocated and resident), `/api/status` has the six
  figures (`memory`: allocated, active, metadata, resident, mapped,
  retained), and `bgp_memory_bytes{kind}` exports them. jemalloc is built
  with statistics for this, and `bgp-web` reads them through
  `tikv-jemalloc-ctl`.
* `scripts/profile/full-table.sh` gains `COLLECTOR=1`, a collector
  session over IPv6 that takes both families with ADD-PATH, and `HEAP=1`,
  which builds the daemon with jemalloc's heap profiler (the new
  `heap-profiling` feature of the daemon crate, not for production) and
  turns the profile into a flame graph with jeprof; and it reports
  memory with the table in and again 30 s later. The feeder
  (`crates/bench/examples/feed.rs`) plays the collector with
  `FEED_COLLECTOR=1`.

Measured in the VM, four peers' full tables and a collector sent every
path (4.9 million routes advertised): 1,723 MiB resident before, 1,286
MiB after, with jemalloc's allocated figure from 1,500 MiB to 946 MiB.
Without the collector the same tables take 630 MiB, so the collector's
bookkeeping is now about 50 bytes a route advertised. Of the resident
figure, some 400 MiB was pages jemalloc had freed and not yet returned:
it returns them on allocator activity only, and a router that has its
tables in allocates little. jemalloc is now built with its background
thread on (the `background_threads` feature), which returns them on
time: 1,020 MiB resident 30 s after the load, against 1,286 MiB without
it.

Documentation: `memory` in the HTTP API reference, `bgp_memory_bytes`
in the metrics reference and the monitoring guide, the script's new
settings in the testing guide.

Testing: the ADD-PATH integration tests pass, including the collector
ones; the heap profile and the figures above are from
`PEERS=4 COLLECTOR=1 HEAP=1 scripts/profile/full-table.sh` in the VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 22:51:11 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d7d7d39bb9 metrics: export each session's TCP connection figures
ci / docs (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The kernel's view of a session's TCP connection, which the status page
shows on request since the last change, could not be looked at after
the fact: a feed that crawled overnight left no record of whether the
peer's window or our own side held it up.

This change samples it at every scrape of `/metricz`: for each
established session, one `NETLINK_SOCK_DIAG` round trip, and twenty
gauges labelled by peer, `bgp_peer_tcp_*`: smoothed, least and variance
of the round-trip time; the congestion window and segment size; the
peer's receive window and ours; the send queue, the part of it not yet
sent, the receive queue and the segments in flight; retransmitted and
lost segments; the busy time and the time held up by the peer's window
and by our send buffer, cumulative, so that their rates give the share
of time a session is held up; octets sent and received; and the
delivery and pacing rates. A session the kernel has no connection for
loses its series, as does one that ends.

`SessionHandle::metric_label` is the one place a peer's metric label is
made (its name, or its address), used by the advertiser's gauge and here.

Documentation: the metrics reference lists them; the monitoring guide
gives the query for the share of time a session is held up by the
neighbor's receive window, which is the question the overnight feed
raised.

Testing: a daemon in the Linux VM with one session shows the series and
drops them when the session ends; the lab's basics phase checks the
scrape for the FRR session's round-trip time; the workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 12:12:53 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 99a47da2af status: a session's TCP connection as the kernel sees it; a route in full 
When a session advertised a full table to a route collector at two
thousand routes a second, nothing on the status page said whether the
daemon or the peer set that pace. The answer is in the kernel: the
peer's receive window, the queues, and how long the connection was held
up by the window or by the send buffer, which `ss -ti` shows.

This change asks the kernel the same way `ss` does, over the socket
diagnostics netlink family (`NETLINK_SOCK_DIAG`, `SOCK_DIAG_BY_FAMILY`
with the connection's addresses and ports), and reads `tcp_info` and
the congestion control name from the answer. No external program is
run and no unsafe code is needed: the request and the answer are byte
layouts of the kernel's ABI, read field by field, with fields an older
kernel does not send left at zero. A socket on a link-local address is
found by its interface, which the request names.

* `bgp_fib::sockdiag`: the request, the parser, `TcpConnection`, and
  `tcp_connection(local, peer)`; `Ok(None)` where there is no such
  connection or the system is not Linux.
* `GET /api/tcp?peer=ADDRESS` in the HTTP API: the connection of the
  session with that peer, with times in milliseconds, the two windows
  named by what they allow, and the held-up times also as shares of the
  busy time.
* The status page: a "tcp" link on each established session, which
  expands a row below it with the figures, re-asked on every refresh
  while open. The group rows' span is corrected for the "sent" column.

Also on the status page: the attributes of a received route that have
no column of their own were badges with their value in a tooltip, which
is hard to read. Clicking the attrs cell now expands a row under the
route with every attribute by its name (ORIGIN, AS_PATH, NEXT_HOP, MED,
LOCAL_PREF, COMMUNITIES, LARGE_COMMUNITIES, AGGREGATOR and the rest),
the path identifier, the selection rank and the RPKI and ASPA states;
the badges stay. Both expansions use the terms of `ss` and the RFCs
rather than sentences.

Documentation: the endpoint in the HTTP API reference, and the link and
the expansion in the monitoring guide.

Testing: unit tests of the request and of the parser on a reply built
field by field, and of a "no such socket" and a refused answer; a
Linux-only test that asks the kernel about a loopback connection of the
test's own and about one that does not exist; the lab's basics phase
checks the endpoint for the FRR session; the page was checked by hand
against the daemon in the Linux VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 11:41:47 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 efde277fd8 Test: both families reach a collector over one IPv6 session 
A collector at an exchange is fed over one IPv6 session that carries
both families, with ADD-PATH for both and IPv4 routes with IPv6 next
hops (RFC 8950). The integration test for that has the collector on
IPv6 loopback with those capabilities and two feeders sending 20,000
IPv4 and 1,000 IPv6 prefixes each, and checks that every route of both
families arrives with its identifier. The test helper now reads the
prefixes that come and go in the multiprotocol attributes too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:37:57 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9b211b61e9 export: count routes as they go out, and say when the table is walked again 
The number of routes advertised to a peer was written once a flush was
over, after every UPDATE of it had been handed to the session. The first
flush to a route collector is the whole table, two million routes or
more to a peer that reads at the pace of the Internet, so the status
page showed the peer holding one route for as long as that took, which
read as nothing going out.

Now the count is written after each batch of a flush, so that the page
and `bgp_peer_advertised_routes` show the table going out.

Also, an exporter that had the whole table and is given it again,
because the table changed faster than the exporter read it and the
journal let the oldest changes go, now logs that it is going through the
table again, at info. Until now that was invisible, and a collector
session during a full-table load from several peers is where it can
happen: the exporter is held up sending while the shards' journals,
cut back to twice their rows, run past its place.

Testing: a new integration test has a collector connected before two
feeders send 40,000 prefixes each, and checks it is told all 80,000
routes with their identifiers, which takes the exporter's initial walk
of the table through many batches; the advertiser and ADD-PATH tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:31:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ec5c9be454 export: advertise every route for a prefix with ADD-PATH (all_paths) 
Before this change, a session on which ADD-PATH (RFC 7911) was
negotiated in the sending direction was sent path identifiers, but only
the best route for each prefix. A route collector such as bgp.tools
wants every route a router has accepted, from every neighbor, and from
one session: without this the collector saw one path per prefix, nearly
always the same neighbor's.

This change adds `all_paths` to the export policy. On a session with it,
and ADD-PATH send negotiated for a family, every accepted route for a
prefix of that family is advertised, each with a path identifier of its
own; for a family without ADD-PATH send the best route goes as before.
`all_paths` needs `learned_routes` (a configuration error otherwise) and
the peer's `add_path` to offer sending. The export policy, the role
rules, the well-known communities and the export limits apply to every
route alike, and the export limits count every route.

Implementation, in the advertiser:

* An exporter with `all_paths` follows the table's journal of all routes
  (`RibTable::reader_of_all_routes`, until now unused) instead of the
  best-route journal, and is given all the routes of a prefix whenever
  one of them changes (`News::Routes`).
* What the peer holds is `Announced`: the set of prefixes for a
  best-route session, as before; or for an all-paths session, for each
  prefix the routes announced, keyed by the peer they came from and that
  peer's own identifier, with the identifier they went out with and the
  form they went out in. A route keeps its identifier for as long as it
  is announced, goes out again only if it changed, and is withdrawn by
  its identifier; a new route gets the lowest identifier not in use for
  the prefix, from 1.
* Verdicts carry either kind (`Verdict::Best`, `Verdict::All`), and a
  reset of the table puts what was announced in doubt until heard of
  again, as before, with the same identifiers kept across the reset.
* The record for an all-paths session is about 50 bytes a route, some
  tens of megabytes for a full table from several neighbors; the
  best-route bookkeeping is unchanged in size.

Also, so that a collector session can be watched: the number of routes
advertised to each peer and not withdrawn since, every route counted
where several are sent for a prefix. The exporter keeps it in the
session's counters after every change; it is `advertised_routes` in
`/api/peers`, the "sent" column of the status page's neighbor table,
and the gauge `bgp_peer_advertised_routes{peer}`.

Documentation: a "Feeding a Route Collector" section in the guide to
passing routes on, with the configuration for a collector; `all_paths`
in the configuration reference; the ADD-PATH sections of the neighbor
guide and the standards reference; the limitation that was removed;
the new field, column and metric in the HTTP API, monitoring and
metrics references.

Testing: an integration test with two feeders and a collector that
negotiated ADD-PATH receive, which checks the two routes with their
identifiers, the withdrawal of one by its identifier, a changed route
keeping its identifier and a new route taking the free one; a
configuration test for `all_paths` without `learned_routes`; a new
`collector` phase of the interoperability lab in which FRR, as a route
collector, is told both of BIRD's routes for a prefix from BIRD's two
sessions and holds them apart, and loses one when its session is
disabled, with the advertised count checked at each step. The workspace tests, the kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:16:48 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 03f4f58ecb rib: give a row room for every peer's route at its first growth 
A row's routes are in a vector that grew by one route at a time, so with
four peers sending full tables every row's vector was reallocated three
times as their routes arrived, and the reallocations were 7% of the
daemon's CPU time in a profile of a four-peer load.

This change has the first growth of a row take it to as many routes as
the table has peers, at most 8 (`FIRST_ROOM`), since a prefix that one
full table has, the others nearly always have too; after that it grows
by a quarter as before. A table with one peer never grows a row and is
unchanged. A route server's clients do not all announce every prefix,
hence the cap: a row of a prefix that fewer peers announce has at most
seven spare slots of 16 bytes.

Measured in the Linux VM with four peers sending the synthetic table of
1.23 million prefixes: the kernel is in step 1.9 s after the first route
instead of 2.2 s, the daemon's CPU time over the load is 5% lower, and
it is 604 MiB resident instead of about 630, since there are no
reallocation leftovers. With one peer nothing changes. Memory per route
in the benchmarks, allocations counted: 77 bytes before and after with
the synthetic table from four peers; 64 before and 65 after with the
RIPE RIS dump's eight largest peers, whose load went from 2.8 s to
1.8 s on one thread, the allocation counting making every reallocation
dearer than it is in the daemon.

Also tried and not adopted, with the new `JEMALLOC_CONF` setting of the
profiling script: transparent huge pages for the allocator
(`thp:always,metadata_thp:always`). Over three pairs of runs with one
peer the table arrived in 0.66, 1.17 and 0.70 s with them against 0.87,
0.75 and 0.87 s without, the CPU time was within the noise, and the
daemon was 60 to 100 MiB larger. An arena for the routes would not help
either: jemalloc's size classes already give rows exact 16, 32, 48 and
64 byte allocations, and the cost was the number of reallocations, not
the allocator.

Testing: the table tests, including the one that grows a row past
65,535 routes, pass; the kernel tests and the interoperability lab pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:34:23 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 48da8cc7d2 rib: size the routing table for the routes the peers are expected to send 
The routing table's shards started empty and grew as routes came in,
each growth moving every row of the shard into a table twice the size.
In a profile of the daemon taking in a full table, the growth showed as
up to a seventh of the session's time, and the daemon's resident memory
had the old and the new table of each shard side by side for a moment.

This change adds `RibTable::expect_prefixes`, which reserves room for
that many prefixes spread over the shards, and has the server call it at
startup with the largest `expected_routes` among the configured peers:
the prefixes of several full tables are mostly the same ones. Without
`expected_routes` nothing changes. The `full_table` benchmark does the
same unless `BENCH_PRESIZE=0`.

Measured, the saving in time is within the run-to-run noise: in the
Linux VM one peer's table is in the routing table in 0.81 s instead of
0.89 s, and the kernel is in step at about the same time as before;
three pairs of runs of the benchmark on a Mac average 0.33 s against
0.35 s. The profile's attribution to the growth was mostly sampling skid.
The memory saving is real: 442 MiB resident with one peer's table in
instead of 483 MiB, and 626 MiB instead of 632 MiB with four.

Also: the profiling script printed nothing for the kernel's change
counts because it asked `/metrics` for them; the endpoint is `/metricz`.
With that fixed, a four-peer load shows 1.25 million installs for 1.23
million prefixes: the journal already gives the kernel task only the
latest change of a prefix when it lags, so no further coalescing of
installs is called for.

Documentation: `expected_routes` in the configuration reference, the
neighbor guide and the proto comment say that the table is sized by it;
the testing guide's VM results are brought up to date.

Testing: a table test checks that the room is there and is kept; the
workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:13:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9e9e4f90ef fib: take best routes from the table 16384 at a time 
The kernel task read changes of best route from the routing table 1024
at a time. A poll of the table does two binary searches and takes a read
lock in every shard, however few events the shard has; with the 128
shards of an 8-core machine, that overhead was paid for every 8 events a
shard, and the profile of a full-table load showed 8% to 9% of the
daemon's CPU time in the reader's searches and locks.

This change raises the batch to 16384. In the Linux VM, with four peers
sending the synthetic table of 1.23 million prefixes, the kernel is in
step 2.2 s after the first route instead of 2.9 s, and the daemon's CPU
time over the load falls 13%; the reader's share drops from 9% to under
1%. With one peer the kernel-side share falls the same way and the CPU
time 3%. A batch holds up to 16384 small records between polls, which is
about a megabyte.

Testing: the fib unit tests pass; measured with
scripts/profile/full-table.sh with one and four peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 5915c8e562 profile: configure the peers and report the kernel's change counts 
The full-table profiling script took its peers as unconfigured ones under
the permissive policy, which is not how a router is set up. Now it writes
a peer block for each of them, with `expected_routes` set to the table's
size, so that the default import policy and anything that sizes itself by
the expected routes are in the measurement. The few bogon prefixes in the
synthetic table are refused by that policy, as the output says.

After the table is in, the script also prints
`bgp_kernel_route_changes_total`, so that a run shows how many installs
and removals the kernel was asked for and not only how many routes it
ended up with.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1bc71aaac5 fib, bench: profile the daemon taking in a full table; count blackholes instead of scanning 
Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.

This change:

* Keeps counts of blackhole routes and of routes kept unconfirmed from
  an earlier run in `Plan`, in step with the map through `Plan::put`
  and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
  cost nothing. `kept()` had the same scan; it ran only while routes of
  an earlier run were being kept, which is exactly during a full-table
  load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
  new `profiling` Cargo profile (release with line tables) and frame
  pointers, runs it in a network namespace joined by a veth pair to a
  second namespace, feeds it the synthetic table from there as one or
  more neighbors, samples it with `perf`, and writes a flame graph
  (inferno), flat profiles by self and inclusive time, the time until
  the routes are in the table and in the kernel, and the daemon's
  resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
  and `OUT` control it. From macOS it runs in the Linux VM through
  scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
  from the peers' addresses, completes the handshakes, writes each
  neighbor's table, drains what the daemon sends back and holds the
  sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
  connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
  "Profiling" section, and the daemon's numbers in the VM.

Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.

Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:03:43 +02:00
69 changed files with 7422 additions and 1573 deletions
Expand all files Collapse all files

No files matched your search

Cargo.lock
Generated
+31 -10
View File
@@ -157,7 +157,7 @@ checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]] [[package]]
name = "bgp" name = "bgp"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-server", "bgp-server",
"bgp-web", "bgp-web",
@@ -172,7 +172,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-bench" name = "bgp-bench"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-integration-tests", "bgp-integration-tests",
"bgp-packet", "bgp-packet",
@@ -182,12 +182,14 @@ dependencies = [
"bytes", "bytes",
"flate2", "flate2",
"stats_alloc", "stats_alloc",
"tikv-jemallocator",
"tokio", "tokio",
"tracing-subscriber",
] ]
[[package]] [[package]]
name = "bgp-fib" name = "bgp-fib"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-packet", "bgp-packet",
"bgp-primitives", "bgp-primitives",
@@ -205,7 +207,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-integration-tests" name = "bgp-integration-tests"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-fib", "bgp-fib",
"bgp-packet", "bgp-packet",
@@ -224,7 +226,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-packet" name = "bgp-packet"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-primitives", "bgp-primitives",
"bitfield", "bitfield",
@@ -240,7 +242,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-primitives" name = "bgp-primitives"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bytes", "bytes",
"nom", "nom",
@@ -252,7 +254,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-rib" name = "bgp-rib"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-packet", "bgp-packet",
"bgp-primitives", "bgp-primitives",
@@ -267,7 +269,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-rpki" name = "bgp-rpki"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-primitives", "bgp-primitives",
"bytes", "bytes",
@@ -280,7 +282,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-server" name = "bgp-server"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"bgp-fib", "bgp-fib",
"bgp-packet", "bgp-packet",
@@ -288,6 +290,7 @@ dependencies = [
"bgp-rib", "bgp-rib",
"bgp-rpki", "bgp-rpki",
"bytes", "bytes",
"foldhash",
"futures", "futures",
"nix", "nix",
"prometheus", "prometheus",
@@ -302,7 +305,7 @@ dependencies = [
[[package]] [[package]]
name = "bgp-web" name = "bgp-web"
version = "0.2.5" version = "0.2.6"
dependencies = [ dependencies = [
"axum", "axum",
"bgp-fib", "bgp-fib",
@@ -315,6 +318,7 @@ dependencies = [
"nix", "nix",
"prometheus", "prometheus",
"serde", "serde",
"tikv-jemalloc-ctl",
"tokio", "tokio",
"tokio-stream", "tokio-stream",
"tracing", "tracing",
@@ -1098,6 +1102,12 @@ dependencies = [
"windows-targets", "windows-targets",
] ]
[[package]]
name = "paste"
version = "1.0.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a"
[[package]] [[package]]
name = "percent-encoding" name = "percent-encoding"
version = "2.3.2" version = "2.3.2"
@@ -1679,6 +1689,17 @@ dependencies = [
"cfg-if", "cfg-if",
] ]
[[package]]
name = "tikv-jemalloc-ctl"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a184c43b8ab2f41df2733b55556e3f5f632f4aeaa205b1bb018f574b7f5f142"
dependencies = [
"libc",
"paste",
"tikv-jemalloc-sys",
]
[[package]] [[package]]
name = "tikv-jemalloc-sys" name = "tikv-jemalloc-sys"
version = "0.7.1+5.3.1-0-g81034ce1f1373e37dc865038e1bc8eeecf559ce8" version = "0.7.1+5.3.1-0-g81034ce1f1373e37dc865038e1bc8eeecf559ce8"
Cargo.toml
+12 -3
View File
@@ -12,7 +12,7 @@ edition = "2024"
homepage = "https://rayhaan.ch" homepage = "https://rayhaan.ch"
license = "Apache-2.0" license = "Apache-2.0"
repository = "https://git.rayhaan.net/rayhaan/ferrix" repository = "https://git.rayhaan.net/rayhaan/ferrix"
version = "0.2.5" version = "0.2.6"
# Every crate opts in with `[lints] workspace = true`. CI turns warnings into # Every crate opts in with `[lints] workspace = true`. CI turns warnings into
# errors, so a lint added here is enforced. # errors, so a lint added here is enforced.
@@ -63,5 +63,14 @@ hashbrown = "0.17"
axum = "0.8.9" axum = "0.8.9"
tokio-stream = { version = "0.1.19", features = ["sync"] } tokio-stream = { version = "0.1.19", features = ["sync"] }
reqwest = { version = "0.13.5", default-features = false, features = ["json"] } reqwest = { version = "0.13.5", default-features = false, features = ["json"] }
# The daemon's memory allocator; see bin/bgp/src/main.rs. # The daemon's memory allocator; see bin/bgp/src/main.rs. With its
tikv-jemallocator = "0.7.0" # statistics, which the status page and the metrics show.
tikv-jemallocator = { version = "0.7.0", features = ["stats"] }
tikv-jemalloc-ctl = { version = "0.7.0", features = ["stats"] }
# `cargo build --profile profiling`: the release build, with the line tables
# a profiler needs to name functions. See scripts/profile/.
[profile.profiling]
inherits = "release"
debug = "line-tables-only"
strip = "none"
bin/bgp/Cargo.toml
+11
View File
@@ -23,5 +23,16 @@ tracing-subscriber.workspace = true
[target.'cfg(not(target_env = "msvc"))'.dependencies] [target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator.workspace = true tikv-jemallocator.workspace = true
# jemalloc's background thread returns freed memory to the kernel on time
# (see the memory figures on the status page); jemalloc has it on Linux
# only, and warns at every start elsewhere if asked for it.
[target.'cfg(target_os = "linux")'.dependencies]
tikv-jemallocator = { workspace = true, features = ["background_threads"] }
[features]
# jemalloc's heap profiler, for scripts/profile/full-table.sh with
# HEAP=1: not for a production build, where it costs memory and time.
heap-profiling = ["tikv-jemallocator/profiling"]
[lints] [lints]
workspace = true workspace = true
crates/bench/Cargo.toml
+8
View File
@@ -16,11 +16,15 @@ bytes.workspace = true
flate2 = "1" flate2 = "1"
stats_alloc = { version = "0.1.10", optional = true } stats_alloc = { version = "0.1.10", optional = true }
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator.workspace = true
[dev-dependencies] [dev-dependencies]
bgp-integration-tests.workspace = true bgp-integration-tests.workspace = true
bgp-rib.workspace = true bgp-rib.workspace = true
bgp-server.workspace = true bgp-server.workspace = true
tokio.workspace = true tokio.workspace = true
tracing-subscriber.workspace = true
[features] [features]
# Counts every allocation, to report how much memory a table holds. The # Counts every allocation, to report how much memory a table holds. The
@@ -42,3 +46,7 @@ harness = false
[lints] [lints]
workspace = true workspace = true
[[bench]]
name = "mesh"
harness = false
crates/bench/benches/full_table.rs
+8 -18
View File
@@ -3,7 +3,8 @@
//! //!
//! Run with `cargo bench -p bgp-bench --bench full_table`. Set //! Run with `cargo bench -p bgp-bench --bench full_table`. Set
//! `BENCH_PEERS` (default 4) and `BENCH_SCALE_DOWN` (default 1; 10 makes the //! `BENCH_PEERS` (default 4) and `BENCH_SCALE_DOWN` (default 1; 10 makes the
//! table ten times smaller) to change the size. The table is synthetic (see //! table ten times smaller) to change the size. `BENCH_PRESIZE=0` leaves
//! the table to grow as it fills, as it does without `expected_routes`. The table is synthetic (see
//! the crate documentation); `real_table` runs on a real one. //! the crate documentation); `real_table` runs on a real one.
//! //!
//! This is a stopwatch around whole operations at full size, not a //! This is a stopwatch around whole operations at full size, not a
@@ -17,6 +18,7 @@ use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
use bgp_bench::memory::{Region, describe}; use bgp_bench::memory::{Region, describe};
use bgp_bench::report::{mib, per_second, row};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context}; use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context};
use bgp_integration_tests::test_peer::TestPeer; use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer; use bgp_integration_tests::test_server::TestServer;
@@ -25,23 +27,6 @@ use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, PeerInfo, RibTable}; use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
use bytes::BytesMut; use bytes::BytesMut;
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
}
fn main() { fn main() {
let peers = env("BENCH_PEERS", 4) as u32; let peers = env("BENCH_PEERS", 4) as u32;
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1)); let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
@@ -109,6 +94,8 @@ fn main() {
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 90) .handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 90)
.await .await
.expect("a session"); .expect("a session");
// The same bytes `table_stream` would make, from the messages
// already encoded for the decoding above.
let stream: Vec<u8> = messages.iter().flat_map(|m| m.iter().copied()).collect(); let stream: Vec<u8> = messages.iter().flat_map(|m| m.iter().copied()).collect();
senders.push((session, stream)); senders.push((session, stream));
} }
@@ -148,6 +135,9 @@ fn main() {
fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) { fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) {
let region = Region::begin(); let region = Region::begin();
let table = RibTable::new(threads * 16); let table = RibTable::new(threads * 16);
if env("BENCH_PRESIZE", 1) == 1 {
table.expect_prefixes(routes / wire.len());
}
let peers: Vec<PeerIndex> = (0..wire.len()) let peers: Vec<PeerIndex> = (0..wire.len())
.map(|peer| { .map(|peer| {
table table
crates/bench/benches/mesh.rs
+659
View File
@@ -0,0 +1,659 @@
//! Two routers of one AS, each taking in full tables from its own peers
//! and sending the other every path (ADD-PATH both ways, `all_paths`), as
//! a mesh of routers that each hold the network's whole view do. How
//! long until each router has the other's routes is what this measures:
//! the exporter reading the table's journal while sessions fill the
//! table, which nothing else exercises at this size.
//!
//! Then a new peer takes the first router's table, as a collector or a
//! router coming back does: how soon its first UPDATE arrives and how
//! long the table takes are measured too. It is a peer of the routers'
//! AS over IPv4, so what it gets is the first router's IPv4 routes from
//! its own peers: the other router's are not reflected to it, and IPv6
//! routes need an IPv6 session.
//!
//! Run with `cargo bench -p bgp-bench --bench mesh`. `BENCH_PEERS`
//! (default 4) feed the first router and `BENCH_PEERS_B` (default 2) the
//! second; `BENCH_SCALE_DOWN` shrinks the table. The routers peer over
//! IPv6 loopback and the feeders come over IPv4 loopback; the new peer
//! comes from 127.0.0.2, which Linux has and macOS does not, so that
//! part is skipped on macOS. With `BENCH_RECEIVER_EARLY=1` the new peer
//! comes as the feeders start instead, so that its exporter goes
//! through the table while the table is being filled, as a router's
//! do after a restart. `BENCH_COLLECTORS` (default 0) more peers of the
//! first router, from 127.0.1.0/24, are peers in the sense of RFC 9234
//! told of learned routes, and the feeders' routes are marked as having
//! come across from a peer, so none of them goes: as many exporters
//! following the table while it fills and passing nearly all of it
//! over, as a router with many peers at exchanges has.
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::report::row;
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, table_stream};
use bgp_integration_tests::peer_config;
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::attributes::PathAttribute;
use bgp_packet::message::msg_type;
use bgp_packet::open::SendReceive;
use bgp_server::config::PeerConfig;
use bgp_server::family::SUPPORTED;
use bgp_server::policy::{ExportPolicy, ImportPolicy, PeerRole};
use bgp_server::session::SessionRegistry;
use tokio::io::{AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::tcp::OwnedWriteHalf;
use tokio::task::JoinHandle;
/// An AS number that is in none of the synthetic paths.
const ASN: u32 = 4_200_000_000;
/// Where the peer that takes the whole table connects from.
const RECEIVER: Ipv4Addr = Ipv4Addr::new(127, 0, 0, 2);
/// Where the collectors connect from: this and the addresses after it.
fn collector_address(n: u32) -> Ipv4Addr {
Ipv4Addr::new(127, 0, 1, n as u8)
}
/// The collectors' AS numbers start here.
const COLLECTOR_ASN: u32 = 65000;
/// How often the tables are looked at while they fill.
const POLL: Duration = Duration::from_millis(5);
/// What the routers send each other, and the receiver: every path.
fn every_path() -> Arc<ExportPolicy> {
Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
})
}
fn free_port() -> u16 {
std::net::TcpListener::bind((Ipv6Addr::LOCALHOST, 0))
.expect("a port")
.local_addr()
.expect("an address")
.port()
}
/// A router of the mesh, listening for feeders on IPv4 loopback and for
/// the other router on IPv6 loopback at `port`, which it dials at
/// `other_port`, knowing it as `other_name`.
async fn router(
other_name: &'static str,
id: u8,
port: u16,
other_port: u16,
with_receiver: bool,
collectors: u32,
) -> TestServer {
TestServer::start_with(ASN, |c| {
// Two routers of one AS must not share a BGP identifier.
c.router_id = Ipv4Addr::new(192, 0, 2, id);
c.cluster_id = c.router_id;
c.listen_endpoints = vec![
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 0),
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), port),
];
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Both),
import: ImportPolicy::permissive(),
export: every_path(),
..peer_config(
other_name,
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), other_port),
ASN,
)
}];
if with_receiver {
c.peers.push(PeerConfig {
add_path: Some(SendReceive::Send),
export: every_path(),
..peer_config("receiver", SocketAddr::new(RECEIVER.into(), 179), ASN)
});
}
for n in 1..=collectors {
c.peers.push(PeerConfig {
role: Some(PeerRole::Peer),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(collector_address(n).into(), 179),
COLLECTOR_ASN + n,
)
});
}
c.connect_retry = Duration::from_millis(200);
})
.await
.expect("a router")
}
/// Connects `count` feeders to `router`, numbered from `first`, and
/// returns their sessions with the bytes each is to send.
async fn feeders(
router: &TestServer,
prefixes: &Prefixes,
shape: Shape,
first: u32,
count: u32,
marked: bool,
) -> Vec<(TestPeer, Vec<u8>)> {
let mut sessions = Vec::new();
for peer in first..first + count {
let stream = if marked {
// As having come across from a peer (RFC 9234): not for the
// collectors, which are peers too.
let mut updates = announcements(prefixes, shape, peer);
for update in &mut updates {
update
.path_attributes
.push(PathAttribute::OnlyToCustomer(64512 + peer));
}
encoded(&updates)
.iter()
.flat_map(|m| m.iter().copied())
.collect()
} else {
table_stream(prefixes, shape, peer)
};
let mut session = TestPeer::connect(router.addr).await.expect("a connection");
// No hold timer: a feeder has nothing to say after its table.
session
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 0)
.await
.expect("a session");
sessions.push((session, stream));
}
sessions
}
/// Every session of `registry`, one line each, for the panic when a
/// deadline passes: where the routes have got to, and what each session
/// was to be sent.
fn sessions_report(registry: &SessionRegistry) -> Vec<String> {
registry
.list()
.iter()
.map(|s| {
format!(
"{} {:?} export {:?} add_path_send {:?} routes {} rejected {} advertised {}",
s.peer,
s.name,
s.export,
s.add_path_send,
s.counters.routes(),
s.counters.rejected(),
s.counters.advertised()
)
})
.collect()
}
/// How big a run is, from the environment.
struct Settings {
peers_a: u32,
peers_b: u32,
shape: Shape,
collectors: u32,
}
impl Settings {
fn from_env() -> Self {
Self {
peers_a: env("BENCH_PEERS", 4) as u32,
peers_b: env("BENCH_PEERS_B", 2) as u32,
shape: Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1)),
collectors: env("BENCH_COLLECTORS", 0) as u32,
}
}
/// How many routes A's own feeders bring, and B's.
fn own_routes(&self) -> (usize, usize) {
let per_peer = self.shape.prefixes();
(
per_peer * self.peers_a as usize,
per_peer * self.peers_b as usize,
)
}
/// How many routes the receiver is to be sent: A's IPv4 routes from
/// its own peers.
fn receiver_routes(&self) -> usize {
self.shape.ipv4_prefixes * self.peers_a as usize
}
}
/// The two routers, up and peered, with the collectors' sessions once
/// they are connected.
struct Mesh {
settings: Settings,
prefixes: Prefixes,
a: TestServer,
b: TestServer,
/// The collectors' connections, drained, held until the end.
collectors: Vec<(OwnedWriteHalf, JoinHandle<()>)>,
}
/// The feeders sending, from when they began.
struct Feeding {
started: Instant,
/// Each hands its session back once its table is written.
sending: Vec<JoinHandle<TestPeer>>,
}
/// When each router had what: its own peers' routes, and the other's too.
struct Tables {
a_own: Duration,
b_own: Duration,
a_all: Duration,
b_all: Duration,
}
impl Mesh {
/// Starts the routers, with the collectors and receiver configured at
/// A, and waits for them to peer.
async fn start(settings: Settings, prefixes: Prefixes) -> Self {
let (port_a, port_b) = (free_port(), free_port());
let a = router("b", 1, port_a, port_b, true, settings.collectors).await;
let b = router("a", 2, port_b, port_a, false, 0).await;
a.wait_for_sessions(1).await.expect("the mesh session at A");
b.wait_for_sessions(1).await.expect("the mesh session at B");
Self {
settings,
prefixes,
a,
b,
collectors: Vec::new(),
}
}
/// Connects the collectors to A. They are there before the feeders, as
/// a router's peers are before its tables arrive after a restart. An
/// address that is not this machine's ends the connecting, with a
/// word.
async fn connect_collectors(&mut self) {
for n in 1..=self.settings.collectors {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(collector_address(n).into(), 0)) {
println!(
"collectors: skipped, {} is not ours ({e})",
collector_address(n)
);
break;
}
let stream = socket.connect(self.a.addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(COLLECTOR_ASN + n, Ipv4Addr::new(192, 0, 3, n as u8), 0)
.await
.expect("A takes a collector's session");
self.collectors.push(session.drain());
}
if !self.collectors.is_empty() {
self.a
.wait_for_sessions(1 + self.collectors.len())
.await
.expect("the collectors' sessions at A");
}
}
/// Connects the feeders to both routers and starts them sending.
async fn feed(&self) -> Feeding {
let Settings {
peers_a,
peers_b,
shape,
..
} = self.settings;
let marked = !self.collectors.is_empty();
let feed_a = feeders(&self.a, &self.prefixes, shape, 1, peers_a, marked).await;
let feed_b = feeders(&self.b, &self.prefixes, shape, 1 + peers_a, peers_b, false).await;
let started = Instant::now();
let sending = feed_a
.into_iter()
.chain(feed_b)
.map(|(mut session, stream)| {
tokio::spawn(async move {
session.send_raw(&stream).await.expect("the router reads");
session
})
})
.collect();
Feeding { started, sending }
}
/// The receiver taking A's table while it fills, if `BENCH_RECEIVER_EARLY`
/// asks for that: as the feeders start (1), or once A has its own
/// peers' routes and B's are still coming (2), as a router's sessions
/// come up after a restart while its peers' tables arrive.
fn early_receiver(&self, deadline: Duration) -> Option<JoinHandle<()>> {
let when = env("BENCH_RECEIVER_EARLY", 0);
if when == 0 {
return None;
}
let (own_a, _) = self.settings.own_routes();
let routes = self.settings.receiver_routes();
let (addr, registry, rib) = (
self.a.addr,
self.a.registry.clone(),
Arc::clone(&self.a.rib),
);
Some(tokio::spawn(async move {
while when == 2 && rib.len() < own_a {
tokio::time::sleep(POLL).await;
}
resync(addr, registry, routes, deadline).await
}))
}
/// Waits until both routers have every route, noting when each had
/// what; or panics once `deadline` has passed since `started`.
async fn wait_for_tables(&self, started: Instant, deadline: Duration) -> Tables {
let (own_a, own_b) = self.settings.own_routes();
let mut a_own = None;
let mut b_own = None;
let mut a_all = None;
let mut b_all = None;
let (a_all, b_all) = loop {
tokio::time::sleep(POLL).await;
let (at_a, at_b) = (self.a.rib.len(), self.b.rib.len());
let now = started.elapsed();
if a_own.is_none() && at_a >= own_a {
a_own = Some(now);
}
if b_own.is_none() && at_b >= own_b {
b_own = Some(now);
}
if a_all.is_none() && at_a >= own_a + own_b {
a_all = Some(now);
}
if b_all.is_none() && at_b >= own_a + own_b {
b_all = Some(now);
}
if let (Some(a_all), Some(b_all)) = (a_all, b_all) {
break (a_all, b_all);
}
if now > deadline {
panic!(
"not all routes arrived: A has {at_a} {:?}, B has {at_b} {:?}",
sessions_report(&self.a.registry),
sessions_report(&self.b.registry)
);
}
};
Tables {
a_own: a_own.unwrap_or_default(),
b_own: b_own.unwrap_or_default(),
a_all,
b_all,
}
}
/// One row for each of the four moments of `tables`, with the rate the
/// routes that made the moment came at.
fn report(&self, tables: &Tables) {
let (own_a, own_b) = self.settings.own_routes();
let rate = |when: Duration, routes: usize| {
format!("{:.2} M routes/s", routes as f64 / when.as_secs_f64() / 1e6)
};
row(
"A has its own peers' routes",
tables.a_own,
rate(tables.a_own, own_a),
);
row(
"B has its own peers' routes",
tables.b_own,
rate(tables.b_own, own_b),
);
row(
"A has B's routes too (B's exporter, A's session)",
tables.a_all,
rate(tables.a_all, own_b),
);
row(
"B has A's routes too (A's exporter, B's session)",
tables.b_all,
rate(tables.b_all, own_a),
);
}
}
impl Feeding {
/// Waits for every feeder to have written its table, and returns the
/// sessions, which are to stay up until the end: a feeder going would
/// have its routes withdrawn from both tables meanwhile.
async fn finished(self) -> Vec<TestPeer> {
let mut feeders = Vec::new();
for session in self.sending {
feeders.push(session.await.expect("a feeder is done"));
}
feeders
}
}
fn main() {
// The routers' logs, with `BENCH_LOG` as a `RUST_LOG` filter.
if let Ok(filter) = std::env::var("BENCH_LOG") {
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.init();
}
let settings = Settings::from_env();
let deadline = Duration::from_secs(env("BENCH_DEADLINE", 600) as u64);
println!(
"== Two routers of AS{ASN}, every path both ways: A fed by {} peers, B by {}, {} prefixes each",
settings.peers_a,
settings.peers_b,
settings.shape.prefixes()
);
let prefixes = Prefixes::generate(settings.shape);
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
let mut mesh = Mesh::start(settings, prefixes).await;
mesh.connect_collectors().await;
let early_receiver = mesh.early_receiver(deadline);
let feeding = mesh.feed().await;
let tables = mesh.wait_for_tables(feeding.started, deadline).await;
mesh.report(&tables);
let feeders = feeding.finished().await;
match early_receiver {
Some(receiver) => receiver.await.expect("the receiver is done"),
None => {
resync(
mesh.a.addr,
mesh.a.registry.clone(),
mesh.settings.receiver_routes(),
deadline,
)
.await
}
}
drop(feeders);
drop(mesh);
});
}
/// The new peer taking a router's table: counting the UPDATEs it is sent
/// while keeping the session alive.
struct Receiver {
started: Instant,
/// When the first UPDATE arrived, in microseconds after `started`; 0
/// until it has.
first: Arc<AtomicU64>,
updates: Arc<AtomicU64>,
counting: JoinHandle<()>,
keepalives: JoinHandle<()>,
}
impl Receiver {
/// A session with the router at `addr` from [`RECEIVER`], as a
/// collector of every path; or `None`, with a word, where that address
/// is not this machine's to connect from.
async fn connect(addr: SocketAddr) -> Option<TestPeer> {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(RECEIVER.into(), 0)) {
println!("A new peer taking A's routes: skipped, {RECEIVER} is not ours ({e})");
return None;
}
let stream = socket.connect(addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake_with(
ASN,
Ipv4Addr::new(192, 0, 2, 9),
90,
collector_capabilities(ASN, &SUPPORTED, false),
)
.await
.expect("A takes the receiver's session");
Some(session)
}
/// Starts counting what `session` is sent, from now. Every millionth
/// UPDATE is noted, as is a NOTIFICATION or the connection closing:
/// what went wrong, if the table never all arrives.
fn count_updates(session: TestPeer) -> Self {
let started = Instant::now();
let (reader, mut writer) = session.into_stream().into_split();
// KEEPALIVEs, so that the session outlives the hold time.
let keepalives = tokio::spawn(async move {
let keepalive = TestPeer::keepalive_bytes();
loop {
tokio::time::sleep(Duration::from_secs(20)).await;
if writer.write_all(&keepalive).await.is_err() {
break;
}
}
});
let first = Arc::new(AtomicU64::new(0));
let updates = Arc::new(AtomicU64::new(0));
let counting = tokio::spawn({
let (first, updates) = (Arc::clone(&first), Arc::clone(&updates));
async move {
let mut reader = BufReader::with_capacity(1 << 20, reader);
let mut header = [0u8; 19];
let mut body = vec![0u8; 4096];
while reader.read_exact(&mut header).await.is_ok() {
let length = usize::from(u16::from_be_bytes([header[16], header[17]]));
if reader.read_exact(&mut body[..length - 19]).await.is_err() {
break;
}
match header[18] {
msg_type::UPDATE => {
let _ = first.compare_exchange(
0,
started.elapsed().as_micros() as u64,
Ordering::Relaxed,
Ordering::Relaxed,
);
let n = updates.fetch_add(1, Ordering::Relaxed) + 1;
if n % 1_000_000 == 0 {
println!(
" the receiver has read {n} UPDATEs after {:.2} s",
started.elapsed().as_secs_f64()
);
}
}
msg_type::NOTIFICATION => println!(
"the receiver was sent a NOTIFICATION, error {} subcode {}, after {} UPDATEs",
body[0],
body[1],
updates.load(Ordering::Relaxed)
),
_ => {}
}
}
println!(
"the receiver's connection closed after {} UPDATEs",
updates.load(Ordering::Relaxed)
);
}
});
Self {
started,
first,
updates,
counting,
keepalives,
}
}
/// Waits until the router whose sessions are in `registry` has
/// advertised `routes` routes to the receiver, and returns how many it
/// has; or panics once `deadline` has passed.
async fn wait_for_advertised(
&self,
registry: &SessionRegistry,
routes: usize,
deadline: Duration,
) -> u64 {
loop {
tokio::time::sleep(POLL).await;
let advertised = registry
.list()
.iter()
.find(|s| s.peer.addr.ip() == RECEIVER)
.map_or(0, |s| s.counters.advertised());
if advertised as usize >= routes {
return advertised;
}
if self.started.elapsed() > deadline {
panic!(
"the receiver has {advertised} of {routes} routes and read {} UPDATEs; sessions at A: {:#?}",
self.updates.load(Ordering::Relaxed),
sessions_report(registry)
);
}
}
}
/// Prints the row for the table being all there, `advertised` routes
/// of it, and stops counting.
fn report(self, advertised: u64) {
let took = self.started.elapsed();
self.counting.abort();
self.keepalives.abort();
let first = Duration::from_micros(self.first.load(Ordering::Relaxed));
row(
"A new peer has A's IPv4 routes (A's exporter)",
took,
format!(
"{:.2} M routes/s first UPDATE after {:.0} ms, {} UPDATEs",
advertised as f64 / took.as_secs_f64() / 1e6,
first.as_secs_f64() * 1e3,
self.updates.load(Ordering::Relaxed),
),
);
}
}
/// A new peer takes the table of the router at `addr`, whose sessions
/// are in `registry`: `routes` routes with every path. How long until
/// the first UPDATE, and until all of it is sent.
async fn resync(addr: SocketAddr, registry: SessionRegistry, routes: usize, deadline: Duration) {
let Some(session) = Receiver::connect(addr).await else {
return;
};
let receiver = Receiver::count_updates(session);
let advertised = receiver
.wait_for_advertised(&registry, routes, deadline)
.await;
receiver.report(advertised);
}
crates/bench/benches/real_table.rs
+6 -16
View File
@@ -11,30 +11,20 @@ use std::net::Ipv4Addr;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::{Duration, Instant}; use std::time::Instant;
use bgp_bench::memory::Region; use bgp_bench::memory::Region;
use bgp_bench::mrt::{Dump, parse_attributes}; use bgp_bench::mrt::{Dump, parse_attributes};
use bgp_bench::report::{mib, per_second, row};
use bgp_packet::message::{NlriNextHop, PathAttribute}; use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{Event, PeerIndex, PeerInfo, RibTable}; use bgp_rib::table::{Event, PeerIndex, PeerInfo, RibTable};
/// Bytes allocated and not freed since `region` began; zero when memory /// Bytes allocated and not freed since `region` began; zero when memory
/// is not being counted (see `bgp_bench::memory`). /// is not being counted (see `bgp_bench::memory`), and zero too in the
fn retained(region: &Region) -> i64 { /// odd case of more freed than allocated, which is nothing held.
region.held().unwrap_or(0) fn retained(region: &Region) -> usize {
} usize::try_from(region.held().unwrap_or(0)).unwrap_or(0)
fn mib(bytes: i64) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
format!("{:.2} M/s", count as f64 / elapsed.as_secs_f64() / 1e6)
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
} }
fn dump_path() -> Option<PathBuf> { fn dump_path() -> Option<PathBuf> {
crates/bench/benches/transit.rs
+1
View File
@@ -42,6 +42,7 @@ fn run(
let server = TestServer::start_with(4_200_000_000, |config| { let server = TestServer::start_with(4_200_000_000, |config| {
config.export = Arc::new(ExportPolicy { config.export = Arc::new(ExportPolicy {
learned_routes: !matches!(passing, Passing::Not), learned_routes: !matches!(passing, Passing::Not),
all_paths: false,
marks: match passing { marks: match passing {
Passing::Dressed => Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(), Passing::Dressed => Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
_ => Marks::default(), _ => Marks::default(),
crates/bench/examples/feed.rs
+181
View File
@@ -0,0 +1,181 @@
//! Feeds the synthetic Internet routing table (see the crate documentation)
//! to a running daemon over TCP, as one or more peers would, and holds the
//! sessions open afterwards. This is the load for profiling the daemon as
//! it takes in a full table; `scripts/profile/full-table.sh` runs it under
//! `perf`.
//!
//! Settings, from the environment:
//!
//! - `FEED_TO`: where the daemon listens (default `127.0.0.1:1790`).
//! - `FEED_PEERS`: how many peers send the table (default 1).
//! - `FEED_SCALE_DOWN`: as `BENCH_SCALE_DOWN`; 10 makes the table ten
//! times smaller (default 1).
//! - `FEED_FROM`: the first three octets of an IPv4 /24, "192.0.2". Peer N
//! connects from .N, which is also the next hop of its IPv4 routes.
//! Without it, peers connect from wherever the system puts them.
//! - `FEED_ASN`: the first peer's AS number (default 64513); the others
//! follow on from it.
//! - `FEED_HOLD_SECS`: how long the sessions stay open after the table is
//! sent (default 30).
//! - `FEED_COLLECTOR=1`: before the peers, one more session, as a route
//! collector would have: over IPv6, to `FEED_COLLECTOR_TO` (default
//! `[::1]:1790`) from `FEED_COLLECTOR_FROM` if set, offering to receive
//! several routes for a prefix (ADD-PATH) for both families and IPv4
//! routes with IPv6 next hops, and reading and dropping whatever it is
//! sent. Its AS number is 212232.
//!
//! What is printed: when the sessions are up and how long the writes took.
//! The writes end when the kernel's socket buffers have taken the last
//! bytes, which is before the daemon has read them: when the table is in
//! is for the daemon's API to say.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::{Duration, Instant};
use bgp_bench::report::mib;
use bgp_bench::{Prefixes, Shape, env, table_stream};
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_primitives::constants::AfiSafi;
use tokio::io::AsyncWriteExt;
use tokio::net::TcpSocket;
use tokio::net::tcp::OwnedWriteHalf;
/// How often KEEPALIVEs go out while the sessions are held open: well
/// inside the 90 s hold time.
const KEEPALIVE_EVERY: Duration = Duration::from_secs(20);
#[tokio::main]
async fn main() {
let to: SocketAddr = std::env::var("FEED_TO")
.unwrap_or_else(|_| "127.0.0.1:1790".into())
.parse()
.expect("FEED_TO is an address and port");
let peers = env("FEED_PEERS", 1) as u32;
let shape = Shape::INTERNET.scaled_down(env("FEED_SCALE_DOWN", 1));
let from = std::env::var("FEED_FROM").ok();
let first_asn = env("FEED_ASN", 64513) as u32;
let hold = Duration::from_secs(env("FEED_HOLD_SECS", 30) as u64);
let started = Instant::now();
let prefixes = Prefixes::generate(shape);
let tables: Vec<Vec<u8>> = (1..=peers)
.map(|peer| table_stream(&prefixes, shape, peer))
.collect();
let routes = shape.prefixes() * peers as usize;
let bytes: usize = tables.iter().map(Vec::len).sum();
println!(
"table: {} IPv4 + {} IPv6 prefixes from each of {peers} peer(s): {routes} routes, {:.1} MiB on the wire, made in {:.2} s",
shape.ipv4_prefixes,
shape.ipv6_prefixes,
mib(bytes),
started.elapsed().as_secs_f64()
);
// The collector first, so that it is there as the table arrives, as a
// router's collector session is after a restart.
let collector = env("FEED_COLLECTOR", 0) == 1;
let mut draining = Vec::new();
let mut collector_writer = None;
if collector {
let to: SocketAddr = std::env::var("FEED_COLLECTOR_TO")
.unwrap_or_else(|_| "[::1]:1790".into())
.parse()
.expect("FEED_COLLECTOR_TO is an IPv6 address and port");
let socket = TcpSocket::new_v6().expect("a socket");
if let Ok(from) = std::env::var("FEED_COLLECTOR_FROM") {
let address: std::net::Ipv6Addr = from.parse().expect("an IPv6 address");
socket
.bind(SocketAddr::new(address.into(), 0))
.expect("the collector's address is ours to connect from");
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake_with(
212_232,
Ipv4Addr::new(10, 0, 0, 200),
90,
collector_capabilities(
212_232,
&[AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST],
true,
),
)
.await
.expect("the daemon takes the collector's session");
let (writer, sink) = session.drain();
draining.push(sink);
println!("collector session established");
collector_writer = Some(writer);
}
let mut sessions = Vec::new();
for peer in 1..=peers {
let socket = TcpSocket::new_v4().expect("a socket");
if let Some(from) = &from {
let address: Ipv4Addr = format!("{from}.{peer}")
.parse()
.expect("FEED_FROM is the first three octets of an IPv4 address");
socket
.bind(SocketAddr::new(address.into(), 0))
.expect("the peer's address is ours to connect from");
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(
first_asn + peer - 1,
Ipv4Addr::new(10, 0, 0, peer as u8),
90,
)
.await
.expect("the daemon takes the session");
// Whatever the daemon sends back, KEEPALIVEs as a rule, is read
// and dropped, so that the writer is never held up.
let (writer, sink) = session.drain();
draining.push(sink);
sessions.push(writer);
}
println!("{peers} session(s) established; sending");
let started = Instant::now();
let senders: Vec<_> = sessions
.into_iter()
.zip(tables)
.map(|(mut writer, table)| {
tokio::spawn(async move {
let started = Instant::now();
writer.write_all(&table).await.expect("the daemon reads");
(writer, started.elapsed())
})
})
.collect();
let mut writers: Vec<OwnedWriteHalf> = collector_writer.into_iter().collect();
for (index, sender) in senders.into_iter().enumerate() {
let (writer, took) = sender.await.expect("the sender runs");
println!(
"peer {}: table written in {:.3} s",
index + 1,
took.as_secs_f64()
);
writers.push(writer);
}
let elapsed = started.elapsed();
println!(
"all {routes} routes written in {:.3} s: {:.2} M routes/s into the sockets",
elapsed.as_secs_f64(),
routes as f64 / elapsed.as_secs_f64() / 1e6
);
let keepalive = TestPeer::keepalive_bytes();
let deadline = Instant::now() + hold;
while Instant::now() < deadline {
tokio::time::sleep(KEEPALIVE_EVERY.min(deadline - Instant::now())).await;
for writer in &mut writers {
// A session the daemon closed is left alone.
let _ = writer.write_all(&keepalive).await;
}
}
println!("sessions closed after {} s", hold.as_secs());
drop(draining);
}
crates/bench/examples/packing.rs
+71
View File
@@ -0,0 +1,71 @@
//! How many UPDATEs, and how many octets, a table takes when the routes
//! that share attributes are grouped only within slices of the table
//! taken in shard order, for a few slice sizes: what streaming a table
//! out in slices costs against grouping it whole.
use std::collections::HashMap;
use std::hash::{BuildHasher, RandomState};
use std::path::PathBuf;
use bgp_bench::mrt::Dump;
fn main() {
let path = std::env::var("BENCH_MRT")
.map(PathBuf::from)
.unwrap_or_else(|_| "crates/bench/data/rrc03-bview.20260915.0000.gz".into());
let dump = Dump::read(&path).expect("a readable dump");
let peers = dump.largest_peers(usize::MAX);
let full: Vec<usize> = peers
.iter()
.copied()
.filter(|&p| dump.routes[p].len() > 800_000)
.collect();
println!("{} peers with a full table", full.len());
let hasher = RandomState::new();
for (what, chosen) in [
("one full table", vec![full[0]]),
("every full table, every path", full.clone()),
] {
// (hash of prefix, attribute set, prefix octets) in shard order.
let mut routes: Vec<(u64, u32, usize)> = chosen
.iter()
.flat_map(|&p| dump.routes[p].iter())
.map(|(prefix, set)| {
(
hasher.hash_one(prefix),
*set,
1 + usize::from(prefix.length()).div_ceil(8) + 4,
)
})
.collect();
routes.sort_unstable_by_key(|r| r.0);
println!("== {what}: {} routes", routes.len());
for slice in [4_096usize, 16_384, 65_536, 262_144, usize::MAX] {
let mut updates = 0usize;
let mut octets = 0usize;
for chunk in routes.chunks(slice.min(routes.len())) {
let mut groups: HashMap<u32, usize> = HashMap::new();
for &(_, set, prefix_octets) in chunk {
*groups.entry(set).or_insert(0) += prefix_octets;
}
for (set, prefix_octets) in groups {
// One UPDATE per 4096 octets of prefixes, roughly.
let attrs = dump.attribute_sets[set as usize].len();
let messages = prefix_octets.div_ceil(4000 - attrs.min(3000)).max(1);
updates += messages;
octets += messages * (23 + attrs) + prefix_octets;
}
}
println!(
" slice {:>9}: {:>9} UPDATEs, {:>5.1} MB on the wire",
if slice == usize::MAX {
"whole".to_string()
} else {
slice.to_string()
},
updates,
octets as f64 / 1e6
);
}
}
}
crates/bench/src/lib.rs
+83
View File
@@ -16,6 +16,14 @@
pub mod memory; pub mod memory;
pub mod mrt; pub mod mrt;
pub mod report;
// The benchmarks allocate as the daemon does (bin/bgp/src/main.rs), with
// jemalloc; the C library's allocator would put its own locks in the
// profile. Not when memory is counted: that needs the counting one.
#[cfg(all(not(feature = "count-memory"), not(target_env = "msvc")))]
#[global_allocator]
static ALLOCATOR: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
use std::collections::HashSet; use std::collections::HashSet;
use std::net::{Ipv4Addr, Ipv6Addr}; use std::net::{Ipv4Addr, Ipv6Addr};
@@ -259,6 +267,16 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
updates updates
} }
/// The whole table as the peer numbered `peer` sends it, one stream of
/// bytes: [`announcements`], [`encoded`] and run together, for a session
/// that writes the table in one go.
pub fn table_stream(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<u8> {
encoded(&announcements(prefixes, shape, peer))
.iter()
.flat_map(|message| message.iter().copied())
.collect()
}
/// `updates` as they are on the wire, on a session with four-octet AS /// `updates` as they are on the wire, on a session with four-octet AS
/// numbers. /// numbers.
pub fn encoded(updates: &[UpdateMessage]) -> Vec<BytesMut> { pub fn encoded(updates: &[UpdateMessage]) -> Vec<BytesMut> {
@@ -293,3 +311,68 @@ pub fn env(name: &str, default: usize) -> usize {
.and_then(|v| v.parse().ok()) .and_then(|v| v.parse().ok())
.unwrap_or(default) .unwrap_or(default)
} }
#[cfg(test)]
mod tests {
use super::*;
/// A small table, enough for every prefix length and attribute to
/// come up.
fn shape() -> Shape {
Shape::INTERNET.scaled_down(1000)
}
/// The benchmarks are compared from one run to the next, so the table
/// they are fed must not change between runs: the generator is
/// seeded, and nothing in it depends on where things are in memory.
#[test]
fn the_table_is_the_same_every_run() {
let shape = shape();
let once = Prefixes::generate(shape);
let again = Prefixes::generate(shape);
assert_eq!(once.ipv4, again.ipv4);
assert_eq!(once.ipv6, again.ipv6);
assert_eq!(once.ipv4.len(), shape.ipv4_prefixes);
assert_eq!(once.ipv6.len(), shape.ipv6_prefixes);
for peer in [1, 2] {
assert_eq!(
table_stream(&once, shape, peer),
table_stream(&again, shape, peer)
);
}
assert_ne!(
table_stream(&once, shape, 1),
table_stream(&once, shape, 2),
"each peer has attributes of its own"
);
}
/// What a peer of the benchmarks sends is well formed, and says what
/// it was meant to: every UPDATE reads back as it was made.
#[test]
fn every_announcement_decodes_to_itself() {
let shape = shape();
let prefixes = Prefixes::generate(shape);
let updates = announcements(&prefixes, shape, 1);
let ctx = wire_context();
let announced: usize = updates
.iter()
.map(|update| {
update.announced_routes.len()
+ update
.path_attributes
.iter()
.map(|attribute| match attribute {
PathAttribute::MpReachNlri(reach) => reach.prefixes.len(),
_ => 0,
})
.sum::<usize>()
})
.sum();
assert_eq!(announced, shape.prefixes());
for (update, wire) in updates.iter().zip(encoded(&updates)) {
let decoded = Message::decode(&ctx, &wire).expect("well formed");
assert_eq!(decoded, Message::Update(update.clone()));
}
}
}
crates/bench/src/mrt.rs
+16 -26
View File
@@ -10,7 +10,7 @@ use std::io::{self, BufReader, Read};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr}; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::path::Path; use std::path::Path;
use bgp_packet::message::{NlriNextHop, PathAttribute}; use bgp_packet::message::{NlriNextHop, PathAttribute, attr_type, split_attribute};
use bgp_primitives::constants::IpVersion; use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
use flate2::read::MultiGzDecoder; use flate2::read::MultiGzDecoder;
@@ -21,7 +21,6 @@ const TABLE_DUMP_V2: u16 = 13;
const PEER_INDEX_TABLE: u16 = 1; const PEER_INDEX_TABLE: u16 = 1;
const RIB_IPV4_UNICAST: u16 = 2; const RIB_IPV4_UNICAST: u16 = 2;
const RIB_IPV6_UNICAST: u16 = 4; const RIB_IPV6_UNICAST: u16 = 4;
const MP_REACH_NLRI: u8 = 14;
/// A peer of the collector. /// A peer of the collector.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@@ -205,32 +204,23 @@ pub fn parse_attributes(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<N
let mut attributes = Vec::new(); let mut attributes = Vec::new();
let mut next_hop = None; let mut next_hop = None;
while !wire.is_empty() { while !wire.is_empty() {
if wire.get(1) == Some(&MP_REACH_NLRI) { let (raw, rest) = split_attribute(wire).ok()?;
let extended = wire[0] & 0x10 != 0; if raw.type_code == attr_type::MP_REACH_NLRI {
let (length, header) = if extended { // The value is the next hop's length and the next hop, and
( // nothing of the family, reserved octet and prefixes that
usize::from(u16::from_be_bytes([*wire.get(2)?, *wire.get(3)?])), // follow in an UPDATE, so the attribute's own parser would
4, // refuse it.
) let (length, address) = raw.value.split_first()?;
} else { if usize::from(*length) != address.len() {
(usize::from(*wire.get(2)?), 3) return None;
}
next_hop = Some(NlriNextHop::from_bytes(address)?);
} else {
let ([], attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()? else {
return None;
}; };
let value = wire.get(header..header + length)?; attributes.push(attribute);
let address = |bytes: &[u8]| Some(Ipv6Addr::from(<[u8; 16]>::try_from(bytes).ok()?));
next_hop = match value {
[4, a, b, c, d] => Some(NlriNextHop::Ipv4(Ipv4Addr::new(*a, *b, *c, *d))),
[16, rest @ ..] => Some(NlriNextHop::Ipv6(address(rest)?)),
[32, rest @ ..] if rest.len() == 32 => Some(NlriNextHop::Ipv6WithLl {
global: address(&rest[..16])?,
link_local: address(&rest[16..])?,
}),
_ => return None,
};
wire = &wire[header + length..];
continue;
} }
let (rest, attribute) = PathAttribute::from_wire(&ctx, wire).ok()?;
attributes.push(attribute);
wire = rest; wire = rest;
} }
Some((attributes, next_hop)) Some((attributes, next_hop))
crates/bench/src/report.rs
+31
View File
@@ -0,0 +1,31 @@
//! How the benchmarks print what they measured: one row a measurement,
//! what was measured in a column wide enough for the longest of them, the
//! time, and whatever else there is to say about it. The rows are what a
//! run is compared against, so their shape is settled here once.
use std::time::Duration;
/// How wide the column that says what was measured is. The widest row is
/// the mesh benchmark's, so that is what every benchmark lines up to.
pub const WIDTH: usize = 52;
/// `bytes` in mebibytes.
pub fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
/// `count` things in `elapsed`, as a rate: millions a second where there
/// are that many, thousands otherwise.
pub fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
/// Prints one row: `what` took `elapsed`, and `detail` says more.
pub fn row(what: &str, elapsed: Duration, detail: impl std::fmt::Display) {
println!("{what:<WIDTH$} {:>8.3} s {detail}", elapsed.as_secs_f64());
}
crates/fib/src/kernel.rs
+94 -49
View File
@@ -34,8 +34,11 @@ use crate::routes::KernelRoutes;
use crate::topology::Topology; use crate::topology::Topology;
use crate::{Fib, FibConfig, FibError, Stop}; use crate::{Fib, FibConfig, FibError, Stop};
/// How many best routes are taken from the routing table at a time. /// How many best routes are taken from the routing table at a time. A
const BATCH: usize = 1024; /// poll of the table costs two binary searches and a lock in every shard,
/// however few events it brings back; with a few hundred shards, a batch
/// of a thousand had that overhead for every eight events a shard.
const BATCH: usize = 16384;
/// How many requests are sent to the kernel in one write. /// How many requests are sent to the kernel in one write.
const WRITE_BATCH: usize = 512; const WRITE_BATCH: usize = 512;
@@ -495,9 +498,53 @@ struct Task {
/// them means looking at every route, and is not done once there are /// them means looking at every route, and is not done once there are
/// none to find. /// none to find.
keeping: bool, keeping: bool,
/// What `follow` takes out of the table, and what it makes of it. Kept
/// from one wake to the next so that their room is not found anew each
/// time the table has something, which is many times a second while
/// it fills.
news: Vec<News>,
ops: Vec<Op>,
} }
impl Task { impl Task {
/// A task that is to keep the kernel in step with `rib`, talking to it
/// over `control` and `resolution` and writing routes over `routes`
/// from a task of its own. Returned with it is where that task
/// reports what became of the routes.
fn new(
rib: Arc<RibTable>,
fib: Fib,
control: Socket,
routes: Socket,
resolution: Option<Socket>,
) -> (Self, mpsc::UnboundedReceiver<Report>) {
let (queue, batches) = mpsc::channel(QUEUE);
let (reports, heard) = mpsc::unbounded_channel();
let config = fib.config();
let task = Self {
config,
rib,
fib,
control,
resolution,
queue: Some(queue),
writer: tokio::spawn(write(routes, batches, reports)),
topology: Topology::new(config.resolution_table, config.resolve_via_default),
resolved: FastHashMap::default(),
plan: Plan::default(),
nexthops: config.nexthop_objects.then(FastHashMap::default),
owned: HashSet::new(),
awaiting: None,
marks: 0,
synced: false,
mended_at: None,
keeping: false,
news: Vec::with_capacity(BATCH),
ops: Vec::with_capacity(BATCH),
};
(task, heard)
}
/// Whether a route the kernel has is one of ours. /// Whether a route the kernel has is one of ours.
fn is_ours(&self, route: &Route) -> bool { fn is_ours(&self, route: &Route) -> bool {
route.protocol == PROTOCOL_BGP route.protocol == PROTOCOL_BGP
@@ -628,6 +675,19 @@ impl Task {
Ok(true) Ok(true)
} }
/// Learns what the kernel has that bears on our routes: the routes
/// that say the way to next hops, its nexthop objects, and the routes
/// of ours it has from before, which are kept for the peers to announce
/// again if `keep` says so. Done at the start, and again whenever what
/// the kernel has is no longer known.
async fn learn(&mut self, keep: bool) -> Result<(), FibError> {
self.learn_routes().await?;
let nexthops = self.kernel_nexthops().await?;
self.adopt_routes(&nexthops, keep).await?;
self.use_nexthops(&nexthops);
Ok(())
}
/// Finds the routes of ours that the kernel has from before. /// Finds the routes of ours that the kernel has from before.
async fn adopt_routes( async fn adopt_routes(
&mut self, &mut self,
@@ -748,17 +808,21 @@ impl Task {
Ok(Ok((change, route))) Ok(Ok((change, route)))
} }
/// Hands `ops` over to be written to the kernel, in order. With /// Hands `ops` over to be written to the kernel, in order, and leaves
/// `whole_table` they complete a whole table, and the kernel is in /// the vector empty for the caller to fill again. With `whole_table`
/// step once it has done them. /// they complete a whole table, and the kernel is in step once it has
async fn send(&mut self, ops: Vec<Op>, whole_table: bool) -> Result<(), FibError> { /// done them.
async fn send(&mut self, ops: &mut Vec<Op>, whole_table: bool) -> Result<(), FibError> {
let mut batch = Batch { let mut batch = Batch {
ops: Vec::with_capacity(ops.len()), ops: Vec::with_capacity(ops.len()),
requests: Vec::with_capacity(ops.len()), requests: Vec::with_capacity(ops.len()),
mark: None, mark: None,
}; };
let mut ops = std::collections::VecDeque::from(ops); // By place rather than by draining: what takes a refused op's
while let Some(op) = ops.pop_front() { // place is put at the end, and is dealt with in its turn.
let mut at = 0;
while let Some(op) = ops.get(at).copied() {
at += 1;
match self.request(&op).await? { match self.request(&op).await? {
Ok(request) => { Ok(request) => {
CHANGES CHANGES
@@ -774,6 +838,7 @@ impl Task {
Err(errno) => ops.extend(self.refused(&op, errno)?), Err(errno) => ops.extend(self.refused(&op, errno)?),
} }
} }
ops.clear();
if whole_table { if whole_table {
self.marks += 1; self.marks += 1;
batch.mark = Some(self.marks); batch.mark = Some(self.marks);
@@ -847,7 +912,12 @@ impl Task {
/// hands over what brings the kernel in step with them. Returns whether /// hands over what brings the kernel in step with them. Returns whether
/// the reader has read everything the table has for it. /// the reader has read everything the table has for it.
async fn follow(&mut self, reader: &mut Reader) -> Result<bool, FibError> { async fn follow(&mut self, reader: &mut Reader) -> Result<bool, FibError> {
let mut news = Vec::with_capacity(BATCH); // Taken out of `self` while they are worked through, since working
// through them is what the rest of `self` is for.
let mut news = std::mem::take(&mut self.news);
let mut ops = std::mem::take(&mut self.ops);
news.clear();
ops.clear();
// Only what is needed is taken while the table is held. // Only what is needed is taken while the table is held.
let progress = reader.poll(BATCH, |event| { let progress = reader.poll(BATCH, |event| {
news.push(match event { news.push(match event {
@@ -869,9 +939,8 @@ impl Task {
Event::Routes { .. } => return, Event::Routes { .. } => return,
}); });
}); });
let mut ops = Vec::with_capacity(news.len());
let mut whole_table = false; let mut whole_table = false;
for item in news { for item in news.drain(..) {
match item { match item {
News::Reset => { News::Reset => {
self.plan.reset(); self.plan.reset();
@@ -925,7 +994,10 @@ impl Task {
News::Withdrawn(prefix) => ops.extend(self.plan.withdrawn(prefix)), News::Withdrawn(prefix) => ops.extend(self.plan.withdrawn(prefix)),
} }
} }
self.send(ops, whole_table).await?; let sent = self.send(&mut ops, whole_table).await;
self.news = news;
self.ops = ops;
sent?;
Ok(progress.caught_up) Ok(progress.caught_up)
} }
@@ -954,7 +1026,7 @@ impl Task {
} }
Report::Refused { op, errno } => { Report::Refused { op, errno } => {
let instead = self.refused(&op, errno)?; let instead = self.refused(&op, errno)?;
self.send(instead.into_iter().collect(), false).await?; self.send(&mut instead.into_iter().collect(), false).await?;
// A refused route may have named a nexthop object that // A refused route may have named a nexthop object that
// went away. Working the routes out again mends that. // went away. Working the routes out again mends that.
let due = matches!(op, Op::Install { .. }) let due = matches!(op, Op::Install { .. })
@@ -1023,18 +1095,14 @@ impl Task {
self.flush(heard).await?; self.flush(heard).await?;
self.synced = false; self.synced = false;
self.plan = Plan::default(); self.plan = Plan::default();
self.learn_routes().await?;
let nexthops = self.kernel_nexthops().await?;
// While routes of an earlier run are being kept for the peers to // While routes of an earlier run are being kept for the peers to
// announce again, they are kept through this too. // announce again, they are kept through this too.
self.adopt_routes(&nexthops, self.keeping).await?; self.learn(self.keeping).await
self.use_nexthops(&nexthops);
Ok(())
} }
/// Waits until the kernel has done everything handed over so far. /// Waits until the kernel has done everything handed over so far.
async fn flush(&mut self, heard: &mut mpsc::UnboundedReceiver<Report>) -> Result<(), FibError> { async fn flush(&mut self, heard: &mut mpsc::UnboundedReceiver<Report>) -> Result<(), FibError> {
self.send(vec![], true).await?; self.send(&mut Vec::new(), true).await?;
while let Some(mark) = self.awaiting { while let Some(mark) = self.awaiting {
match heard.recv().await { match heard.recv().await {
Some(Report::Done(done)) if done == mark => self.awaiting = None, Some(Report::Done(done)) if done == mark => self.awaiting = None,
@@ -1087,8 +1155,8 @@ impl Task {
self.plan = Plan::default(); self.plan = Plan::default();
self.adopt_routes(&HashMap::new(), false).await?; self.adopt_routes(&HashMap::new(), false).await?;
} }
let ops = self.plan.clear(); let mut ops = self.plan.clear();
self.send(ops, false).await?; self.send(&mut ops, false).await?;
self.finish_writer().await self.finish_writer().await
} }
@@ -1150,38 +1218,15 @@ pub(crate) async fn run(
resolution, resolution,
mut hints, mut hints,
} = sockets; } = sockets;
let (queue, batches) = mpsc::channel(QUEUE); let (mut task, mut heard) = Task::new(rib, fib, control, routes, resolution);
let (reports, mut heard) = mpsc::unbounded_channel(); let config = task.config;
let config = fib.config();
let mut task = Task {
config,
rib,
fib,
control,
resolution,
queue: Some(queue),
writer: tokio::spawn(write(routes, batches, reports)),
topology: Topology::new(config.resolution_table, config.resolve_via_default),
resolved: FastHashMap::default(),
plan: Plan::default(),
nexthops: config.nexthop_objects.then(FastHashMap::default),
owned: HashSet::new(),
awaiting: None,
marks: 0,
synced: false,
mended_at: None,
keeping: false,
};
// The socket for news was bound before this: what changes while the // The socket for news was bound before this: what changes while the
// interfaces are being learned is told there. // interfaces are being learned is told there.
task.learn_interfaces().await?; task.learn_interfaces().await?;
task.learn_routes().await?;
let nexthops = task.kernel_nexthops().await?;
// Kept for a while, if there is to be any keeping: our table is empty // Kept for a while, if there is to be any keeping: our table is empty
// until the peers have sent theirs again. // until the peers have sent theirs again.
let keep = !config.stale_time.is_zero(); let keep = !config.stale_time.is_zero();
task.adopt_routes(&nexthops, keep).await?; task.learn(keep).await?;
task.use_nexthops(&nexthops);
let mut release_at = None; let mut release_at = None;
if task.plan.installed() > 0 { if task.plan.installed() > 0 {
info!( info!(
@@ -1281,12 +1326,12 @@ pub(crate) async fn run(
// taken out only to be put back. // taken out only to be put back.
while !task.follow(&mut reader).await? {} while !task.follow(&mut reader).await? {}
task.keeping = false; task.keeping = false;
let stale = task.plan.release(); let mut stale = task.plan.release();
info!( info!(
removed = stale.len(), removed = stale.len(),
"Routes of an earlier run that were not announced again are removed" "Routes of an earlier run that were not announced again are removed"
); );
task.send(stale, false).await?; task.send(&mut stale, false).await?;
task.publish(false); task.publish(false);
} }
ended = &mut task.writer => return Err(writer_error(ended)), ended = &mut task.writer => return Err(writer_error(ended)),
crates/fib/src/lib.rs
+1
View File
@@ -43,6 +43,7 @@ mod kernel;
pub mod netlink; pub mod netlink;
pub mod plan; pub mod plan;
pub mod routes; pub mod routes;
pub mod sockdiag;
pub mod topology; pub mod topology;
use std::net::IpAddr; use std::net::IpAddr;
crates/fib/src/netlink.rs
+22 -9
View File
@@ -29,9 +29,13 @@ pub const KIND_UNICAST: u8 = 1;
/// `RTN_BLACKHOLE`: a route that discards the packets for its prefix. /// `RTN_BLACKHOLE`: a route that discards the packets for its prefix.
pub const KIND_BLACKHOLE: u8 = 6; pub const KIND_BLACKHOLE: u8 = 6;
const HEADER_LEN: usize = 16; /// The netlink header: length, type, flags, sequence number and the
/// sender's port, four octets each but for the two-octet type and flags.
/// The socket diagnostics family (`sockdiag`) frames its messages the
/// same way.
pub(crate) const HEADER_LEN: usize = 16;
mod message_type { pub(crate) mod message_type {
pub const ERROR: u16 = 2; pub const ERROR: u16 = 2;
pub const DONE: u16 = 3; pub const DONE: u16 = 3;
pub const NEW_LINK: u16 = 16; pub const NEW_LINK: u16 = 16;
@@ -48,7 +52,7 @@ mod message_type {
pub const GET_NEXTHOP: u16 = 106; pub const GET_NEXTHOP: u16 = 106;
} }
mod flag { pub(crate) mod flag {
pub const REQUEST: u16 = 0x1; pub const REQUEST: u16 = 0x1;
pub const ACK: u16 = 0x4; pub const ACK: u16 = 0x4;
/// Send the sender what everyone else is told of the change. /// Send the sender what everyone else is told of the change.
@@ -76,8 +80,8 @@ pub mod group {
/// joined once the socket is bound. /// joined once the socket is bound.
pub const GROUP_NEXTHOP: u32 = 32; pub const GROUP_NEXTHOP: u32 = 32;
const AF_INET: u8 = 2; pub(crate) const AF_INET: u8 = 2;
const AF_INET6: u8 = 10; pub(crate) const AF_INET6: u8 = 10;
/// `RT_TABLE_COMPAT`: the table's number is in an attribute, being too /// `RT_TABLE_COMPAT`: the table's number is in an attribute, being too
/// large for the octet the fixed part has for it. /// large for the octet the fixed part has for it.
@@ -219,22 +223,31 @@ pub enum NetlinkError {
Truncated { message_type: u16 }, Truncated { message_type: u16 },
} }
fn ne_u16(bytes: &[u8]) -> u16 { // The kernel writes its numbers in the machine's byte order. Each reader
// takes the first octets of what it is given, so that a caller can pass
// the rest of a buffer rather than cut the field out first.
pub(crate) fn ne_u16(bytes: &[u8]) -> u16 {
u16::from_ne_bytes([bytes[0], bytes[1]]) u16::from_ne_bytes([bytes[0], bytes[1]])
} }
fn ne_u32(bytes: &[u8]) -> u32 { pub(crate) fn ne_u32(bytes: &[u8]) -> u32 {
u32::from_ne_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]) u32::from_ne_bytes([bytes[0], bytes[1], bytes[2], bytes[3]])
} }
pub(crate) fn ne_u64(bytes: &[u8]) -> u64 {
u64::from_ne_bytes([
bytes[0], bytes[1], bytes[2], bytes[3], bytes[4], bytes[5], bytes[6], bytes[7],
])
}
/// Rounds a length up to the four octets everything is aligned to. /// Rounds a length up to the four octets everything is aligned to.
fn aligned(length: usize) -> usize { pub(crate) fn aligned(length: usize) -> usize {
(length + 3) & !3 (length + 3) & !3
} }
/// The attributes in `bytes`: each its type and value. Stops at what is /// The attributes in `bytes`: each its type and value. Stops at what is
/// not an attribute. /// not an attribute.
fn attributes(mut bytes: &[u8]) -> impl Iterator<Item = (u16, &[u8])> { pub(crate) fn attributes(mut bytes: &[u8]) -> impl Iterator<Item = (u16, &[u8])> {
std::iter::from_fn(move || { std::iter::from_fn(move || {
let header = bytes.first_chunk::<4>()?; let header = bytes.first_chunk::<4>()?;
let length = usize::from(ne_u16(header)); let length = usize::from(ne_u16(header));
crates/fib/src/plan.rs
+122 -22
View File
@@ -317,6 +317,14 @@ pub struct Plan {
unresolved_counts: [usize; Unresolved::ALL.len()], unresolved_counts: [usize; Unresolved::ALL.len()],
/// Prefixes whose route the kernel refused. /// Prefixes whose route the kernel refused.
refused: HashSet<IpPrefix>, refused: HashSet<IpPrefix>,
/// How many entries of `installed` are blackhole routes, and how many
/// are kept from an earlier run without having been confirmed. Kept in
/// step with the map by [`Plan::put`] and [`Plan::take`], so that
/// asking costs nothing however large the table: the kernel task asks
/// after every batch of changes, and a scan of a million entries each
/// time was a third of its work.
blackholes: usize,
kept_unconfirmed: usize,
} }
impl Plan { impl Plan {
@@ -327,10 +335,38 @@ impl Plan {
/// How many of the routes the kernel has drop their packets. /// How many of the routes the kernel has drop their packets.
pub fn blackholes(&self) -> usize { pub fn blackholes(&self) -> usize {
self.installed self.blackholes
.values() }
.filter(|entry| entry.target == Some(Target::Discard))
.count() /// What `entry` counts for: one if it is a blackhole route, and one if
/// it is kept from an earlier run without having been confirmed.
fn weigh(entry: &Entry) -> (usize, usize) {
(
usize::from(entry.target == Some(Target::Discard)),
usize::from(entry.kept && !entry.confirmed),
)
}
/// Puts `entry` in as the route of `prefix`, in place of what was
/// there, keeping the counts in step.
fn put(&mut self, prefix: IpPrefix, entry: Entry) {
let (blackhole, kept) = Self::weigh(&entry);
self.blackholes += blackhole;
self.kept_unconfirmed += kept;
if let Some(old) = self.installed.insert(prefix, entry) {
let (blackhole, kept) = Self::weigh(&old);
self.blackholes -= blackhole;
self.kept_unconfirmed -= kept;
}
}
/// Takes the route of `prefix` out, keeping the counts in step.
fn take(&mut self, prefix: &IpPrefix) -> Option<Entry> {
let old = self.installed.remove(prefix)?;
let (blackhole, kept) = Self::weigh(&old);
self.blackholes -= blackhole;
self.kept_unconfirmed -= kept;
Some(old)
} }
/// How many prefixes have a best route that could not be given to the /// How many prefixes have a best route that could not be given to the
@@ -383,7 +419,7 @@ impl Plan {
/// have sent theirs again, and to remove everything meanwhile would /// have sent theirs again, and to remove everything meanwhile would
/// stop the machine forwarding for as long as that takes. /// stop the machine forwarding for as long as that takes.
pub fn adopt(&mut self, prefix: IpPrefix, target: Option<Target>, keep: bool) { pub fn adopt(&mut self, prefix: IpPrefix, target: Option<Target>, keep: bool) {
self.installed.insert( self.put(
prefix, prefix,
Entry { Entry {
target, target,
@@ -396,32 +432,36 @@ impl Plan {
/// How many routes are being kept from an earlier run without the /// How many routes are being kept from an earlier run without the
/// routing table having confirmed them. /// routing table having confirmed them.
pub fn kept(&self) -> usize { pub fn kept(&self) -> usize {
self.installed self.kept_unconfirmed
.values()
.filter(|entry| entry.kept && !entry.confirmed)
.count()
} }
/// The time for keeping unconfirmed routes of an earlier run is over: /// The time for keeping unconfirmed routes of an earlier run is over:
/// those the routing table still does not have go. /// those the routing table still does not have go.
pub fn release(&mut self) -> Vec<Op> { pub fn release(&mut self) -> Vec<Op> {
let mut stale = Vec::new(); let mut stale = Vec::new();
let mut blackholes_gone = 0;
self.installed.retain(|prefix, entry| { self.installed.retain(|prefix, entry| {
let goes = entry.kept && !entry.confirmed; let goes = entry.kept && !entry.confirmed;
entry.kept = false; entry.kept = false;
if goes { if goes {
stale.push(Op::Remove { prefix: *prefix }); stale.push(Op::Remove { prefix: *prefix });
blackholes_gone += usize::from(entry.target == Some(Target::Discard));
} }
!goes !goes
}); });
// What stays was confirmed, and is now not kept either.
self.kept_unconfirmed = 0;
self.blackholes -= blackholes_gone;
stale stale
} }
/// The whole table follows: nothing is known to be wanted until it is /// The whole table follows: nothing is known to be wanted until it is
/// heard of again. /// heard of again.
pub fn reset(&mut self) { pub fn reset(&mut self) {
self.kept_unconfirmed = 0;
for entry in self.installed.values_mut() { for entry in self.installed.values_mut() {
entry.confirmed = false; entry.confirmed = false;
self.kept_unconfirmed += usize::from(entry.kept);
} }
self.forget_unresolved_and_refused(); self.forget_unresolved_and_refused();
} }
@@ -442,20 +482,26 @@ impl Plan {
confirmed: true, confirmed: true,
kept: false, kept: false,
}); });
let before = Self::weigh(entry);
entry.confirmed = true; entry.confirmed = true;
if entry.target == Some(target) { let op = if entry.target == Some(target) {
return None; None
} } else {
entry.target = Some(target); entry.target = Some(target);
Some(Op::Install { prefix, target }) Some(Op::Install { prefix, target })
};
let after = Self::weigh(entry);
self.blackholes += after.0;
self.blackholes -= before.0;
self.kept_unconfirmed += after.1;
self.kept_unconfirmed -= before.1;
op
} }
Err(reason) => { Err(reason) => {
self.set_unresolved(prefix, Some(reason)); self.set_unresolved(prefix, Some(reason));
// Packets must not go on following a route that is no // Packets must not go on following a route that is no
// longer the best. // longer the best.
self.installed self.take(&prefix).map(|_| Op::Remove { prefix })
.remove(&prefix)
.map(|_| Op::Remove { prefix })
} }
} }
} }
@@ -464,9 +510,7 @@ impl Plan {
pub fn withdrawn(&mut self, prefix: IpPrefix) -> Option<Op> { pub fn withdrawn(&mut self, prefix: IpPrefix) -> Option<Op> {
self.set_unresolved(prefix, None); self.set_unresolved(prefix, None);
self.refused.remove(&prefix); self.refused.remove(&prefix);
self.installed self.take(&prefix).map(|_| Op::Remove { prefix })
.remove(&prefix)
.map(|_| Op::Remove { prefix })
} }
/// The whole table has been given: what was not in it goes. /// The whole table has been given: what was not in it goes.
@@ -480,7 +524,7 @@ impl Plan {
stale stale
.into_iter() .into_iter()
.map(|prefix| { .map(|prefix| {
self.installed.remove(&prefix); self.take(&prefix);
Op::Remove { prefix } Op::Remove { prefix }
}) })
.collect() .collect()
@@ -495,7 +539,7 @@ impl Plan {
if self.installed.get(prefix)?.target != Some(*target) { if self.installed.get(prefix)?.target != Some(*target) {
return None; return None;
} }
self.installed.remove(prefix); self.take(prefix);
self.refused.insert(*prefix); self.refused.insert(*prefix);
// What the kernel refused to replace, it still has. // What the kernel refused to replace, it still has.
Some(Op::Remove { prefix: *prefix }) Some(Op::Remove { prefix: *prefix })
@@ -508,6 +552,8 @@ impl Plan {
/// Everything goes: the changes that empty the kernel of our routes. /// Everything goes: the changes that empty the kernel of our routes.
pub fn clear(&mut self) -> Vec<Op> { pub fn clear(&mut self) -> Vec<Op> {
self.forget_unresolved_and_refused(); self.forget_unresolved_and_refused();
self.blackholes = 0;
self.kept_unconfirmed = 0;
self.installed self.installed
.drain() .drain()
.map(|(prefix, _)| Op::Remove { prefix }) .map(|(prefix, _)| Op::Remove { prefix })
@@ -1163,6 +1209,60 @@ mod tests {
] ]
} }
/// What the counts would be if the entries were scanned.
fn scanned(plan: &Plan) -> (usize, usize) {
plan.installed
.values()
.map(Plan::weigh)
.fold((0, 0), |(b, k), (eb, ek)| (b + eb, k + ek))
}
#[test]
fn the_counts_keep_step_with_the_routes() {
let mut plan = Plan::default();
let p = |n: u8| prefix(&format!("203.0.113.{n}/32"));
let via = target("192.0.2.2", 2);
let check = |plan: &Plan| assert_eq!((plan.blackholes(), plan.kept()), scanned(plan));
// Adopted from an earlier run: two kept, one a blackhole.
plan.adopt(p(1), Some(Target::Discard), true);
plan.adopt(p(2), Some(via), true);
plan.adopt(p(3), Some(via), false);
check(&plan);
// Heard of again, as a blackhole and as a route by way of a
// gateway; one changes kind.
plan.best(p(1), Ok(Target::Discard));
plan.best(p(2), Ok(Target::Discard));
plan.best(p(4), Ok(Target::Discard));
check(&plan);
plan.best(p(4), Ok(via));
check(&plan);
// The whole table again: everything unconfirmed, then confirmed.
plan.reset();
check(&plan);
plan.best(p(1), Ok(Target::Discard));
plan.synced();
check(&plan);
plan.reset();
plan.release();
check(&plan);
// Gone for want of a next hop, withdrawn, refused, adopted over.
plan.best(p(1), Err(Unresolved::Missing));
plan.best(p(5), Ok(Target::Discard));
plan.withdrawn(p(5));
plan.best(p(6), Ok(Target::Discard));
plan.refused_by_kernel(&Op::Install {
prefix: p(6),
target: Target::Discard,
});
plan.adopt(p(7), Some(Target::Discard), true);
plan.adopt(p(7), Some(via), true);
check(&plan);
let had = plan.installed();
assert_eq!(plan.clear().len(), had);
assert_eq!((plan.blackholes(), plan.kept()), (0, 0));
}
proptest! { proptest! {
/// Whatever happens to the best routes, and whatever an earlier /// Whatever happens to the best routes, and whatever an earlier
/// run left behind, a kernel that does as it is asked ends up with /// run left behind, a kernel that does as it is asked ends up with
crates/fib/src/sockdiag.rs
+420
View File
@@ -0,0 +1,420 @@
//! What the kernel knows about one of our TCP connections: the round-trip
//! time, the windows, the queues and what has held the connection up,
//! asked over the socket diagnostics netlink family (`NETLINK_SOCK_DIAG`,
//! as `ss -ti` asks). For the status page, when someone wants to know
//! why a session is slow.
use std::io;
use std::net::{IpAddr, SocketAddr};
use crate::netlink::{
AF_INET, AF_INET6, HEADER_LEN, attributes, flag, message_type, ne_u16, ne_u32, ne_u64,
};
/// A TCP connection as the kernel sees it. Times are in microseconds,
/// sizes in octets, rates in octets per second, unless said otherwise.
/// Fields the running kernel does not report are zero.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct TcpConnection {
/// The TCP state, by its number (1 is ESTABLISHED); see
/// [`TcpConnection::state_name`].
pub state: u8,
/// The congestion control algorithm, if the kernel named it.
pub congestion: Option<String>,
/// Octets received and not yet read by us.
pub recv_queue: u32,
/// Octets written by us and not yet acknowledged by the peer.
pub send_queue: u32,
pub rtt: u32,
pub rtt_variance: u32,
pub min_rtt: u32,
pub rto: u32,
pub mss: u32,
pub pmtu: u32,
/// The congestion window, in segments.
pub cwnd: u32,
/// The slow start threshold, in segments; `None` if there is none.
pub ssthresh: Option<u32>,
/// What the peer lets us send: its receive window.
pub send_window: u32,
/// What we let the peer send.
pub receive_window: u32,
pub receive_space: u32,
pub receive_ssthresh: u32,
pub send_window_scale: u8,
pub receive_window_scale: u8,
/// Segments sent and not acknowledged.
pub unacked: u32,
/// Segments retransmitted and still out.
pub retrans: u32,
pub lost: u32,
pub total_retrans: u32,
pub bytes_retrans: u64,
/// Octets in the socket's buffer that are not sent yet.
pub not_sent: u32,
pub pacing_rate: u64,
pub delivery_rate: u64,
pub bytes_sent: u64,
pub bytes_acked: u64,
pub bytes_received: u64,
pub segments_out: u32,
pub segments_in: u32,
/// How long the connection had something to send.
pub busy: u64,
/// Of that, how long it was held up by the peer's receive window.
pub limited_by_receive_window: u64,
/// And how long by our own send buffer being full.
pub limited_by_send_buffer: u64,
pub delivered: u32,
pub reordering: u32,
/// Milliseconds since data was last sent, received, and since the
/// last acknowledgement came.
pub last_data_sent_ms: u32,
pub last_data_received_ms: u32,
pub last_ack_received_ms: u32,
}
impl TcpConnection {
pub fn state_name(&self) -> &'static str {
match self.state {
1 => "established",
2 => "syn-sent",
3 => "syn-received",
4 => "fin-wait-1",
5 => "fin-wait-2",
6 => "time-wait",
7 => "close",
8 => "close-wait",
9 => "last-ack",
10 => "listen",
11 => "closing",
_ => "unknown",
}
}
}
const SOCK_DIAG_BY_FAMILY: u16 = 20;
const IPPROTO_TCP: u8 = 6;
const INET_DIAG_INFO: u16 = 2;
const INET_DIAG_CONG: u16 = 4;
/// `idiag_ext` bits asking for those two.
const EXTENSIONS: u8 = (1 << (INET_DIAG_INFO - 1)) | (1 << (INET_DIAG_CONG - 1));
const NO_COOKIE: u32 = u32::MAX;
const REQUEST_LEN: usize = HEADER_LEN + 56;
const DIAG_MSG_LEN: usize = 72;
/// The request for the connection from `local` to `peer`, with `sequence`.
pub fn request(local: SocketAddr, peer: SocketAddr, sequence: u32) -> Vec<u8> {
let family = match local {
SocketAddr::V4(_) => AF_INET,
SocketAddr::V6(_) => AF_INET6,
};
let mut buf = Vec::with_capacity(REQUEST_LEN);
buf.extend_from_slice(&(REQUEST_LEN as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&flag::REQUEST.to_ne_bytes());
buf.extend_from_slice(&sequence.to_ne_bytes());
buf.extend_from_slice(&0u32.to_ne_bytes());
// inet_diag_req_v2
buf.extend_from_slice(&[family, IPPROTO_TCP, EXTENSIONS, 0]);
buf.extend_from_slice(&u32::MAX.to_ne_bytes()); // any state
// inet_diag_sockid: ports in network order, addresses as 16 octets.
buf.extend_from_slice(&local.port().to_be_bytes());
buf.extend_from_slice(&peer.port().to_be_bytes());
buf.extend_from_slice(&address_octets(local.ip()));
buf.extend_from_slice(&address_octets(peer.ip()));
// The interface, which a socket on a link-local address is bound to
// and is found by.
let interface = match local {
SocketAddr::V6(v6) => v6.scope_id(),
SocketAddr::V4(_) => 0,
};
buf.extend_from_slice(&interface.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
debug_assert_eq!(buf.len(), REQUEST_LEN);
buf
}
fn address_octets(address: IpAddr) -> [u8; 16] {
let mut octets = [0u8; 16];
match address {
IpAddr::V4(v4) => octets[..4].copy_from_slice(&v4.octets()),
IpAddr::V6(v6) => octets = v6.octets(),
}
octets
}
/// What the kernel answered.
#[derive(Debug, PartialEq, Eq)]
pub enum Answer {
Connection(Box<TcpConnection>),
/// No such connection.
None,
/// The kernel refused, with this error number.
Refused(i32),
}
/// Reads the kernel's answer to a [`request`].
pub fn parse(buf: &[u8]) -> io::Result<Answer> {
let header = buf
.get(..HEADER_LEN)
.ok_or_else(|| io::Error::other("short netlink answer"))?;
let length = ne_u32(&header[0..4]) as usize;
let kind = ne_u16(&header[4..6]);
let message = buf
.get(HEADER_LEN..length.min(buf.len()))
.ok_or_else(|| io::Error::other("short netlink answer"))?;
if kind == message_type::ERROR {
let errno = -(message
.first_chunk::<4>()
.map_or(0, |e| i32::from_ne_bytes(*e)));
// ENOENT: no such socket.
return Ok(if errno == 2 {
Answer::None
} else {
Answer::Refused(errno)
});
}
if kind != SOCK_DIAG_BY_FAMILY || message.len() < DIAG_MSG_LEN {
return Err(io::Error::other("not a socket diagnostic"));
}
let mut connection = TcpConnection {
state: message[1],
recv_queue: ne_u32(&message[56..60]),
send_queue: ne_u32(&message[60..64]),
..TcpConnection::default()
};
for (kind, value) in attributes(&message[DIAG_MSG_LEN..]) {
match kind {
INET_DIAG_INFO => fill_from_tcp_info(&mut connection, value),
INET_DIAG_CONG => {
let name = value.split(|b| *b == 0).next().unwrap_or(&[]);
connection.congestion = Some(String::from_utf8_lossy(name).into_owned());
}
_ => {}
}
}
Ok(Answer::Connection(Box::new(connection)))
}
/// Fills in what `tcp_info` (linux/tcp.h) has, as far as the kernel sent
/// it: the structure has grown over the years, and a field that is not
/// there is left at zero.
fn fill_from_tcp_info(c: &mut TcpConnection, info: &[u8]) {
let u32_at = |at: usize| info.get(at..at + 4).map_or(0, ne_u32);
let u64_at = |at: usize| info.get(at..at + 8).map_or(0, ne_u64);
c.send_window_scale = info.get(6).map_or(0, |b| b & 0x0f);
c.receive_window_scale = info.get(6).map_or(0, |b| b >> 4);
c.rto = u32_at(8);
c.mss = u32_at(16);
c.unacked = u32_at(24);
c.lost = u32_at(32);
c.retrans = u32_at(36);
c.last_data_sent_ms = u32_at(44);
c.last_data_received_ms = u32_at(52);
c.last_ack_received_ms = u32_at(56);
c.pmtu = u32_at(60);
c.receive_ssthresh = u32_at(64);
c.rtt = u32_at(68);
c.rtt_variance = u32_at(72);
c.ssthresh = Some(u32_at(76)).filter(|t| *t < 0x7fff_ffff);
c.cwnd = u32_at(80);
c.reordering = u32_at(88);
c.receive_space = u32_at(96);
c.total_retrans = u32_at(100);
c.pacing_rate = u64_at(104);
c.bytes_acked = u64_at(120);
c.bytes_received = u64_at(128);
c.segments_out = u32_at(136);
c.segments_in = u32_at(140);
c.not_sent = u32_at(144);
c.min_rtt = u32_at(148);
c.delivery_rate = u64_at(160);
c.busy = u64_at(168);
c.limited_by_receive_window = u64_at(176);
c.limited_by_send_buffer = u64_at(184);
c.delivered = u32_at(192);
c.bytes_sent = u64_at(200);
c.bytes_retrans = u64_at(208);
c.send_window = u32_at(228);
c.receive_window = u32_at(232);
}
/// Asks the kernel about the TCP connection from `local` to `peer`.
/// `Ok(None)` if there is no such connection, or on an operating system
/// that cannot be asked.
#[cfg(target_os = "linux")]
pub async fn tcp_connection(
local: SocketAddr,
peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
use netlink_sys::protocols::NETLINK_SOCK_DIAG;
use netlink_sys::{AsyncSocket, AsyncSocketExt, TokioSocket};
let mut socket = TokioSocket::new(NETLINK_SOCK_DIAG)?;
socket
.socket_mut()
.bind(&netlink_sys::SocketAddr::new(0, 0))?;
let sequence = 1;
socket.send(&request(local, peer, sequence)).await?;
let mut buf = bytes::BytesMut::with_capacity(4096);
tokio::time::timeout(std::time::Duration::from_secs(2), socket.recv(&mut buf))
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "the kernel did not answer"))??;
match parse(&buf)? {
Answer::Connection(connection) => Ok(Some(*connection)),
Answer::None => Ok(None),
Answer::Refused(errno) => Err(io::Error::from_raw_os_error(errno)),
}
}
#[cfg(not(target_os = "linux"))]
pub async fn tcp_connection(
_local: SocketAddr,
_peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
Ok(None)
}
#[cfg(test)]
mod tests {
use super::*;
/// A reply as the kernel writes it, for an established connection,
/// with the first `info_len` octets of a `tcp_info` of the full
/// modern length: an older kernel sends a shorter one.
fn reply(info_len: usize) -> Vec<u8> {
let mut info = vec![0u8; 248];
info[0] = 1; // established
info[6] = 0x7a; // snd_wscale 10, rcv_wscale 7
let put32 = |info: &mut Vec<u8>, at: usize, v: u32| {
info[at..at + 4].copy_from_slice(&v.to_ne_bytes())
};
let put64 = |info: &mut Vec<u8>, at: usize, v: u64| {
info[at..at + 8].copy_from_slice(&v.to_ne_bytes())
};
put32(&mut info, 16, 1448); // mss
put32(&mut info, 24, 12); // unacked
put32(&mut info, 68, 2300); // rtt
put32(&mut info, 76, 0x7fff_ffff); // no ssthresh
put32(&mut info, 80, 42); // cwnd
put32(&mut info, 144, 65536); // notsent
put64(&mut info, 176, 900_000); // rwnd limited
put64(&mut info, 168, 1_000_000); // busy
put32(&mut info, 228, 131072); // snd_wnd
put32(&mut info, 232, 262144); // rcv_wnd
info.truncate(info_len);
let cong = b"cubic\0";
let mut message = vec![0u8; DIAG_MSG_LEN];
message[1] = 1;
message[60..64].copy_from_slice(&80_000u32.to_ne_bytes()); // wqueue
// attributes: INFO, then CONG
let mut attrs = Vec::new();
attrs.extend_from_slice(&((4 + info.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_INFO.to_ne_bytes());
attrs.extend_from_slice(&info);
attrs.extend_from_slice(&((4 + cong.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_CONG.to_ne_bytes());
attrs.extend_from_slice(cong);
while attrs.len() % 4 != 0 {
attrs.push(0);
}
let length = HEADER_LEN + message.len() + attrs.len();
let mut buf = Vec::new();
buf.extend_from_slice(&(length as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&0u16.to_ne_bytes()); // flags
buf.extend_from_slice(&1u32.to_ne_bytes()); // sequence
buf.extend_from_slice(&0u32.to_ne_bytes()); // port
buf.extend_from_slice(&message);
buf.extend_from_slice(&attrs);
buf
}
#[test]
fn a_reply_is_read_field_by_field() {
let Answer::Connection(c) = parse(&reply(248)).unwrap() else {
panic!("a connection");
};
assert_eq!(c.state_name(), "established");
assert_eq!(c.congestion.as_deref(), Some("cubic"));
assert_eq!(c.send_queue, 80_000);
assert_eq!((c.send_window_scale, c.receive_window_scale), (10, 7));
assert_eq!((c.mss, c.unacked, c.rtt, c.cwnd), (1448, 12, 2300, 42));
assert_eq!(c.ssthresh, None);
assert_eq!(c.not_sent, 65536);
assert_eq!((c.busy, c.limited_by_receive_window), (1_000_000, 900_000));
assert_eq!((c.send_window, c.receive_window), (131072, 262144));
}
/// Linux 4.x sends 104 octets of `tcp_info`, up to `tcpi_total_retrans`:
/// what is there is read, and what came later is left at zero rather
/// than read from the attribute after it.
#[test]
fn a_short_tcp_info_from_an_old_kernel_leaves_later_fields_zero() {
let Answer::Connection(c) = parse(&reply(104)).unwrap() else {
panic!("a connection");
};
assert_eq!(c.congestion.as_deref(), Some("cubic"));
assert_eq!((c.mss, c.rtt, c.cwnd), (1448, 2300, 42));
assert_eq!((c.busy, c.send_window), (0, 0));
}
#[test]
fn no_such_connection_and_a_refusal_are_told_apart() {
let error = |errno: i32| {
let mut buf = vec![0u8; 20];
buf[0..4].copy_from_slice(&20u32.to_ne_bytes());
buf[4..6].copy_from_slice(&message_type::ERROR.to_ne_bytes());
buf[16..20].copy_from_slice(&(-errno).to_ne_bytes());
buf
};
assert_eq!(parse(&error(2)).unwrap(), Answer::None);
assert_eq!(parse(&error(13)).unwrap(), Answer::Refused(13));
}
#[test]
fn a_request_names_the_connection() {
let local: SocketAddr = "[2001:db8::1]:179".parse().unwrap();
let peer: SocketAddr = "[2001:db8::2]:40000".parse().unwrap();
let request = request(local, peer, 7);
assert_eq!(request.len(), REQUEST_LEN);
assert_eq!(request[16], AF_INET6);
assert_eq!(&request[24..26], &179u16.to_be_bytes());
assert_eq!(&request[26..28], &40000u16.to_be_bytes());
assert_eq!(
&request[28..44],
&"2001:db8::1"
.parse::<std::net::Ipv6Addr>()
.unwrap()
.octets()
);
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn the_kernel_tells_of_a_connection_of_ours() {
use tokio::io::AsyncWriteExt;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let mut client = tokio::net::TcpStream::connect(listener.local_addr().unwrap())
.await
.unwrap();
let (_server, _) = listener.accept().await.unwrap();
client.write_all(&[0u8; 1000]).await.unwrap();
let connection = tcp_connection(client.local_addr().unwrap(), client.peer_addr().unwrap())
.await
.unwrap()
.expect("the kernel knows it");
assert_eq!(connection.state_name(), "established");
assert!(connection.mss > 0 && connection.cwnd > 0);
assert!(connection.bytes_sent >= 1000 || connection.bytes_acked >= 1000);
let none = tcp_connection(
"127.0.0.1:1".parse().unwrap(),
"127.0.0.1:2".parse().unwrap(),
)
.await
.unwrap();
assert!(none.is_none());
}
}
crates/integration-tests/src/test_peer.rs
+82 -5
View File
@@ -2,16 +2,55 @@ use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration; use std::time::Duration;
use bgp_packet::message::{BGP_HEADER_LEN, Message, NotificationMessage, UpdateMessage}; use bgp_packet::message::{BGP_HEADER_LEN, Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{Capability, OpenMessage, OpenOption}; use bgp_packet::open::{
AddPathEntry, Capability, ExtendedNextHopEntry, OpenMessage, OpenOption, SendReceive,
};
use bgp_primitives::constants::{Afi, AfiSafi, Safi}; use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::ParserContext; use bgp_primitives::parser::ParserContext;
use bytes::BytesMut; use bytes::BytesMut;
use eyre::{Result, bail, eyre}; use eyre::{Result, bail, eyre};
use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::tcp::OwnedWriteHalf;
use tokio::net::{TcpListener, TcpStream}; use tokio::net::{TcpListener, TcpStream};
use tokio::task::JoinHandle;
use crate::TIMEOUT; use crate::TIMEOUT;
/// The capabilities a route collector offers in its OPEN: four-octet AS
/// numbers as `asn`, each of `families`, and to be sent several routes
/// for a prefix (ADD-PATH, RFC 7911) in each of them; with
/// `ipv4_over_ipv6`, IPv4 routes with IPv6 next hops too (RFC 8950), as a
/// collector on an IPv6-only session wants. For the tests and benchmarks
/// that play a collector taking a router's whole table.
pub fn collector_capabilities(
asn: u32,
families: &[AfiSafi],
ipv4_over_ipv6: bool,
) -> Vec<Capability> {
let mut capabilities = vec![Capability::FourByteAsn { asn }];
capabilities.extend(families.iter().copied().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: families
.iter()
.map(|family| AddPathEntry {
afi: family.afi,
safi: family.safi,
send_recv: SendReceive::Receive,
})
.collect(),
});
if ipv4_over_ipv6 {
capabilities.push(Capability::ExtendedNextHop {
entries: vec![ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
}],
});
}
capabilities
}
/// A minimal BGP peer for use in tests. Connects over TCP and can exchange /// A minimal BGP peer for use in tests. Connects over TCP and can exchange
/// framed BGP messages using the `bgp-packet` wire format. /// framed BGP messages using the `bgp-packet` wire format.
pub struct TestPeer { pub struct TestPeer {
@@ -51,6 +90,38 @@ impl TestPeer {
} }
} }
/// The connection itself, for a test or a tool that goes on without
/// the framing, once the handshake is done.
pub fn into_stream(self) -> TcpStream {
self.stream
}
/// The connection with a task reading and dropping whatever the server
/// sends over it, and the half that is left for writing: for a peer
/// that has nothing to hear, as one feeding a table or taking one by
/// the megabyte has not, and whose writer must never be held up by
/// what it has not read. The task ends with the connection, or when
/// its handle is dropped.
pub fn drain(self) -> (OwnedWriteHalf, JoinHandle<()>) {
let (mut reader, writer) = self.stream.into_split();
let sink = tokio::spawn(async move {
let mut sink = vec![0u8; 1 << 16];
while matches!(reader.read(&mut sink).await, Ok(n) if n > 0) {}
});
(writer, sink)
}
/// A KEEPALIVE as it goes on the wire, for whoever writes to the
/// connection without the framing and has to keep the session alive
/// past its hold time.
pub fn keepalive_bytes() -> Vec<u8> {
let mut wire = BytesMut::new();
Message::KeepAlive
.to_wire(&ParserContext::default(), &mut wire)
.expect("a KEEPALIVE is nineteen octets");
wire.to_vec()
}
/// Our end of the connection: what the server sees the peer as. /// Our end of the connection: what the server sees the peer as.
pub fn local_addr(&self) -> Result<SocketAddr> { pub fn local_addr(&self) -> Result<SocketAddr> {
Ok(self.stream.local_addr()?) Ok(self.stream.local_addr()?)
@@ -139,7 +210,9 @@ impl TestPeer {
pub async fn recv_open(&mut self) -> Result<OpenMessage> { pub async fn recv_open(&mut self) -> Result<OpenMessage> {
match self.recv().await? { match self.recv().await? {
Message::Open(o) => { Message::Open(o) => {
// Enable 4-octet ASN parsing if the peer advertised it. // AS numbers are four octets wide only if both sides
// offered it (RFC 6793 §4.1): the server's offer is read
// here, ours was noted when the OPEN went out.
let four_byte = o.options.iter().any(|opt| { let four_byte = o.options.iter().any(|opt| {
matches!(opt, OpenOption::Capabilities(caps) matches!(opt, OpenOption::Capabilities(caps)
if caps.iter().any(|c| matches!(c, Capability::FourByteAsn { .. }))) if caps.iter().any(|c| matches!(c, Capability::FourByteAsn { .. })))
@@ -237,7 +310,8 @@ impl TestPeer {
async fn finish_handshake(&mut self) -> Result<OpenMessage> { async fn finish_handshake(&mut self) -> Result<OpenMessage> {
let server_open = self.recv_open().await?; let server_open = self.recv_open().await?;
// Expect server KEEPALIVE // The server's KEEPALIVE says it accepted our OPEN; anything else
// is its refusal, or a server out of step.
match self.recv().await? { match self.recv().await? {
Message::KeepAlive => {} Message::KeepAlive => {}
other => { other => {
@@ -247,12 +321,15 @@ impl TestPeer {
)); ));
} }
} }
// Send our KEEPALIVE // Ours says the same of its OPEN, and the session is established.
self.send(&Message::KeepAlive).await?; self.send(&Message::KeepAlive).await?;
Ok(server_open) Ok(server_open)
} }
/// Send raw bytes without framing — useful for testing invalid input. /// Writes `bytes` as they are, with no framing of its own: for a test
/// that sends what no message would be, and for the benchmarks, which
/// send a table encoded once as one stream rather than message by
/// message.
pub async fn send_raw(&mut self, bytes: &[u8]) -> Result<()> { pub async fn send_raw(&mut self, bytes: &[u8]) -> Result<()> {
self.stream.write_all(bytes).await?; self.stream.write_all(bytes).await?;
Ok(()) Ok(())
crates/integration-tests/src/test_server.rs
+6 -5
View File
@@ -29,6 +29,9 @@ pub struct TestServer {
handle: JoinHandle<Result<(), ServerError>>, handle: JoinHandle<Result<(), ServerError>>,
/// The single IPv4 loopback address the server is bound to. /// The single IPv4 loopback address the server is bound to.
pub addr: SocketAddr, pub addr: SocketAddr,
/// Every address it is bound to, in the order configured: `addr`
/// first.
pub addrs: Vec<SocketAddr>,
/// The server's routing table. /// The server's routing table.
pub rib: Arc<RibTable>, pub rib: Arc<RibTable>,
/// Handle to the server's locally originated routes. /// Handle to the server's locally originated routes.
@@ -85,11 +88,8 @@ impl TestServer {
customize(&mut config); customize(&mut config);
let bound = BgpServer::bind(config).await?; let bound = BgpServer::bind(config).await?;
let addr = bound let addrs = bound.local_addrs();
.local_addrs() let addr = *addrs.first().expect("at least one listener");
.into_iter()
.next()
.expect("at least one listener");
let rib = Arc::clone(&bound.rib); let rib = Arc::clone(&bound.rib);
let local_rib = bound.local_rib.clone(); let local_rib = bound.local_rib.clone();
let registry = bound.registry.clone(); let registry = bound.registry.clone();
@@ -102,6 +102,7 @@ impl TestServer {
Ok(Self { Ok(Self {
handle, handle,
addr, addr,
addrs,
rib, rib,
local_rib, local_rib,
registry, registry,
crates/integration-tests/tests/add_path.rs
+329 -4
View File
@@ -1,21 +1,26 @@
//! End-to-end tests of ADD-PATH (RFC 7911): a peer sending several routes //! End-to-end tests of ADD-PATH (RFC 7911): a peer sending several routes
//! for one prefix, told apart by path identifiers. //! for one prefix, told apart by path identifiers.
use std::net::{Ipv4Addr, SocketAddr}; use std::collections::BTreeMap;
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal}; use bgp_integration_tests::announcement::{Announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer; use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_integration_tests::test_server::TestServer; use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{peer_config, prefix, wait_until}; use bgp_integration_tests::{peer_config, prefix, wait_until};
use bgp_packet::message::PathAttribute; use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::open::{AddPathEntry, Capability, OpenMessage, SendReceive}; use bgp_packet::open::{AddPathEntry, Capability, OpenMessage, SendReceive};
use bgp_primitives::constants::AfiSafi; use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::Nlri; use bgp_primitives::ip_prefix::Nlri;
use bgp_rib::{LocalRoute, LocalRouteAttrs}; use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig; use bgp_server::config::PeerConfig;
use bgp_server::family::SUPPORTED; use bgp_server::family::SUPPORTED;
use bgp_server::policy::ImportPolicy; use bgp_server::policy::{ExportPolicy, ImportPolicy};
use eyre::Result; use eyre::Result;
use serde_json::Value; use serde_json::Value;
@@ -242,3 +247,323 @@ async fn test_prefixes_are_sent_with_path_identifiers_when_agreed() -> Result<()
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]); assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
Ok(()) Ok(())
} }
const COLLECTOR_ASN: u32 = 65100;
/// What a collector holds: for each prefix and path identifier, the
/// route's AS path.
type Held = BTreeMap<(String, u32), Vec<u32>>;
/// Reads what the collector is told, into `held`, until `done` says so.
async fn told(
collector: &mut TestPeer,
held: &mut Held,
done: impl Fn(&Held) -> bool,
) -> Result<()> {
while !done(held) {
let update = collector.recv_update().await?;
let path = update
.path_attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.unwrap_or_default();
for nlri in &update.withdrawn_routes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
for nlri in &update.announced_routes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
// And what came and went in the multiprotocol attributes.
for attribute in &update.path_attributes {
match attribute {
PathAttribute::MpReachNlri(reach) => {
for nlri in &reach.prefixes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
}
PathAttribute::MpUnreachNlri(unreach) => {
for nlri in &unreach.prefixes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
}
_ => {}
}
}
}
Ok(())
}
/// The routes held for 203.0.113.0/24, by path identifier.
fn paths(held: &Held) -> Vec<(u32, Vec<u32>)> {
held.iter()
.map(|((prefix, id), path)| {
assert_eq!(prefix, "203.0.113.0/24");
(*id, path.clone())
})
.collect()
}
/// A collector that agreed to be sent several routes for a prefix, and
/// whose export policy says to send it every route, is told of every
/// peer's route for a prefix, each with a path identifier of its own that
/// stays with the route; a route that goes is withdrawn by its identifier.
#[tokio::test]
async fn test_every_route_goes_to_a_collector_told_apart_by_path_identifiers() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// The collector is the configured peer at the IPv4 loopback
// address; the feeders come over IPv6 loopback as peers of no
// account.
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let route = |path: &[u32]| Announcement::of([prefix("203.0.113.0/24").into()], path);
let feeder = async |n: u32| -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
Ok(peer)
};
let mut a = feeder(0).await?;
let mut b = feeder(1).await?;
server.wait_for_sessions(2).await?;
a.announce(route(&[PEER_ASN, 64510])).await?;
b.announce(route(&[PEER_ASN + 1, 64510])).await?;
server.wait_for_routes(2).await?;
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, false),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
// Both routes, told apart.
let mut held = Held::new();
told(&mut collector, &mut held, |held| held.len() == 2).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 1, 64510]),
]
);
// The second peer's route goes: withdrawn by its identifier, the
// first's untouched.
b.announce(withdrawal([prefix("203.0.113.0/24").into()]))
.await?;
told(&mut collector, &mut held, |held| held.len() == 1).await?;
assert_eq!(paths(&held), [(1, vec![LOCAL_ASN, PEER_ASN, 64510])]);
// The first peer's route changes: it goes out again with the same
// identifier. A third peer's route gets the lowest free one.
a.announce(route(&[PEER_ASN, 64511, 64510])).await?;
let mut c = feeder(2).await?;
server.wait_for_sessions(4).await?;
c.announce(route(&[PEER_ASN + 2, 64510])).await?;
let changed = |held: &Held| {
held.len() == 2
&& held.get(&("203.0.113.0/24".to_string(), 1))
== Some(&vec![LOCAL_ASN, PEER_ASN, 64511, 64510])
};
told(&mut collector, &mut held, changed).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64511, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 2, 64510]),
]
);
Ok(())
}
/// A collector that is connected before the table arrives is told the
/// whole table, however large: the walk of the table that a new follower
/// starts with takes many batches with a table of some size, and goes on
/// while the feeders keep sending.
#[tokio::test]
async fn test_a_large_table_reaches_a_collector_that_was_there_first() -> Result<()> {
const PREFIXES: u32 = 40_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, false),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
server.wait_for_sessions(1).await?;
// Two feeders, each sending the same prefixes a hundred to an UPDATE.
let nth = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..PREFIXES).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| nth(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * PREFIXES as usize
})
.await?;
let ids: std::collections::BTreeSet<u32> = held.keys().map(|(_, id)| *id).collect();
assert_eq!(ids.into_iter().collect::<Vec<_>>(), [1, 2]);
Ok(())
}
/// As a router at an exchange feeds a collector: one IPv6 session that
/// carries both families, with ADD-PATH for both and IPv4 routes with
/// IPv6 next hops (RFC 8950), and a table of some size from two feeders.
#[tokio::test]
async fn test_both_families_reach_a_collector_over_one_ipv6_session() -> Result<()> {
const V4: u32 = 20_000;
const V6: u32 = 1_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut collector = TestPeer::connect(server.addrs[1]).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, true),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
server.wait_for_sessions(1).await?;
let session = &server.registry.list()[0];
assert!(session.extended_next_hop, "RFC 8950 agreed");
assert_eq!(session.add_path_send.iter().count(), 2);
let v4 = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let v6 = |n: u32| prefix(&format!("2001:db8:{:x}::/48", 0x1000 + n));
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..V4).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| v4(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
for start in (0..V6).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
let update = UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN + n as u32, 64510]),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::9".parse().unwrap()),
prefixes: (start..start + 100).map(|i| v6(i).into()).collect(),
}),
],
announced_routes: vec![],
};
feeder.send(&Message::Update(update)).await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * (V4 + V6) as usize
})
.await?;
let v6_routes = held
.keys()
.filter(|(prefix, _)| prefix.contains(':'))
.count();
assert_eq!(v6_routes, 2 * V6 as usize);
Ok(())
}
crates/integration-tests/tests/deadlock.rs
+92
View File
@@ -0,0 +1,92 @@
//! A peer that sends us a table while not reading what we send it must
//! not stop us reading it: two routers sending each other their tables at
//! once would otherwise each wait on the other for ever.
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::Announcement;
use bgp_integration_tests::prefix;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_server::policy::ExportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 64601;
const ROUTES: u32 = 100_000;
fn nth(n: u32) -> bgp_primitives::ip_prefix::IpPrefix {
let [a, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("{a}.{b}.{c}.0/24"))
}
#[tokio::test]
async fn a_peer_that_does_not_read_is_still_read() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
});
})
.await?;
// One peer's table, for the server to pass on.
let mut first = server
.established(64701, Ipv4Addr::new(10, 0, 0, 1))
.await?;
// A route to an UPDATE, each with attributes of its own, as a real
// table has them: passing it on is then megabytes, not a few hundred
// UPDATEs that the socket buffers would take whole.
for i in 0..ROUTES {
first
.announce(
Announcement::of([nth(i).into()], &[64701, 64510])
.with_communities(&[((i >> 16) as u16 + 1, i as u16)]),
)
.await?;
}
server.wait_for_routes(ROUTES as usize).await?;
// A second peer, which the server starts sending that table to as
// soon as it is up, sends a table of its own and reads nothing back:
// the server's writes to it fill its socket and stop. The server has
// to go on reading it all the same. The peer's receive buffer is
// small, as a real link's window would be, so that this happens with
// the first few hundred kilobytes and not after the loopback's
// megabytes.
let socket = tokio::net::TcpSocket::new_v4()?;
socket.set_recv_buffer_size(16 * 1024)?;
let stream = socket.connect(server.addr).await?;
let mut second = TestPeer::from_stream(stream);
second
.handshake(64702, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
server.wait_for_sessions(2).await?;
// Sent over a couple of seconds, so that the server's export to the
// peer is well under way while the peer is still sending.
let sending = async {
for start in (0..ROUTES).step_by(100) {
second
.announce(Announcement::of(
(start..start + 100).map(|i| nth(i).into()),
&[64702, 64510],
))
.await?;
if start % 2000 == 0 {
tokio::time::sleep(Duration::from_millis(40)).await;
}
}
Ok::<_, eyre::Report>(())
};
tokio::time::timeout(Duration::from_secs(30), sending)
.await
.expect("the server reads the peer's table though the peer reads nothing")?;
tokio::time::timeout(
Duration::from_secs(30),
server.wait_for_routes(2 * ROUTES as usize),
)
.await
.expect("every route is in the table")?;
Ok(())
}
crates/integration-tests/tests/http_api.rs
+21
View File
@@ -52,6 +52,27 @@ async fn test_status_and_peers() -> Result<()> {
peers[0]["address_families"], peers[0]["address_families"],
serde_json::json!(["IPv4 unicast", "IPv6 unicast"]) serde_json::json!(["IPv4 unicast", "IPv6 unicast"])
); );
// The session's TCP connection, by the address the peer list gives.
// What the kernel says of it depends on the platform; that it was
// asked, and why it had nothing to say if it had not, does not.
let peer_addr = peers[0]["peer_addr"].as_str().unwrap();
let tcp = get_json(&client, &format!("{base}/api/tcp?peer={peer_addr}")).await?;
let available = tcp["available"].as_bool().expect("available is a bool");
if available {
assert!(tcp["connection"].is_object(), "{tcp}");
} else {
assert!(tcp["reason"].is_string(), "{tcp}");
}
// The address alone names the peer too, as it does elsewhere.
let by_address = get_json(&client, &format!("{base}/api/tcp?peer=127.0.0.1")).await?;
assert_eq!(by_address["available"], tcp["available"]);
// A peer there is no session with is a bad request, not an answer.
let nobody = client
.get(format!("{base}/api/tcp?peer=nobody"))
.send()
.await?;
assert_eq!(nobody.status(), 400);
Ok(()) Ok(())
} }
crates/integration-tests/tests/transit.rs
+2
View File
@@ -27,6 +27,7 @@ async fn transit(customize: impl FnOnce(&mut bgp_server::config::Config)) -> Res
TestServer::start_with(LOCAL_ASN, |config| { TestServer::start_with(LOCAL_ASN, |config| {
config.export = Arc::new(ExportPolicy { config.export = Arc::new(ExportPolicy {
learned_routes: true, learned_routes: true,
all_paths: false,
..ExportPolicy::default() ..ExportPolicy::default()
}); });
customize(config); customize(config);
@@ -264,6 +265,7 @@ async fn test_communities_say_who_is_told() -> Result<()> {
}); });
config.export = Arc::new(ExportPolicy { config.export = Arc::new(ExportPolicy {
learned_routes: true, learned_routes: true,
all_paths: false,
marks: Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(), marks: Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
prepend: 1, prepend: 1,
}); });
crates/packet/src/attributes.rs
+15 -3
View File
@@ -18,7 +18,6 @@ use bgp_primitives::constants::{AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri}; use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext}; use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext};
use crate::message::write_nlri;
use crate::wire::{ use crate::wire::{
Parsed, list, malformed, parse_afi_safi, parse_asn, put_afi_safi, put_asn, put_length, records, Parsed, list, malformed, parse_afi_safi, parse_asn, put_afi_safi, put_asn, put_length, records,
whole, whole,
@@ -833,6 +832,19 @@ pub(crate) fn parse_nlri_list(
list(buf, |buf| parse_nlri(version, with_path_id, buf)) list(buf, |buf| parse_nlri(version, with_path_id, buf))
} }
/// Writes prefixes as `parse_nlri_list` reads them: for each, its path
/// identifier if the session uses them for the family (RFC 7911 §3), its
/// length in bits, and the bytes that length covers.
pub fn put_nlri_list(nlri: &[Nlri], with_path_id: bool, out: &mut BytesMut) {
for n in nlri {
if with_path_id {
out.put_u32(n.path_id);
}
out.put_u8(n.prefix.length());
out.put(n.prefix.bytes());
}
}
/// The IP version of the prefixes that make up the NLRI of `family`. /// The IP version of the prefixes that make up the NLRI of `family`.
/// ///
/// Any family can be named in MP_REACH_NLRI and MP_UNREACH_NLRI, but only /// Any family can be named in MP_REACH_NLRI and MP_UNREACH_NLRI, but only
@@ -897,7 +909,7 @@ impl MpReachNlriPathAttribute {
out.put_u8(self.next_hop.wire_len()); out.put_u8(self.next_hop.wire_len());
self.next_hop.to_wire(out); self.next_hop.to_wire(out);
out.put_u8(0); out.put_u8(0);
write_nlri(&self.prefixes, ctx.has_add_path(self.family), out); put_nlri_list(&self.prefixes, ctx.has_add_path(self.family), out);
} }
} }
@@ -919,6 +931,6 @@ impl MpUnreachNlriPathAttribute {
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) { pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
put_afi_safi(self.family, out); put_afi_safi(self.family, out);
write_nlri(&self.prefixes, ctx.has_add_path(self.family), out); put_nlri_list(&self.prefixes, ctx.has_add_path(self.family), out);
} }
} }
crates/packet/src/message.rs
+78 -39
View File
@@ -15,7 +15,7 @@ use crate::errors::DecodeError;
use crate::errors::MsgHeaderSubcode; use crate::errors::MsgHeaderSubcode;
use crate::errors::OpenMsgSubcode; use crate::errors::OpenMsgSubcode;
use crate::open::OpenMessage; use crate::open::OpenMessage;
use crate::wire::{fill_length, put_length}; use crate::wire::{body_has_path_id, fill_length, put_length};
/// A sentinel AS number used to denote that the actual AS number is provided as /// A sentinel AS number used to denote that the actual AS number is provided as
/// a 4 byte value in the capabilities instead. /// a 4 byte value in the capabilities instead.
@@ -30,10 +30,15 @@ pub const BGP_HEADER_LEN: u16 = 19;
/// Maximum BGP message length per RFC 4271 §4.1. /// Maximum BGP message length per RFC 4271 §4.1.
pub const BGP_MAX_MSG_LEN: u16 = 4096; pub const BGP_MAX_MSG_LEN: u16 = 4096;
const BGP_MARKER: [u8; 16] = [0xFF; 16]; /// The sixteen octets of ones that open every message (RFC 4271 §4.1).
/// Whoever frames a message by hand wants [`begin_message`] and
/// [`finish_message`] rather than this.
pub const BGP_MARKER: [u8; 16] = [0xFF; 16];
/// BGP message type codes (RFC 4271 §4.1, RFC 2918). /// BGP message type codes (RFC 4271 §4.1, RFC 2918), for whoever frames
mod msg_type { /// a message with [`begin_message`] or tells messages apart by their
/// header.
pub mod msg_type {
pub const OPEN: u8 = 1; pub const OPEN: u8 = 1;
pub const UPDATE: u8 = 2; pub const UPDATE: u8 = 2;
pub const NOTIFICATION: u8 = 3; pub const NOTIFICATION: u8 = 3;
@@ -162,23 +167,10 @@ impl Message {
/// cannot be encoded, `out` is left as it was. /// cannot be encoded, `out` is left as it was.
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> { pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
let start = out.len(); let start = out.len();
out.put_slice(&BGP_MARKER); let length_at = begin_message(out, self.type_code());
// The length, filled in once the body is written. It counts the let written = self
// header too. .body_to_wire(ctx, out)
let length_at = out.len(); .and_then(|()| finish_message(out, length_at));
out.put_u16(0);
out.put_u8(self.type_code());
let written = self.body_to_wire(ctx, out).and_then(|()| {
let length = out.len() - start;
put_length(
out,
length_at,
2,
length,
BGP_MAX_MSG_LEN.into(),
"BGP message",
)
});
if let Err(e) = written { if let Err(e) = written {
out.truncate(start); out.truncate(start);
return Err(e); return Err(e);
@@ -215,6 +207,35 @@ impl Message {
} }
} }
/// Writes the header of a message of type `kind` to the end of `out`,
/// with its length left at zero, and returns where the length goes. The
/// body follows, and then [`finish_message`] fills the length in: for
/// whoever builds a message out of pieces that are not a [`Message`], as
/// an exporter streaming a table does.
pub fn begin_message(out: &mut BytesMut, kind: u8) -> usize {
out.put_slice(&BGP_MARKER);
let length_at = out.len();
out.put_u16(0);
out.put_u8(kind);
length_at
}
/// Fills in the length of the message whose header [`begin_message`] put
/// at `length_at`, counting the header too, once the body is written. A
/// message longer than [`BGP_MAX_MSG_LEN`] cannot be sent, and is an
/// error; `out` is left for the caller to cut back.
pub fn finish_message(out: &mut BytesMut, length_at: usize) -> Result<(), EncodeError> {
let length = out.len() - length_at + BGP_MARKER.len();
put_length(
out,
length_at,
2,
length,
BGP_MAX_MSG_LEN.into(),
"BGP message",
)
}
/// Decodes the body of an OPEN, with the NOTIFICATION that reports a /// Decodes the body of an OPEN, with the NOTIFICATION that reports a
/// failure (RFC 4271 §6.2). /// failure (RFC 4271 §6.2).
fn decode_open(body: &[u8]) -> Result<OpenMessage, DecodeError> { fn decode_open(body: &[u8]) -> Result<OpenMessage, DecodeError> {
@@ -256,7 +277,10 @@ pub struct Decoded {
pub message: Message, pub message: Message,
/// The errors in an UPDATE that were handled without resetting the /// The errors in an UPDATE that were handled without resetting the
/// session (RFC 7606). Empty for a well-formed UPDATE and for every /// session (RFC 7606). Empty for a well-formed UPDATE and for every
/// other message. /// other message. These are
/// [`DecodedUpdate::errors`](crate::update_decode::DecodedUpdate::errors),
/// named here for the one message that can have them, since this is
/// the result for any message.
pub update_errors: Vec<UpdateError>, pub update_errors: Vec<UpdateError>,
/// See [`crate::update_decode::DecodedUpdate::attributes_wire`]. /// See [`crate::update_decode::DecodedUpdate::attributes_wire`].
pub attributes_wire: Option<Vec<u8>>, pub attributes_wire: Option<Vec<u8>>,
@@ -332,14 +356,13 @@ impl UpdateMessage {
/// Writes the body of the UPDATE (what follows the 19-octet header). /// Writes the body of the UPDATE (what follows the 19-octet header).
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> { pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
// The prefixes in the body are IPv4 unicast (RFC 4271 §4.3). let with_path_id = body_has_path_id(ctx);
let with_path_id = ctx.has_add_path(AfiSafi::IPV4_UNICAST);
// Each field's two-octet length is filled in once the field is // Each field's two-octet length is filled in once the field is
// written. // written.
let withdrawn = out.len(); let withdrawn = out.len();
out.put_u16(0); out.put_u16(0);
write_nlri(&self.withdrawn_routes, with_path_id, out); put_nlri_list(&self.withdrawn_routes, with_path_id, out);
fill_length(out, withdrawn, 2, u16::MAX.into(), "withdrawn routes")?; fill_length(out, withdrawn, 2, u16::MAX.into(), "withdrawn routes")?;
let attributes = out.len(); let attributes = out.len();
@@ -350,23 +373,14 @@ impl UpdateMessage {
fill_length(out, attributes, 2, u16::MAX.into(), "path attributes")?; fill_length(out, attributes, 2, u16::MAX.into(), "path attributes")?;
// The NLRI take up the rest of the message. // The NLRI take up the rest of the message.
write_nlri(&self.announced_routes, with_path_id, out); put_nlri_list(&self.announced_routes, with_path_id, out);
Ok(()) Ok(())
} }
} }
/// Writes prefixes as an UPDATE names them: for each, its path identifier /// The name [`put_nlri_list`] went by before it moved next to the parser
/// if the session uses them for the family (RFC 7911 §3), its length in /// it is the pair of; the exporter still uses it.
/// bits, and the bytes that length covers. pub use crate::attributes::put_nlri_list as write_nlri;
pub(crate) fn write_nlri(nlri: &[Nlri], with_path_id: bool, out: &mut BytesMut) {
for n in nlri {
if with_path_id {
out.put_u32(n.path_id);
}
out.put_u8(n.prefix.length());
out.put(n.prefix.bytes());
}
}
/// A parser failure as text for a log line. /// A parser failure as text for a log line.
pub(crate) fn describe(e: &nom::Err<BgpParserError<&[u8]>>) -> String { pub(crate) fn describe(e: &nom::Err<BgpParserError<&[u8]>>) -> String {
@@ -409,7 +423,7 @@ mod tests {
str::FromStr, str::FromStr,
}; };
use bytes::BytesMut; use bytes::{BufMut, BytesMut};
use eyre::{Result, eyre}; use eyre::{Result, eyre};
use crate::message::{ use crate::message::{
@@ -1133,6 +1147,31 @@ mod tests {
Ok(()) Ok(())
} }
/// A message framed by hand, as the exporter frames an UPDATE it
/// builds from pieces, is the message `to_wire` writes; and one that
/// grows past the maximum is refused with its length filled in.
#[test]
fn a_message_framed_by_hand_is_the_one_to_wire_writes() {
use crate::message::{begin_message, finish_message, msg_type};
let ctx = default_v6_context();
let mut whole = BytesMut::new();
Message::Update(UpdateMessage::end_of_rib(AfiSafi::IPV4_UNICAST))
.to_wire(&ctx, &mut whole)
.unwrap();
let mut by_hand = BytesMut::new();
let length_at = begin_message(&mut by_hand, msg_type::UPDATE);
assert_eq!(length_at, 16);
by_hand.put_u32(0);
finish_message(&mut by_hand, length_at).unwrap();
assert_eq!(by_hand, whole);
let mut too_long = BytesMut::new();
let length_at = begin_message(&mut too_long, msg_type::UPDATE);
too_long.put_bytes(0, 4096);
assert!(finish_message(&mut too_long, length_at).is_err());
}
#[test] #[test]
fn test_message_rejects_bad_marker() { fn test_message_rejects_bad_marker() {
let ctx = default_v6_context(); let ctx = default_v6_context();
crates/packet/src/update_decode.rs
+13 -8
View File
@@ -19,7 +19,7 @@
//! be acted on, with treat-as-withdraw and attribute discard already carried //! be acted on, with treat-as-withdraw and attribute discard already carried
//! out, together with a report of what was wrong, for the log. //! out, together with a report of what was wrong, for the log.
use bgp_primitives::constants::{AfiSafi, IpVersion}; use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::Nlri; use bgp_primitives::ip_prefix::Nlri;
use bgp_primitives::parser::ParserContext; use bgp_primitives::parser::ParserContext;
@@ -29,6 +29,7 @@ use crate::message::{
MpUnreachNlriPathAttribute, NotAnAttribute, PathAttribute, RawAttribute, UpdateMessage, MpUnreachNlriPathAttribute, NotAnAttribute, PathAttribute, RawAttribute, UpdateMessage,
defined_flags, describe, parse_nlri_list, split_attribute, defined_flags, describe, parse_nlri_list, split_attribute,
}; };
use crate::wire::body_has_path_id;
/// How an error in an UPDATE was handled, short of resetting the session. /// How an error in an UPDATE was handled, short of resetting the session.
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -62,7 +63,9 @@ pub struct UpdateError {
pub struct DecodedUpdate { pub struct DecodedUpdate {
/// The UPDATE after treat-as-withdraw and attribute discard. /// The UPDATE after treat-as-withdraw and attribute discard.
pub update: UpdateMessage, pub update: UpdateMessage,
/// Empty when the UPDATE was well formed. /// What was wrong, in the order met; empty when the UPDATE was well
/// formed. Where a message of any type was decoded this is
/// [`Decoded::update_errors`](crate::message::Decoded::update_errors).
pub errors: Vec<UpdateError>, pub errors: Vec<UpdateError>,
/// The attributes in `update`, as they were on the wire, for a table /// The attributes in `update`, as they were on the wire, for a table
/// that keeps attributes in that form and need not write them out /// that keeps attributes in that form and need not write them out
@@ -102,9 +105,7 @@ fn reset(subcode: UpdateMsgSubcode, data: Vec<u8>, detail: String) -> DecodeErro
/// error here cannot be recovered from: the prefixes are what /// error here cannot be recovered from: the prefixes are what
/// treat-as-withdraw would act on. /// treat-as-withdraw would act on.
fn prefixes(ctx: &ParserContext, field: &str, buf: &[u8]) -> Result<Vec<Nlri>, DecodeError> { fn prefixes(ctx: &ParserContext, field: &str, buf: &[u8]) -> Result<Vec<Nlri>, DecodeError> {
// The prefixes in the body are IPv4 unicast (RFC 4271 §4.3). match parse_nlri_list(IpVersion::Ipv4, body_has_path_id(ctx), buf) {
let with_path_id = ctx.has_add_path(AfiSafi::IPV4_UNICAST);
match parse_nlri_list(IpVersion::Ipv4, with_path_id, buf) {
Ok((_, prefixes)) => Ok(prefixes), Ok((_, prefixes)) => Ok(prefixes),
Err(e) => Err(reset( Err(e) => Err(reset(
UpdateMsgSubcode::InvalidNetworkField, UpdateMsgSubcode::InvalidNetworkField,
@@ -234,7 +235,10 @@ struct AttributeReader<'a> {
attributes: Vec<PathAttribute>, attributes: Vec<PathAttribute>,
errors: Vec<UpdateError>, errors: Vec<UpdateError>,
/// The bytes of the attributes that are kept and are a property of a /// The bytes of the attributes that are kept and are a property of a
/// route: all but MP_REACH_NLRI and MP_UNREACH_NLRI. /// route: all but MP_REACH_NLRI and MP_UNREACH_NLRI. Left empty on a
/// session with two-octet AS numbers, whose bytes no table keeps (see
/// [`DecodedUpdate::attributes_wire`]), so that nothing is copied or
/// reserved for them.
kept_wire: Vec<u8>, kept_wire: Vec<u8>,
/// The type codes met so far, one bit each. /// The type codes met so far, one bit each.
seen: [u64; 4], seen: [u64; 4],
@@ -246,7 +250,7 @@ impl<'a> AttributeReader<'a> {
ctx, ctx,
attributes: Vec::new(), attributes: Vec::new(),
errors: Vec::new(), errors: Vec::new(),
kept_wire: Vec::with_capacity(wire_len), kept_wire: Vec::with_capacity(if ctx.four_octet_asn { wire_len } else { 0 }),
seen: [0; 4], seen: [0; 4],
} }
} }
@@ -291,7 +295,7 @@ impl<'a> AttributeReader<'a> {
route_property, route_property,
} = self.judge(&raw)? } = self.judge(&raw)?
{ {
if route_property { if route_property && self.ctx.four_octet_asn {
self.kept_wire.extend_from_slice(raw.wire); self.kept_wire.extend_from_slice(raw.wire);
} }
self.attributes.push(attribute); self.attributes.push(attribute);
@@ -494,6 +498,7 @@ mod tests {
use std::net::Ipv4Addr; use std::net::Ipv4Addr;
use crate::message::PathAttributeFlags; use crate::message::PathAttributeFlags;
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
use super::*; use super::*;
crates/packet/src/wire.rs
+10 -1
View File
@@ -9,7 +9,7 @@ use nom::IResult;
use nom::number::complete::{be_u8, be_u16, be_u32}; use nom::number::complete::{be_u8, be_u16, be_u32};
use bgp_primitives::constants::{Afi, AfiSafi, Safi}; use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::{BgpParserError, EncodeError}; use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext};
use crate::message::AS_TRANS; use crate::message::AS_TRANS;
@@ -91,6 +91,15 @@ pub(crate) fn put_asn(four_octet: bool, asn: u32, out: &mut BytesMut) {
} }
} }
/// Whether the prefixes in the body of an UPDATE carry path identifiers
/// on a session with `ctx`. The body holds IPv4 unicast routes and nothing
/// else (RFC 4271 §4.3), so that is the family whose ADD-PATH agreement
/// decides it; every other family's prefixes are in MP_REACH_NLRI and
/// MP_UNREACH_NLRI, which name their family.
pub(crate) fn body_has_path_id(ctx: &ParserContext) -> bool {
ctx.has_add_path(AfiSafi::IPV4_UNICAST)
}
/// Parses an address family as the multiprotocol attributes and most /// Parses an address family as the multiprotocol attributes and most
/// capabilities name it: a two-octet AFI and a one-octet SAFI. /// capabilities name it: a two-octet AFI and a one-octet SAFI.
pub(crate) fn parse_afi_safi(buf: &[u8]) -> Parsed<'_, AfiSafi> { pub(crate) fn parse_afi_safi(buf: &[u8]) -> Parsed<'_, AfiSafi> {
crates/rib/src/table/attrs.rs
+112 -34
View File
@@ -13,7 +13,7 @@
use std::hash::BuildHasher; use std::hash::BuildHasher;
use std::sync::{Arc, Mutex, MutexGuard, PoisonError}; use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use bgp_packet::message::{NlriNextHop, PathAttribute, attr_type, split_attribute}; use bgp_packet::message::{NlriNextHop, PathAttribute, RawAttribute, attr_type, split_attribute};
use bgp_primitives::parser::{EncodeError, ParserContext}; use bgp_primitives::parser::{EncodeError, ParserContext};
use bytes::{BufMut, BytesMut}; use bytes::{BufMut, BytesMut};
use foldhash::fast::RandomState; use foldhash::fast::RandomState;
@@ -21,6 +21,10 @@ use hashbrown::HashTable;
const OPTIONAL: u8 = 0x80; const OPTIONAL: u8 = 0x80;
/// The type of an AS_PATH segment whose AS numbers are in order; the other
/// type, AS_SET, is an unordered handful (RFC 4271 §4.3).
const AS_SEQUENCE: u8 = 2;
/// What route selection reads from a set of attributes (RFC 4271 §9.1.2.2). /// What route selection reads from a set of attributes (RFC 4271 §9.1.2.2).
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SelectionKey { pub struct SelectionKey {
@@ -116,7 +120,6 @@ impl SelectionKey {
/// Reads an AS_PATH value with four-octet AS numbers. /// Reads an AS_PATH value with four-octet AS numbers.
fn scan_as_path(mut segments: &[u8]) -> Option<PathScan> { fn scan_as_path(mut segments: &[u8]) -> Option<PathScan> {
const AS_SEQUENCE: u8 = 2;
let mut hops = 0usize; let mut hops = 0usize;
let (mut first, mut last) = (0, 0); let (mut first, mut last) = (0, 0);
let mut at_start = true; let mut at_start = true;
@@ -150,7 +153,6 @@ fn scan_as_path(mut segments: &[u8]) -> Option<PathScan> {
/// AS numbers to `out`, and returns whether any segment is not a sequence. /// AS numbers to `out`, and returns whether any segment is not a sequence.
/// Stops at what is not a segment. /// Stops at what is not a segment.
fn read_as_path(mut segments: &[u8], out: &mut Vec<u32>) -> bool { fn read_as_path(mut segments: &[u8], out: &mut Vec<u32>) -> bool {
const AS_SEQUENCE: u8 = 2;
let mut has_set = false; let mut has_set = false;
while let [kind, count, rest @ ..] = segments { while let [kind, count, rest @ ..] = segments {
let Some((asns, rest)) = rest.split_at_checked(usize::from(*count) * 4) else { let Some((asns, rest)) = rest.split_at_checked(usize::from(*count) * 4) else {
@@ -187,43 +189,58 @@ impl AttrSet {
&self.wire &self.wire
} }
/// The attributes one by one, as they lie in the stored bytes. The
/// bytes were checked when the set was made, so this is all of them;
/// it stops at the first that does not split as an attribute all the
/// same, rather than go on from the middle of one.
fn raw_attributes(&self) -> impl Iterator<Item = RawAttribute<'_>> {
let mut wire = &*self.wire;
std::iter::from_fn(move || {
let (attribute, rest) = split_attribute(wire).ok()?;
wire = rest;
Some(attribute)
})
}
/// The value of the attribute of type `type_code`, if the set has one.
fn attribute(&self, type_code: u8) -> Option<&[u8]> {
self.raw_attributes()
.find(|attribute| attribute.type_code == type_code)
.map(|attribute| attribute.value)
}
/// The route's next hop: the one that came in MP_REACH_NLRI, or else /// The route's next hop: the one that came in MP_REACH_NLRI, or else
/// the NEXT_HOP attribute. Read off the stored bytes without building /// the NEXT_HOP attribute. Read off the stored bytes without building
/// anything else, for those who ask it of every route. /// anything else, for those who ask it of every route.
pub fn next_hop(&self) -> Option<NlriNextHop> { pub fn next_hop(&self) -> Option<NlriNextHop> {
let mut plain = None; let mut plain = None;
let mut wire = &*self.wire; for attribute in self.raw_attributes() {
while let Ok((attribute, rest)) = split_attribute(wire) {
match attribute.type_code { match attribute.type_code {
attr_type::MP_REACH_NLRI => { attr_type::MP_REACH_NLRI => return decode_next_hop(attribute.value),
// Kept as the next hop's length and the next hop.
let (length, address) = attribute.value.split_first()?;
return NlriNextHop::from_bytes(address)
.filter(|_| usize::from(*length) == address.len());
}
attr_type::NEXT_HOP => plain = NlriNextHop::from_bytes(attribute.value), attr_type::NEXT_HOP => plain = NlriNextHop::from_bytes(attribute.value),
_ => {} _ => {}
} }
wire = rest;
} }
plain plain
} }
/// Whether the route carries the OTC attribute (RFC 9234): it came
/// down from a provider or across from a peer, and is to go on down
/// only. Read off the stored bytes without building anything else,
/// for those who ask it of every route: a session to a peer or a
/// provider asks it of most of the table, to pass over them.
pub fn is_only_to_customer(&self) -> bool {
self.attribute(attr_type::ONLY_TO_CUSTOMER).is_some()
}
/// Whether the route carries the BLACKHOLE community (RFC 7999), which /// Whether the route carries the BLACKHOLE community (RFC 7999), which
/// asks for the packets to its prefix to be dropped. Read off the /// asks for the packets to its prefix to be dropped. Read off the
/// stored bytes without building anything else, for those who ask it /// stored bytes without building anything else, for those who ask it
/// of every route. /// of every route.
pub fn is_blackhole(&self) -> bool { pub fn is_blackhole(&self) -> bool {
const BLACKHOLE: [u8; 4] = [0xFF, 0xFF, 0x02, 0x9A]; const BLACKHOLE: [u8; 4] = [0xFF, 0xFF, 0x02, 0x9A];
let mut wire = &*self.wire; self.attribute(attr_type::COMMUNITIES)
while let Ok((attribute, rest)) = split_attribute(wire) { .is_some_and(|communities| communities.as_chunks::<4>().0.contains(&BLACKHOLE))
if attribute.type_code == attr_type::COMMUNITIES {
let (communities, _) = attribute.value.as_chunks::<4>();
return communities.contains(&BLACKHOLE);
}
wire = rest;
}
false
} }
/// Puts the AS numbers of the route's path in `out`, in place of what /// Puts the AS numbers of the route's path in `out`, in place of what
@@ -237,14 +254,10 @@ impl AttrSet {
/// else, for those who ask it of every route. /// else, for those who ask it of every route.
pub fn as_path(&self, out: &mut Vec<u32>) -> bool { pub fn as_path(&self, out: &mut Vec<u32>) -> bool {
out.clear(); out.clear();
let mut wire = &*self.wire; match self.attribute(attr_type::AS_PATH) {
while let Ok((attribute, rest)) = split_attribute(wire) { Some(segments) => read_as_path(segments, out),
if attribute.type_code == attr_type::AS_PATH { None => false,
return read_as_path(attribute.value, out);
}
wire = rest;
} }
false
} }
/// The attributes, parsed, and the next hop if it came in /// The attributes, parsed, and the next hop if it came in
@@ -291,7 +304,8 @@ fn encode(
Ok(()) Ok(())
} }
/// Writes a next hop as an MP_REACH_NLRI that holds nothing else. /// Writes a next hop as an MP_REACH_NLRI that holds nothing else: the
/// next hop's length, and the next hop.
fn encode_next_hop(next_hop: &NlriNextHop, out: &mut BytesMut) { fn encode_next_hop(next_hop: &NlriNextHop, out: &mut BytesMut) {
out.put_u8(OPTIONAL); out.put_u8(OPTIONAL);
out.put_u8(attr_type::MP_REACH_NLRI); out.put_u8(attr_type::MP_REACH_NLRI);
@@ -300,6 +314,13 @@ fn encode_next_hop(next_hop: &NlriNextHop, out: &mut BytesMut) {
next_hop.to_wire(out); next_hop.to_wire(out);
} }
/// Reads the next hop out of the value of an MP_REACH_NLRI as
/// [`encode_next_hop`] writes it. `None` if the value is not one.
fn decode_next_hop(value: &[u8]) -> Option<NlriNextHop> {
let (length, address) = value.split_first()?;
NlriNextHop::from_bytes(address).filter(|_| usize::from(*length) == address.len())
}
/// Parses path attributes in the form [`AttrSet`] keeps and table dumps /// Parses path attributes in the form [`AttrSet`] keeps and table dumps
/// use (RFC 6396 §4.3.4): four-octet AS numbers, and an MP_REACH_NLRI that /// use (RFC 6396 §4.3.4): four-octet AS numbers, and an MP_REACH_NLRI that
/// holds only a next hop. `None` if they are malformed. /// holds only a next hop. `None` if they are malformed.
@@ -310,11 +331,7 @@ fn parse(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
while !wire.is_empty() { while !wire.is_empty() {
let (raw, rest) = split_attribute(wire).ok()?; let (raw, rest) = split_attribute(wire).ok()?;
if raw.type_code == attr_type::MP_REACH_NLRI { if raw.type_code == attr_type::MP_REACH_NLRI {
let (length, address) = raw.value.split_first()?; next_hop = Some(decode_next_hop(raw.value)?);
if usize::from(*length) != address.len() {
return None;
}
next_hop = Some(NlriNextHop::from_bytes(address)?);
} else { } else {
let (_, attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()?; let (_, attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()?;
attributes.push(attribute); attributes.push(attribute);
@@ -634,6 +651,67 @@ mod tests {
); );
} }
/// The OTC attribute is read off the stored bytes, wherever in them it
/// lies.
#[test]
fn otc_is_told_from_the_bytes() {
let store = AttrStore::new();
let without = store.intern(&attributes(vec![3333]), None).unwrap();
assert!(!without.is_only_to_customer());
let mut with = attributes(vec![3333]);
with.insert(1, PathAttribute::OnlyToCustomer(3333));
assert!(store.intern(&with, None).unwrap().is_only_to_customer());
// After COMMUNITIES, which is the last of the usual attributes.
let mut after = attributes(vec![3333]);
after.push(PathAttribute::OnlyToCustomer(3333));
assert!(store.intern(&after, None).unwrap().is_only_to_customer());
}
/// What a session received, with attributes of our own added after it,
/// is one set with the parsed form of the whole; the bytes are not
/// parsed to find that out, so the bytes must say the same thing.
#[test]
fn received_attributes_with_ours_added_are_the_same_set_as_the_parsed_form() {
let store = AttrStore::new();
let ctx = four_octet();
let received = attributes(vec![65001, 65002]);
let mut wire = BytesMut::new();
encode(&received, None, &mut wire).unwrap();
let ours = [
PathAttribute::OnlyToCustomer(65001),
PathAttribute::LocalPref(LocalPrefPathAttribute(150)),
];
let mut added = BytesMut::new();
for attribute in &ours {
attribute.to_wire(&ctx, &mut added).unwrap();
}
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let mut whole = received.clone();
whole.extend(ours.iter().cloned());
let parsed = store.intern(&whole, Some(&next_hop)).unwrap();
let from_bytes = store
.intern_received_with(&wire, &added, Some(&next_hop))
.unwrap();
assert!(Arc::ptr_eq(&parsed, &from_bytes));
assert!(from_bytes.is_only_to_customer());
assert_eq!(from_bytes.next_hop(), Some(next_hop));
// Without a next hop too.
let parsed = store.intern(&whole, None).unwrap();
let from_bytes = store.intern_received_with(&wire, &added, None).unwrap();
assert!(Arc::ptr_eq(&parsed, &from_bytes));
// An attribute of ours of a type the peer sent is not a set.
let mut repeated = BytesMut::new();
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(1))
.to_wire(&ctx, &mut repeated)
.unwrap();
assert!(store.intern_received_with(&wire, &repeated, None).is_none());
}
/// Two of the same attribute are not a set: its readers could each /// Two of the same attribute are not a set: its readers could each
/// take a different one. /// take a different one.
#[test] #[test]
crates/rib/src/table/journal.rs
+284 -206
View File
@@ -95,7 +95,7 @@ struct Entry {
} }
/// One journal of one shard. /// One journal of one shard.
struct Journal { pub(super) struct Journal {
kind: Kind, kind: Kind,
/// The sequence number of the latest change. Counts every change this /// The sequence number of the latest change. Counts every change this
/// journal is for, whether or not anyone was reading. /// journal is for, whether or not anyone was reading.
@@ -171,25 +171,35 @@ impl Journal {
} }
} }
/// Forgets that `entry`'s prefix had gone, if `entry` is what said so. /// Voids the note that `entry`'s prefix had gone, if `entry` is what
fn drop_entry(gone: &mut HashMap<IpPrefix, u64, RandomState>, entry: &Entry) { /// said so: an entry that is dropped must not leave the note behind,
/// or the prefix would be given out as gone on the strength of a later
/// entry that says something else.
fn void_gone_note(gone: &mut HashMap<IpPrefix, u64, RandomState>, entry: &Entry) {
if gone.get(&entry.prefix) == Some(&entry.seq) { if gone.get(&entry.prefix) == Some(&entry.seq) {
gone.remove(&entry.prefix); gone.remove(&entry.prefix);
} }
} }
/// What `entry` says, if it still says something: it is the latest /// What `entry` says: something only if it is the latest change of its
/// change of its prefix. The row, or `None` for a prefix that has gone. /// prefix.
fn live<'a>( fn says<'a>(&self, entry: &Entry, rows: &'a HashTable<Row>, hasher: &RandomState) -> Says<'a> {
&self,
entry: &Entry,
rows: &'a HashTable<Row>,
hasher: &RandomState,
) -> Option<Option<&'a Row>> {
let hash = hasher.hash_one(entry.prefix); let hash = hasher.hash_one(entry.prefix);
match rows.find(hash, |row| row.prefix == entry.prefix) { match rows.find(hash, |row| row.prefix == entry.prefix) {
Some(row) => (row.seqs.of(self.kind) == entry.seq).then_some(Some(row)), Some(row) if *row.seqs.of(self.kind) == entry.seq => Says::Row(row),
None => (self.gone.get(&entry.prefix) == Some(&entry.seq)).then_some(None), None if self.gone.get(&entry.prefix) == Some(&entry.seq) => Says::Gone,
_ => Says::Nothing,
}
}
/// Where a reader whose cursor is `cursor` is to read from.
fn place_of(&self, cursor: u64) -> Place {
if cursor < self.floor {
Place::Lagged
} else if cursor == self.seq {
Place::Nothing
} else {
Place::From(self.entries.partition_point(|entry| entry.seq <= cursor))
} }
} }
@@ -199,7 +209,26 @@ impl Journal {
if self.entries.len() <= self.limits.tidy_at { if self.entries.len() <= self.limits.tidy_at {
return; return;
} }
// What every reader has passed can go. self.drop_passed();
// Entries for prefixes that have a row are at most as many as the
// rows once the dead ones are out. Entries for prefixes that have
// gone are not counted in: they are what a reader that has stopped
// reading makes pile up. Twice that, so that the pass over every
// entry that finding the dead ones takes is made only once the
// journal has doubled, and the cost per change stays small.
let limit = 2 * (rows.len() + self.limits.slack);
if self.entries.len() <= limit {
return;
}
self.drop_dead(rows, hasher);
if self.entries.len() <= limit {
return;
}
self.drop_oldest_half();
}
/// Drops what every reader has passed.
fn drop_passed(&mut self) {
let passed = self let passed = self
.cursors .cursors
.iter() .iter()
@@ -210,40 +239,57 @@ impl Journal {
if front.seq > passed { if front.seq > passed {
break; break;
} }
Self::drop_entry(&mut self.gone, front); Self::void_gone_note(&mut self.gone, front);
self.entries.pop_front(); self.entries.pop_front();
} }
}
// Entries for prefixes that have a row are at most as many as the /// Drops the dead entries: those whose prefix has changed again since,
// rows once the dead ones are out. Entries for prefixes that have /// so that a later entry says what they said. This looks every entry
// gone are not counted in: they are what a reader that has stopped /// up.
// reading makes pile up. fn drop_dead(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
let limit = 2 * (rows.len() + self.limits.slack);
if self.entries.len() <= limit {
return;
}
// Dead entries: their prefix has changed again since. This looks
// every entry up, but only once the journal has doubled, so the
// cost per change stays small.
let mut entries = std::mem::take(&mut self.entries); let mut entries = std::mem::take(&mut self.entries);
entries.retain(|entry| self.live(entry, rows, hasher).is_some()); entries.retain(|entry| !matches!(self.says(entry, rows, hasher), Says::Nothing));
self.entries = entries; self.entries = entries;
}
if self.entries.len() <= limit { /// Drops the older half of the entries without regard to the readers,
return; /// for a journal that is too long even without its dead entries: a
} /// reader is not reading while prefixes come and go, or more prefixes
// Still too long: a reader is not reading while prefixes come and /// went at once than are left. A reader that had not read what goes
// go, or more prefixes went at once than are left. The older half /// loses its place. Half, so that readers that are only a little
// goes, and a reader that had not read it loses its place. Half, /// behind keep theirs, and so that getting a long journal down to size
// so that readers that are only a little behind keep theirs, and /// takes few passes.
// so that getting a long journal down to size takes few passes. fn drop_oldest_half(&mut self) {
for entry in self.entries.drain(..self.entries.len() / 2) { for entry in self.entries.drain(..self.entries.len() / 2) {
Self::drop_entry(&mut self.gone, &entry); Self::void_gone_note(&mut self.gone, &entry);
self.floor = entry.seq; self.floor = entry.seq;
} }
} }
} }
/// What an entry says; see [`Journal::says`].
enum Says<'a> {
/// The prefix's row, whose latest change the entry is.
Row(&'a Row),
/// The prefix has no route any more, and the entry is what says so.
Gone,
/// Nothing: the prefix has changed again since, and a later entry has
/// it.
Nothing,
}
/// Where a reader is to read a journal from; see [`Journal::place_of`].
enum Place {
/// Entries the reader had not read were dropped; it has lost its
/// place.
Lagged,
/// The reader has read everything up to the latest change.
Nothing,
/// The index of the first entry the reader has not read.
From(usize),
}
/// Which of a shard's two journals. /// Which of a shard's two journals.
#[derive(Debug, Clone, Copy, PartialEq, Eq)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum Kind { pub(super) enum Kind {
@@ -253,29 +299,36 @@ pub(super) enum Kind {
All, All,
} }
/// The sequence numbers a row remembers, one for each journal: of the /// One of something for each [`Kind`] of journal.
/// latest change of its best route, and of any of its routes. #[derive(Debug, Clone, Copy, Default)]
#[derive(Debug, Clone, Copy)] pub(super) struct ByKind<T> {
pub(super) struct Seqs { best: T,
best: u64, all: T,
all: u64,
} }
impl Seqs { impl<T> ByKind<T> {
fn of(self, kind: Kind) -> u64 { pub(super) fn of(&self, kind: Kind) -> &T {
match kind { match kind {
Kind::Best => self.best, Kind::Best => &self.best,
Kind::All => self.all, Kind::All => &self.all,
}
}
pub(super) fn of_mut(&mut self, kind: Kind) -> &mut T {
match kind {
Kind::Best => &mut self.best,
Kind::All => &mut self.all,
} }
} }
} }
/// The sequence numbers a row remembers, one for each journal: of the
/// latest change of its best route, and of any of its routes.
pub(super) type Seqs = ByKind<u64>;
/// The two journals of a shard, written together: every change to a row is /// The two journals of a shard, written together: every change to a row is
/// a change of one of its routes, and some are a change of its best route. /// a change of one of its routes, and some are a change of its best route.
pub(super) struct Journals { pub(super) type Journals = ByKind<Journal>;
best: Journal,
all: Journal,
}
impl Journals { impl Journals {
pub(super) fn new(limits: Limits) -> Self { pub(super) fn new(limits: Limits) -> Self {
@@ -285,18 +338,13 @@ impl Journals {
} }
} }
fn of(&self, kind: Kind) -> &Journal { /// Copies the journals' sequence numbers to `latest`, which the table
match kind { /// keeps outside the shard's lock for readers to look at
Kind::Best => &self.best, /// ([`RibTable::latest_seq`]). Called with the lock held, after every
Kind::All => &self.all, /// change to the shard.
} pub(super) fn publish(&self, latest: &ByKind<AtomicU64>) {
} latest.best.store(self.best.seq, Ordering::Release);
latest.all.store(self.all.seq, Ordering::Release);
fn of_mut(&mut self, kind: Kind) -> &mut Journal {
match kind {
Kind::Best => &mut self.best,
Kind::All => &mut self.all,
}
} }
/// The time by the table's clock that the changes made from now on /// The time by the table's clock that the changes made from now on
@@ -362,18 +410,11 @@ pub(super) struct Mark(u64, u64);
pub struct Routes<'a>(&'a Row); pub struct Routes<'a>(&'a Row);
impl<'a> Routes<'a> { impl<'a> Routes<'a> {
/// The routes: at least one, since a prefix with none has no row, and
/// is given out as [`Event::Withdrawn`] instead.
pub fn iter(&self) -> impl Iterator<Item = RouteRef<'a>> + use<'a> { pub fn iter(&self) -> impl Iterator<Item = RouteRef<'a>> + use<'a> {
self.0.routes() self.0.routes()
} }
/// How many there are: at least one.
pub fn len(&self) -> usize {
self.0.paths.len()
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
} }
impl std::fmt::Debug for Routes<'_> { impl std::fmt::Debug for Routes<'_> {
@@ -424,19 +465,15 @@ pub struct Progress {
} }
enum Phase { enum Phase {
/// Giving out the whole table; `next` is the shard to do next, and /// Giving out the whole table; `next` is the shard to do next. `upto`
/// `announced` whether [`Event::Reset`] has been given. `upto` is the /// is the time on the table's clock the table is given out as of, and
/// time on the table's clock the table is given out as of: the shards /// is taken when [`Event::Reset`] is given, so `None` until then: the
/// are gone through one at a time while the table changes, and the /// shards are gone through one at a time while the table changes, and
/// place taken in each shard's journal is as of that one time, so that /// the place taken in each shard's journal is as of that one time, so
/// what follows is consistent from shard to shard. A change after it /// that what follows is consistent from shard to shard. A change after
/// that a later shard's rows already show is read again from the /// it that a later shard's rows already show is read again from the
/// journal; that is harmless. /// journal; that is harmless.
Resync { Resync { next: usize, upto: Option<u64> },
next: usize,
announced: bool,
upto: u64,
},
/// Following the journals; `next` is the shard to read next, and /// Following the journals; `next` is the shard to read next, and
/// `upto` the time by the table's clock up to which this pass reads /// `upto` the time by the table's clock up to which this pass reads
/// (see [`Entry::stamp`]). /// (see [`Entry::stamp`]).
@@ -447,9 +484,11 @@ impl Phase {
/// The beginning of a resynchronisation. /// The beginning of a resynchronisation.
const RESYNC: Self = Self::Resync { const RESYNC: Self = Self::Resync {
next: 0, next: 0,
announced: false, upto: None,
upto: 0,
}; };
/// The beginning of a pass over the journals.
const FOLLOW: Self = Self::Follow { next: 0, upto: 0 };
} }
/// Follows the routes of a table: the best of each prefix, or all of them. /// Follows the routes of a table: the best of each prefix, or all of them.
@@ -466,7 +505,10 @@ pub struct Reader {
/// is for the shard's benefit. /// is for the shard's benefit.
cursors: Vec<u64>, cursors: Vec<u64>,
phase: Phase, phase: Phase,
/// The table's version when the reader was last found caught up. /// The table's version as it was when the current pass over the
/// journals began. [`Reader::changed`] waits while the version is still
/// this one: whatever was written since the pass began has moved it
/// on, and is there to be read.
seen: u64, seen: u64,
caught_up: bool, caught_up: bool,
} }
@@ -482,7 +524,7 @@ impl RibTable {
/// those who want the news and not the table. If it falls too far /// those who want the news and not the table. If it falls too far
/// behind it is reset like any other, and is then given the table. /// behind it is reset like any other, and is then given the table.
pub fn reader_from_now(self: &Arc<Self>) -> Reader { pub fn reader_from_now(self: &Arc<Self>) -> Reader {
let mut reader = self.register(Kind::Best, Phase::Follow { next: 0, upto: 0 }); let mut reader = self.register(Kind::Best, Phase::FOLLOW);
for (index, shard) in self.shards.iter().enumerate() { for (index, shard) in self.shards.iter().enumerate() {
let shard = read(shard); let shard = read(shard);
reader.set_cursor(index, &shard, shard.journals.best.seq); reader.set_cursor(index, &shard, shard.journals.best.seq);
@@ -598,107 +640,128 @@ impl Reader {
pub fn poll(&mut self, budget: usize, mut visit: impl FnMut(Event<'_>)) -> Progress { pub fn poll(&mut self, budget: usize, mut visit: impl FnMut(Event<'_>)) -> Progress {
// A call that may give nothing out would never get anywhere. // A call that may give nothing out would never get anywhere.
let budget = budget.max(1); let budget = budget.max(1);
let shards = self.table.shards.len(); // Held once for the call: a count taken on the table for every
// shard of every pass, by every reader, was the most of what a
// router with many peers did.
let table = Arc::clone(&self.table);
let mut events = 0; let mut events = 0;
loop { loop {
match self.phase { let done = match self.phase {
Phase::Resync { Phase::Resync { next, upto } => {
next, self.resync_step(&table, next, upto, budget, &mut events, &mut visit)
announced,
upto,
} => {
let upto = if announced {
upto
} else {
self.forget_place();
visit(Event::Reset);
events += 1;
self.table.settled()
};
if next == shards {
visit(Event::Synced);
events += 1;
self.phase = Phase::Follow { next: 0, upto: 0 };
continue;
}
events += self.scan(next, upto, &mut visit);
self.phase = Phase::Resync {
next: next + 1,
announced: true,
upto,
};
if events >= budget {
return self.progress(events, false);
}
} }
Phase::Follow { next, upto } => { Phase::Follow { next, upto } => {
// Read before the pass: a change made during it shows as self.follow_pass(&table, next, upto, budget, &mut events, &mut visit)
// a newer version, and `changed` then returns at once. }
// And the pass reads up to one time in every shard };
// alike, chosen so that about the budget is read. if let Some(progress) = done {
let (version, upto) = if next == 0 { return progress;
( }
self.table.version.load(Ordering::Acquire), }
self.time_for(budget - events), }
)
} else { /// One step of a resynchronisation: the shard `next`, or the end of it
(self.seen, upto) /// once every shard has been given out. Returns what to tell the
}; /// caller if the budget is spent, and `None` to go on.
self.seen = version; fn resync_step(
let mut shard = next; &mut self,
let mut lagged = false; table: &RibTable,
let mut consumed_any = false; next: usize,
let mut more_somewhere = false; upto: Option<u64>,
while shard < shards { budget: usize,
if events >= budget { events: &mut usize,
self.phase = Phase::Follow { next: shard, upto }; visit: &mut impl FnMut(Event<'_>),
return self.progress(events, false); ) -> Option<Progress> {
} let upto = upto.unwrap_or_else(|| {
match self.drain(shard, upto, &mut visit) { self.forget_place();
Drained::Lagged => { visit(Event::Reset);
lagged = true; *events += 1;
break; table.settled()
} });
Drained::Read { if next == table.shards.len() {
events: read, visit(Event::Synced);
consumed, *events += 1;
more, self.phase = Phase::FOLLOW;
} => { return None;
events += read; }
consumed_any |= consumed; *events += self.scan(table, next, upto, visit);
more_somewhere |= more; self.phase = Phase::Resync {
} next: next + 1,
} upto: Some(upto),
shard += 1; };
} (*events >= budget).then(|| self.progress(*events, false))
if lagged { }
self.phase = Phase::RESYNC;
continue; /// A pass over the journals from shard `next` on, or the rest of one
} /// that an earlier call cut short. Returns what to tell the caller if
self.phase = Phase::Follow { next: 0, upto: 0 }; /// the pass is to be the call's last, and `None` to go round again.
if next == 0 && !consumed_any && more_somewhere { fn follow_pass(
// A whole pass got nowhere, though there is more: &mut self,
// it is all beyond the earliest change still under table: &RibTable,
// way, which no pass can read past until it ends. next: usize,
// Asking again at once would spin until then. The upto: u64,
// change wakes the readers when it ends. budget: usize,
return self.progress(events, true); events: &mut usize,
} visit: &mut impl FnMut(Event<'_>),
if next != 0 || more_somewhere { ) -> Option<Progress> {
// This pass took up where an earlier call stopped, // Read before the pass: a change made during it shows as a newer
// so the shards before that point have not been // version, and `changed` then returns at once. And the pass reads
// looked at in this call; or some shard had more // up to one time in every shard alike, chosen so that about the
// than its share, or changes later than this // budget is read.
// pass's time. Go round again from the first. let (version, upto) = if next == 0 {
if events >= budget { (
return self.progress(events, false); table.version.load(Ordering::Acquire),
} self.time_for(budget - *events),
continue; )
} } else {
return self.progress(events, true); (self.seen, upto)
};
self.seen = version;
let mut consumed_any = false;
let mut more_somewhere = false;
for shard in next..table.shards.len() {
if *events >= budget {
self.phase = Phase::Follow { next: shard, upto };
return Some(self.progress(*events, false));
}
// Nothing new in this shard, by the number that is read
// without its lock: it is left alone, and so are the sessions
// writing to it.
if self.cursors[shard] == table.latest_seq(shard, self.kind) {
continue;
}
match self.drain(table, shard, upto, visit) {
Drained::Lagged => {
self.phase = Phase::RESYNC;
return None;
}
Drained::Read {
events: read,
consumed,
more,
} => {
*events += read;
consumed_any |= consumed;
more_somewhere |= more;
} }
} }
} }
self.phase = Phase::FOLLOW;
if next == 0 && !consumed_any && more_somewhere {
// A whole pass got nowhere, though there is more: it is all
// beyond the earliest change still under way, which no pass
// can read past until it ends. Asking again at once would spin
// until then. The change wakes the readers when it ends.
return Some(self.progress(*events, true));
}
if next != 0 || more_somewhere {
// This pass took up where an earlier call stopped, so the
// shards before that point have not been looked at in this
// call; or some shard had more than its share, or changes
// later than this pass's time. Go round again from the first.
return (*events >= budget).then(|| self.progress(*events, false));
}
Some(self.progress(*events, true))
} }
fn progress(&mut self, events: usize, caught_up: bool) -> Progress { fn progress(&mut self, events: usize, caught_up: bool) -> Progress {
@@ -706,6 +769,11 @@ impl Reader {
Progress { events, caught_up } Progress { events, caught_up }
} }
/// Whether the last [`Reader::poll`] read everything there was.
pub fn caught_up(&self) -> bool {
self.caught_up
}
/// Waits until there may be something to read. Returns at once if the /// Waits until there may be something to read. Returns at once if the
/// last [`Reader::poll`] did not catch up. /// last [`Reader::poll`] did not catch up.
pub async fn changed(&mut self) { pub async fn changed(&mut self) {
@@ -760,8 +828,13 @@ impl Reader {
/// as of `upto` on the table's clock, in one hold of the shard: what /// as of `upto` on the table's clock, in one hold of the shard: what
/// changed after that time is in the journal after that place, whether /// changed after that time is in the journal after that place, whether
/// or not the rows given out show it already. /// or not the rows given out show it already.
fn scan(&mut self, index: usize, upto: u64, visit: &mut impl FnMut(Event<'_>)) -> usize { fn scan(
let table = Arc::clone(&self.table); &mut self,
table: &RibTable,
index: usize,
upto: u64,
visit: &mut impl FnMut(Event<'_>),
) -> usize {
let shard = read(&table.shards[index]); let shard = read(&table.shards[index]);
for row in &shard.rows { for row in &shard.rows {
visit(self.event_for(row)); visit(self.event_for(row));
@@ -797,13 +870,15 @@ impl Reader {
// many changes were made with it. // many changes were made with it.
let mut earliest = None; let mut earliest = None;
for (index, shard) in self.table.shards.iter().enumerate() { for (index, shard) in self.table.shards.iter().enumerate() {
let shard = read(shard);
let journal = shard.journals.of(self.kind);
let cursor = self.cursors[index]; let cursor = self.cursors[index];
if cursor < journal.floor || cursor == journal.seq { if cursor == self.table.latest_seq(index, self.kind) {
continue; continue;
} }
let start = journal.entries.partition_point(|entry| entry.seq <= cursor); let shard = read(shard);
let journal = shard.journals.of(self.kind);
let Place::From(start) = journal.place_of(cursor) else {
continue;
};
if let Some(next) = journal.entries.get(start) { if let Some(next) = journal.entries.get(start) {
earliest = Some(earliest.map_or(next.stamp, |e: u64| e.min(next.stamp))); earliest = Some(earliest.map_or(next.stamp, |e: u64| e.min(next.stamp)));
} }
@@ -817,23 +892,27 @@ impl Reader {
/// Reads a shard's journal from this reader's place, no further than /// Reads a shard's journal from this reader's place, no further than
/// the changes made by `upto` on the table's clock. /// the changes made by `upto` on the table's clock.
fn drain(&mut self, index: usize, upto: u64, visit: &mut impl FnMut(Event<'_>)) -> Drained { fn drain(
let table = Arc::clone(&self.table); &mut self,
table: &RibTable,
index: usize,
upto: u64,
visit: &mut impl FnMut(Event<'_>),
) -> Drained {
let shard = read(&table.shards[index]); let shard = read(&table.shards[index]);
let journal = shard.journals.of(self.kind); let journal = shard.journals.of(self.kind);
let mut cursor = self.cursors[index]; let mut cursor = self.cursors[index];
if cursor < journal.floor { let start = match journal.place_of(cursor) {
return Drained::Lagged; Place::Lagged => return Drained::Lagged,
} Place::Nothing => {
if cursor == journal.seq { return Drained::Read {
return Drained::Read { events: 0,
events: 0, consumed: false,
consumed: false, more: false,
more: false, };
}; }
} Place::From(start) => start,
};
let start = journal.entries.partition_point(|entry| entry.seq <= cursor);
let mut events = 0; let mut events = 0;
let mut more = false; let mut more = false;
for entry in journal.entries.range(start..) { for entry in journal.entries.range(start..) {
@@ -842,13 +921,12 @@ impl Reader {
break; break;
} }
cursor = entry.seq; cursor = entry.seq;
let event = match journal.live(entry, &shard.rows, &shard.hasher) { let event = match journal.says(entry, &shard.rows, &shard.hasher) {
Some(Some(row)) => self.event_for(row), Says::Row(row) => self.event_for(row),
Some(None) => Event::Withdrawn { Says::Gone => Event::Withdrawn {
prefix: entry.prefix, prefix: entry.prefix,
}, },
// The prefix has changed again; a later entry has it. Says::Nothing => continue,
None => continue,
}; };
visit(event); visit(event);
events += 1; events += 1;
crates/rib/src/table/journal_tests.rs
+106 -5
View File
@@ -8,7 +8,7 @@ use proptest::prelude::*;
use super::super::tests::{Fixture, Route, external, internal, prefix, route}; use super::super::tests::{Fixture, Route, external, internal, prefix, route};
use super::super::{AttrSet, PeerIndex, PeerInfo, RibTable}; use super::super::{AttrSet, PeerIndex, PeerInfo, RibTable};
use super::{Event, Limits, Reader}; use super::{Event, Kind, Limits, Reader, read};
/// What a follower would keep: its own copy of the best routes, brought /// What a follower would keep: its own copy of the best routes, brought
/// up to date from the events it is given. /// up to date from the events it is given.
@@ -53,7 +53,10 @@ impl Mirror {
} }
Event::Routes { prefix, routes } => { Event::Routes { prefix, routes } => {
self.routes_events += 1; self.routes_events += 1;
assert!(!routes.is_empty(), "a prefix with no routes is withdrawn"); assert!(
routes.iter().next().is_some(),
"a prefix with no routes is withdrawn"
);
self.routes.insert( self.routes.insert(
prefix, prefix,
sorted( sorted(
@@ -115,9 +118,6 @@ impl Mirror {
); );
} }
} }
}
impl Mirror {
/// For the mirror of a follower of all routes: it holds every route in /// For the mirror of a follower of all routes: it holds every route in
/// the table, and knows which is best. /// the table, and knows which is best.
#[track_caller] #[track_caller]
@@ -213,6 +213,107 @@ fn a_reader_from_now_is_given_the_changes_and_not_the_table() {
assert_eq!((best, withdrawn), (vec![new], vec![old])); assert_eq!((best, withdrawn), (vec![new], vec![old]));
} }
/// The number a reader checks without a shard's lock must be the journal's
/// own after every kind of change, whether or not anyone is reading: a
/// reader skips a shard on the strength of it, and a shard with news that
/// said it had none would never be read until the next change.
#[test]
fn the_lock_free_number_is_the_journals_after_every_kind_of_change() {
let mut fixture = Fixture::new(&[external(1), external(2)]);
let prefixes: Vec<IpPrefix> = (0..16).map(nth_prefix).collect();
let agree = |table: &RibTable| {
for (index, shard) in table.shards.iter().enumerate() {
let shard = read(shard);
for kind in [Kind::Best, Kind::All] {
assert_eq!(
table.latest_seq(index, kind),
shard.journals.of(kind).seq,
"shard {index}, {kind:?}"
);
}
}
};
// Without a reader, with one of each kind, and with both.
for readers in 0..4 {
let _best = (readers & 1 != 0).then(|| fixture.table.reader());
let _all = (readers & 2 != 0).then(|| fixture.table.reader_of_all_routes());
agree(&fixture.table);
for p in &prefixes {
fixture.announce(0, p, &Route::via(&[1]));
}
agree(&fixture.table);
// The same prefixes again, with other attributes.
for p in &prefixes {
fixture.announce(0, p, &Route::via(&[1, 2]));
}
agree(&fixture.table);
// A second route for each, then the last route of one taken away.
for p in &prefixes {
fixture.announce(1, p, &Route::via(&[2, 3]));
}
agree(&fixture.table);
fixture.withdraw(0, &prefixes[0]);
agree(&fixture.table);
fixture.withdraw(1, &prefixes[0]);
agree(&fixture.table);
fixture.table.remove_peer(fixture.peers[1]);
agree(&fixture.table);
fixture.peers[1] = fixture.table.add_peer(external(2)).unwrap();
fixture
.table
.mark_stale(fixture.peers[0], &[IpVersion::Ipv4]);
agree(&fixture.table);
fixture
.table
.sweep_stale(fixture.peers[0], &[IpVersion::Ipv4]);
agree(&fixture.table);
assert!(
fixture.table.is_empty(),
"round {readers} left routes behind"
);
}
}
/// A shard that had nothing new for a reader is passed over without its
/// lock. The next change to it must bring the reader back.
#[test]
fn a_shard_passed_over_is_read_again_once_it_has_news() {
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(0, &prefix("10.0.0.0/8"), &Route::via(&[1]));
let mut events = Vec::new();
let progress = reader.poll(100, |event| events.push(format!("{event:?}")));
assert!(progress.caught_up);
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].starts_with("Best"), "{events:?}");
}
/// As [`a_shard_passed_over_is_read_again_once_it_has_news`], for a reader
/// that begins where the journals are rather than with the table.
#[test]
fn a_shard_passed_over_by_a_reader_from_now_is_read_again_once_it_has_news() {
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
fixture.announce(0, &prefix("192.0.2.0/24"), &Route::via(&[1]));
let mut reader = fixture.table.reader_from_now();
assert!(reader.poll(100, |_| panic!("no event expected")).caught_up);
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(0, &prefix("10.0.0.0/8"), &Route::via(&[1]));
let mut events = Vec::new();
let progress = reader.poll(100, |event| events.push(format!("{event:?}")));
assert!(progress.caught_up);
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].starts_with("Best"), "{events:?}");
}
/// A change to a route that is not the best is nothing to a follower of /// A change to a route that is not the best is nothing to a follower of
/// the best routes, and is what a follower of all routes is there for. /// the best routes, and is what a follower of all routes is there for.
#[test] #[test]
crates/rib/src/table/mod.rs
+88 -28
View File
@@ -193,13 +193,16 @@ impl Row {
.position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id) .position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id)
} }
fn push(&mut self, path: Path, path_id: u32) { /// Adds a route. `peers` is how many peers the table has: a prefix
/// that one of them announces, the others mostly announce too, so a
/// row is given room for that many at its first growth.
fn push(&mut self, path: Path, path_id: u32, peers: usize) {
if path_id != 0 && self.path_ids.is_empty() { if path_id != 0 && self.path_ids.is_empty() {
self.path_ids = vec![0; self.paths.len()]; self.path_ids = vec![0; self.paths.len()];
} }
push_tight(&mut self.paths, path); push_tight(&mut self.paths, path, peers);
if !self.path_ids.is_empty() { if !self.path_ids.is_empty() {
push_tight(&mut self.path_ids, path_id); push_tight(&mut self.path_ids, path_id, peers);
} }
} }
@@ -310,10 +313,21 @@ struct Shard {
journals: journal::Journals, journals: journal::Journals,
} }
/// The routing table. Shared between threads by reference; every method /// A shard, and beside its lock what is read without taking it.
/// takes `&self`. struct ShardSlot {
lock: RwLock<Shard>,
/// The sequence number of each of the shard's journals as of the last
/// change to the shard, stored by the change while it still holds the
/// lock ([`RibTable::change`]). A reader looks at it first, and passes
/// over a shard with nothing new for it without stopping the shard's
/// writers.
latest: journal::ByKind<AtomicU64>,
}
/// The routing table. Shared between threads by reference: nothing takes
/// `&mut self`, and the readers ([`Reader`]) keep an `Arc` of their own.
pub struct RibTable { pub struct RibTable {
shards: Box<[RwLock<Shard>]>, shards: Box<[ShardSlot]>,
hasher: RandomState, hasher: RandomState,
attrs: AttrStore, attrs: AttrStore,
/// Which peer numbers are in use. Changing the peer list also takes /// Which peer numbers are in use. Changing the peer list also takes
@@ -354,15 +368,16 @@ impl RibTable {
let hasher = RandomState::default(); let hasher = RandomState::default();
Self { Self {
shards: (0..shards.max(1)) shards: (0..shards.max(1))
.map(|_| { .map(|_| ShardSlot {
RwLock::new(Shard { lock: RwLock::new(Shard {
rows: HashTable::new(), rows: HashTable::new(),
routes: 0, routes: 0,
peers: Vec::new(), peers: Vec::new(),
hasher: hasher.clone(), hasher: hasher.clone(),
scratch: select::Scratch::default(), scratch: select::Scratch::default(),
journals: journal::Journals::new(limits), journals: journal::Journals::new(limits),
}) }),
latest: journal::ByKind::default(),
}) })
.collect(), .collect(),
hasher, hasher,
@@ -513,7 +528,7 @@ impl RibTable {
/// How many routes of `peer` are stale; see [`RibTable::mark_stale`]. /// How many routes of `peer` are stale; see [`RibTable::mark_stale`].
/// Looks at every prefix in the table. /// Looks at every prefix in the table.
#[cfg(test)] #[cfg(test)]
pub(super) fn count_stale(&self, peer: PeerIndex) -> usize { fn count_stale(&self, peer: PeerIndex) -> usize {
self.shards self.shards
.iter() .iter()
.map(|shard| { .map(|shard| {
@@ -613,18 +628,22 @@ impl RibTable {
/// journal one of them follows was written to. /// journal one of them follows was written to.
fn change<'a, T>( fn change<'a, T>(
&'a self, &'a self,
parts: impl IntoIterator<Item = (&'a RwLock<Shard>, T)>, parts: impl IntoIterator<Item = (&'a ShardSlot, T)>,
mut apply: impl FnMut(&mut Shard, T), mut apply: impl FnMut(&mut Shard, T),
) { ) {
let change = self.begin_change(); let change = self.begin_change();
let mut news = false; let mut news = false;
for (shard, part) in parts { for (slot, part) in parts {
let mut shard = write(shard); let mut shard = write(slot);
shard.journals.set_stamp(change.stamp); shard.journals.set_stamp(change.stamp);
let before = shard.journals.mark(); let before = shard.journals.mark();
apply(&mut shard, part); apply(&mut shard, part);
shard.tidy_journals(); shard.tidy_journals();
news |= shard.journals.has_news_since(before); news |= shard.journals.has_news_since(before);
// Once per change, not once per route; and before the lock is
// let go, so that a reader that sees the new number and takes
// the lock finds the entries behind it.
shard.journals.publish(&slot.latest);
} }
drop(change); drop(change);
if news { if news {
@@ -648,9 +667,18 @@ impl RibTable {
} }
} }
/// The sequence number of shard `shard`'s journal of `kind` as of the
/// last change to the shard, without the shard's lock.
fn latest_seq(&self, shard: usize, kind: journal::Kind) -> u64 {
self.shards[shard]
.latest
.of(kind)
.load(std::sync::atomic::Ordering::Acquire)
}
/// The time on the clock up to which every change is complete in every /// The time on the clock up to which every change is complete in every
/// shard: everything before the earliest one still under way. /// shard: everything before the earliest one still under way.
pub(super) fn settled(&self) -> u64 { fn settled(&self) -> u64 {
let in_progress = self let in_progress = self
.in_progress .in_progress
.lock() .lock()
@@ -670,17 +698,31 @@ impl RibTable {
self.len() == 0 self.len() == 0
} }
/// Makes room for `prefixes` prefixes, spread over the shards, so that
/// a table of that size goes in without the shards growing as it
/// does. Growing a shard moves every row it has into a bigger table;
/// over a full table that was a seventh of what a session spent
/// taking it in. Room that is already there is kept.
pub fn expect_prefixes(&self, prefixes: usize) {
let each = prefixes.div_ceil(self.shards.len());
for shard in &self.shards {
let mut shard = write(shard);
let hasher = shard.hasher.clone();
let additional = each.saturating_sub(shard.rows.len());
shard
.rows
.reserve(additional, |row| hasher.hash_one(row.prefix));
}
}
/// How many prefixes have at least one route. /// How many prefixes have at least one route.
pub fn prefix_count(&self) -> usize { pub fn prefix_count(&self) -> usize {
self.shards.iter().map(|shard| read(shard).rows.len()).sum() self.shards.iter().map(|shard| read(shard).rows.len()).sum()
} }
/// The best route for `prefix`: who it is from, and its attributes. /// The best route for `prefix`: who it is from, and its attributes.
pub fn best(&self, prefix: &IpPrefix) -> Option<(PeerIndex, Arc<AttrSet>)> { pub fn best(&self, prefix: &IpPrefix) -> Option<Route> {
self.with_row(prefix, |row| { self.with_row(prefix, |row| row.route(row.best()).to_owned())
let best = &row.paths[row.best()];
(best.peer(), Arc::clone(&best.attrs))
})
} }
/// Every route for `prefix`. /// Every route for `prefix`.
@@ -778,14 +820,14 @@ impl RibTable {
} }
} }
fn read(shard: &RwLock<Shard>) -> RwLockReadGuard<'_, Shard> { fn read(shard: &ShardSlot) -> RwLockReadGuard<'_, Shard> {
// A panic under the lock cannot leave a row half changed in a way that // A panic under the lock cannot leave a row half changed in a way that
// is unsafe to read; the poison is ignored, as for the server's tables. // is unsafe to read; the poison is ignored, as for the server's tables.
shard.read().unwrap_or_else(PoisonError::into_inner) shard.lock.read().unwrap_or_else(PoisonError::into_inner)
} }
fn write(shard: &RwLock<Shard>) -> RwLockWriteGuard<'_, Shard> { fn write(shard: &ShardSlot) -> RwLockWriteGuard<'_, Shard> {
shard.write().unwrap_or_else(PoisonError::into_inner) shard.lock.write().unwrap_or_else(PoisonError::into_inner)
} }
/// A change to the table that is under way; see [`RibTable::begin_change`]. /// A change to the table that is under way; see [`RibTable::begin_change`].
@@ -838,12 +880,30 @@ fn spread(hash: u64, count: usize) -> usize {
((stirred * count as u64) >> 32) as usize ((stirred * count as u64) >> 32) as usize
} }
/// The most room a row is given at its first growth, however many peers
/// there are: a route server's clients do not all announce every prefix.
const FIRST_ROOM: usize = 8;
/// Adds to a vector without the doubling that `push` does: a row of a few /// Adds to a vector without the doubling that `push` does: a row of a few
/// hundred routes should not hold room for as many again. Grows by a /// hundred routes should not hold room for as many again. The first
/// quarter, which keeps adding cheap on average. /// growth is to `expected` (at most [`FIRST_ROOM`]), since with several
fn push_tight<T>(vec: &mut Vec<T>, value: T) { /// full tables nearly every row gets that many routes, and growing a row
/// one route at a time was a twelfth of what taking in four tables cost.
/// After that it grows by a quarter, which keeps adding cheap on average.
fn push_tight<T>(vec: &mut Vec<T>, value: T, expected: usize) {
if vec.len() == vec.capacity() { if vec.len() == vec.capacity() {
vec.reserve_exact((vec.len() / 4).max(1)); let first = expected.min(FIRST_ROOM);
let more = if vec.capacity() < first {
first - vec.len()
} else if vec.capacity() < FIRST_ROOM {
// A peer that sends several routes for a prefix (ADD-PATH)
// takes a row past the peers' count: doubled while small,
// and from there by a quarter.
vec.capacity()
} else {
(vec.len() / 4).max(1)
};
vec.reserve_exact(more);
} }
vec.push(value); vec.push(value);
} }
@@ -885,7 +945,7 @@ impl Shard {
let (outcome, best_changed) = match row.position(peer, path_id) { let (outcome, best_changed) = match row.position(peer, path_id) {
None => { None => {
row.push(path, path_id); row.push(path, path_id, self.peers.len());
self.routes += 1; self.routes += 1;
let outcome = Outcome { let outcome = Outcome {
added: 1, added: 1,
crates/rib/src/table/tests.rs
+31 -3
View File
@@ -162,7 +162,7 @@ impl Fixture {
/// Which peer (by order added) the best route for `prefix` is from. /// Which peer (by order added) the best route for `prefix` is from.
fn best(&self, prefix: &IpPrefix) -> Option<usize> { fn best(&self, prefix: &IpPrefix) -> Option<usize> {
let (peer, _) = self.table.best(prefix)?; let peer = self.table.best(prefix)?.peer;
self.peers.iter().position(|p| *p == peer) self.peers.iter().position(|p| *p == peer)
} }
} }
@@ -957,13 +957,13 @@ fn a_row_can_hold_more_routes_than_a_sixteen_bit_index_counts() {
let mut scratch = select::Scratch::default(); let mut scratch = select::Scratch::default();
let count = usize::from(u16::MAX) + 10; let count = usize::from(u16::MAX) + 10;
for path_id in 2..=count as u32 { for path_id in 2..=count as u32 {
row.push(path(&worse), path_id); row.push(path(&worse), path_id, peers.len());
} }
// Equal routes from one peer: the lowest identifier is the best. // Equal routes from one peer: the lowest identifier is the best.
assert!(!select::after_add(&mut row, &peers, &mut scratch)); assert!(!select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), 0); assert_eq!(row.best(), 0);
row.push(path(&better), count as u32 + 1); row.push(path(&better), count as u32 + 1, peers.len());
assert!(select::after_add(&mut row, &peers, &mut scratch)); assert!(select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), count); assert_eq!(row.best(), count);
assert_eq!(row.route(row.best()).path_id, count as u32 + 1); assert_eq!(row.route(row.best()).path_id, count as u32 + 1);
@@ -1002,3 +1002,31 @@ fn routes_can_be_picked_out_of_the_table() {
); );
assert!(table.routes_where(0, |_| true).is_empty()); assert!(table.routes_where(0, |_| true).is_empty());
} }
#[test]
fn a_table_told_what_to_expect_has_the_room_before_the_routes_come() {
let table = RibTable::new(8);
table.expect_prefixes(100_000);
let room: usize = table
.shards
.iter()
.map(|shard| read(shard).rows.capacity())
.sum();
assert!(room >= 100_000, "room for {room} prefixes");
// Asking again for less, or for the same, takes nothing away.
table.expect_prefixes(10);
let still: usize = table
.shards
.iter()
.map(|shard| read(shard).rows.capacity())
.sum();
assert_eq!(still, room);
// And it is a table like any other.
let peer = table.add_peer(external(1)).unwrap();
let attrs = table
.attrs()
.intern(&Route::via(&[64512]).attributes(), None)
.unwrap();
table.announce(peer, &attrs, &[Nlri::from(prefix("192.0.2.0/24"))]);
assert_eq!(table.prefix_count(), 1);
}
crates/server/Cargo.toml
+1
View File
@@ -8,6 +8,7 @@ repository.workspace = true
version.workspace = true version.workspace = true
[dependencies] [dependencies]
foldhash.workspace = true
bgp-packet.workspace = true bgp-packet.workspace = true
bgp-primitives.workspace = true bgp-primitives.workspace = true
bgp-rib.workspace = true bgp-rib.workspace = true
crates/server/proto/config.proto
+11 -1
View File
@@ -407,7 +407,9 @@ message Peer {
// the table with it, and shows an established session with fewer than // the table with it, and shows an established session with fewer than
// half as needing attention: a peer that is up but has sent next to // half as needing attention: a peer that is up but has sent next to
// nothing looks, without this, exactly like one that is fine. Zero, the // nothing looks, without this, exactly like one that is fine. Zero, the
// default, compares nothing. // default, compares nothing. The routing table is also made with room
// for the largest of these from the start, so that it need not grow,
// and so use twice its memory for a moment, while the tables come in.
uint32 expected_routes = 17; uint32 expected_routes = 17;
} }
@@ -440,6 +442,14 @@ message ExportPolicy {
// Pass on the best routes learned from other peers. // Pass on the best routes learned from other peers.
bool learned_routes = 1; bool learned_routes = 1;
// Pass on every route learned from other peers, not only the best of
// each prefix, each with a path identifier (ADD-PATH, RFC 7911). For a
// route collector, which wants to see every path. Needs learned_routes,
// and the peer's add_path to offer sending; for a family the peer did
// not agree to receive several routes for, the best goes as usual. The
// export policy and the limits apply to every route alike.
bool all_paths = 6;
// Communities put on every route announced to the peer: which router // Communities put on every route announced to the peer: which router
// it went through, say. "64601:100", or a well-known name. // it went through, say. "64601:100", or a well-known name.
repeated string add_communities = 2; repeated string add_communities = 2;
crates/server/src/address.rs
+14
View File
@@ -54,6 +54,14 @@ impl From<IpAddr> for PeerAddress {
} }
} }
/// Whether `wanted`, as someone typed it, names the peer at `address`
/// that is configured as `name`, if it is: by that name, or by the address
/// as [`PeerAddress::is_called`] reads it. The one rule for every place a
/// peer is asked for by name.
pub fn names_peer(wanted: &str, name: Option<&str>, address: PeerAddress) -> bool {
name == Some(wanted) || address.is_called(wanted)
}
/// The address, and the interface by name if there is one: `fe80::1%eth0`. /// The address, and the interface by name if there is one: `fe80::1%eth0`.
/// An interface that is no longer there goes by its number. /// An interface that is no longer there goes by its number.
impl fmt::Display for PeerAddress { impl fmt::Display for PeerAddress {
@@ -325,6 +333,12 @@ mod tests {
assert!(there.is_called("fe80::1")); assert!(there.is_called("fe80::1"));
assert!(!there.is_called("fe80::1%other")); assert!(!there.is_called("fe80::1%other"));
assert!(!there.is_called("fe80::2")); assert!(!there.is_called("fe80::2"));
// By its configured name as well, where it has one.
assert!(names_peer("upstream", Some("upstream"), there));
assert!(names_peer("fe80::1", Some("upstream"), there));
assert!(names_peer("fe80::1", None, there));
assert!(!names_peer("upstream", None, there));
assert!(!names_peer("other", Some("upstream"), there));
assert_eq!( assert_eq!(
socket_text(peer.to_dial().unwrap()), socket_text(peer.to_dial().unwrap()),
format!("[fe80::1%{name}]:179") format!("[fe80::1%{name}]:179")
crates/server/src/advertise.rs
+1267 -318
View File
File diff suppressed because it is too large. Load diff
crates/server/src/config.rs
+183 -138
View File
@@ -11,6 +11,7 @@ use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs}; use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_rpki::RpkiConfig; use bgp_rpki::RpkiConfig;
use protobuf::EnumOrUnknown;
use crate::address::{Endpoint, PeerAddress}; use crate::address::{Endpoint, PeerAddress};
use crate::error::{ConfigError, ExportProblem, ImportProblem, OriginateProblem, PeerProblem}; use crate::error::{ConfigError, ExportProblem, ImportProblem, OriginateProblem, PeerProblem};
@@ -141,20 +142,28 @@ impl Config {
let listen_endpoints = parse_listen_endpoints(&proto.listen_endpoints)?; let listen_endpoints = parse_listen_endpoints(&proto.listen_endpoints)?;
let defaults = PeerDefaults { let defaults = PeerDefaults {
export: Arc::new(match proto.export.as_ref() { export: Arc::new(
Some(export) => parse_export(export)?, proto
None => ExportPolicy::default(), .export
}), .as_ref()
export_limits: match proto.export_limits.as_ref() { .map(parse_export)
Some(limits) => parse_export_limits(limits) .transpose()?
.map_err(ConfigError::ExportLimits)? .unwrap_or_default(),
.map(Arc::new), ),
None => None, export_limits: proto
}, .export_limits
graceful_restart: match proto.graceful_restart.as_ref() { .as_ref()
Some(own) => GracefulRestartConfig::default().overridden_by(own)?, .map(parse_export_limits)
None => GracefulRestartConfig::default(), .transpose()
}, .map_err(ConfigError::ExportLimits)?
.flatten()
.map(Arc::new),
graceful_restart: proto
.graceful_restart
.as_ref()
.map(|own| GracefulRestartConfig::default().overridden_by(own))
.transpose()?
.unwrap_or_default(),
}; };
let mut peers = parse_peers(proto.peers, proto.local_asn, &defaults)?; let mut peers = parse_peers(proto.peers, proto.local_asn, &defaults)?;
check_roles( check_roles(
@@ -179,16 +188,7 @@ impl Config {
None => None, None => None,
}; };
let rpki = match proto.rpki.as_ref() { let rpki = proto.rpki.as_ref().map(parse_rpki).transpose()?;
Some(rpki) => Some(parse_rpki(rpki)?),
None => None,
};
let connect_retry = if proto.connect_retry_seconds == 0 {
DEFAULT_CONNECT_RETRY
} else {
Duration::from_secs(proto.connect_retry_seconds as u64)
};
Ok(Self { Ok(Self {
router_id, router_id,
@@ -197,7 +197,7 @@ impl Config {
hold_time_secs: DEFAULT_HOLD_TIME_SECS, hold_time_secs: DEFAULT_HOLD_TIME_SECS,
listen_endpoints, listen_endpoints,
peers, peers,
connect_retry, connect_retry: seconds(proto.connect_retry_seconds).unwrap_or(DEFAULT_CONNECT_RETRY),
collision_detect_established_state: proto.collision_detect_established_state, collision_detect_established_state: proto.collision_detect_established_state,
accept_unconfigured_peers: proto.accept_unconfigured_peers, accept_unconfigured_peers: proto.accept_unconfigured_peers,
originate, originate,
@@ -398,20 +398,37 @@ fn parse_identifier(text: &str) -> Option<Ipv4Addr> {
.filter(|id| !id.is_unspecified()) .filter(|id| !id.is_unspecified())
} }
/// A number the configuration has as zero is a field left out: the
/// protocol buffer has no way of telling the two apart, and nothing
/// configured here is meaningfully zero.
fn given(n: u32) -> Option<u32> {
(n > 0).then_some(n)
}
/// A time given in seconds, if it was given.
fn seconds(n: u32) -> Option<Duration> {
given(n).map(|n| Duration::from_secs(n.into()))
}
/// A list of prefixes, if any are listed: an empty list leaves the setting
/// out rather than allowing nothing.
fn parse_prefixes_if_any(
ranges: &[proto::PrefixRange],
) -> Result<Option<PrefixSet>, ImportProblem> {
if ranges.is_empty() {
return Ok(None);
}
parse_prefixes(ranges).map(Some)
}
fn parse_kernel(kernel: &proto::Kernel) -> FibConfig { fn parse_kernel(kernel: &proto::Kernel) -> FibConfig {
let defaults = FibConfig::default(); let defaults = FibConfig::default();
FibConfig { FibConfig {
// The kernel has no table 0 to put routes in. // The kernel has no table 0 to put routes in.
table: match kernel.table { table: given(kernel.table).unwrap_or(defaults.table),
0 => defaults.table,
table => table,
},
metric: kernel.metric.unwrap_or(defaults.metric), metric: kernel.metric.unwrap_or(defaults.metric),
nexthop_objects: !kernel.without_nexthop_objects, nexthop_objects: !kernel.without_nexthop_objects,
resolution_table: match kernel.resolution_table { resolution_table: given(kernel.resolution_table).unwrap_or(defaults.resolution_table),
0 => defaults.resolution_table,
table => table,
},
resolve_via_default: kernel.resolve_via_default_route, resolve_via_default: kernel.resolve_via_default_route,
stale_time: kernel.stale_seconds.map_or(defaults.stale_time, |seconds| { stale_time: kernel.stale_seconds.map_or(defaults.stale_time, |seconds| {
Duration::from_secs(seconds.into()) Duration::from_secs(seconds.into())
@@ -428,8 +445,12 @@ fn parse_prepend(prepend: u32) -> Option<u8> {
} }
fn parse_export(export: &proto::ExportPolicy) -> Result<ExportPolicy, ExportProblem> { fn parse_export(export: &proto::ExportPolicy) -> Result<ExportPolicy, ExportProblem> {
if export.all_paths && !export.learned_routes {
return Err(ExportProblem::AllPathsWithoutLearnedRoutes);
}
Ok(ExportPolicy { Ok(ExportPolicy {
learned_routes: export.learned_routes, learned_routes: export.learned_routes,
all_paths: export.all_paths,
marks: Marks::parse( marks: Marks::parse(
&export.remove_communities, &export.remove_communities,
&export.add_communities, &export.add_communities,
@@ -448,12 +469,8 @@ fn parse_export_limits(
limits: &proto::ExportLimits, limits: &proto::ExportLimits,
) -> Result<Option<ExportLimits>, ImportProblem> { ) -> Result<Option<ExportLimits>, ImportProblem> {
let limits = ExportLimits { let limits = ExportLimits {
prefixes: if limits.prefixes.is_empty() { prefixes: parse_prefixes_if_any(&limits.prefixes)?,
None max_prefixes: given(limits.max_prefixes),
} else {
Some(parse_prefixes(&limits.prefixes)?)
},
max_prefixes: Some(limits.max_prefixes).filter(|limit| *limit > 0),
}; };
Ok((limits != ExportLimits::default()).then_some(limits)) Ok((limits != ExportLimits::default()).then_some(limits))
} }
@@ -468,38 +485,17 @@ fn parse_peer(peer: Peer, defaults: &PeerDefaults) -> Result<PeerConfig, PeerPro
return Err(PeerProblem::RemoteAsn); return Err(PeerProblem::RemoteAsn);
} }
let hold_time_secs = match peer.hold_time_seconds { let hold_time_secs = parse_hold_time(peer.hold_time_seconds)?;
None => DEFAULT_HOLD_TIME_SECS, let local_address = (!peer.local_address.is_empty())
// RFC 4271 §4.2: zero, or at least three seconds. .then(|| {
Some(1 | 2) => return Err(PeerProblem::HoldTime), IpAddr::from_str(&peer.local_address)
Some(secs) => u16::try_from(secs).map_err(|_| PeerProblem::HoldTime)?, .map_err(|_| PeerProblem::LocalAddress(peer.local_address.clone()))
}; })
.transpose()?;
let local_address = if peer.local_address.is_empty() { if local_address.is_some_and(|ip| ip.is_ipv4() != endpoint.ip.is_ipv4()) {
None return Err(PeerProblem::AddressFamilyMismatch);
} else { }
let ip = IpAddr::from_str(&peer.local_address) let families = parse_families(&peer.address_families)?;
.map_err(|_| PeerProblem::LocalAddress(peer.local_address.clone()))?;
if ip.is_ipv4() != endpoint.ip.is_ipv4() {
return Err(PeerProblem::AddressFamilyMismatch);
}
Some(ip)
};
let families = if peer.address_families.is_empty() {
FamilySet::supported()
} else {
peer.address_families
.iter()
.map(|family| match family.enum_value() {
Ok(proto::AddressFamily::IPV4_UNICAST) => Ok(AfiSafi::IPV4_UNICAST),
Ok(proto::AddressFamily::IPV6_UNICAST) => Ok(AfiSafi::IPV6_UNICAST),
Ok(proto::AddressFamily::ADDRESS_FAMILY_UNSPECIFIED) | Err(_) => {
Err(PeerProblem::AddressFamily)
}
})
.collect::<Result<_, _>>()?
};
let add_path = match peer.add_path.enum_value() { let add_path = match peer.add_path.enum_value() {
Ok(proto::AddPath::ADD_PATH_OFF) => None, Ok(proto::AddPath::ADD_PATH_OFF) => None,
@@ -509,20 +505,14 @@ fn parse_peer(peer: Peer, defaults: &PeerDefaults) -> Result<PeerConfig, PeerPro
Err(_) => return Err(PeerProblem::AddPath), Err(_) => return Err(PeerProblem::AddPath),
}; };
let import = match peer.import.as_ref() { let import = peer
Some(import) => parse_import(import)?, .import
None => ImportPolicy::default(), .as_ref()
}; .map(parse_import)
.transpose()?
.unwrap_or_default();
let role = match peer.role.enum_value() { let role = parse_role(peer.role)?;
Ok(proto::Role::ROLE_UNSPECIFIED) => None,
Ok(proto::Role::PROVIDER) => Some(PeerRole::Provider),
Ok(proto::Role::CUSTOMER) => Some(PeerRole::Customer),
Ok(proto::Role::PEER) => Some(PeerRole::Peer),
Ok(proto::Role::ROUTE_SERVER) => Some(PeerRole::RouteServer),
Ok(proto::Role::ROUTE_SERVER_CLIENT) => Some(PeerRole::RouteServerClient),
Err(_) => return Err(PeerProblem::Role),
};
if peer.require_role && role.is_none() { if peer.require_role && role.is_none() {
return Err(PeerProblem::RequireRoleWithoutRole); return Err(PeerProblem::RequireRoleWithoutRole);
} }
@@ -546,24 +536,74 @@ fn parse_peer(peer: Peer, defaults: &PeerDefaults) -> Result<PeerConfig, PeerPro
import, import,
role, role,
require_role: peer.require_role, require_role: peer.require_role,
export: match peer.export.as_ref() { export: peer
Some(own) => Arc::new(parse_export(own)?), .export
None => Arc::clone(&defaults.export), .as_ref()
}, .map(parse_export)
export_limits: match peer.export_limits.as_ref() { .transpose()?
Some(own) => parse_export_limits(own) .map_or_else(|| Arc::clone(&defaults.export), Arc::new),
.map_err(PeerProblem::ExportLimits)? // A block of the peer's own takes the place of the top-level one,
.map(Arc::new), // an empty block included: that lifts the limits.
None => defaults.export_limits.clone(), export_limits: peer
}, .export_limits
graceful_restart: match peer.graceful_restart.as_ref() { .as_ref()
Some(own) => defaults.graceful_restart.overridden_by(own)?, .map(parse_export_limits)
None => defaults.graceful_restart, .transpose()
}, .map_err(PeerProblem::ExportLimits)?
.map_or_else(|| defaults.export_limits.clone(), |own| own.map(Arc::new)),
graceful_restart: peer
.graceful_restart
.as_ref()
.map(|own| defaults.graceful_restart.overridden_by(own))
.transpose()?
.unwrap_or(defaults.graceful_restart),
groups: peer.groups, groups: peer.groups,
reflector_client: peer.route_reflector_client, reflector_client: peer.route_reflector_client,
passive: peer.passive, passive: peer.passive,
expected_routes: Some(peer.expected_routes).filter(|n| *n > 0), expected_routes: given(peer.expected_routes),
})
}
/// The hold time a peer is offered: the default where none is given.
fn parse_hold_time(seconds: Option<u32>) -> Result<u16, PeerProblem> {
match seconds {
None => Ok(DEFAULT_HOLD_TIME_SECS),
// RFC 4271 §4.2: zero, or at least three seconds.
Some(1 | 2) => Err(PeerProblem::HoldTime),
Some(secs) => u16::try_from(secs).map_err(|_| PeerProblem::HoldTime),
}
}
/// The address families offered to a peer: every one there is, unless
/// some are listed.
fn parse_families(
families: &[EnumOrUnknown<proto::AddressFamily>],
) -> Result<FamilySet, PeerProblem> {
if families.is_empty() {
return Ok(FamilySet::supported());
}
families
.iter()
.map(|family| match family.enum_value() {
Ok(proto::AddressFamily::IPV4_UNICAST) => Ok(AfiSafi::IPV4_UNICAST),
Ok(proto::AddressFamily::IPV6_UNICAST) => Ok(AfiSafi::IPV6_UNICAST),
Ok(proto::AddressFamily::ADDRESS_FAMILY_UNSPECIFIED) | Err(_) => {
Err(PeerProblem::AddressFamily)
}
})
.collect()
}
/// What the peer is to us, if that was said.
fn parse_role(role: EnumOrUnknown<proto::Role>) -> Result<Option<PeerRole>, PeerProblem> {
Ok(match role.enum_value() {
Ok(proto::Role::ROLE_UNSPECIFIED) => None,
Ok(proto::Role::PROVIDER) => Some(PeerRole::Provider),
Ok(proto::Role::CUSTOMER) => Some(PeerRole::Customer),
Ok(proto::Role::PEER) => Some(PeerRole::Peer),
Ok(proto::Role::ROUTE_SERVER) => Some(PeerRole::RouteServer),
Ok(proto::Role::ROUTE_SERVER_CLIENT) => Some(PeerRole::RouteServerClient),
Err(_) => return Err(PeerProblem::Role),
}) })
} }
@@ -580,7 +620,6 @@ fn parse_rpki(rpki: &proto::Rpki) -> Result<RpkiConfig, ConfigError> {
port => parse_endpoint(&cache.address, port), port => parse_endpoint(&cache.address, port),
}) })
.collect::<Result<_, _>>()?; .collect::<Result<_, _>>()?;
let seconds = |seconds: u32| (seconds > 0).then(|| Duration::from_secs(seconds.into()));
Ok(RpkiConfig { Ok(RpkiConfig {
caches, caches,
refresh: seconds(rpki.refresh_seconds), refresh: seconds(rpki.refresh_seconds),
@@ -591,20 +630,20 @@ fn parse_rpki(rpki: &proto::Rpki) -> Result<RpkiConfig, ConfigError> {
fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProblem> { fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProblem> {
let defaults = ImportPolicy::default(); let defaults = ImportPolicy::default();
// A length of zero is the field left out. let longest = |field, value: u32, version: IpVersion, default: u8| {
let longest = |field, value: u32, version: IpVersion, default: u8| match value { given(value).map_or(Ok(default), |value| {
0 => Ok(default), u8::try_from(value)
value => u8::try_from(value) .ok()
.ok() .filter(|length| *length <= version.bits())
.filter(|length| *length <= version.bits()) .ok_or(ImportProblem::PrefixLength {
.ok_or(ImportProblem::PrefixLength { field,
field, value,
value, longest: version.bits(),
longest: version.bits(), })
}), })
}; };
let prefixes = parse_prefixes(&import.prefixes)?; let prefixes = parse_prefixes_if_any(&import.prefixes)?;
let rules = import let rules = import
.rules .rules
.iter() .iter()
@@ -618,11 +657,8 @@ fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProb
.collect::<Result<Vec<_>, _>>()?; .collect::<Result<Vec<_>, _>>()?;
Ok(ImportPolicy { Ok(ImportPolicy {
max_prefixes: Some(import.max_prefixes).filter(|limit| *limit > 0), max_prefixes: given(import.max_prefixes),
hold_down: match import.hold_down_seconds { hold_down: seconds(import.hold_down_seconds).unwrap_or(defaults.hold_down),
0 => defaults.hold_down,
seconds => Duration::from_secs(seconds.into()),
},
accept_bogons: import.accept_bogons, accept_bogons: import.accept_bogons,
accept_default_route: import.accept_default_route, accept_default_route: import.accept_default_route,
max_ipv4_length: longest( max_ipv4_length: longest(
@@ -642,6 +678,7 @@ fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProb
accept_aspa_invalid: import.accept_aspa_invalid, accept_aspa_invalid: import.accept_aspa_invalid,
accept_as_sets: import.accept_as_sets, accept_as_sets: import.accept_as_sets,
accept_third_party_next_hop: import.accept_third_party_next_hop, accept_third_party_next_hop: import.accept_third_party_next_hop,
accept_own_host_next_hop: false,
accept_blackholes: import.accept_blackholes, accept_blackholes: import.accept_blackholes,
own_prefixes: Arc::default(), own_prefixes: Arc::default(),
originated: Arc::default(), originated: Arc::default(),
@@ -655,13 +692,13 @@ fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProb
local_pref: import.local_pref, local_pref: import.local_pref,
rules, rules,
reflection: None, reflection: None,
max_as_path_length: match import.max_as_path_length { max_as_path_length: given(import.max_as_path_length)
0 => defaults.max_as_path_length, .map(|length| u16::try_from(length).map_err(|_| ImportProblem::PathLength(length)))
length => u16::try_from(length).map_err(|_| ImportProblem::PathLength(length))?, .transpose()?
}, .unwrap_or(defaults.max_as_path_length),
prefixes: Some(prefixes).filter(|prefixes| !prefixes.is_empty()), prefixes,
origin_asns: Some(import.origin_asns.iter().copied().collect()) origin_asns: (!import.origin_asns.is_empty())
.filter(|asns: &std::collections::HashSet<u32>| !asns.is_empty()), .then(|| import.origin_asns.iter().copied().collect()),
}) })
} }
@@ -676,17 +713,17 @@ fn parse_prefixes(ranges: &[proto::PrefixRange]) -> Result<PrefixSet, ImportProb
problem, problem,
})?; })?;
let longest = prefix.address_family().bits(); let longest = prefix.address_family().bits();
let max_length = match range.max_length { // Left out, the prefix itself is the longest allowed.
0 => prefix.length(), let max_length = given(range.max_length).map_or(Ok(prefix.length()), |max_length| {
max_length => u8::try_from(max_length) u8::try_from(max_length)
.ok() .ok()
.filter(|max| (prefix.length()..=longest).contains(max)) .filter(|max| (prefix.length()..=longest).contains(max))
.ok_or(ImportProblem::MaxLength { .ok_or(ImportProblem::MaxLength {
prefix: range.prefix.clone(), prefix: range.prefix.clone(),
max_length, max_length,
longest, longest,
})?, })
}; })?;
prefixes.allow(prefix, max_length); prefixes.allow(prefix, max_length);
} }
Ok(prefixes) Ok(prefixes)
@@ -695,21 +732,16 @@ fn parse_prefixes(ranges: &[proto::PrefixRange]) -> Result<PrefixSet, ImportProb
fn parse_rule(rule: &proto::ImportRule) -> Result<ImportRule, ImportProblem> { fn parse_rule(rule: &proto::ImportRule) -> Result<ImportRule, ImportProblem> {
Ok(ImportRule { Ok(ImportRule {
communities: Patterns::parse(&rule.match_communities)?, communities: Patterns::parse(&rule.match_communities)?,
prefixes: if rule.match_prefixes.is_empty() { prefixes: parse_prefixes_if_any(&rule.match_prefixes)?,
None
} else {
Some(parse_prefixes(&rule.match_prefixes)?)
},
local_pref: rule.local_pref, local_pref: rule.local_pref,
marks: Marks::parse(&[], &rule.add_communities, &rule.add_large_communities)?, marks: Marks::parse(&[], &rule.add_communities, &rule.add_large_communities)?,
}) })
} }
fn parse_port(port: u32) -> Result<u16, ConfigError> { fn parse_port(port: u32) -> Result<u16, ConfigError> {
match port { given(port).map_or(Ok(DEFAULT_BGP_PORT), |port| {
0 => Ok(DEFAULT_BGP_PORT), u16::try_from(port).map_err(|_| ConfigError::Port(port))
port => u16::try_from(port).map_err(|_| ConfigError::Port(port)), })
}
} }
fn parse_endpoint(address: &str, port: u32) -> Result<SocketAddr, ConfigError> { fn parse_endpoint(address: &str, port: u32) -> Result<SocketAddr, ConfigError> {
@@ -1510,6 +1542,19 @@ mod tests {
/// What a peer is expected to send is kept for the status page; zero, /// What a peer is expected to send is kept for the status page; zero,
/// the default, is nothing to compare with. /// the default, is nothing to compare with.
#[test]
fn all_paths_needs_learned_routes() {
let without = "export { all_paths: true }";
let error = rejected(&format!("{BASE}\n{without}"));
assert!(
error.to_string().contains("all_paths needs learned_routes"),
"{error}"
);
let with = "export { learned_routes: true all_paths: true }";
let config = parse(&format!("{BASE}\n{with}")).expect("in order");
assert!(config.export.all_paths);
}
#[test] #[test]
fn expected_routes_are_kept_when_given() { fn expected_routes_are_kept_when_given() {
let config = parse(&format!( let config = parse(&format!(
crates/server/src/connection.rs
+620 -165
View File
File diff suppressed because it is too large. Load diff
crates/server/src/dialer.rs
+12 -18
View File
@@ -18,7 +18,7 @@ use tracing::{info, warn};
use crate::collision::Direction; use crate::collision::Direction;
use crate::config::PeerConfig; use crate::config::PeerConfig;
use crate::connection::{CLOSED_AT_OUR_REQUEST, PeerSettings, ServerContext, run_session}; use crate::connection::{PeerSettings, ServerContext, run_session};
use crate::session::DialState; use crate::session::DialState;
/// Upper bound for exponential backoff between connection attempts. /// Upper bound for exponential backoff between connection attempts.
@@ -102,7 +102,9 @@ async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry:
registry.set_dial_state(&peer.name, DialState::Connecting); registry.set_dial_state(&peer.name, DialState::Connecting);
info!(peer = %peer.name, endpoint = %peer.endpoint, "Connecting to peer"); info!(peer = %peer.name, endpoint = %peer.endpoint, "Connecting to peer");
let ended = match tokio::time::timeout(CONNECT_TIMEOUT, connect(&peer)).await { // Why the attempt ended, and whether that is a failure worth a
// line in the log.
let (reason, failed) = match tokio::time::timeout(CONNECT_TIMEOUT, connect(&peer)).await {
Ok(Ok(stream)) => match stream.peer_addr() { Ok(Ok(stream)) => match stream.peer_addr() {
Ok(peer_addr) => { Ok(peer_addr) => {
info!(peer = %peer.name, endpoint = %peer.endpoint, "TCP connected, starting session"); info!(peer = %peer.name, endpoint = %peer.endpoint, "TCP connected, starting session");
@@ -121,25 +123,17 @@ async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry:
if end.established { if end.established {
backoff = connect_retry; backoff = connect_retry;
} }
if end.result.is_ok() { let failed = end.result.is_err();
Ok(()) (end.reason(), failed)
} else {
Err(end.reason())
}
} }
Err(e) => Err(format!("connection lost at once: {e}")), Err(e) => (format!("connection lost at once: {e}"), true),
}, },
Ok(Err(e)) => Err(format!("connect failed: {e}")), Ok(Err(e)) => (format!("connect failed: {e}"), true),
Err(_) => Err(format!("connect timed out after {CONNECT_TIMEOUT:?}")), Err(_) => (format!("connect timed out after {CONNECT_TIMEOUT:?}"), true),
};
let reason = match ended {
Ok(()) => CLOSED_AT_OUR_REQUEST.to_string(),
Err(reason) => {
warn!(peer = %peer.name, endpoint = %peer.endpoint, err = %reason, "Session attempt ended");
reason
}
}; };
if failed {
warn!(peer = %peer.name, endpoint = %peer.endpoint, err = %reason, "Session attempt ended");
}
// Not all at the same moment, when many peers went away together. // Not all at the same moment, when many peers went away together.
let wait = jittered(backoff); let wait = jittered(backoff);
crates/server/src/error.rs
+9 -3
View File
@@ -41,6 +41,9 @@ pub enum SessionError {
Decode(#[from] DecodeError), Decode(#[from] DecodeError),
#[error("could not encode a message: {0}")] #[error("could not encode a message: {0}")]
Encode(#[from] EncodeError), Encode(#[from] EncodeError),
/// A received UPDATE's attributes could not be kept in the table.
#[error(transparent)]
Unstorable(#[from] crate::routes::Unstorable),
/// The peer sent nothing for as long as the pre-negotiation hold time. /// The peer sent nothing for as long as the pre-negotiation hold time.
#[error("timed out waiting for a message in the {0} state")] #[error("timed out waiting for a message in the {0} state")]
Timeout(State), Timeout(State),
@@ -101,9 +104,10 @@ impl SessionError {
pub fn notification(&self) -> Option<NotificationMessage> { pub fn notification(&self) -> Option<NotificationMessage> {
let error = match self { let error = match self {
Self::Io(_) | Self::SendHoldTimerExpired(_) | Self::NotifiedByPeer(_) => return None, Self::Io(_) | Self::SendHoldTimerExpired(_) | Self::NotifiedByPeer(_) => return None,
// Something of ours could not be written out, which is a bug // Something of ours could not be written out, or the peer's
// here: as for a panic, the session is reset. // routes could not be kept, which is a bug here: as for a
Self::Encode(_) => BgpError::Cease(CeaseSubcode::AdminReset), // panic, the session is reset.
Self::Encode(_) | Self::Unstorable(_) => BgpError::Cease(CeaseSubcode::AdminReset),
Self::Decode(e) => { Self::Decode(e) => {
return Some(NotificationMessage { return Some(NotificationMessage {
error: e.notification.clone(), error: e.notification.clone(),
@@ -336,6 +340,8 @@ pub enum ExportProblem {
Marks(#[from] crate::marks::MarksProblem), Marks(#[from] crate::marks::MarksProblem),
#[error("as_path_prepend of {value} is more than {max}")] #[error("as_path_prepend of {value} is more than {max}")]
Prepend { value: u32, max: u32 }, Prepend { value: u32, max: u32 },
#[error("all_paths needs learned_routes")]
AllPathsWithoutLearnedRoutes,
} }
/// What is wrong with one `originate` entry. /// What is wrong with one `originate` entry.
crates/server/src/export.rs
+289 -8
View File
@@ -35,10 +35,13 @@ use std::net::{IpAddr, Ipv4Addr};
use bgp_packet::message::{ use bgp_packet::message::{
AsPathAttribute, AsPathSegment, LocalPrefPathAttribute, MpReachNlriPathAttribute, AsPathAttribute, AsPathSegment, LocalPrefPathAttribute, MpReachNlriPathAttribute,
MpUnreachNlriPathAttribute, NextHopPathAttribute, NlriNextHop, PathAttribute, UpdateMessage, MpUnreachNlriPathAttribute, NextHopPathAttribute, NlriNextHop, PathAttribute, UpdateMessage,
attr_type, begin_message, finish_message, msg_type, put_nlri_list,
}; };
use bgp_primitives::constants::{AfiSafi, IpVersion}; use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::Nlri; use bgp_primitives::ip_prefix::Nlri;
use bgp_primitives::parser::ParserContext;
use bgp_rib::table::{DEFAULT_LOCAL_PREF, PeerIndex}; use bgp_rib::table::{DEFAULT_LOCAL_PREF, PeerIndex};
use bytes::{BufMut, Bytes, BytesMut};
use crate::advertise::{Towards, ipv6_next_hop}; use crate::advertise::{Towards, ipv6_next_hop};
use crate::marks::{self, Control, Named, communities_of, well_known_allow}; use crate::marks::{self, Control, Named, communities_of, well_known_allow};
@@ -47,7 +50,7 @@ use crate::policy::{ExportPolicy, PeerRole};
/// The session a route may go out on, as far as that decides whether it /// The session a route may go out on, as far as that decides whether it
/// does and as what. /// does and as what.
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub struct Egress<'a> { pub(crate) struct Egress<'a> {
pub towards: Towards, pub towards: Towards,
pub peer_asn: u32, pub peer_asn: u32,
/// The peer's number in the routing table: what its own routes there /// The peer's number in the routing table: what its own routes there
@@ -66,7 +69,7 @@ pub struct Egress<'a> {
/// Route reflection (RFC 4456), as far as one session goes. /// Route reflection (RFC 4456), as far as one session goes.
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub struct Reflection { pub(crate) struct Reflection {
/// The cluster this router reflects routes for. /// The cluster this router reflects routes for.
pub cluster_id: Ipv4Addr, pub cluster_id: Ipv4Addr,
/// Whether the peer is a route reflector client of ours. /// Whether the peer is a route reflector client of ours.
@@ -78,7 +81,7 @@ pub struct Reflection {
/// Where a learned route came from. /// Where a learned route came from.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct Source { pub(crate) struct Source {
pub peer: PeerIndex, pub peer: PeerIndex,
/// Whether that peer is in our own AS. /// Whether that peer is in our own AS.
pub ibgp: bool, pub ibgp: bool,
@@ -92,7 +95,7 @@ pub struct Source {
/// A route ready to be announced on a session: its attributes, but for /// A route ready to be announced on a session: its attributes, but for
/// the ones that depend on which prefixes go with them. /// the ones that depend on which prefixes go with them.
#[derive(Debug, Clone, PartialEq, Eq)] #[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outgoing { pub(crate) struct Outgoing {
pub attributes: Vec<PathAttribute>, pub attributes: Vec<PathAttribute>,
pub next_hop: NlriNextHop, pub next_hop: NlriNextHop,
} }
@@ -393,15 +396,155 @@ pub(crate) fn finish(attributes: &mut Vec<PathAttribute>, towards: &Towards) {
/// lengths every UPDATE has. /// lengths every UPDATE has.
const UPDATE_ROOM: usize = 4096 - 19 - 4; const UPDATE_ROOM: usize = 4096 - 19 - 4;
/// The most octets a route's attributes may take on the wire for the
/// route to go out at all. A message is 4,096 octets at most (RFC 4271
/// §4.1) and must hold the attributes, the next hop and at least one
/// prefix; three quarters of it leaves [`UPDATE_ROOM`] a thousand octets
/// for the next hop and the prefixes, which is some forty IPv6 prefixes
/// with path identifiers a message, enough to keep a table's UPDATEs few.
pub(crate) const ATTRIBUTES_AT_MOST: usize = 3072;
/// The most octets a prefix takes in an UPDATE: a path identifier, a /// The most octets a prefix takes in an UPDATE: a path identifier, a
/// length, and the address. /// length, and the address.
fn nlri_octets(routes: IpVersion) -> usize { fn nlri_octets(routes: IpVersion) -> usize {
4 + 1 + usize::from(routes.bits()) / 8 4 + 1 + usize::from(routes.bits()) / 8
} }
/// A route's attributes as they go out on one session, encoded once and
/// kept: what every UPDATE announcing a route with them carries, less the
/// next hop, which goes with the prefixes. `at_next_hop` and
/// `at_mp_reach` are where in `bytes` a NEXT_HOP (type 3) and an
/// MP_REACH_NLRI (type 14) go, so that the attributes stay in order of
/// type code.
#[derive(Debug)]
pub(crate) struct EncodedAttributes {
bytes: Bytes,
at_next_hop: usize,
at_mp_reach: usize,
}
impl EncodedAttributes {
/// Encodes `attributes`, which are `finish`ed, for a session with
/// `four_octet_asn`. `None` if they cannot be encoded.
pub(crate) fn encode(attributes: &[PathAttribute], four_octet_asn: bool) -> Option<Self> {
let ctx = ParserContext {
four_octet_asn,
ibgp: false,
add_path: Vec::new(),
};
let mut out = BytesMut::with_capacity(256);
let mut at_next_hop = None;
let mut at_mp_reach = None;
for attribute in attributes {
let code = attribute.type_code();
if code > attr_type::NEXT_HOP && at_next_hop.is_none() {
at_next_hop = Some(out.len());
}
if code > attr_type::MP_REACH_NLRI && at_mp_reach.is_none() {
at_mp_reach = Some(out.len());
}
attribute.to_wire(&ctx, &mut out).ok()?;
}
Some(Self {
at_next_hop: at_next_hop.unwrap_or(out.len()),
at_mp_reach: at_mp_reach.unwrap_or(out.len()),
bytes: out.freeze(),
})
}
pub(crate) fn octets(&self) -> usize {
self.bytes.len()
}
/// The UPDATE messages, complete with their headers, that announce
/// `routes`, all of IP version `version`, with these attributes and
/// `next_hop`: as few as they fit in, in pieces of [`ENCODED_PIECE`]
/// octets or so, each of whole messages. `ctx` is the session's, and
/// says whether path identifiers go with the family's prefixes
/// (RFC 7911).
///
/// What `announcements`, kept under test as the reference, builds as
/// messages, built as bytes: the attributes are not parsed, cloned or
/// encoded again for each message.
pub(crate) fn announcements(
&self,
next_hop: NlriNextHop,
version: IpVersion,
routes: &[Nlri],
ctx: &ParserContext,
) -> Vec<BytesMut> {
// Attributes that would leave no room for a prefix do not go out
// at all; see `Learned::dressed`.
debug_assert!(self.octets() < ATTRIBUTES_AT_MOST);
let room = UPDATE_ROOM.saturating_sub(self.octets() + NEXT_HOP_ROOM);
let at_once = (room / nlri_octets(version)).max(1);
let in_body = matches!(next_hop, NlriNextHop::Ipv4(_)) && version == IpVersion::Ipv4;
let with_path_id = ctx.add_path.contains(&AfiSafi::unicast(version));
let mut pieces = Vec::new();
let mut out = BytesMut::with_capacity(ENCODED_PIECE.min(routes.len() * 64 + 4096));
for routes in routes.chunks(at_once) {
{
let length_at = begin_message(&mut out, msg_type::UPDATE);
// No withdrawn routes.
out.put_u16(0);
let attributes_at = out.len();
out.put_u16(0);
if in_body {
let NlriNextHop::Ipv4(next_hop) = next_hop else {
unreachable!("in_body means an IPv4 next hop");
};
out.put_slice(&self.bytes[..self.at_next_hop]);
PathAttribute::NextHop(NextHopPathAttribute(next_hop))
.to_wire(ctx, &mut out)
.expect("a next hop encodes");
out.put_slice(&self.bytes[self.at_next_hop..]);
} else {
out.put_slice(&self.bytes[..self.at_mp_reach]);
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::unicast(version),
next_hop,
prefixes: routes.to_vec(),
})
.to_wire(ctx, &mut out)
.expect("a chunk that fits encodes");
out.put_slice(&self.bytes[self.at_mp_reach..]);
}
let attributes_len = out.len() - attributes_at - 2;
out[attributes_at..attributes_at + 2]
.copy_from_slice(&(attributes_len as u16).to_be_bytes());
if in_body {
put_nlri_list(routes, with_path_id, &mut out);
}
// As many prefixes went in as fit beside the attributes.
finish_message(&mut out, length_at).expect("a chunk that fits is a message");
}
if out.len() >= ENCODED_PIECE {
pieces.push(out.split());
}
}
if !out.is_empty() {
pieces.push(out);
}
pieces
}
}
/// What the next hop takes, in its own attribute or with the prefixes,
/// and a little to spare.
const NEXT_HOP_ROOM: usize = 64;
/// How many octets of UPDATEs go to a session in one piece: what the
/// session hands its writer at a time. A piece holds whole messages,
/// so it runs over by at most one.
pub(crate) const ENCODED_PIECE: usize = 64 * 1024;
/// The UPDATEs that announce `routes`, all of IP version `version`, as /// The UPDATEs that announce `routes`, all of IP version `version`, as
/// `outgoing`: as few as they fit in. `attributes_octets` is how long /// `outgoing`: as few as they fit in. `attributes_octets` is how long
/// `outgoing`'s attributes are on this session's wire. /// `outgoing`'s attributes are on this session's wire.
///
/// What [`EncodedAttributes::announcements`] does, built as messages:
/// kept as what its bytes are checked against.
#[cfg(test)]
pub(crate) fn announcements( pub(crate) fn announcements(
outgoing: &Outgoing, outgoing: &Outgoing,
version: IpVersion, version: IpVersion,
@@ -409,9 +552,6 @@ pub(crate) fn announcements(
towards: &Towards, towards: &Towards,
attributes_octets: usize, attributes_octets: usize,
) -> Vec<UpdateMessage> { ) -> Vec<UpdateMessage> {
// What the next hop takes, in its own attribute or with the prefixes,
// and a little to spare.
const NEXT_HOP_ROOM: usize = 64;
let room = UPDATE_ROOM.saturating_sub(attributes_octets + NEXT_HOP_ROOM); let room = UPDATE_ROOM.saturating_sub(attributes_octets + NEXT_HOP_ROOM);
let at_once = (room / nlri_octets(version)).max(1); let at_once = (room / nlri_octets(version)).max(1);
let in_body = matches!(outgoing.next_hop, NlriNextHop::Ipv4(_)) && version == IpVersion::Ipv4; let in_body = matches!(outgoing.next_hop, NlriNextHop::Ipv4(_)) && version == IpVersion::Ipv4;
@@ -484,7 +624,7 @@ mod tests {
use std::net::Ipv4Addr; use std::net::Ipv4Addr;
use bgp_packet::message::{ use bgp_packet::message::{
CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, Message,
MultiExitDiscPathAttribute, OriginPathAttribute, PathAttributeFlags, MultiExitDiscPathAttribute, OriginPathAttribute, PathAttributeFlags,
}; };
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
@@ -529,6 +669,7 @@ mod tests {
fn all_routes() -> ExportPolicy { fn all_routes() -> ExportPolicy {
ExportPolicy { ExportPolicy {
learned_routes: true, learned_routes: true,
all_paths: false,
..ExportPolicy::default() ..ExportPolicy::default()
} }
} }
@@ -548,6 +689,145 @@ mod tests {
PathAttribute::Communities(CommunitiesPathAttribute(communities.to_vec())) PathAttribute::Communities(CommunitiesPathAttribute(communities.to_vec()))
} }
/// What `EncodedAttributes::announcements` writes is, byte for byte,
/// what the messages `announcements` builds encode to: with the next
/// hop in its own attribute (IPv4) and in MP_REACH_NLRI (IPv6, and
/// IPv4 with an IPv6 next hop, RFC 8950), with and without path
/// identifiers, to two-octet and four-octet AS speakers (so with the
/// attributes around NEXT_HOP, MP_REACH_NLRI and AS4_PATH in order),
/// and over as many messages when the routes do not fit in one.
#[test]
fn encoded_announcements_are_the_messages_on_the_wire() {
use std::net::Ipv6Addr;
// The routes' IP version, their next hop, and whether the session
// takes IPv4 routes with IPv6 next hops.
let layouts = [
(
IpVersion::Ipv4,
NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 1)),
false,
),
(
IpVersion::Ipv6,
NlriNextHop::Ipv6WithLl {
global: Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1),
link_local: Ipv6Addr::new(0xfe80, 0, 0, 0, 0, 0, 0, 1),
},
false,
),
(
IpVersion::Ipv4,
NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1)),
true,
),
];
for four_octet_asn in [true, false] {
for with_path_id in [false, true] {
for (version, next_hop, extended_next_hop) in layouts {
let towards = Towards {
four_octet_asn,
extended_next_hop,
..towards(None, false)
};
let attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![4_200_000_000, 3356, 64700]),
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(10)),
communities(&[(3356, 100), (64700, 1)]),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: 64700,
data_1: 1,
data_2: 2,
}],
}),
];
let routes: Vec<Nlri> = (0..30_000u32)
.map(|i| Nlri {
prefix: match version {
IpVersion::Ipv4 => IpPrefix::new(
version,
Ipv4Addr::from(0x0a00_0000 + i * 256).octets(),
24,
),
IpVersion::Ipv6 => IpPrefix::new(
version,
Ipv6Addr::new(0x2001, 0xdb8, i as u16, 0, 0, 0, 0, 0).octets(),
48,
),
}
.unwrap(),
path_id: if with_path_id { i + 1 } else { 0 },
})
.collect();
let ctx = ParserContext {
four_octet_asn,
ibgp: false,
add_path: if with_path_id {
vec![AfiSafi::unicast(version)]
} else {
Vec::new()
},
};
let mut finished = attributes.clone();
finish(&mut finished, &towards);
let encoded = EncodedAttributes::encode(&finished, four_octet_asn).unwrap();
let bytes = encoded.announcements(next_hop, version, &routes, &ctx);
let outgoing = Outgoing {
attributes,
next_hop,
};
let messages =
announcements(&outgoing, version, &routes, &towards, encoded.octets());
let expected: Vec<BytesMut> = messages
.into_iter()
.map(|update| {
let mut out = BytesMut::new();
Message::Update(update).to_wire(&ctx, &mut out).unwrap();
out
})
.collect();
let what = format!(
"{version:?} with next hop {next_hop:?}, path ids {with_path_id}, \
four-octet {four_octet_asn}"
);
assert!(expected.len() > 1, "{what}: more than one message");
for update in &expected {
assert!(update.len() <= 4096, "{what}: a message fits");
}
// In pieces of whole messages, each of about the size
// meant, all but the last at least that.
assert!(bytes.len() > 1, "{what}: more than one piece");
for (n, piece) in bytes.iter().enumerate() {
assert!(
n == bytes.len() - 1
|| (ENCODED_PIECE..ENCODED_PIECE + 4096).contains(&piece.len()),
"{what}: piece {n} is {} octets",
piece.len()
);
let mut at = 0;
while at < piece.len() {
let length =
usize::from(u16::from_be_bytes([piece[at + 16], piece[at + 17]]));
at += length;
}
assert_eq!(at, piece.len(), "{what}: piece {n} is of whole messages");
}
let got: Vec<u8> = bytes
.iter()
.flat_map(|piece| piece.iter().copied())
.collect();
let want: Vec<u8> = expected.iter().flat_map(|m| m.iter().copied()).collect();
assert_eq!(got, want, "{what}: the same octets");
}
}
}
}
struct Case<'a> { struct Case<'a> {
policy: &'a ExportPolicy, policy: &'a ExportPolicy,
role: Option<PeerRole>, role: Option<PeerRole>,
@@ -922,6 +1202,7 @@ mod tests {
|texts: &[&str]| -> Vec<String> { texts.iter().map(|text| text.to_string()).collect() }; |texts: &[&str]| -> Vec<String> { texts.iter().map(|text| text.to_string()).collect() };
let policy = ExportPolicy { let policy = ExportPolicy {
learned_routes: true, learned_routes: true,
all_paths: false,
marks: Marks::parse(&strings(&["64601:900"]), &strings(&["64601:5"]), &[]).unwrap(), marks: Marks::parse(&strings(&["64601:900"]), &strings(&["64601:5"]), &[]).unwrap(),
prepend: 2, prepend: 2,
}; };
crates/server/src/marks.rs
+15 -1
View File
@@ -15,7 +15,8 @@
use bgp_packet::community::{CommunityError, parse_large, parse_standard}; use bgp_packet::community::{CommunityError, parse_large, parse_standard};
use bgp_packet::message::{ use bgp_packet::message::{
CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, PathAttribute, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
LocalPrefPathAttribute, PathAttribute,
}; };
/// A standard community, or all those with one half or both left open: /// A standard community, or all those with one half or both left open:
@@ -186,6 +187,19 @@ impl Marks {
} }
} }
/// Marks `attributes` as a policy or one of its rules says: with `marks`
/// on their communities, and with `local_pref` as their LOCAL_PREF if one
/// is given, in place of whatever they had.
pub fn mark(attributes: &mut Vec<PathAttribute>, marks: &Marks, local_pref: Option<u32>) {
marks.apply(attributes);
if let Some(local_pref) = local_pref {
put(
attributes,
PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)),
);
}
}
/// Takes the standard communities out of `attributes`. /// Takes the standard communities out of `attributes`.
fn take_communities(attributes: &mut Vec<PathAttribute>) -> Vec<(u16, u16)> { fn take_communities(attributes: &mut Vec<PathAttribute>) -> Vec<(u16, u16)> {
let at = attributes let at = attributes
crates/server/src/metrics.rs
+293 -168
View File
@@ -2,235 +2,296 @@
use std::sync::LazyLock; use std::sync::LazyLock;
use bgp_fib::sockdiag::TcpConnection;
use bgp_packet::message::Message; use bgp_packet::message::Message;
#[cfg(test)]
use prometheus::core::Collector;
use prometheus::{ use prometheus::{
IntCounter, IntCounterVec, IntGauge, IntGaugeVec, register_int_counter, GaugeVec, IntCounter, IntCounterVec, IntGauge, IntGaugeVec, register_gauge_vec,
register_int_counter_vec, register_int_gauge, register_int_gauge_vec, register_int_counter, register_int_counter_vec, register_int_gauge, register_int_gauge_vec,
}; };
pub static MESSAGES_SENT: LazyLock<IntCounterVec> = LazyLock::new(|| { /// What a declared metric puts on the registry: one collector for a
register_int_counter_vec!( /// metric, and one for each gauge of a group of them.
#[cfg(test)]
trait Collectors {
fn collectors(&self) -> Vec<Box<dyn Collector>>;
}
#[cfg(test)]
impl<T: Collector + Clone + 'static> Collectors for T {
fn collectors(&self) -> Vec<Box<dyn Collector>> {
vec![Box::new(self.clone())]
}
}
/// Declares the metrics: a static for each, made on first use, and
/// [`init`], which forces every one of them so that nothing is left out
/// of a scrape for not having been used yet. Each is declared once, here,
/// so that the two cannot drift apart.
macro_rules! metrics {
($(
$(#[$meta:meta])*
$name:ident: $type:ty = $register:expr;
)*) => {
$(
$(#[$meta])*
pub static $name: LazyLock<$type> =
LazyLock::new(|| $register.expect("metric registration"));
)*
/// Every metric declared above, as the collectors the registry
/// holds, for the tests to check it against.
#[cfg(test)]
fn declared() -> Vec<Box<dyn Collector>> {
let mut all = Vec::new();
$(all.extend($name.collectors());)*
all
}
/// Forces registration of every metric so they appear in scrapes
/// before their first observation: a counter that is not there
/// until it first counts hides that first count from `increase()`.
pub fn init() {
$(LazyLock::force(&$name);)*
BUILD_INFO
.with_label_values(&[env!("CARGO_PKG_VERSION")])
.set(1);
}
};
}
metrics! {
MESSAGES_SENT: IntCounterVec = register_int_counter_vec!(
"bgp_messages_sent_total", "bgp_messages_sent_total",
"BGP messages sent, by type", "BGP messages sent, by type",
&["type"] &["type"]
) );
.expect("metric registration") MESSAGES_RECEIVED: IntCounterVec = register_int_counter_vec!(
});
pub static MESSAGES_RECEIVED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_messages_received_total", "bgp_messages_received_total",
"BGP messages received, by type", "BGP messages received, by type",
&["type"] &["type"]
) );
.expect("metric registration") UPDATE_ERRORS: IntCounterVec = register_int_counter_vec!(
});
pub static UPDATE_ERRORS: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_update_errors_total", "bgp_update_errors_total",
"Errors in received UPDATE messages that did not reset the session, by how they were handled (RFC 7606)", "Errors in received UPDATE messages that did not reset the session, by how they were handled (RFC 7606)",
&["handling"] &["handling"]
) );
.expect("metric registration") SESSIONS_ESTABLISHED_TOTAL: IntCounter = register_int_counter!(
});
pub static SESSIONS_ESTABLISHED_TOTAL: LazyLock<IntCounter> = LazyLock::new(|| {
register_int_counter!(
"bgp_sessions_established_total", "bgp_sessions_established_total",
"Sessions that reached Established" "Sessions that reached Established"
) );
.expect("metric registration") SESSIONS_ESTABLISHED: IntGauge = register_int_gauge!(
});
pub static SESSIONS_ESTABLISHED: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_sessions_established", "bgp_sessions_established",
"Sessions currently in Established" "Sessions currently in Established"
) );
.expect("metric registration") ROUTES_REJECTED: IntCounterVec = register_int_counter_vec!(
});
pub static ROUTES_REJECTED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_routes_rejected_total", "bgp_routes_rejected_total",
"Routes announced by a peer and refused by its import policy, by why", "Routes announced by a peer and refused by its import policy, by why",
&["peer", "reason"] &["peer", "reason"]
) );
.expect("metric registration") PEER_ROUTES: IntGaugeVec = register_int_gauge_vec!(
});
pub static PEER_ROUTES: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_peer_routes", "bgp_peer_routes",
"Routes of an established peer that are in the routing table", "Routes of an established peer that are in the routing table",
&["peer"] &["peer"]
) );
.expect("metric registration") PEER_ADVERTISED: IntGaugeVec = register_int_gauge_vec!(
}); "bgp_peer_advertised_routes",
"Routes announced to an established peer and not withdrawn since",
pub static PEER_MAX_PREFIXES: LazyLock<IntGaugeVec> = LazyLock::new(|| { &["peer"]
register_int_gauge_vec!( );
PEER_TCP: PeerTcp = PeerTcp::new();
/// The allocator's account of the daemon's memory, by `kind`: allocated,
/// active, metadata, resident, mapped, retained (jemalloc's `stats.*`).
MEMORY: IntGaugeVec = register_int_gauge_vec!(
"bgp_memory_bytes",
"The daemon's memory as its allocator accounts for it, by kind",
&["kind"]
);
PEER_MAX_PREFIXES: IntGaugeVec = register_int_gauge_vec!(
"bgp_peer_max_prefixes", "bgp_peer_max_prefixes",
"The most routes a configured peer may have (import.max_prefixes), where there is a limit", "The most routes a configured peer may have (import.max_prefixes), where there is a limit",
&["peer"] &["peer"]
) );
.expect("metric registration") PREFIX_LIMIT_EXCEEDED: IntCounterVec = register_int_counter_vec!(
});
pub static PREFIX_LIMIT_EXCEEDED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_prefix_limit_exceeded_total", "bgp_prefix_limit_exceeded_total",
"Sessions ended because the peer went over its limit of routes", "Sessions ended because the peer went over its limit of routes",
&["peer"] &["peer"]
) );
.expect("metric registration") EXPORT_LIMIT_EXCEEDED: IntCounterVec = register_int_counter_vec!(
});
pub static EXPORT_LIMIT_EXCEEDED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_export_limit_exceeded_total", "bgp_export_limit_exceeded_total",
"Sessions ended because more routes were to be announced to the peer than its export limit allows", "Sessions ended because more routes were to be announced to the peer than its export limit allows",
&["peer"] &["peer"]
) );
.expect("metric registration") RPKI_VRPS: IntGaugeVec = register_int_gauge_vec!(
});
pub static RPKI_VRPS: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_rpki_vrps", "bgp_rpki_vrps",
"Validated ROA payloads in use for route origin validation, by address family", "Validated ROA payloads in use for route origin validation, by address family",
&["family"] &["family"]
) );
.expect("metric registration") RPKI_ASPAS: IntGauge = register_int_gauge!(
});
pub static RPKI_ASPAS: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rpki_aspas", "bgp_rpki_aspas",
"Customer ASes with an ASPA in use for AS path verification" "Customer ASes with an ASPA in use for AS path verification"
) );
.expect("metric registration") RPKI_UPDATES: IntCounter = register_int_counter!(
}); "bgp_rpki_updates_total",
"Times the VRPs or ASPAs changed"
pub static RPKI_UPDATES: LazyLock<IntCounter> = LazyLock::new(|| { );
register_int_counter!("bgp_rpki_updates_total", "Times the VRPs or ASPAs changed") CONNECTIONS_REFUSED: IntCounterVec = register_int_counter_vec!(
.expect("metric registration")
});
pub static CONNECTIONS_REFUSED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_connections_refused_total", "bgp_connections_refused_total",
"Incoming connections closed without a session, by why", "Incoming connections closed without a session, by why",
&["reason"] &["reason"]
) );
.expect("metric registration") COLLISIONS: IntCounterVec = register_int_counter_vec!(
});
pub static COLLISIONS: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_connection_collisions_total", "bgp_connection_collisions_total",
"Connection collisions resolved, by which connection was closed", "Connection collisions resolved, by which connection was closed",
&["closed"] &["closed"]
) );
.expect("metric registration") LOCAL_ROUTES: IntGauge = register_int_gauge!(
}); "bgp_local_routes",
"Locally originated routes"
pub static LOCAL_ROUTES: LazyLock<IntGauge> = LazyLock::new(|| { );
register_int_gauge!("bgp_local_routes", "Locally originated routes") RIB_ROUTES: IntGauge = register_int_gauge!(
.expect("metric registration")
});
pub static RIB_ROUTES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_routes", "bgp_rib_routes",
"Routes learned from peers, counting each peer's separately" "Routes learned from peers, counting each peer's separately"
) );
.expect("metric registration") RIB_PREFIXES: IntGauge = register_int_gauge!(
});
pub static RIB_PREFIXES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_prefixes", "bgp_rib_prefixes",
"Prefixes with at least one route, and so with a best route" "Prefixes with at least one route, and so with a best route"
) );
.expect("metric registration") RIB_ATTRIBUTE_SETS: IntGauge = register_int_gauge!(
});
pub static RIB_ATTRIBUTE_SETS: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_attribute_sets", "bgp_rib_attribute_sets",
"Distinct sets of path attributes held for the routes" "Distinct sets of path attributes held for the routes"
) );
.expect("metric registration") RIB_JOURNAL_ENTRIES: IntGauge = register_int_gauge!(
});
pub static RIB_JOURNAL_ENTRIES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_journal_entries", "bgp_rib_journal_entries",
"Changes of best route waiting for a follower to read them" "Changes of best route waiting for a follower to read them"
) );
.expect("metric registration") ROUTES_EXPORTED: IntCounterVec = register_int_counter_vec!(
});
pub static ROUTES_EXPORTED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_routes_exported_total", "bgp_routes_exported_total",
"Routes learned from one peer and announced to, or withdrawn from, another", "Routes learned from one peer and announced to, or withdrawn from, another",
&["change"] &["change"]
) );
.expect("metric registration") /// Whether each configured peer's session is up: what to alert on when a
}); /// peer goes away, which a count of sessions cannot say.
PEER_UP: IntGaugeVec = register_int_gauge_vec!(
/// Whether each configured peer's session is up: what to alert on when a
/// peer goes away, which a count of sessions cannot say.
pub static PEER_UP: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_peer_up", "bgp_peer_up",
"Whether the session with a configured peer is Established (1) or not (0)", "Whether the session with a configured peer is Established (1) or not (0)",
&["peer"] &["peer"]
) );
.expect("metric registration") BUILD_INFO: IntGaugeVec = register_int_gauge_vec!(
});
pub static BUILD_INFO: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_build_info", "bgp_build_info",
"Always 1; the version of the daemon is in the label", "Always 1; the version of the daemon is in the label",
&["version"] &["version"]
) );
.expect("metric registration") }
});
/// Forces registration of every metric so they appear in scrapes before /// The type of a gauge of [`PeerTcp`] by what it holds: a time, in
/// their first observation: a counter that is not there until it first /// seconds, or a count.
/// counts hides that first count from `increase()`. macro_rules! tcp_gauge {
pub fn init() { (seconds) => {
LazyLock::force(&MESSAGES_SENT); GaugeVec
LazyLock::force(&MESSAGES_RECEIVED); };
LazyLock::force(&UPDATE_ERRORS); (count) => {
LazyLock::force(&SESSIONS_ESTABLISHED_TOTAL); IntGaugeVec
LazyLock::force(&SESSIONS_ESTABLISHED); };
LazyLock::force(&ROUTES_REJECTED); }
LazyLock::force(&PEER_ROUTES);
LazyLock::force(&PEER_MAX_PREFIXES); /// Registers a gauge of [`PeerTcp`], with the peer as its one label.
LazyLock::force(&PREFIX_LIMIT_EXCEEDED); macro_rules! register_tcp_gauge {
LazyLock::force(&EXPORT_LIMIT_EXCEEDED); (seconds, $name:expr, $help:expr) => {
LazyLock::force(&RPKI_VRPS); register_gauge_vec!($name, $help, &["peer"])
LazyLock::force(&RPKI_ASPAS); };
LazyLock::force(&RPKI_UPDATES); (count, $name:expr, $help:expr) => {
LazyLock::force(&CONNECTIONS_REFUSED); register_int_gauge_vec!($name, $help, &["peer"])
LazyLock::force(&COLLISIONS); };
LazyLock::force(&LOCAL_ROUTES); }
LazyLock::force(&RIB_ROUTES);
LazyLock::force(&RIB_PREFIXES); /// What the kernel's figure becomes on the gauge: microseconds become
LazyLock::force(&RIB_ATTRIBUTE_SETS); /// seconds, and a count is clamped into what a gauge holds.
LazyLock::force(&RIB_JOURNAL_ENTRIES); macro_rules! tcp_gauge_value {
LazyLock::force(&ROUTES_EXPORTED); (seconds, $microseconds:expr) => {
LazyLock::force(&PEER_UP); $microseconds as f64 / 1e6
BUILD_INFO };
.with_label_values(&[env!("CARGO_PKG_VERSION")]) (count, $count:expr) => {
.set(1); i64::try_from($count).unwrap_or(i64::MAX)
};
}
/// Declares the gauges of [`PeerTcp`], each named once with its help text
/// and the field of [`TcpConnection`] it is set from: the struct, how it is
/// made, and what sets and removes a peer's series all follow from the one
/// list. The gauge `name` is `bgp_peer_tcp_name`.
macro_rules! peer_tcp {
($connection:ident; $(
$name:ident: $kind:ident from $source:ident = $help:literal,
)*) => {
/// A session's TCP connection as the kernel reports it (`TCP_INFO`),
/// sampled when the metrics are scraped. Seconds, octets, segments, and
/// octets per second; the busy, limited and byte figures are cumulative
/// over the connection's life.
pub struct PeerTcp {
$(pub $name: tcp_gauge!($kind),)*
}
#[cfg(test)]
impl Collectors for PeerTcp {
fn collectors(&self) -> Vec<Box<dyn Collector>> {
vec![$(Box::new(self.$name.clone()),)*]
}
}
impl PeerTcp {
fn new() -> prometheus::Result<Self> {
Ok(Self {
$($name: register_tcp_gauge!(
$kind,
concat!("bgp_peer_tcp_", stringify!($name)),
$help
)?,)*
})
}
/// Sets the peer's gauges from what the kernel reports of its
/// connection.
pub fn set(&self, label: &str, $connection: &TcpConnection) {
let label = [label];
$(
self.$name
.with_label_values(&label)
.set(tcp_gauge_value!($kind, u64::from($connection.$source)));
)*
}
/// Forgets the peer's series, when its session has ended.
pub fn remove(&self, label: &str) {
$(let _ = self.$name.remove_label_values(&[label]);)*
}
}
};
}
peer_tcp! {
connection;
rtt_seconds: seconds from rtt = "Smoothed round-trip time of the session's TCP connection",
min_rtt_seconds: seconds from min_rtt = "Least round-trip time seen on the connection",
rtt_variance_seconds: seconds from rtt_variance = "Round-trip time variance of the connection",
cwnd_segments: count from cwnd = "Congestion window, in segments",
mss_bytes: count from mss = "Maximum segment size",
send_window_bytes: count from send_window = "The peer's receive window: what it lets us send",
receive_window_bytes: count from receive_window = "Our receive window: what we let the peer send",
send_queue_bytes: count from send_queue = "Octets written and not yet acknowledged by the peer",
not_sent_bytes: count from not_sent = "Octets written and not yet sent",
receive_queue_bytes: count from recv_queue = "Octets received and not yet read",
unacked_segments: count from unacked = "Segments sent and not yet acknowledged",
retransmitted_segments: count from total_retrans = "Segments retransmitted over the connection's life",
lost_segments: count from lost = "Segments thought lost now",
busy_seconds: seconds from busy = "Time the connection had something to send, over its life",
receive_window_limited_seconds: seconds from limited_by_receive_window = "Of the busy time, time held up by the peer's receive window",
send_buffer_limited_seconds: seconds from limited_by_send_buffer = "Of the busy time, time held up by our send buffer being full",
sent_bytes: count from bytes_sent = "Octets sent over the connection's life",
received_bytes: count from bytes_received = "Octets received over the connection's life",
delivery_rate_bytes_per_second: count from delivery_rate = "Delivery rate the kernel measured most recently",
pacing_rate_bytes_per_second: count from pacing_rate = "Rate the kernel paces the connection at",
} }
pub fn message_type_name(msg: &Message) -> &'static str { pub fn message_type_name(msg: &Message) -> &'static str {
@@ -242,3 +303,67 @@ pub fn message_type_name(msg: &Message) -> &'static str {
Message::RouteRefresh(_) => "route_refresh", Message::RouteRefresh(_) => "route_refresh",
} }
} }
#[cfg(test)]
mod tests {
use std::collections::HashSet;
use super::*;
/// Every metric family the declaration lists is on the registry once
/// `init` has run, each under a name of its own: the registry refuses
/// a second collector for any of them. The registry is the process's,
/// so what else is on it is not looked at.
#[test]
fn init_registers_every_declared_metric_family() {
init();
let declared = declared();
let names: HashSet<String> = declared
.iter()
.flat_map(|collector| collector.desc())
.map(|desc| desc.fq_name.clone())
.collect();
assert_eq!(names.len(), declared.len(), "a name is declared twice");
for collector in declared {
assert!(matches!(
prometheus::register(collector),
Err(prometheus::Error::AlreadyReg)
));
}
// A metric without labels is in a scrape from the start, which is
// what `init` is for; one with labels is there once it has one.
let scraped: HashSet<String> = prometheus::gather()
.iter()
.map(|family| family.name().to_string())
.collect();
for name in [
"bgp_sessions_established",
"bgp_rib_routes",
"bgp_build_info",
] {
assert!(scraped.contains(name), "{name} is not in a scrape");
}
}
/// The kernel's microseconds become seconds, and its counts are set as
/// they are; a peer whose session ended loses every series.
#[test]
fn a_connection_sets_and_a_removal_clears_every_gauge() {
let tcp = &*PEER_TCP;
let connection = TcpConnection {
rtt: 1500,
cwnd: 10,
bytes_sent: 1 << 40,
..TcpConnection::default()
};
tcp.set("metrics-test", &connection);
let label = ["metrics-test"];
assert_eq!(tcp.rtt_seconds.with_label_values(&label).get(), 0.0015);
assert_eq!(tcp.cwnd_segments.with_label_values(&label).get(), 10);
assert_eq!(tcp.sent_bytes.with_label_values(&label).get(), 1 << 40);
tcp.remove("metrics-test");
// There is nothing left to remove.
assert!(tcp.rtt_seconds.remove_label_values(&label).is_err());
assert!(tcp.sent_bytes.remove_label_values(&label).is_err());
}
}
crates/server/src/policy.rs
+47 -34
View File
@@ -29,9 +29,7 @@ use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::sync::{Arc, LazyLock}; use std::sync::{Arc, LazyLock};
use std::time::Duration; use std::time::Duration;
use bgp_packet::message::{ use bgp_packet::message::{AS_TRANS, AsPathAttribute, NlriNextHop, PathAttribute};
AS_TRANS, AsPathAttribute, LocalPrefPathAttribute, NlriNextHop, PathAttribute,
};
use bgp_packet::open::BgpRole; use bgp_packet::open::BgpRole;
use bgp_primitives::constants::IpVersion; use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix; use bgp_primitives::ip_prefix::IpPrefix;
@@ -91,6 +89,10 @@ pub struct ImportPolicy {
/// who never asked for it. A route server's routes always have its /// who never asked for it. A route server's routes always have its
/// clients as next hops, and are taken either way. /// clients as next hops, and are taken either way.
pub accept_third_party_next_hop: bool, pub accept_third_party_next_hop: bool,
/// Take a next hop that is on this host: a loopback address, or one of
/// our own. For routers that share a host, as those of a test do. Not
/// a setting; only [`ImportPolicy::permissive`] has it.
pub accept_own_host_next_hop: bool,
/// Take routes with the BLACKHOLE community for single addresses, and /// Take routes with the BLACKHOLE community for single addresses, and
/// have the kernel drop the traffic to them (RFC 7999). /// have the kernel drop the traffic to them (RFC 7999).
pub accept_blackholes: bool, pub accept_blackholes: bool,
@@ -170,13 +172,7 @@ impl ImportRule {
} }
pub fn apply(&self, attributes: &mut Vec<PathAttribute>) { pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
self.marks.apply(attributes); marks::mark(attributes, &self.marks, self.local_pref);
if let Some(local_pref) = self.local_pref {
marks::put(
attributes,
PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)),
);
}
} }
} }
@@ -199,6 +195,7 @@ impl Default for ImportPolicy {
accept_aspa_invalid: false, accept_aspa_invalid: false,
accept_as_sets: false, accept_as_sets: false,
accept_third_party_next_hop: false, accept_third_party_next_hop: false,
accept_own_host_next_hop: false,
accept_blackholes: false, accept_blackholes: false,
own_prefixes: Arc::default(), own_prefixes: Arc::default(),
originated: Arc::default(), originated: Arc::default(),
@@ -228,6 +225,7 @@ impl ImportPolicy {
accept_aspa_invalid: true, accept_aspa_invalid: true,
accept_as_sets: true, accept_as_sets: true,
accept_third_party_next_hop: true, accept_third_party_next_hop: true,
accept_own_host_next_hop: true,
accept_blackholes: false, accept_blackholes: false,
..Self::default() ..Self::default()
} }
@@ -250,13 +248,7 @@ impl ImportPolicy {
if let Some(scrub) = self.scrub { if let Some(scrub) = self.scrub {
scrub.apply(attributes); scrub.apply(attributes);
} }
self.marks.apply(attributes); marks::mark(attributes, &self.marks, self.local_pref);
if let Some(local_pref) = self.local_pref {
marks::put(
attributes,
PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)),
);
}
} }
} }
@@ -266,6 +258,11 @@ pub struct ExportPolicy {
/// Pass on the best routes learned from other peers, and not only /// Pass on the best routes learned from other peers, and not only
/// what we originate. /// what we originate.
pub learned_routes: bool, pub learned_routes: bool,
/// Pass on every route learned from other peers, not only the best
/// of each prefix, told apart by path identifiers (ADD-PATH, RFC
/// 7911), for the families the peer agreed to receive them for; a
/// route collector asks for this. For other families the best goes.
pub all_paths: bool,
/// Communities taken off and put on every route announced. /// Communities taken off and put on every route announced.
pub marks: Marks, pub marks: Marks,
/// How many more times our AS goes on the path. /// How many more times our AS goes on the path.
@@ -649,11 +646,24 @@ impl ImportPolicy {
return Err(Rejection::BogonPrefix); return Err(Rejection::BogonPrefix);
} }
} }
self.listed_or_ours(prefix, PrefixSet::allows)
}
/// Whether `prefix` is one the peer may announce by its list, where it
/// has one, and else not one of ours. `inside` says what being in a
/// set means: [`PrefixSet::allows`] for a route, which is bound by the
/// lengths allowed, [`PrefixSet::covers`] for a blackhole request,
/// which is for one address whatever the lengths.
fn listed_or_ours(
&self,
prefix: &IpPrefix,
inside: fn(&PrefixSet, &IpPrefix) -> bool,
) -> Result<(), Rejection> {
match &self.prefixes { match &self.prefixes {
Some(allowed) if !allowed.allows(prefix) => Err(Rejection::PrefixNotAllowed), Some(allowed) if !inside(allowed, prefix) => Err(Rejection::PrefixNotAllowed),
// Said to be this peer's to announce, ours or not. // Said to be this peer's to announce, ours or not.
Some(_) => Ok(()), Some(_) => Ok(()),
None if self.own_prefixes.allows(prefix) => Err(Rejection::OwnPrefix), None if inside(&self.own_prefixes, prefix) => Err(Rejection::OwnPrefix),
None => Ok(()), None => Ok(()),
} }
} }
@@ -679,12 +689,7 @@ impl ImportPolicy {
if !self.accept_bogons && is_bogon_prefix(prefix) { if !self.accept_bogons && is_bogon_prefix(prefix) {
return Err(Rejection::BogonPrefix); return Err(Rejection::BogonPrefix);
} }
match &self.prefixes { self.listed_or_ours(prefix, PrefixSet::covers)
Some(allowed) if !allowed.covers(prefix) => Err(Rejection::PrefixNotAllowed),
Some(_) => Ok(()),
None if self.own_prefixes.covers(prefix) => Err(Rejection::OwnPrefix),
None => Ok(()),
}
} }
/// Judges the next hop of routes of IP version `routes`. /// Judges the next hop of routes of IP version `routes`.
@@ -722,10 +727,10 @@ impl ImportPolicy {
let mut is_peer = false; let mut is_peer = false;
for address in named { for address in named {
let usable = !address.is_unspecified() let usable = !address.is_unspecified()
&& !address.is_loopback()
&& !address.is_multicast() && !address.is_multicast()
&& address != IpAddr::V4(Ipv4Addr::BROADCAST) && address != IpAddr::V4(Ipv4Addr::BROADCAST)
&& address != from.local_address; && (self.accept_own_host_next_hop
|| (!address.is_loopback() && address != from.local_address));
if !usable { if !usable {
return Err(Rejection::NextHopUnusable); return Err(Rejection::NextHopUnusable);
} }
@@ -1543,13 +1548,7 @@ mod tests {
); );
// Nowhere at all, whatever the policy. // Nowhere at all, whatever the policy.
let lenient = ImportPolicy::permissive(); let lenient = ImportPolicy::permissive();
for nowhere in [ for nowhere in ["0.0.0.0", "224.0.0.5", "255.255.255.255"] {
"0.0.0.0",
"127.0.0.1",
"224.0.0.5",
"255.255.255.255",
"192.0.2.1",
] {
for policy in [&policy, &lenient] { for policy in [&policy, &lenient] {
assert_eq!( assert_eq!(
check(policy, &v4(nowhere), &EXTERNAL), check(policy, &v4(nowhere), &EXTERNAL),
@@ -1558,6 +1557,20 @@ mod tests {
); );
} }
} }
// This machine: nowhere for traffic to go, unless the policy is
// the permissive one, for routers that share a host.
for ourselves in ["127.0.0.1", "192.0.2.1"] {
assert_eq!(
check(&policy, &v4(ourselves), &EXTERNAL),
Err(Rejection::NextHopUnusable),
"{ourselves}"
);
assert_eq!(
check(&lenient, &v4(ourselves), &EXTERNAL),
Ok(()),
"{ourselves}"
);
}
// Where it may be someone else: said so, a route server, our own AS. // Where it may be someone else: said so, a route server, our own AS.
assert_eq!(check(&lenient, &v4("192.0.2.3"), &EXTERNAL), Ok(())); assert_eq!(check(&lenient, &v4("192.0.2.3"), &EXTERNAL), Ok(()));
for from in [ for from in [
crates/server/src/reach.rs
+14 -40
View File
@@ -29,9 +29,9 @@ use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri}; use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_rib::table::{PeerIndex, RibTable}; use bgp_rib::table::{PeerIndex, RibTable};
use tokio::task::JoinHandle; use tokio::task::JoinHandle;
use tracing::{info, warn};
use crate::address::PeerAddress; use crate::address::PeerAddress;
use crate::rpki::{by_rib_peer, group_by_peer, revalidate_all};
use crate::session::SessionRegistry; use crate::session::SessionRegistry;
/// How long after the interfaces change the table is looked through: /// How long after the interfaces change the table is looked through:
@@ -130,37 +130,18 @@ pub(crate) fn spawn_revalidator(
continue; continue;
}; };
let sessions = registry.list(); let sessions = registry.list();
// By peer, for looking up as often as there are routes. let neighbors = by_rib_peer(
let mut neighbors: Vec<Option<Neighbor>> = Vec::new(); sessions
for session in &sessions { .iter()
let at = session.rib_peer.as_usize(); .map(|session| (session.rib_peer, neighbor_at(session.peer.addr))),
if neighbors.len() <= at {
neighbors.resize(at + 1, None);
}
neighbors[at] = Some(neighbor_at(session.peer.addr));
}
let table = Arc::clone(&rib);
// Looks at every route: not for a thread that sessions run on.
let found =
tokio::task::spawn_blocking(move || now_unreachable(&table, &topology, &neighbors))
.await;
let mut suspects = match found {
Ok(suspects) => suspects,
Err(e) => {
warn!(err = %e, "Looking for routes whose next hop is gone failed");
HashMap::new()
}
};
info!(
now_unreachable = suspects.values().map(Vec::len).sum::<usize>(),
sessions = sessions.len(),
"The machine's interfaces or routes changed; looking at next hops again"
); );
// Every session, also those with nothing found: each has its revalidate_all(
// held routes to look at again. sessions,
for session in sessions { &rib,
session.revalidate(suspects.remove(&session.rib_peer).unwrap_or_default()); "The machine's interfaces or routes changed; looking at next hops again",
} move |rib| now_unreachable(rib, &topology, &neighbors),
)
.await;
} }
}) })
} }
@@ -211,14 +192,7 @@ fn now_unreachable(
.is_ok() .is_ok()
}) })
}); });
let mut by_peer: HashMap<PeerIndex, Vec<Nlri>> = HashMap::new(); group_by_peer(routes)
for route in routes {
by_peer.entry(route.peer).or_default().push(Nlri {
prefix: route.prefix,
path_id: route.path_id,
});
}
by_peer
} }
#[cfg(test)] #[cfg(test)]
@@ -370,7 +344,7 @@ mod tests {
} }
fn best(&self, route: &str) -> Option<PeerIndex> { fn best(&self, route: &str) -> Option<PeerIndex> {
self.table.best(&prefix(route)).map(|(peer, _)| peer) self.table.best(&prefix(route)).map(|best| best.peer)
} }
} }
crates/server/src/restart.rs
+31 -26
View File
@@ -244,6 +244,30 @@ pub struct Kept {
pub until: SystemTime, pub until: SystemTime,
} }
impl Kept {
/// What was kept of a peer: `routes` routes of the IP versions
/// `versions`, under `rib_peer`, of the peer at `addr` called `name`.
/// Which restart they are of and until when they are kept are the
/// table's to say, in [`Restarts::keep`].
pub(crate) fn new(
addr: SocketAddr,
name: Option<String>,
rib_peer: PeerIndex,
versions: Vec<IpVersion>,
routes: usize,
) -> Self {
Self {
id: 0,
addr,
rib_peer,
name,
versions,
routes,
until: SystemTime::UNIX_EPOCH,
}
}
}
/// What there is of graceful restart while the server runs: the routes /// What there is of graceful restart while the server runs: the routes
/// kept of peers that are restarting, and how far the peers are with /// kept of peers that are restarting, and how far the peers are with
/// sending their routes after this daemon started. /// sending their routes after this daemon started.
@@ -298,33 +322,14 @@ impl Restarts {
}) })
} }
/// Notes that the routes of `peer` under `rib_peer` have been kept /// Notes that `kept` says what routes of `peer` have been kept, until
/// (`routes` of them, of the IP versions `versions`) until `for_at_most` /// `for_at_most` from now, and has them removed then unless the peer
/// from now, and has them removed then unless the peer is back. /// is back.
#[allow(clippy::too_many_arguments)] pub(crate) fn keep(self: &Arc<Self>, peer: PeerKey, mut kept: Kept, for_at_most: Duration) {
pub(crate) fn keep(
self: &Arc<Self>,
peer: PeerKey,
addr: SocketAddr,
name: Option<String>,
rib_peer: PeerIndex,
versions: Vec<IpVersion>,
routes: usize,
for_at_most: Duration,
) {
let id = self.ids.fetch_add(1, Ordering::Relaxed); let id = self.ids.fetch_add(1, Ordering::Relaxed);
self.kept.locked().insert( kept.id = id;
peer, kept.until = SystemTime::now() + for_at_most;
Kept { self.kept.locked().insert(peer, kept);
id,
addr,
rib_peer,
name,
versions,
routes,
until: SystemTime::now() + for_at_most,
},
);
let restarts = Arc::clone(self); let restarts = Arc::clone(self);
tokio::spawn(async move { tokio::spawn(async move {
tokio::time::sleep(for_at_most).await; tokio::time::sleep(for_at_most).await;
crates/server/src/routes.rs
+260 -1
View File
@@ -21,6 +21,14 @@ use crate::rpki::Checks;
/// than a peer sending the whole Internet has; a bound all the same. /// than a peer sending the whole Internet has; a bound all the same.
const MAX_HELD: usize = 200_000; const MAX_HELD: usize = 200_000;
/// Why an UPDATE could not be applied: the attributes its routes were to
/// be kept with could not be written in the form the table keeps them in.
/// Attributes that were decoded from a message always can be, so this is
/// a bug here, and not anything the peer did.
#[derive(Debug, thiserror::Error)]
#[error("could not keep the routes' attributes: {0}")]
pub struct Unstorable(#[from] pub(crate) EncodeError);
/// What applying an UPDATE did. /// What applying an UPDATE did.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub(crate) struct Applied { pub(crate) struct Applied {
@@ -113,7 +121,7 @@ pub(crate) fn apply(
held: &mut Held, held: &mut Held,
update: &UpdateMessage, update: &UpdateMessage,
attributes_wire: Option<&[u8]>, attributes_wire: Option<&[u8]>,
) -> Result<Applied, EncodeError> { ) -> Result<Applied, Unstorable> {
let mut applying = Applying { let mut applying = Applying {
table, table,
import, import,
@@ -598,3 +606,254 @@ pub(crate) fn revalidate(
} }
applied applied
} }
#[cfg(test)]
mod tests {
use std::net::{IpAddr, Ipv4Addr};
use bgp_packet::message::{AsPathAttribute, NextHopPathAttribute, OriginPathAttribute};
use bgp_rib::table::PeerInfo;
use bgp_rpki::{Vrp, VrpTable};
use super::*;
use crate::rejections::RejectionLog;
const PEER_ASN: u32 = 65001;
const PEER_ADDRESS: Ipv4Addr = Ipv4Addr::new(192, 0, 2, 1);
fn prefix(text: &str) -> IpPrefix {
IpPrefix::try_from(text).unwrap()
}
fn nlri(text: &str) -> Nlri {
prefix(text).into()
}
fn attrs_of(route: &str) -> Arc<AttrSet> {
RibTable::new(1)
.attrs()
.intern(&announcement(route).path_attributes, None)
.unwrap()
}
/// An UPDATE as the peer announces `route`, originating it itself.
fn announcement(route: &str) -> UpdateMessage {
UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(PEER_ADDRESS)),
],
announced_routes: vec![nlri(route)],
}
}
/// VRPs that allow `route` to be originated by `asn` and nobody else.
fn vrps(route: &str, asn: u32) -> VrpTable {
let prefix = prefix(route);
VrpTable::new([Vrp {
prefix,
max_length: prefix.length(),
asn,
}])
}
/// A table with the peer in it, and what is needed to apply its
/// UPDATEs by a permissive policy, checking only against `vrps`.
struct Fixture {
table: RibTable,
peer: PeerIndex,
policy: ImportPolicy,
rejections: RejectionLog,
}
impl Fixture {
fn new() -> Self {
let table = RibTable::new(1);
let peer = table
.add_peer(PeerInfo {
asn: PEER_ASN,
router_id: Ipv4Addr::new(10, 0, 0, 2),
address: PEER_ADDRESS.into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap();
Self {
table,
peer,
policy: ImportPolicy::permissive(),
rejections: RejectionLog::default(),
}
}
fn with_import<T>(
&self,
vrps: Option<&VrpTable>,
held: &mut Held,
work: impl FnOnce(&RibTable, &Import<'_>, &mut Held) -> T,
) -> T {
let import = Import {
peer: self.peer,
from: RouteSource {
peer_asn: PEER_ASN,
local_asn: 65000,
ibgp: false,
four_octet_asn: true,
role: None,
peer_address: PEER_ADDRESS.into(),
local_address: Ipv4Addr::new(192, 0, 2, 9).into(),
extended_next_hop: false,
},
policy: &self.policy,
checks: Checks {
vrps,
..Checks::NONE
},
rejections: &self
.rejections
.of_peer(IpAddr::from(PEER_ADDRESS).into(), "test"),
};
work(&self.table, &import, held)
}
fn announce(&self, vrps: Option<&VrpTable>, held: &mut Held, route: &str) -> Applied {
self.with_import(vrps, held, |table, import, held| {
apply(table, import, held, &announcement(route), None).unwrap()
})
}
fn revalidate(
&self,
vrps: Option<&VrpTable>,
held: &mut Held,
suspects: &[Nlri],
) -> Applied {
self.with_import(vrps, held, |table, import, held| {
revalidate(table, import, held, suspects)
})
}
}
/// Past the bound, a new route is dropped and the overflow noted, so
/// that the peer can be asked for it later; one already held is still
/// replaced, since that takes no more room.
#[test]
fn holding_stops_at_the_bound_and_says_so() {
let mut held = Held::default();
let attrs = attrs_of("198.51.100.0/24");
for n in 0..MAX_HELD as u32 {
let [a, b, c, d] = (0x0a00_0000 + n).to_be_bytes();
held.hold(
IpPrefix::new(IpVersion::Ipv4, [a, b, c, d], 32)
.unwrap()
.into(),
&attrs,
);
}
assert_eq!(held.len(), MAX_HELD);
assert!(!held.take_overflowed());
held.hold(nlri("198.51.100.0/24"), &attrs);
assert_eq!(held.len(), MAX_HELD);
assert!(held.take_overflowed());
assert!(
!held.take_overflowed(),
"noted once, until it happens again"
);
let other = attrs_of("203.0.113.0/24");
held.hold(nlri("10.0.0.0/32"), &other);
assert_eq!(held.len(), MAX_HELD);
assert!(!held.take_overflowed());
assert!(Arc::ptr_eq(&held.routes[&nlri("10.0.0.0/32")], &other));
}
#[test]
fn forgetting_takes_out_what_is_named_and_nothing_else() {
let mut held = Held::default();
let attrs = attrs_of("198.51.100.0/24");
held.hold(nlri("198.51.100.0/24"), &attrs);
held.hold(nlri("203.0.113.0/24"), &attrs);
held.forget(&[nlri("198.51.100.0/24"), nlri("192.0.2.0/24")]);
assert_eq!(held.len(), 1);
assert!(held.routes.contains_key(&nlri("203.0.113.0/24")));
held.forget(&[nlri("203.0.113.0/24")]);
held.forget(&[nlri("203.0.113.0/24")]);
assert_eq!(held.len(), 0);
}
/// A route the VRPs find invalid stays out of the table but is not
/// lost: when the VRPs change in its favour it goes in, without the
/// peer having to send it again.
#[test]
fn a_held_route_is_released_once_the_checks_pass_it() {
let fixture = Fixture::new();
let mut held = Held::default();
let route = "198.51.100.0/24";
let against = vrps(route, PEER_ASN + 1);
let applied = fixture.announce(Some(&against), &mut held, route);
assert_eq!(applied.rejected, 1);
assert_eq!(applied.outcome.added, 0);
assert!(fixture.table.is_empty());
assert_eq!(held.len(), 1);
// Nothing has changed in the route's favour yet.
let applied = fixture.revalidate(Some(&against), &mut held, &[]);
assert_eq!(applied, Applied::default());
assert_eq!(held.len(), 1);
let for_it = vrps(route, PEER_ASN);
let applied = fixture.revalidate(Some(&for_it), &mut held, &[]);
assert_eq!(applied.outcome.added, 1);
assert_eq!(applied.rejected, 0);
assert_eq!(held.len(), 0);
assert_eq!(fixture.table.routes(&prefix(route)).len(), 1);
}
/// The other way round: a route in the table that the VRPs come to
/// find invalid is taken out and held, and comes back once they
/// change again.
#[test]
fn a_route_turned_invalid_is_held_and_comes_back() {
let fixture = Fixture::new();
let mut held = Held::default();
let route = "198.51.100.0/24";
let for_it = vrps(route, PEER_ASN);
let applied = fixture.announce(Some(&for_it), &mut held, route);
assert_eq!(applied.outcome.added, 1);
assert_eq!(fixture.table.len(), 1);
let against = vrps(route, PEER_ASN + 1);
let applied = fixture.revalidate(Some(&against), &mut held, &[nlri(route)]);
assert_eq!(applied.rejected, 1);
assert_eq!(applied.outcome.removed, 1);
assert!(fixture.table.is_empty());
assert_eq!(held.len(), 1);
let applied = fixture.revalidate(Some(&for_it), &mut held, &[]);
assert_eq!(applied.outcome.added, 1);
assert_eq!(held.len(), 0);
assert_eq!(fixture.table.len(), 1);
}
/// Without anything to check against there is nothing to look at
/// again: what is held stays held, and what is in the table stays.
#[test]
fn revalidation_without_checks_changes_nothing() {
let fixture = Fixture::new();
let mut held = Held::default();
let route = "198.51.100.0/24";
fixture.announce(Some(&vrps(route, PEER_ASN + 1)), &mut held, route);
assert_eq!(held.len(), 1);
let applied = fixture.revalidate(None, &mut held, &[nlri(route)]);
assert_eq!(applied, Applied::default());
assert_eq!(held.len(), 1);
assert!(fixture.table.is_empty());
}
}
crates/server/src/rpki.rs
+203 -66
View File
@@ -22,7 +22,7 @@ use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri}; use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_rib::table::{AttrSet, PeerIndex, RibTable}; use bgp_rib::table::{AttrSet, PeerIndex, RibTable, Route};
use bgp_rpki::{Algorithm, AspaTable, Changed, PathValidity, Rpki, Update, Validity, VrpTable}; use bgp_rpki::{Algorithm, AspaTable, Changed, PathValidity, Rpki, Update, Validity, VrpTable};
use tokio::task::JoinHandle; use tokio::task::JoinHandle;
use tracing::{info, warn}; use tracing::{info, warn};
@@ -239,52 +239,92 @@ pub(crate) fn spawn_revalidator(
RPKI_UPDATES.inc(); RPKI_UPDATES.inc();
tokio::time::sleep(SETTLE).await; tokio::time::sleep(SETTLE).await;
let sessions: Vec<(SessionHandle, Subject)> = registry // The sessions whose routes are subject to the checks at all.
let sessions: Vec<SessionHandle> = registry
.list() .list()
.into_iter() .into_iter()
.filter_map(|session| { .filter(|session| session.validated().is_some())
let subject = session.validated()?;
Some((session, subject))
})
.collect(); .collect();
// By peer, for looking up as often as there are routes. let subjects = by_rib_peer(
let mut subjects: Vec<Option<Subject>> = Vec::new(); sessions
for (session, subject) in &sessions { .iter()
let at = session.rib_peer.as_usize(); .filter_map(|session| Some((session.rib_peer, session.validated()?))),
if subjects.len() <= at {
subjects.resize(at + 1, None);
}
subjects[at] = Some(*subject);
}
let (vrps, aspas) = (update.vrps.len(), update.aspas.len());
let rib = Arc::clone(&rib);
// Looks at every route, or at every prefix at least: not for a
// thread that sessions run on.
let found =
tokio::task::spawn_blocking(move || newly_invalid(&rib, &update, &subjects)).await;
let mut suspects = match found {
Ok(suspects) => suspects,
Err(e) => {
warn!(err = %e, "Looking for routes that are now invalid failed");
HashMap::new()
}
};
info!(
vrps,
aspas,
now_invalid = suspects.values().map(Vec::len).sum::<usize>(),
sessions = sessions.len(),
"RPKI data changed; revalidating routes"
); );
// Every session, also those with nothing found: each has its let what = format!(
// held routes to look at again. "RPKI data changed, to {} VRPs and {} ASPAs; revalidating routes",
for (session, _) in sessions { update.vrps.len(),
session.revalidate(suspects.remove(&session.rib_peer).unwrap_or_default()); update.aspas.len()
} );
revalidate_all(sessions, &rib, &what, move |rib| {
newly_invalid(rib, &update, &subjects)
})
.await;
} }
}) })
} }
/// `what` of each session, by its peer's number in the table, for looking
/// up as often as there are routes; `None` where no session has that
/// number.
pub(crate) fn by_rib_peer<T>(sessions: impl IntoIterator<Item = (PeerIndex, T)>) -> Vec<Option<T>> {
let mut by_peer: Vec<Option<T>> = Vec::new();
for (peer, what) in sessions {
let at = peer.as_usize();
if by_peer.len() <= at {
by_peer.resize_with(at + 1, || None);
}
by_peer[at] = Some(what);
}
by_peer
}
/// `routes` as what each session is to take out of the table, by peer.
pub(crate) fn group_by_peer(
routes: impl IntoIterator<Item = Route>,
) -> HashMap<PeerIndex, Vec<Nlri>> {
let mut by_peer: HashMap<PeerIndex, Vec<Nlri>> = HashMap::new();
for route in routes {
by_peer.entry(route.peer).or_default().push(Nlri {
prefix: route.prefix,
path_id: route.path_id,
});
}
by_peer
}
/// One round of revalidation, for whichever data changed: `find` looks
/// through the table for the routes that are to go, by peer, and every one
/// of `sessions` is told of its own, also those with nothing found, since
/// each has its held routes to look at again. `what` is for the log.
pub(crate) async fn revalidate_all<F>(
sessions: Vec<SessionHandle>,
rib: &Arc<RibTable>,
what: &str,
find: F,
) where
F: FnOnce(&RibTable) -> HashMap<PeerIndex, Vec<Nlri>> + Send + 'static,
{
let table = Arc::clone(rib);
// Looks at every route, or at every prefix at least: not for a thread
// that sessions run on.
let found = tokio::task::spawn_blocking(move || find(&table)).await;
let mut suspects = match found {
Ok(suspects) => suspects,
Err(e) => {
warn!(err = %e, "Looking through the table for routes to take out failed");
HashMap::new()
}
};
info!(
found = suspects.values().map(Vec::len).sum::<usize>(),
sessions = sessions.len(),
"{what}"
);
for session in sessions {
session.revalidate(suspects.remove(&session.rib_peer).unwrap_or_default());
}
}
/// The routes in the table that `data` finds invalid, by peer. `subjects` /// The routes in the table that `data` finds invalid, by peer. `subjects`
/// says, for each peer by its number in the table, which checks its routes /// says, for each peer by its number in the table, which checks its routes
/// are subject to. /// are subject to.
@@ -306,42 +346,54 @@ fn newly_invalid(
path.iter().any(|asn| changed.aspa_customers.contains(asn)) path.iter().any(|asn| changed.aspa_customers.contains(asn))
}; };
let path = RefCell::new(Vec::new()); let path = RefCell::new(Vec::new());
// What each peer's routes are checked against: the data, for the
// checks its routes are subject to.
let checks: Vec<Option<Checks<'_>>> = subjects
.iter()
.map(|subject| {
subject.map(|subject| Checks {
vrps: subject.origin.then_some(&*data.vrps),
aspas: subject.path.map(|check| (&*data.aspas, check)),
next_hops: None,
})
})
.collect();
let routes = rib.routes_where(usize::MAX, |route| { let routes = rib.routes_where(usize::MAX, |route| {
let Some(Some(subject)) = subjects.get(route.peer.as_usize()) else { let Some(Some(checks)) = checks.get(route.peer.as_usize()) else {
return false; return false;
}; };
if subject.origin let path = &mut *path.borrow_mut();
&& vrps_may_matter // Only the checks whose verdict the change can have altered for
&& changed.is_none_or(|changed| changed.vrps.covers(&route.prefix)) // this route are made.
{ let checks = Checks {
let origin = route.attrs.key().origin(); vrps: checks.vrps.filter(|_| {
if data.vrps.validate(&route.prefix, origin) == Validity::Invalid { vrps_may_matter && changed.is_none_or(|changed| changed.vrps.covers(&route.prefix))
return true; }),
} aspas: checks.aspas.filter(|_| {
} aspas_may_matter
if let Some(check) = subject.path && changed.is_none_or(|changed| {
&& aspas_may_matter route.attrs.as_path(path);
{ on_changed_path(changed, path)
let path = &mut *path.borrow_mut(); })
let has_set = route.attrs.as_path(path); }),
return changed.is_none_or(|changed| on_changed_path(changed, path)) next_hops: None,
&& check.verify(&data.aspas, path, has_set) == PathValidity::Invalid; };
} checks.rejection(&route.prefix, route.attrs, path).is_some()
false
}); });
let mut by_peer: HashMap<PeerIndex, Vec<Nlri>> = HashMap::new(); group_by_peer(routes)
for route in routes {
by_peer.entry(route.peer).or_default().push(Nlri {
prefix: route.prefix,
path_id: route.path_id,
});
}
by_peer
} }
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::net::Ipv4Addr;
use bgp_packet::message::{
AsPathAttribute, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
};
use bgp_rib::table::PeerInfo;
use bgp_rpki::Vrp;
use super::*; use super::*;
fn aspas() -> AspaTable { fn aspas() -> AspaTable {
@@ -463,4 +515,89 @@ mod tests {
}) })
); );
} }
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// Where it is known which VRPs came or went, only the routes those
/// cover are looked at: the others are as valid or invalid as they
/// were, and were dealt with then. Without that knowledge every route
/// is looked at; and a peer whose routes are not subject to the check
/// has none found.
#[test]
fn only_routes_a_change_can_have_made_invalid_are_found() {
let table = RibTable::new(2);
let peer = table
.add_peer(PeerInfo {
asn: 65001,
router_id: Ipv4Addr::new(10, 0, 0, 1),
address: Ipv4Addr::new(192, 0, 2, 1).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap();
let announce = |route: &str, origin: u32| {
let attrs = table
.attrs()
.intern(
&[
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65001, origin]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
None,
)
.unwrap();
table.announce(peer, &attrs, &[prefix(route)]);
};
announce("10.0.0.0/24", 65010);
announce("10.1.0.0/24", 65011);
// Both prefixes are someone else's to originate.
let vrp = |covered: &str| Vrp {
prefix: prefix(covered),
max_length: 24,
asn: 65009,
};
let update = Update {
version: 2,
vrps: Arc::new(VrpTable::new([vrp("10.0.0.0/24"), vrp("10.1.0.0/24")])),
aspas: Arc::new(AspaTable::default()),
changed: Some(Changed {
vrps: Arc::new(VrpTable::new([vrp("10.1.0.0/24")])),
aspa_customers: Arc::default(),
}),
};
let origins = by_rib_peer([(
peer,
Subject {
origin: true,
path: None,
},
)]);
let found = newly_invalid(&table, &update, &origins);
assert_eq!(
found.get(&peer).map(Vec::as_slice),
Some(&[Nlri::from(prefix("10.1.0.0/24"))][..])
);
let whole = Update {
changed: None,
..update
};
let found = newly_invalid(&table, &whole, &origins);
let routes = found.get(&peer).expect("the peer's routes");
assert_eq!(routes.len(), 2);
assert!(routes.contains(&Nlri::from(prefix("10.0.0.0/24"))));
let paths_only = by_rib_peer([(
peer,
Subject {
origin: false,
path: None,
},
)]);
assert!(newly_invalid(&table, &whole, &paths_only).is_empty());
}
} }
crates/server/src/server.rs
+11 -5
View File
@@ -118,7 +118,7 @@ impl BgpServer {
for route in &config.originate { for route in &config.originate {
local_rib.announce(route.clone()).await?; local_rib.announce(route.clone()).await?;
} }
let configured = config let configured: Vec<ConfiguredPeer> = config
.peers .peers
.iter() .iter()
.map(|p| ConfiguredPeer { .map(|p| ConfiguredPeer {
@@ -134,6 +134,11 @@ impl BgpServer {
// first routes are. // first routes are.
let (rpki, rpki_client) = config.rpki.clone().map(Rpki::spawn).unzip(); let (rpki, rpki_client) = config.rpki.clone().map(Rpki::spawn).unzip();
let rib = Arc::new(RibTable::new(table_shards())); let rib = Arc::new(RibTable::new(table_shards()));
// Sized for the largest table a peer is expected to send: the
// prefixes of the others are mostly the same ones.
if let Some(expected) = configured.iter().filter_map(|p| p.expected_routes).max() {
rib.expect_prefixes(expected as usize);
}
// Started now, so that a machine on which it cannot be done stops // Started now, so that a machine on which it cannot be done stops
// the daemon from starting. There is nothing to install yet. // the daemon from starting. There is nothing to install yet.
let (kernel, kernel_task) = match config.kernel { let (kernel, kernel_task) = match config.kernel {
@@ -232,7 +237,7 @@ impl BgpServer {
}; };
match result { match result {
Ok((stream, peer_addr)) => { Ok((stream, peer_addr)) => {
match admit(&self.config, &context, peer_addr.into(), &connections) { match screen(&self.config, &context, peer_addr.into(), &connections) {
Ok((peer, slot)) => { Ok((peer, slot)) => {
let context = Arc::clone(&context); let context = Arc::clone(&context);
tokio::spawn(serve(stream, peer_addr, context, peer, slot)); tokio::spawn(serve(stream, peer_addr, context, peer, slot));
@@ -456,9 +461,10 @@ fn listen(addr: SocketAddr) -> io::Result<TcpListener> {
socket.listen(LISTEN_BACKLOG) socket.listen(LISTEN_BACKLOG)
} }
/// Decides whether to take a connection from `address`, and if so with what /// Screens a connection from `address`: whether to take it, and if so
/// settings. /// with what settings. Not the collision check, which is
fn admit( /// [`crate::collision::CollisionTable::admit`].
fn screen(
config: &Config, config: &Config,
context: &ServerContext, context: &ServerContext,
address: PeerAddress, address: PeerAddress,
crates/server/src/session.rs
+42 -7
View File
@@ -17,6 +17,7 @@ use std::time::{Duration, SystemTime};
use bgp_packet::message::Message; use bgp_packet::message::Message;
use bgp_primitives::ip_prefix::Nlri; use bgp_primitives::ip_prefix::Nlri;
use bgp_rib::table::PeerIndex; use bgp_rib::table::PeerIndex;
use bytes::BytesMut;
use tokio::sync::{broadcast, mpsc, watch}; use tokio::sync::{broadcast, mpsc, watch};
use crate::address::{Endpoint, PeerAddress, socket_text}; use crate::address::{Endpoint, PeerAddress, socket_text};
@@ -93,6 +94,10 @@ pub struct SessionCounters {
/// Routes of the peer's kept from its last session, which it has not /// Routes of the peer's kept from its last session, which it has not
/// sent again yet (graceful restart). /// sent again yet (graceful restart).
stale: AtomicU64, stale: AtomicU64,
/// Routes announced to the peer and not withdrawn since: what it
/// holds from us, every route counted where it is sent several for
/// a prefix.
advertised: AtomicU64,
} }
impl SessionCounters { impl SessionCounters {
@@ -112,6 +117,14 @@ impl SessionCounters {
self.held.store(held as u64, Ordering::Relaxed); self.held.store(held as u64, Ordering::Relaxed);
} }
pub fn advertised(&self) -> u64 {
self.advertised.load(Ordering::Relaxed)
}
pub(crate) fn set_advertised(&self, advertised: usize) {
self.advertised.store(advertised as u64, Ordering::Relaxed);
}
pub fn stale(&self) -> u64 { pub fn stale(&self) -> u64 {
self.stale.load(Ordering::Relaxed) self.stale.load(Ordering::Relaxed)
} }
@@ -187,7 +200,7 @@ pub struct SessionHandle {
pub reflector_client: bool, pub reflector_client: bool,
/// The peer's BGP Identifier. /// The peer's BGP Identifier.
pub peer_router_id: Ipv4Addr, pub peer_router_id: Ipv4Addr,
outbound: mpsc::Sender<Message>, outbound: mpsc::Sender<Outbound>,
/// Where the session is asked to end, if it listens for that. /// Where the session is asked to end, if it listens for that.
ending: Option<mpsc::UnboundedSender<SessionError>>, ending: Option<mpsc::UnboundedSender<SessionError>>,
/// Which of the RPKI's checks the peer's routes are subject to, if /// Which of the RPKI's checks the peer's routes are subject to, if
@@ -207,7 +220,7 @@ impl SessionHandle {
local_addr: SocketAddr, local_addr: SocketAddr,
negotiated: &Negotiated, negotiated: &Negotiated,
rib_peer: PeerIndex, rib_peer: PeerIndex,
outbound: mpsc::Sender<Message>, outbound: mpsc::Sender<Outbound>,
) -> Self { ) -> Self {
Self { Self {
peer, peer,
@@ -282,6 +295,14 @@ impl SessionHandle {
} }
} }
/// What the peer goes by in the metrics: its configured name, or its
/// address.
pub fn metric_label(&self) -> String {
self.name
.clone()
.unwrap_or_else(|| crate::address::PeerAddress::from(self.peer.addr).to_string())
}
/// Whether `other` is a handle to this same session, and not to another /// Whether `other` is a handle to this same session, and not to another
/// session with the same peer. /// session with the same peer.
pub fn is_same_session(&self, other: &SessionHandle) -> bool { pub fn is_same_session(&self, other: &SessionHandle) -> bool {
@@ -292,10 +313,28 @@ impl SessionHandle {
/// space if the session's writer is behind. /// space if the session's writer is behind.
pub async fn send(&self, msg: Message) -> Result<(), SessionClosed> { pub async fn send(&self, msg: Message) -> Result<(), SessionClosed> {
self.outbound self.outbound
.send(msg) .send(Outbound::Message(msg))
.await .await
.map_err(|_| SessionClosed(self.peer)) .map_err(|_| SessionClosed(self.peer))
} }
/// Sends the peer a piece of one or more UPDATEs that are already
/// encoded, headers and all, as the exporter builds them.
pub async fn send_encoded(&self, piece: BytesMut) -> Result<(), SessionClosed> {
self.outbound
.send(Outbound::Encoded(piece))
.await
.map_err(|_| SessionClosed(self.peer))
}
}
/// What the rest of the server gives a session to send: a message, or a
/// piece of one or more UPDATEs already encoded, headers and all, which
/// goes to the peer as it is once the session is established.
#[derive(Debug)]
pub enum Outbound {
Message(Message),
Encoded(BytesMut),
} }
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@@ -421,10 +460,6 @@ impl SessionRegistry {
self.shutting_down.load(Ordering::Relaxed) self.shutting_down.load(Ordering::Relaxed)
} }
/// Closes every session politely: each peer is sent Cease /
/// Administrative Shutdown (RFC 4486) so it can drop the session at once
/// instead of discovering a dead TCP connection. Waits up to `grace` for
/// the sessions to finish; returns how many were still open.
/// Closes every connection because the daemon is going, as `leave` /// Closes every connection because the daemon is going, as `leave`
/// says: for good, or to restart (see [`Leave`]). Waits up to `grace` /// says: for good, or to restart (see [`Leave`]). Waits up to `grace`
/// for the sessions to finish; returns how many were still open. /// for the sessions to finish; returns how many were still open.
crates/web/Cargo.toml
+5
View File
@@ -24,5 +24,10 @@ tokio-stream.workspace = true
tracing.workspace = true tracing.workspace = true
nix = { workspace = true, features = ["hostname"] } nix = { workspace = true, features = ["hostname"] }
# The allocator's own account of memory, for the status page and the
# metrics. The allocator itself is the daemon's choice (bin/bgp).
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemalloc-ctl.workspace = true
[lints] [lints]
workspace = true workspace = true
crates/web/src/api.rs
+251 -33
View File
@@ -9,6 +9,7 @@ use axum::Json;
use axum::extract::{Query, State}; use axum::extract::{Query, State};
use axum::http::StatusCode; use axum::http::StatusCode;
use axum::response::{IntoResponse, Response}; use axum::response::{IntoResponse, Response};
use bgp_fib::sockdiag::TcpConnection;
use bgp_packet::community; use bgp_packet::community;
use bgp_packet::message::{ use bgp_packet::message::{
AsPathAttribute, ExtendedCommunity, NlriNextHop, OriginPathAttribute, PathAttribute, AsPathAttribute, ExtendedCommunity, NlriNextHop, OriginPathAttribute, PathAttribute,
@@ -18,7 +19,8 @@ use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{AttrSet, PeerIndex, RibTable, Route, RouteRef, SelectionKey}; use bgp_rib::table::{AttrSet, PeerIndex, RibTable, Route, RouteRef, SelectionKey};
use bgp_rib::{LocalRoute, RibStopped}; use bgp_rib::{LocalRoute, RibStopped};
use bgp_rpki::{CacheState, Update}; use bgp_rpki::{CacheState, Update};
use bgp_server::address::{PeerAddress, socket_text}; use bgp_server::address::{names_peer, socket_text};
use bgp_server::family::FamilySet;
use bgp_server::policy::PeerRole; use bgp_server::policy::PeerRole;
use bgp_server::restart::{Agreed, Restarts}; use bgp_server::restart::{Agreed, Restarts};
use bgp_server::rpki::PathCheck; use bgp_server::rpki::PathCheck;
@@ -39,6 +41,9 @@ fn unix_secs(t: SystemTime) -> u64 {
#[derive(Debug, Serialize)] #[derive(Debug, Serialize)]
pub struct StatusView { pub struct StatusView {
/// The allocator's account of the daemon's memory; `None` where the
/// allocator keeps none.
pub memory: Option<crate::memory::MemoryUse>,
pub router_id: String, pub router_id: String,
pub local_asn: u32, pub local_asn: u32,
pub version: String, pub version: String,
@@ -89,6 +94,8 @@ pub struct PeerView {
/// How many routes the peer announced on this session that its import /// How many routes the peer announced on this session that its import
/// policy refused. /// policy refused.
pub rejected_routes: Option<u64>, pub rejected_routes: Option<u64>,
/// Routes announced to the peer and not withdrawn since.
pub advertised_routes: Option<u64>,
/// How many of the peer's routes are held out of the table as invalid /// How many of the peer's routes are held out of the table as invalid
/// by the RPKI's data, to be taken if that changes. /// by the RPKI's data, to be taken if that changes.
pub held_routes: Option<u64>, pub held_routes: Option<u64>,
@@ -143,50 +150,46 @@ impl From<&Agreed> for GracefulRestartView {
impl From<&SessionHandle> for PeerView { impl From<&SessionHandle> for PeerView {
fn from(s: &SessionHandle) -> Self { fn from(s: &SessionHandle) -> Self {
let families =
|families: &FamilySet| Some(families.iter().map(|f| f.to_string()).collect());
Self { Self {
name: s.name.clone(),
peer_addr: socket_text(s.peer.addr),
peer_asn: s.peer.asn,
state: "Established".into(),
direction: Some(s.direction.as_str().into()), direction: Some(s.direction.as_str().into()),
local_addr: Some(socket_text(s.local_addr)), local_addr: Some(socket_text(s.local_addr)),
established_at_unix: Some(unix_secs(s.established_at)), established_at_unix: Some(unix_secs(s.established_at)),
uptime_secs: Some(s.established_at.elapsed().map(|d| d.as_secs()).unwrap_or(0)), uptime_secs: Some(s.established_at.elapsed().map(|d| d.as_secs()).unwrap_or(0)),
four_octet_asn: Some(s.four_octet_asn), four_octet_asn: Some(s.four_octet_asn),
extended_next_hop: Some(s.extended_next_hop), extended_next_hop: Some(s.extended_next_hop),
address_families: Some(s.families.iter().map(|f| f.to_string()).collect()), address_families: families(&s.families),
add_path_receive: Some(s.add_path_receive.iter().map(|f| f.to_string()).collect()), add_path_receive: families(&s.add_path_receive),
add_path_send: Some(s.add_path_send.iter().map(|f| f.to_string()).collect()), add_path_send: families(&s.add_path_send),
routes: Some(s.counters.routes()), routes: Some(s.counters.routes()),
rejected_routes: Some(s.counters.rejected()), rejected_routes: Some(s.counters.rejected()),
advertised_routes: Some(s.counters.advertised()),
held_routes: Some(s.counters.held()), held_routes: Some(s.counters.held()),
expected_routes: None,
role: s.role.map(PeerRole::as_str), role: s.role.map(PeerRole::as_str),
role_confirmed: s.role.map(|_| s.role_confirmed), role_confirmed: s.role.map(|_| s.role_confirmed),
last_error: None,
retry_at_unix: None,
graceful_restart: s.graceful_restart.as_ref().map(GracefulRestartView::from), graceful_restart: s.graceful_restart.as_ref().map(GracefulRestartView::from),
stale_routes: Some(s.counters.stale()).filter(|stale| *stale > 0), stale_routes: Some(s.counters.stale()).filter(|stale| *stale > 0),
stale_until_unix: None, ..Self::known(
s.name.clone(),
socket_text(s.peer.addr),
s.peer.asn,
"Established",
)
} }
} }
} }
impl PeerView { impl PeerView {
fn from_configured(peer: &ConfiguredPeer, state: &DialState) -> Self { /// What is known of any peer, with or without a session: who it is
let (state_name, last_error, retry_at_unix) = match state { /// and where it stands. Everything else is `None`, for the caller to
DialState::Idle => ("Idle", None, None), /// fill in what it knows besides.
DialState::Connecting => ("Connecting", None, None), fn known(name: Option<String>, peer_addr: String, peer_asn: u32, state: &str) -> Self {
DialState::Backoff { until, last_error } => {
("Backoff", Some(last_error.clone()), Some(unix_secs(*until)))
}
DialState::Passive { last_error } => ("Passive", last_error.clone(), None),
};
Self { Self {
name: Some(peer.name.clone()), name,
peer_addr: peer.endpoint.with_port(), peer_addr,
peer_asn: peer.remote_asn, peer_asn,
state: state_name.into(), state: state.into(),
direction: None, direction: None,
local_addr: None, local_addr: None,
established_at_unix: None, established_at_unix: None,
@@ -198,17 +201,41 @@ impl PeerView {
add_path_send: None, add_path_send: None,
routes: None, routes: None,
rejected_routes: None, rejected_routes: None,
advertised_routes: None,
held_routes: None, held_routes: None,
expected_routes: peer.expected_routes, expected_routes: None,
role: peer.role.map(PeerRole::as_str), role: None,
role_confirmed: None, role_confirmed: None,
last_error, last_error: None,
retry_at_unix, retry_at_unix: None,
graceful_restart: None, graceful_restart: None,
stale_routes: None, stale_routes: None,
stale_until_unix: None, stale_until_unix: None,
} }
} }
fn from_configured(peer: &ConfiguredPeer, state: &DialState) -> Self {
let (state_name, last_error, retry_at_unix) = match state {
DialState::Idle => ("Idle", None, None),
DialState::Connecting => ("Connecting", None, None),
DialState::Backoff { until, last_error } => {
("Backoff", Some(last_error.clone()), Some(unix_secs(*until)))
}
DialState::Passive { last_error } => ("Passive", last_error.clone(), None),
};
Self {
expected_routes: peer.expected_routes,
role: peer.role.map(PeerRole::as_str),
last_error,
retry_at_unix,
..Self::known(
Some(peer.name.clone()),
peer.endpoint.with_port(),
peer.remote_asn,
state_name,
)
}
}
} }
/// Configured peers first (with their live session when established), then /// Configured peers first (with their live session when established), then
@@ -534,9 +561,7 @@ impl PeerNames {
fn matching(&self, wanted: &str) -> Vec<PeerIndex> { fn matching(&self, wanted: &str) -> Vec<PeerIndex> {
self.peers self.peers
.iter() .iter()
.filter(|(_, (peer, name, _))| { .filter(|(_, (peer, name, _))| names_peer(wanted, name.as_deref(), peer.addr.into()))
name.as_deref() == Some(wanted) || PeerAddress::from(peer.addr).is_called(wanted)
})
.map(|(index, _)| *index) .map(|(index, _)| *index)
.collect() .collect()
} }
@@ -602,6 +627,7 @@ impl IntoResponse for Unavailable {
pub async fn status(State(state): State<AppState>) -> Result<Json<StatusView>, Unavailable> { pub async fn status(State(state): State<AppState>) -> Result<Json<StatusView>, Unavailable> {
Ok(Json(StatusView { Ok(Json(StatusView {
memory: crate::memory::memory_use(),
router_id: state.info.router_id.to_string(), router_id: state.info.router_id.to_string(),
local_asn: state.info.local_asn, local_asn: state.info.local_asn,
version: state.info.version.clone(), version: state.info.version.clone(),
@@ -1218,7 +1244,7 @@ pub async fn rejected_routes(
query query
.peer .peer
.as_deref() .as_deref()
.is_none_or(|wanted| record.label() == wanted || address.is_called(wanted)) .is_none_or(|wanted| names_peer(wanted, Some(record.label()), *address))
}) })
.map(|(address, record)| RejectionsView { .map(|(address, record)| RejectionsView {
peer: record.label().to_string(), peer: record.label().to_string(),
@@ -1248,6 +1274,168 @@ pub async fn rejected_routes(
Json(peers) Json(peers)
} }
/// `GET /api/tcp?peer=NAME`: what the kernel knows of the TCP connection of
/// the session with that peer.
#[derive(Debug, Deserialize)]
pub struct TcpQuery {
/// The peer whose session it is: its configured name, or its address.
/// The address with its port, as `peer_addr` of `/api/peers` has it,
/// names it too.
pub peer: String,
}
/// A session's TCP connection, as the kernel reports it (Linux only).
/// Times in milliseconds, sizes in octets, rates in octets per second.
#[derive(Debug, Serialize)]
pub struct TcpView {
pub state: &'static str,
pub congestion: Option<String>,
pub rtt_ms: f64,
pub rtt_variance_ms: f64,
pub min_rtt_ms: f64,
pub rto_ms: f64,
pub mss: u32,
pub pmtu: u32,
pub cwnd_segments: u32,
pub ssthresh_segments: Option<u32>,
/// The peer's receive window: what it lets us send.
pub send_window: u32,
/// Our receive window: what we let the peer send.
pub receive_window: u32,
pub receive_space: u32,
pub send_window_scale: u8,
pub receive_window_scale: u8,
/// Octets written by us and not yet acknowledged, and of those the
/// octets not even sent yet.
pub send_queue: u32,
pub not_sent: u32,
pub recv_queue: u32,
pub unacked_segments: u32,
pub retrans_segments: u32,
pub lost_segments: u32,
pub total_retrans: u32,
pub bytes_retrans: u64,
pub pacing_rate: u64,
pub delivery_rate: u64,
pub bytes_sent: u64,
pub bytes_acked: u64,
pub bytes_received: u64,
pub segments_out: u32,
pub segments_in: u32,
/// How long the connection had something to send, and of that how
/// long it was held up by the peer's window and by our send buffer.
pub busy_ms: f64,
pub limited_by_receive_window_ms: f64,
pub limited_by_send_buffer_ms: f64,
/// The same two, as shares of the busy time, 0 to 1.
pub limited_by_receive_window_share: f64,
pub limited_by_send_buffer_share: f64,
pub last_data_sent_ms: u32,
pub last_data_received_ms: u32,
pub last_ack_received_ms: u32,
}
impl From<&TcpConnection> for TcpView {
fn from(c: &TcpConnection) -> Self {
// The kernel's times are in microseconds.
let ms = |us: u64| us as f64 / 1000.0;
// A connection that was never busy was never held up: a share of
// nothing is nothing, not a division by zero.
let share = |part: u64| {
if c.busy == 0 {
0.0
} else {
part as f64 / c.busy as f64
}
};
Self {
state: c.state_name(),
congestion: c.congestion.clone(),
rtt_ms: ms(c.rtt.into()),
rtt_variance_ms: ms(c.rtt_variance.into()),
min_rtt_ms: ms(c.min_rtt.into()),
rto_ms: ms(c.rto.into()),
mss: c.mss,
pmtu: c.pmtu,
cwnd_segments: c.cwnd,
ssthresh_segments: c.ssthresh,
send_window: c.send_window,
receive_window: c.receive_window,
receive_space: c.receive_space,
send_window_scale: c.send_window_scale,
receive_window_scale: c.receive_window_scale,
send_queue: c.send_queue,
not_sent: c.not_sent,
recv_queue: c.recv_queue,
unacked_segments: c.unacked,
retrans_segments: c.retrans,
lost_segments: c.lost,
total_retrans: c.total_retrans,
bytes_retrans: c.bytes_retrans,
pacing_rate: c.pacing_rate,
delivery_rate: c.delivery_rate,
bytes_sent: c.bytes_sent,
bytes_acked: c.bytes_acked,
bytes_received: c.bytes_received,
segments_out: c.segments_out,
segments_in: c.segments_in,
busy_ms: ms(c.busy),
limited_by_receive_window_ms: ms(c.limited_by_receive_window),
limited_by_send_buffer_ms: ms(c.limited_by_send_buffer),
limited_by_receive_window_share: share(c.limited_by_receive_window),
limited_by_send_buffer_share: share(c.limited_by_send_buffer),
last_data_sent_ms: c.last_data_sent_ms,
last_data_received_ms: c.last_data_received_ms,
last_ack_received_ms: c.last_ack_received_ms,
}
}
}
/// The answer to `/api/tcp`.
#[derive(Debug, Serialize)]
pub struct TcpAnswer {
/// Whether the kernel could be asked and knew the connection.
pub available: bool,
/// Why not, if not.
pub reason: Option<String>,
pub connection: Option<TcpView>,
}
pub async fn tcp(
State(state): State<AppState>,
Query(query): Query<TcpQuery>,
) -> Result<Json<TcpAnswer>, QueryError> {
let session = state
.registry
.list()
.into_iter()
.find(|s| {
names_peer(&query.peer, s.name.as_deref(), s.peer.addr.into())
|| socket_text(s.peer.addr) == query.peer
})
.ok_or_else(|| QueryError::Bad(format!("no session with {}", query.peer)))?;
let unavailable = |reason: String| TcpAnswer {
available: false,
reason: Some(reason),
connection: None,
};
let answer =
match bgp_fib::sockdiag::tcp_connection(session.local_addr, session.peer.addr).await {
Ok(Some(c)) => TcpAnswer {
available: true,
reason: None,
connection: Some(TcpView::from(&c)),
},
Ok(None) => unavailable(if cfg!(target_os = "linux") {
"the kernel has no such connection".to_string()
} else {
"only on Linux".to_string()
}),
Err(e) => unavailable(format!("the kernel could not be asked: {e}")),
};
Ok(Json(answer))
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::net::Ipv4Addr; use std::net::Ipv4Addr;
@@ -1572,4 +1760,34 @@ mod tests {
.map(|(name, value)| (name.to_string(), value.to_string())); .map(|(name, value)| (name.to_string(), value.to_string()));
assert_eq!(other, expected); assert_eq!(other, expected);
} }
/// The kernel's microseconds are shown as milliseconds, and the time
/// held up as a share of the busy time; a connection that was never
/// busy has a share of zero, not of NaN.
#[test]
fn a_tcp_view_has_milliseconds_and_shares_of_the_busy_time() {
let idle = TcpConnection {
rtt: 1500,
busy: 0,
limited_by_receive_window: 0,
..TcpConnection::default()
};
let view = TcpView::from(&idle);
assert_eq!(view.rtt_ms, 1.5);
assert_eq!(view.busy_ms, 0.0);
assert_eq!(view.limited_by_receive_window_share, 0.0);
assert_eq!(view.limited_by_send_buffer_share, 0.0);
let busy = TcpConnection {
busy: 4_000_000,
limited_by_receive_window: 1_000_000,
limited_by_send_buffer: 3_000_000,
..TcpConnection::default()
};
let view = TcpView::from(&busy);
assert_eq!(view.busy_ms, 4000.0);
assert_eq!(view.limited_by_receive_window_ms, 1000.0);
assert_eq!(view.limited_by_receive_window_share, 0.25);
assert_eq!(view.limited_by_send_buffer_share, 0.75);
}
} }
crates/web/src/lib.rs
+4 -7
View File
@@ -7,6 +7,7 @@
mod api; mod api;
mod events; mod events;
mod memory;
mod metrics; mod metrics;
use std::net::Ipv4Addr; use std::net::Ipv4Addr;
@@ -27,11 +28,6 @@ use serde::Serialize;
use tokio::net::TcpListener; use tokio::net::TcpListener;
use tracing::info; use tracing::info;
pub use api::{
AttrSummary, ConfigView, KernelView, LocalRouteView, PeerView, ReceivedRouteView,
RejectedRouteView, RejectionsView, RoutesPage, RpkiView, StatusView,
};
/// Static facts about the router shown in status output. /// Static facts about the router shown in status output.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct RouterInfo { pub struct RouterInfo {
@@ -97,8 +93,8 @@ impl RouterInfo {
} }
} }
/// Best-effort host name without pulling in a dependency: ask the `hostname` /// The machine's host name as the kernel has it, for display; a machine
/// utility, which exists on every platform we run on. /// without one, or with one that is not text, is "unknown host".
fn hostname() -> String { fn hostname() -> String {
nix::unistd::gethostname() nix::unistd::gethostname()
.ok() .ok()
@@ -144,6 +140,7 @@ pub fn router(state: AppState) -> Router {
.route("/api/routes/rejected", get(api::rejected_routes)) .route("/api/routes/rejected", get(api::rejected_routes))
.route("/api/rpki", get(api::rpki)) .route("/api/rpki", get(api::rpki))
.route("/api/kernel", get(api::kernel)) .route("/api/kernel", get(api::kernel))
.route("/api/tcp", get(api::tcp))
.route("/api/events", get(events::stream)) .route("/api/events", get(events::stream))
.route("/metricz", get(metrics::metricz)) .route("/metricz", get(metrics::metricz))
.with_state(state) .with_state(state)
crates/web/src/memory.rs
+46
View File
@@ -0,0 +1,46 @@
//! What the allocator says the daemon holds: jemalloc's own statistics,
//! for the status page and the metrics, where the kernel's resident size
//! alone would not say what is in use and what the allocator keeps back.
use serde::Serialize;
/// jemalloc's account of memory, in octets.
#[derive(Debug, Clone, Copy, Serialize)]
pub struct MemoryUse {
/// Octets in allocations the program holds.
pub allocated: u64,
/// Octets in the pages those allocations are in: `allocated` plus
/// the fragmentation within pages.
pub active: u64,
/// Octets of the allocator's own bookkeeping.
pub metadata: u64,
/// Octets in pages mapped and physically resident, as far as the
/// allocator knows: `active`, the metadata, and pages freed but not
/// yet returned to the kernel.
pub resident: u64,
/// Octets mapped, resident or not.
pub mapped: u64,
/// Octets of address space kept back after being unmapped, which
/// cost nothing physical.
pub retained: u64,
}
#[cfg(not(target_env = "msvc"))]
pub fn memory_use() -> Option<MemoryUse> {
use tikv_jemalloc_ctl::{epoch, stats};
// The statistics are refreshed only when the epoch is advanced.
epoch::advance().ok()?;
Some(MemoryUse {
allocated: stats::allocated::read().ok()? as u64,
active: stats::active::read().ok()? as u64,
metadata: stats::metadata::read().ok()? as u64,
resident: stats::resident::read().ok()? as u64,
mapped: stats::mapped::read().ok()? as u64,
retained: stats::retained::read().ok()? as u64,
})
}
#[cfg(target_env = "msvc")]
pub fn memory_use() -> Option<MemoryUse> {
None
}
crates/web/src/metrics.rs
+33 -1
View File
@@ -5,7 +5,8 @@ use axum::extract::State;
use axum::http::{StatusCode, header}; use axum::http::{StatusCode, header};
use axum::response::IntoResponse; use axum::response::IntoResponse;
use bgp_server::metrics::{ use bgp_server::metrics::{
LOCAL_ROUTES, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES, RIB_ROUTES, LOCAL_ROUTES, MEMORY, PEER_TCP, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES,
RIB_ROUTES,
}; };
use prometheus::{Encoder, TextEncoder}; use prometheus::{Encoder, TextEncoder};
@@ -24,6 +25,20 @@ pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
RIB_PREFIXES.set(gauge(state.rib.prefix_count())); RIB_PREFIXES.set(gauge(state.rib.prefix_count()));
RIB_ATTRIBUTE_SETS.set(gauge(state.rib.attrs().len())); RIB_ATTRIBUTE_SETS.set(gauge(state.rib.attrs().len()));
RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len())); RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len()));
sample_tcp(&state).await;
if let Some(m) = crate::memory::memory_use() {
let int = |n: u64| i64::try_from(n).unwrap_or(i64::MAX);
for (kind, octets) in [
("allocated", m.allocated),
("active", m.active),
("metadata", m.metadata),
("resident", m.resident),
("mapped", m.mapped),
("retained", m.retained),
] {
MEMORY.with_label_values(&[kind]).set(int(octets));
}
}
let mut buf = Vec::new(); let mut buf = Vec::new();
let encoder = TextEncoder::new(); let encoder = TextEncoder::new();
@@ -37,3 +52,20 @@ pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(), Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
} }
} }
/// Asks the kernel about every established session's TCP connection and
/// sets the peer's gauges: one netlink round trip a session, at scrape
/// time, so that there is history to look back on. A session the kernel
/// has no connection for any more loses its series.
async fn sample_tcp(state: &AppState) {
let tcp = &*PEER_TCP;
for session in state.registry.list() {
let label = session.metric_label();
let connection =
bgp_fib::sockdiag::tcp_connection(session.local_addr, session.peer.addr).await;
match connection {
Ok(Some(c)) => tcp.set(&label, &c),
_ => tcp.remove(&label),
}
}
}
crates/web/static/statusz.html
+164 -17
View File
@@ -68,6 +68,17 @@ table.data th { background: var(--head); font-weight: bold; position: sticky; to
table.data > tbody > tr:hover > td { background: var(--hover); } table.data > tbody > tr:hover > td { background: var(--hover); }
table.data td.num, table.data th.num { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; } table.data td.num, table.data th.num { text-align: right; font-variant-numeric: tabular-nums; white-space: nowrap; }
table.data tr.group > td { background: var(--band); font-weight: bold; padding: 3px 6px; } table.data tr.group > td { background: var(--band); font-weight: bold; padding: 3px 6px; }
/* A row of detail under a row of the table: a session's TCP connection, a
route in full. The route's is indented past the rank column. */
table.data tr.detail > td { background: var(--band); padding: 6px 10px 8px 28px; }
table.data tr.route-detail > td { padding-left: 40px; }
table.data td.attrs { cursor: pointer; }
table.data td.attrs:hover { background: var(--hover); }
/* Names and values, two columns, as the detail rows show them. */
.kv-grid { display: grid; grid-template-columns: max-content 1fr; gap: 1px 16px; font-size: 12px; }
.kv-grid > div:nth-child(odd) { color: var(--fg-soft); white-space: nowrap; }
a.tcp-link { color: var(--dim); font-size: 11.5px; text-decoration: none; border: 1px solid var(--line); border-radius: 3px; padding: 0 4px; margin-left: 4px; }
a.tcp-link:hover, a.tcp-link.open { color: var(--link); border-color: var(--link); }
table.data tr.best > td { background: var(--best-bg); } table.data tr.best > td { background: var(--best-bg); }
table.data tr.dim td { color: var(--dim); } table.data tr.dim td { color: var(--dim); }
.asn { cursor: help; } .asn { cursor: help; }
@@ -157,7 +168,7 @@ input, select, button { color-scheme: light dark; }
<div> <div>
<div class="lefthand"> <div class="lefthand">
<b id="s-hostname">...</b>: router ID <b id="s-router-id">...</b>, AS <b id="s-asn">...</b>, Ferrix <span id="s-version">...</span><br> <b id="s-hostname">...</b>: router ID <b id="s-router-id">...</b>, AS <b id="s-asn">...</b>, Ferrix <span id="s-version">...</span><br>
Started <span id="s-started">...</span>, up <span id="s-uptime">...</span><br> Started <span id="s-started">...</span>, up <span id="s-uptime">...</span><span id="s-memory"></span><br>
<span class="dim">On this page: <a href="#peers">peers</a> · <a href="#rpki">RPKI</a> · <a href="#local">originated</a> · <a href="#kernel">kernel</a> · <a href="#received">received, and looking routes up</a> · <a href="#rejected">rejected</a> · <a href="#events">events</a></span><br> <span class="dim">On this page: <a href="#peers">peers</a> · <a href="#rpki">RPKI</a> · <a href="#local">originated</a> · <a href="#kernel">kernel</a> · <a href="#received">received, and looking routes up</a> · <a href="#rejected">rejected</a> · <a href="#events">events</a></span><br>
</div> </div>
<div class="righthand"> <div class="righthand">
@@ -181,7 +192,7 @@ JSON: <a href="/api/status">status</a>, <a href="/api/peers">peers</a>, <a href=
<span class="dim legend">Glyphs: <span class="g">v4</span><span class="g">v6</span> address families · <span class="g">GR</span> graceful restart · <span class="g">ENH</span> IPv4 with IPv6 next hops · <span class="g">AP</span> ADD-PATH · <span class="g amber">2B</span> two-octet AS numbers only · <span class="g">↑</span> provider <span class="g">↓</span> customer <span class="g">↔</span> peer <span class="g">RS</span> route server <span class="g">RSC</span> its client <span class="g">iBGP</span> our AS. Point at one for its meaning; click a peer's name to look at its routes.</span> <span class="dim legend">Glyphs: <span class="g">v4</span><span class="g">v6</span> address families · <span class="g">GR</span> graceful restart · <span class="g">ENH</span> IPv4 with IPv6 next hops · <span class="g">AP</span> ADD-PATH · <span class="g amber">2B</span> two-octet AS numbers only · <span class="g">↑</span> provider <span class="g">↓</span> customer <span class="g">↔</span> peer <span class="g">RS</span> route server <span class="g">RSC</span> its client <span class="g">iBGP</span> our AS. Point at one for its meaning; click a peer's name to look at its routes.</span>
</div> </div>
<table class="data"> <table class="data">
<thead><tr><th>name</th><th>peer</th><th>state</th><th>up for</th><th class="c-opt">our address (next hop)</th><th class="c-opt">session</th><th class="num">routes</th><th class="num" title="Announcements rejected by the import policy since the daemon started, and routes held out of the table as invalid or unreachable now">rejected · held</th><th>detail</th></tr></thead> <thead><tr><th>name</th><th>peer</th><th>state</th><th>up for</th><th class="c-opt">our address (next hop)</th><th class="c-opt">session</th><th class="num">routes</th><th class="num" title="Routes announced to the peer and not withdrawn since; with ADD-PATH, every route sent counts">sent</th><th class="num" title="Announcements rejected by the import policy since the daemon started, and routes held out of the table as invalid or unreachable now">rejected · held</th><th>detail</th></tr></thead>
<tbody id="peers-body"></tbody> <tbody id="peers-body"></tbody>
</table> </table>
@@ -241,7 +252,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
const $ = id => document.getElementById(id); const $ = id => document.getElementById(id);
const esc = s => String(s).replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c])); const esc = s => String(s).replace(/[&<>"']/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
const state = { status: null, peers: [], local: [], received: [], receivedPage: {}, rejected: [], rpki: null, kernel: null }; const state = { status: null, peers: [], local: [], received: [], receivedPage: {}, rejected: [], rpki: null, kernel: null, tcpOpen: new Set(), routeOpen: new Set() };
const DASH = '<span class="dim">–</span>'; const DASH = '<span class="dim">–</span>';
const fmtCount = n => Number(n).toLocaleString('en-US'); const fmtCount = n => Number(n).toLocaleString('en-US');
// A glyph: a small badge with its meaning as the tooltip. // A glyph: a small badge with its meaning as the tooltip.
@@ -268,6 +279,17 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
if (state.status) $('s-uptime').textContent = fmtDur(t - state.status.started_at_unix); if (state.status) $('s-uptime').textContent = fmtDur(t - state.status.started_at_unix);
} }
const fmtWhen = unix => new Date(unix * 1000).toLocaleString(); const fmtWhen = unix => new Date(unix * 1000).toLocaleString();
const fmtOctets = n => n >= 1048576 ? `${(n / 1048576).toFixed(1)} MiB` : n >= 1024 ? `${(n / 1024).toFixed(1)} KiB` : `${n} B`;
const fmtBits = n => { const b = n * 8; return b >= 1e9 ? `${(b / 1e9).toFixed(2)} Gbit/s` : b >= 1e6 ? `${(b / 1e6).toFixed(1)} Mbit/s` : `${(b / 1e3).toFixed(0)} kbit/s`; };
const fmtMs = n => n >= 1000 ? `${(n / 1000).toFixed(1)} s` : `${n.toFixed(n < 10 ? 2 : 0)} ms`;
const ago = ms => ms >= 60000 ? `${Math.round(ms / 60000)}m` : ms >= 1000 ? `${(ms / 1000).toFixed(1)}s` : `${ms}ms`;
// A cell across every column of the table that `body` is the body of,
// and a row of one: for a heading, a note or a detail row. The width is
// the table's header's, so a column added there is covered here.
const fullCell = (body, html, cls = '') => `<td colspan="${body.closest('table').querySelectorAll('thead th').length}"${cls ? ` class="${cls}"` : ''}>${html}</td>`;
const fullRow = (body, html, rowClass = '', cellClass = '') => `<tr${rowClass ? ` class="${rowClass}"` : ''}>${fullCell(body, html, cellClass)}</tr>`;
// Names and values in two columns, for a detail row.
const kvGrid = rows => `<div class="kv-grid mono">${rows.map(([k, v]) => `<div>${k}</div><div>${v}</div>`).join('')}</div>`;
// What a community of ours asks for where control communities are on // What a community of ours asks for where control communities are on
// (control_communities), or ''. They are acted on where a route leaves // (control_communities), or ''. They are acted on where a route leaves
@@ -402,6 +424,8 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
$('s-asn').className = 'asn'; $('s-asn').dataset.asn = s.local_asn; $('s-asn').className = 'asn'; $('s-asn').dataset.asn = s.local_asn;
$('s-version').textContent = s.version; $('s-hostname').textContent = s.hostname; $('s-version').textContent = s.version; $('s-hostname').textContent = s.hostname;
$('s-started').textContent = new Date(s.started_at_unix * 1000).toString().replace(/ \(.*\)$/, ''); $('s-started').textContent = new Date(s.started_at_unix * 1000).toString().replace(/ \(.*\)$/, '');
const m = s.memory;
setHTML($('s-memory'), m ? ` · memory <span class="mono" title="jemalloc: allocated / active / resident / mapped / retained">${fmtOctets(m.allocated)} allocated, ${fmtOctets(m.resident)} resident</span>` : '');
tick(); tick();
} }
@@ -488,8 +512,10 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
'route server': ['RS', 'route server of an Internet exchange'], 'route server': ['RS', 'route server of an Internet exchange'],
'route server client': ['RSC', 'client of a route server we run'], 'route server client': ['RSC', 'client of a route server we run'],
}; };
// A router of our own AS, by its AS number.
const isIbgp = p => p.peer_asn === (state.status || {}).local_asn;
function roleGlyph(p) { function roleGlyph(p) {
if (p.peer_asn === (state.status || {}).local_asn) return glyph('iBGP', 'a router of our own AS'); if (isIbgp(p)) return glyph('iBGP', 'a router of our own AS');
const r = ROLE_GLYPH[p.role]; const r = ROLE_GLYPH[p.role];
if (!r) return glyph('?', 'no role configured', 'amber'); if (!r) return glyph('?', 'no role configured', 'amber');
const confirmed = p.role_confirmed ? '; the peer says so too (RFC 9234)' : p.state === 'Established' ? '; by our configuration, the peer has not said what it is' : ''; const confirmed = p.role_confirmed ? '; the peer says so too (RFC 9234)' : p.state === 'Established' ? '; by our configuration, the peer has not said what it is' : '';
@@ -521,7 +547,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
} }
const ROLE_ORDER = ['iBGP', 'provider', 'route server', 'peer', 'customer', 'route server client', 'none']; const ROLE_ORDER = ['iBGP', 'provider', 'route server', 'peer', 'customer', 'route server client', 'none'];
function peerGroup(p) { function peerGroup(p) {
if (p.peer_asn === (state.status || {}).local_asn) return 'iBGP'; if (isIbgp(p)) return 'iBGP';
return p.role || 'none'; return p.role || 'none';
} }
const GROUP_TITLE = { iBGP: 'Our own routers', provider: 'Providers', 'route server': 'Route servers', peer: 'Peers', customer: 'Customers', 'route server client': 'Route server clients', none: 'Without a role' }; const GROUP_TITLE = { iBGP: 'Our own routers', provider: 'Providers', 'route server': 'Route servers', peer: 'Peers', customer: 'Customers', 'route server client': 'Route server clients', none: 'Without a role' };
@@ -532,7 +558,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
const down = peers.filter(p => peerRank(p) === 0).length; const down = peers.filter(p => peerRank(p) === 0).length;
const waiting = peers.filter(p => p.state === 'Passive').length; const waiting = peers.filter(p => p.state === 'Passive').length;
$('peers-count').textContent = `${up} of ${peers.length} established` + (down ? `, ${down} down` : '') + (waiting ? `, ${waiting} waiting` : ''); $('peers-count').textContent = `${up} of ${peers.length} established` + (down ? `, ${down} down` : '') + (waiting ? `, ${waiting} waiting` : '');
if (!peers.length) { setHTML(b, '<tr><td colspan="9" class="dim">No peers configured and none connected.</td></tr>'); return; } if (!peers.length) { setHTML(b, fullRow(b, 'No peers configured and none connected.', '', 'dim')); return; }
const problemsOnly = $('peers-problems').checked; const problemsOnly = $('peers-problems').checked;
// The IPv4 and IPv6 sessions of one neighbour, named <stem>-v4 and // The IPv4 and IPv6 sessions of one neighbour, named <stem>-v4 and
// <stem>-v6, are one neighbour to the reader: listed together, IPv6 // <stem>-v6, are one neighbour to the reader: listed together, IPv6
@@ -549,7 +575,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
const expected = up && p.expected_routes > 0 const expected = up && p.expected_routes > 0
? ` <span class="${short ? 'amber' : 'dim'}" title="${short ? 'Well under' : 'About'} the ${fmtCount(p.expected_routes)} routes this peer is expected to send (expected_routes)">of ~${fmtCount(p.expected_routes)}</span>` : ''; ? ` <span class="${short ? 'amber' : 'dim'}" title="${short ? 'Well under' : 'About'} the ${fmtCount(p.expected_routes)} routes this peer is expected to send (expected_routes)">of ~${fmtCount(p.expected_routes)}</span>` : '';
const classes = [peerRank(p) === 3 ? 'dim' : '', pair || ''].filter(Boolean).join(' '); const classes = [peerRank(p) === 3 ? 'dim' : '', pair || ''].filter(Boolean).join(' ');
return `<tr class="${classes}" data-peer-row="${esc(p.name || p.peer_addr)}"> return `<tr class="${classes}" data-peer-row="${esc(p.name || p.peer_addr)}" data-peer-addr="${esc(p.peer_addr)}">
<td class="name">${sinceGlyph(p)}${name}</td> <td class="name">${sinceGlyph(p)}${name}</td>
<td class="mono">${esc(p.peer_addr)} <span class="dim">AS${asn(p.peer_asn)}</span> ${roleGlyph(p)}</td> <td class="mono">${esc(p.peer_addr)} <span class="dim">AS${asn(p.peer_asn)}</span> ${roleGlyph(p)}</td>
<td><span class="${STATE_CLASS[p.state] || ''}">${esc(p.state)}</span></td> <td><span class="${STATE_CLASS[p.state] || ''}">${esc(p.state)}</span></td>
@@ -557,8 +583,9 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
<td class="mono c-opt">${up ? esc(p.local_addr) : DASH}</td> <td class="mono c-opt">${up ? esc(p.local_addr) : DASH}</td>
<td class="c-opt">${up ? sessionGlyphs(p) : DASH}</td> <td class="c-opt">${up ? sessionGlyphs(p) : DASH}</td>
<td class="num">${up ? trendGlyph(p) + fmtCount(p.routes) + expected + stale : DASH}</td> <td class="num">${up ? trendGlyph(p) + fmtCount(p.routes) + expected + stale : DASH}</td>
<td class="num">${up ? fmtCount(p.advertised_routes) : DASH}</td>
<td class="num">${rejected}${held}</td> <td class="num">${rejected}${held}</td>
<td>${peerDetail(p)}</td></tr>`; <td>${peerDetail(p)}${up ? tcpLink(p) : ''}</td></tr>`;
}; };
// Rows for a sorted list, with the v4/v6 pairs in it marked. // Rows for a sorted list, with the v4/v6 pairs in it marked.
const rows = list => list.map((p, i) => { const rows = list => list.map((p, i) => {
@@ -574,23 +601,90 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
// they are to us. // they are to us.
const problems = peers.filter(p => peerRank(p) <= 1).sort(byName); const problems = peers.filter(p => peerRank(p) <= 1).sort(byName);
if (problems.length) { if (problems.length) {
html += `<tr class="group"><td colspan="9"><span class="red">Needing attention</span> (${problems.length})</td></tr>` + rows(problems); html += fullRow(b, `<span class="red">Needing attention</span> (${problems.length})`, 'group') + rows(problems);
} }
if (!problemsOnly) { if (!problemsOnly) {
for (const group of ROLE_ORDER) { for (const group of ROLE_ORDER) {
const members = peers.filter(p => peerRank(p) >= 2 && peerGroup(p) === group).sort(byName); const members = peers.filter(p => peerRank(p) >= 2 && peerGroup(p) === group).sort(byName);
if (!members.length) continue; if (!members.length) continue;
html += `<tr class="group"><td colspan="9">${GROUP_TITLE[group]} (${members.length})</td></tr>` + rows(members); html += fullRow(b, `${GROUP_TITLE[group]} (${members.length})`, 'group') + rows(members);
} }
} else if (!problems.length) { } else if (!problems.length) {
html += '<tr><td colspan="9" class="dim">Nothing needs attention.</td></tr>'; html += fullRow(b, 'Nothing needs attention.', '', 'dim');
} }
setHTML(b, html); setHTML(b, html);
renderTcpRows();
const marked = peers.filter(changedSince).length; const marked = peers.filter(changedSince).length;
setHTML($('since-ctl'), marked ? `${glyph('●', 'Rows that changed while this tab was in the background', 'since')} <span class="dim">${marked} changed since you last looked ·</span> <a href="#" id="since-clear">clear</a>` : ''); setHTML($('since-ctl'), marked ? `${glyph('●', 'Rows that changed while this tab was in the background', 'since')} <span class="dim">${marked} changed since you last looked ·</span> <a href="#" id="since-clear">clear</a>` : '');
renderPeerChoices(); renderPeerChoices();
} }
// The TCP connection of a session, as the kernel sees it: a link on the
// row, and a row below it while it is open, asked for afresh on every
// refresh.
function tcpLink(p) {
const open = state.tcpOpen.has(p.peer_addr);
return `<a href="#" class="tcp-link${open ? ' open' : ''}" data-peer="${esc(p.peer_addr)}" title="TCP_INFO for this session's socket">tcp</a>`;
}
function tcpDetail(a) {
if (!a.available) return `<span class="dim">${esc(a.reason || 'n/a')}</span>`;
const c = a.connection;
const pct = share => `${(share * 100).toFixed(0)}%`;
const held = share => share > 0.5 ? 'red' : share > 0.1 ? 'amber' : '';
const kv = (k, v) => `<span class="dim">${k}</span> ${v}`;
const rows = [
['state', `${esc(c.state.toUpperCase())} ${c.congestion ? esc(c.congestion) : ''}`],
['rtt', `${fmtMs(c.rtt_ms)} ${kv('minrtt', fmtMs(c.min_rtt_ms))} ${kv('rttvar', fmtMs(c.rtt_variance_ms))} ${kv('rto', fmtMs(c.rto_ms))}`],
['cwnd', `${fmtCount(c.cwnd_segments)} ${kv('mss', `${c.mss}`)} ${kv('=', fmtOctets(c.cwnd_segments * c.mss))} ${kv('ssthresh', c.ssthresh_segments == null ? '∞' : fmtCount(c.ssthresh_segments))} ${kv('pmtu', `${c.pmtu}`)}`],
['snd_wnd', `${fmtOctets(c.send_window)} ${kv('wscale', `${c.send_window_scale}`)}`],
['rcv_wnd', `${fmtOctets(c.receive_window)} ${kv('rcv_space', fmtOctets(c.receive_space))} ${kv('wscale', `${c.receive_window_scale}`)}`],
['send-q', `${fmtOctets(c.send_queue)} ${kv('notsent', fmtOctets(c.not_sent))} ${kv('unacked', fmtCount(c.unacked_segments))}`],
['recv-q', fmtOctets(c.recv_queue)],
['retrans', `${fmtCount(c.retrans_segments)} ${kv('lost', fmtCount(c.lost_segments))} ${kv('total', `${fmtCount(c.total_retrans)} / ${fmtOctets(c.bytes_retrans)}`)}`],
['rate', `${kv('delivery', fmtBits(c.delivery_rate))} ${kv('pacing', fmtBits(c.pacing_rate))}`],
['limited', `${kv('busy', fmtMs(c.busy_ms))} <span class="${held(c.limited_by_receive_window_share)}">${kv('rwnd', pct(c.limited_by_receive_window_share))}</span> <span class="${held(c.limited_by_send_buffer_share)}">${kv('sndbuf', pct(c.limited_by_send_buffer_share))}</span>`],
['bytes', `${kv('sent', fmtOctets(c.bytes_sent))} ${kv('acked', fmtOctets(c.bytes_acked))} ${kv('received', fmtOctets(c.bytes_received))} ${kv('segs', `${fmtCount(c.segments_out)}↑ ${fmtCount(c.segments_in)}↓`)}`],
['last', `${kv('snd', ago(c.last_data_sent_ms))} ${kv('rcv', ago(c.last_data_received_ms))} ${kv('ack', ago(c.last_ack_received_ms))}`],
];
return kvGrid(rows);
}
async function renderTcpRows() {
const b = $('peers-body');
for (const addr of state.tcpOpen) {
const tr = b.querySelector(`tr[data-peer-addr="${CSS.escape(addr)}"]`);
if (!tr) continue;
let row = tr.nextElementSibling;
if (!row || !row.classList.contains('tcp-row')) {
row = document.createElement('tr');
row.className = 'detail tcp-row';
row.innerHTML = fullCell(b, '<span class="dim">asking the kernel…</span>');
tr.after(row);
}
try {
const answer = await getJSON(`/api/tcp?peer=${encodeURIComponent(addr)}`);
row.innerHTML = fullCell(b, tcpDetail(answer));
} catch (e) {
row.innerHTML = fullCell(b, `<span class="red">${esc(String(e))}</span>`);
}
}
}
document.addEventListener('click', e => {
const link = e.target.closest('a.tcp-link');
if (!link) return;
e.preventDefault();
const addr = link.dataset.peer;
if (state.tcpOpen.has(addr)) {
state.tcpOpen.delete(addr);
link.classList.remove('open');
const tr = link.closest('tr'), next = tr && tr.nextElementSibling;
if (next && next.classList.contains('tcp-row')) next.remove();
} else {
state.tcpOpen.add(addr);
link.classList.add('open');
renderTcpRows();
}
});
// The strip: what needs attention, in one line, or that nothing does. // The strip: what needs attention, in one line, or that nothing does.
function renderStrip() { function renderStrip() {
const items = []; const items = [];
@@ -709,7 +803,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
function renderLocal() { function renderLocal() {
const b = $('local-body'); $('local-count').textContent = state.local.length; const b = $('local-body'); $('local-count').textContent = state.local.length;
if (!state.local.length) { setHTML(b, '<tr><td colspan="7" class="dim">This router originates no routes.</td></tr>'); return; } if (!state.local.length) { setHTML(b, fullRow(b, 'This router originates no routes.', '', 'dim')); return; }
const local = state.status ? state.status.local_asn : '?'; const local = state.status ? state.status.local_asn : '?';
setHTML(b, state.local.map(r => `<tr> setHTML(b, state.local.map(r => `<tr>
<td class="mono">${esc(r.prefix)}</td><td>${esc(r.origin)}</td> <td class="mono">${esc(r.prefix)}</td><td>${esc(r.origin)}</td>
@@ -745,20 +839,73 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
: `${shown}${st ? `, for ${fmtCount(st.received_prefixes)} prefixes` : ''}`; : `${shown}${st ? `, for ${fmtCount(st.received_prefixes)} prefixes` : ''}`;
renderPager(); renderPager();
if (!state.received.length) { if (!state.received.length) {
setHTML(b, `<tr><td colspan="12" class="dim">${looking ? 'No route matches.' : 'No routes received from peers.'}</td></tr>`); return; setHTML(b, fullRow(b, looking ? 'No route matches.' : 'No routes received from peers.', '', 'dim')); return;
} }
setHTML(b, state.received.map(r => { const s = r.summary; return `<tr class="${r.best ? 'best' : ''}"> setHTML(b, state.received.map(r => { const s = r.summary; return `<tr class="${r.best ? 'best' : ''}" data-route="${esc(routeKey(r))}">
<td class="nowrap">${rankGlyph(r)}${r.stale ? glyph('stale', 'Kept from a session of the peer’s that ended (graceful restart), and not sent again since', 'amber') : ''}</td> <td class="nowrap">${rankGlyph(r)}${r.stale ? glyph('stale', 'Kept from a session of the peer’s that ended (graceful restart), and not sent again since', 'amber') : ''}</td>
<td class="mono">${esc(r.prefix)}</td> <td class="mono">${esc(r.prefix)}</td>
<td class="mono">${r.peer_name ? `<a href="#received" class="peer-link" data-peer="${esc(r.peer_name)}">${esc(r.peer_name)}</a> ` : ''}<span class="dim">${esc(r.peer_addr)} AS${asn(r.peer_asn)}${r.path_id ? ` path ${r.path_id}` : ''}</span></td> <td class="mono">${r.peer_name ? `<a href="#received" class="peer-link" data-peer="${esc(r.peer_name)}">${esc(r.peer_name)}</a> ` : ''}<span class="dim">${esc(r.peer_addr)} AS${asn(r.peer_asn)}${r.path_id ? ` path ${r.path_id}` : ''}</span></td>
<td class="mono path">${clamp(asPath(s.as_path))}</td> <td class="mono path">${clamp(asPath(s.as_path))}</td>
<td class="mono">${s.next_hop ? esc(s.next_hop) : DASH}</td><td>${s.origin ? esc(s.origin) : DASH}</td> <td class="mono">${s.next_hop ? esc(s.next_hop) : DASH}</td><td>${s.origin ? esc(s.origin) : DASH}</td>
<td class="num">${s.med ?? DASH}</td><td class="num">${s.local_pref ?? DASH}</td> <td class="num">${s.med ?? DASH}</td><td class="num">${s.local_pref ?? DASH}</td>
<td>${clamp(commList(s.communities))}</td><td>${clamp(commList(s.large_communities, true))}</td><td>${otherList(s.other)}</td> <td>${clamp(commList(s.communities))}</td><td>${clamp(commList(s.large_communities, true))}</td><td class="attrs" title="all attributes">${otherList(s.other)}</td>
<td class="nowrap">${validity('RPKI', r.rpki)} ${validity('ASPA', r.aspa)}</td></tr>`; }).join('')); <td class="nowrap">${validity('RPKI', r.rpki)} ${validity('ASPA', r.aspa)}</td></tr>`; }).join(''));
markTall(b); markTall(b);
renderRouteDetails();
} }
// A route, in full, under its row: every attribute by its name, as the
// RFCs write it, opened by a click on the attrs cell.
const routeKey = r => `${r.prefix}|${r.peer_addr}|${r.path_id}`;
const ATTR_NAMES = {
'atomic aggregate': 'ATOMIC_AGGREGATE', 'aggregator': 'AGGREGATOR', 'extended communities': 'EXTENDED_COMMUNITIES',
'only to customer': 'OTC', 'originator ID': 'ORIGINATOR_ID', 'cluster list': 'CLUSTER_LIST',
};
function routeDetail(r) {
const s = r.summary;
const rows = [
['ORIGIN', s.origin ? esc(s.origin) : DASH],
['AS_PATH', s.as_path.length ? s.as_path.map(asn).join(' ') : '<span class="dim">(empty)</span>'],
['NEXT_HOP', s.next_hop ? esc(s.next_hop) : DASH],
['MED', s.med ?? DASH],
['LOCAL_PREF', s.local_pref ?? DASH],
['COMMUNITIES', s.communities.length ? s.communities.map(esc).join(' ') : DASH],
['LARGE_COMMUNITIES', s.large_communities.length ? s.large_communities.map(esc).join(' ') : DASH],
...(s.other || []).map(a => [esc(ATTR_NAMES[a.name] || a.name), esc(a.value) || 'present']),
];
if (r.path_id) rows.push(['PATH_ID', `${r.path_id}`]);
rows.push(['selection', `${r.best ? 'best' : `rank ${r.rank}`}${r.stale ? ', stale' : ''}`]);
rows.push(['rpki', r.rpki ? esc(r.rpki) : DASH], ['aspa', r.aspa ? esc(r.aspa) : DASH]);
return kvGrid(rows);
}
function renderRouteDetails() {
const b = $('received-body');
for (const key of state.routeOpen) {
const r = state.received.find(r => routeKey(r) === key);
const tr = b.querySelector(`tr[data-route="${CSS.escape(key)}"]`);
if (!r || !tr) continue;
const next = tr.nextElementSibling;
if (next && next.classList.contains('route-detail')) continue;
const row = document.createElement('tr');
row.className = 'detail route-detail';
row.innerHTML = fullCell(b, routeDetail(r));
tr.after(row);
}
}
document.addEventListener('click', e => {
const cell = e.target.closest('#received-body td.attrs');
if (!cell || e.target.closest('a')) return;
const tr = cell.closest('tr'), key = tr.dataset.route;
if (state.routeOpen.has(key)) {
state.routeOpen.delete(key);
const next = tr.nextElementSibling;
if (next && next.classList.contains('route-detail')) next.remove();
} else {
state.routeOpen.add(key);
renderRouteDetails();
}
});
const RPKI_STATE_CLASS = { synchronised: 'green', synchronising: 'amber', connecting: 'red' }; const RPKI_STATE_CLASS = { synchronised: 'green', synchronising: 'amber', connecting: 'red' };
function renderRpki() { function renderRpki() {
const r = state.rpki, b = $('rpki-body'); const r = state.rpki, b = $('rpki-body');
@@ -828,7 +975,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
const total = state.rejected.reduce((n, p) => n + p.total, 0); const total = state.rejected.reduce((n, p) => n + p.total, 0);
const announcements = n => `${fmtCount(n)} announcement${n === 1 ? '' : 's'}`; const announcements = n => `${fmtCount(n)} announcement${n === 1 ? '' : 's'}`;
$('rejected-count').textContent = total ? `${announcements(total)} from ${state.rejected.length} peer${state.rejected.length === 1 ? '' : 's'}${chosen ? `; showing ${chosen}` : ''}` : '0'; $('rejected-count').textContent = total ? `${announcements(total)} from ${state.rejected.length} peer${state.rejected.length === 1 ? '' : 's'}${chosen ? `; showing ${chosen}` : ''}` : '0';
if (!peers.length) { setHTML(b, `<tr><td colspan="11" class="dim">${chosen ? `Nothing ${esc(chosen)} announced has been rejected.` : 'Nothing a peer announced has been rejected.'}</td></tr>`); return; } if (!peers.length) { setHTML(b, fullRow(b, chosen ? `Nothing ${esc(chosen)} announced has been rejected.` : 'Nothing a peer announced has been rejected.', '', 'dim')); return; }
setHTML(b, peers.map(p => { setHTML(b, peers.map(p => {
// Alike: the same prefix, for the same reason, by the same path. // Alike: the same prefix, for the same reason, by the same path.
const alike = new Map(); const alike = new Map();
@@ -841,7 +988,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
const lines = [...alike.values()].sort((a, c) => c.at_unix - a.at_unix).slice(0, REJECTED_SHOWN); const lines = [...alike.values()].sort((a, c) => c.at_unix - a.at_unix).slice(0, REJECTED_SHOWN);
const more = alike.size > REJECTED_SHOWN const more = alike.size > REJECTED_SHOWN
? ` <span class="dim">${REJECTED_SHOWN} of ${alike.size} shown; <a href="/api/routes/rejected?peer=${encodeURIComponent(p.peer)}">the latest ${p.recent.length} as json</a>.</span>` : ''; ? ` <span class="dim">${REJECTED_SHOWN} of ${alike.size} shown; <a href="/api/routes/rejected?peer=${encodeURIComponent(p.peer)}">the latest ${p.recent.length} as json</a>.</span>` : '';
return `<tr class="group"><td colspan="11"><a href="#received" class="peer-link" data-peer="${esc(p.peer)}">${esc(p.peer)}</a> <span class="dim" style="font-weight: normal">${esc(p.peer_addr)}</span>: <span style="font-weight: normal">${announcements(p.total)} rejected</span> ${reasonBar(p.by_reason, p.total)}${more}</td></tr>` + return fullRow(b, `<a href="#received" class="peer-link" data-peer="${esc(p.peer)}">${esc(p.peer)}</a> <span class="dim" style="font-weight: normal">${esc(p.peer_addr)}</span>: <span style="font-weight: normal">${announcements(p.total)} rejected</span> ${reasonBar(p.by_reason, p.total)}${more}`, 'group') +
lines.map(r => { const s = r.summary; return `<tr><td></td> lines.map(r => { const s = r.summary; return `<tr><td></td>
<td class="mono">${esc(r.prefix)}${r.path_id ? ` <span class="dim">path ${r.path_id}</span>` : ''}</td> <td class="mono">${esc(r.prefix)}${r.path_id ? ` <span class="dim">path ${r.path_id}</span>` : ''}</td>
<td class="mono path">${clamp(asPath(r.as_path))}</td><td>${esc(r.reason)}</td> <td class="mono path">${clamp(asPath(r.as_path))}</td><td>${esc(r.reason)}</td>
docs/content/docs/development/testing.md
+82 -5
View File
@@ -99,6 +99,7 @@ test phases.
| `hijacks` | FRR advertises a local prefix, a route with BIRD as third-party next hop, and the exchange LAN prefix; all are rejected | | `hijacks` | FRR advertises a local prefix, a route with BIRD as third-party next hop, and the exchange LAN prefix; all are rejected |
| `nexthops` | A preferred route from FRR with an unresolvable next hop is held, and the route from BIRD is selected and installed; the selection changes when the next hop becomes resolvable, and reverts when it becomes unresolvable | | `nexthops` | A preferred route from FRR with an unresolvable next hop is held, and the route from BIRD is selected and installed; the selection changes when the next hop becomes resolvable, and reverts when it becomes unresolvable |
| `transit` | FRR as customer and BIRD as provider: FRR's routes are advertised to BIRD with the local AS prepended; BIRD's routes are advertised to FRR with the OTC attribute; a control community prevents the advertisement of one FRR route to BIRD; traffic is forwarded between their networks | | `transit` | FRR as customer and BIRD as provider: FRR's routes are advertised to BIRD with the local AS prepended; BIRD's routes are advertised to FRR with the OTC attribute; a control community prevents the advertisement of one FRR route to BIRD; traffic is forwarded between their networks |
| `collector` | FRR as a route collector with ADD-PATH and `all_paths`: BIRD peers twice, FRR receives both of BIRD's routes for a prefix with distinct path identifiers, and one is withdrawn by its identifier when its session is disabled |
| `v6only` | Exchange LAN without IPv4 addresses: one IPv6 session with each router for both address families; IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel; IPv4 forwarding verified with `ping` | | `v6only` | Exchange LAN without IPv4 addresses: one IPv6 session with each router for both address families; IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel; IPv4 forwarding verified with `ping` |
| `linklocal` | Exchange LAN with link-local addresses only, with the same addresses on a second interface: sessions with FRR (by interface, with radvd) and BIRD at `fe80::…%ix0`; both address families in BGP and in the kernel; forwarding verified with `ping`; interface disabled and enabled | | `linklocal` | Exchange LAN with link-local addresses only, with the same addresses on a second interface: sessions with FRR (by interface, with radvd) and BIRD at `fe80::…%ix0`; both address families in BGP and in the kernel; forwarding verified with `ping`; interface disabled and enabled |
@@ -172,11 +173,87 @@ performance: approximately 1 million IPv4 prefixes, mostly /24, and
in UPDATE messages; AS paths of several hops; and communities on most in UPDATE messages; AS paths of several hops; and communities on most
routes. routes.
Table 5 lists results on an Apple M-series computer with a release build. Table 5 lists `full_table` results on an Apple M-series computer with a
release build: 1.23 million prefixes from each of four neighbors, 4.92
million routes.
| Measurement | Result | | Measurement | Result |
|---|---| |---|---|
| 1.23 million routes from one neighbor, accepted into the routing table | 0.47 seconds | | Decoding the UPDATE messages, one thread | 0.43 seconds, 11 million routes per second |
| The same routes received by a second neighbor (`transit`) | 2.1 seconds, approximately 4 routes per UPDATE message | | Decoding, interning attributes, inserting and selecting, one thread | 1.9 seconds, 2.5 million routes per second |
| 1 million IPv4 routes installed in the kernel (Linux virtual machine) | Slightly more than 1 second | | The same, 12 threads | 0.7 seconds, 6.9 million routes per second |
| The same routes removed from the kernel | 0.5 seconds | | Four neighbors send the table over TCP to the in-process server | 0.9 seconds, 5.2 million routes per second |
| Memory per route, attribute sets included (`--features count-memory`) | 77 bytes |
## Profiling
The `scripts/profile/full-table.sh` script measures the daemon itself, as
built for production, while one or more neighbors send it a full routing
table, and records where its CPU time goes. It requires Linux with `perf`,
`inferno` (`cargo install inferno`), `curl`, `jq`, and `sudo`. On another
operating system, run it in the Linux virtual machine:
```bash
scripts/linux-vm.sh scripts/profile/full-table.sh
```
The script builds the daemon with the `profiling` Cargo profile, which is
the release profile with line tables, and with frame pointers, so that
`perf` can attribute samples to functions. It starts the daemon in a
network namespace of its own, connected by a virtual Ethernet pair to a
second namespace in which the load generator
(`crates/bench/examples/feed.rs`) connects as the neighbors and sends the
synthetic table. Routes that the daemon installs go into the first
namespace's routing table and do not affect the host. Table 6 lists the
environment variables that control the run.
| Variable | Effect | Default |
|---|---|---|
| `PEERS` | The number of neighbors that send the table | 1 |
| `KERNEL` | `1` installs the best routes in the kernel; `0` keeps them in the routing table only | 1 |
| `SCALE_DOWN` | Divides the size of the table | 1 |
| `HOLD_SECS` | How long the sessions stay open after the table is sent | 30 |
| `FREQ` | The sampling frequency in hertz | 1999 |
| `COLLECTOR` | `1` adds a route collector session that is sent every path (`all_paths`) | 0 |
| `HEAP` | `1` profiles memory instead: the daemon is built with jemalloc's heap profiler, and `heap.svg` and `heap-top.txt` show where the allocations are | 0 |
| `OUT` | The directory for the results | `~/profile` |
| `JEMALLOC_CONF` | jemalloc options for the daemon (its `MALLOC_CONF`), for example `thp:always` | none |
When run through `scripts/linux-vm.sh`, pass the variables with `env`,
because the virtual machine does not receive the host's environment:
```bash
scripts/linux-vm.sh env PEERS=4 KERNEL=0 scripts/profile/full-table.sh
```
The script prints how long the routes take to arrive in the routing table
and in the kernel, and the daemon's resident memory. It writes the
following files to the results directory:
| File | Contents |
|---|---|
| `flame.svg` | An interactive flame graph of the daemon's threads |
| `top-self.txt` | Functions by the time spent in their own code |
| `top-children.txt` | Functions by the time spent in them and everything they call |
| `stacks.folded` | The collapsed stacks the flame graph is drawn from |
| `perf.data`, `daemon.log`, `feed.log` | The raw samples and the logs |
Table 7 lists results in the Linux virtual machine (8 virtual CPUs on an
Apple M-series computer) with the synthetic table of 1.23 million
prefixes.
| Measurement | Result |
|---|---|
| One neighbor; routes in the routing table | 0.8 seconds, 1.5 million routes per second |
| One neighbor; the best routes installed in the kernel | 2.0 seconds after the first route |
| One neighbor, without the kernel; routes in the routing table | 0.6 seconds, 2.0 million routes per second |
| Four neighbors; 4.92 million routes in the routing table | 1.1 seconds, 4.4 million routes per second |
| Four neighbors; the best routes installed in the kernel | 2.2 seconds after the first route |
| Four neighbors; routes the kernel was asked to install | 1.25 million, for 1.23 million prefixes |
| Resident memory with one neighbor's table installed | 440 MB |
| Resident memory with four neighbors' tables installed | 630 MB |
A session is processed on one thread, so one neighbor's table arrives at
the rate of one CPU core, and several neighbors' tables arrive in
parallel. Installing routes in the kernel takes longer than receiving
them, and continues after the last route has arrived.
docs/content/docs/getting-started/installation.md
+1 -1
View File
@@ -55,7 +55,7 @@ To build and install the daemon:
bgp --version bgp --version
``` ```
The command displays the installed version, for example `bgp 0.2.5`. The command displays the installed version, for example `bgp 0.2.6`.
## Building a Static Executable ## Building a Static Executable
docs/content/docs/getting-started/introduction.md
+2 -2
View File
@@ -55,11 +55,11 @@ Ferrix provides the following capabilities:
## Release Status ## Release Status
> [!IMPORTANT] > [!IMPORTANT]
> Ferrix 0.2.5 is an early release, in production use on one small > Ferrix 0.2.6 is an early release, in production use on one small
> network. Ferrix is tested against FRR, BIRD, and Routinator in an > network. Ferrix is tested against FRR, BIRD, and Routinator in an
> automated interoperability lab and against real Linux kernels. > automated interoperability lab and against real Linux kernels.
Ferrix 0.2.5 does not support the following features: Ferrix 0.2.6 does not support the following features:
- Per-client best path selection for route servers, and ADD-PATH for - Per-client best path selection for route servers, and ADD-PATH for
advertised routes. advertised routes.
docs/content/docs/guides/passing-routes-on.md
+38 -2
View File
@@ -398,12 +398,48 @@ Ferrix advertised (`announced`) and withdrew (`withdrawn`). Ferrix does not
provide a view of the routes advertised to a neighbor (Adj-RIB-Out); to provide a view of the routes advertised to a neighbor (Adj-RIB-Out); to
display them, use the neighbor's own tools. display them, use the neighbor's own tools.
## Feeding a Route Collector
A route collector, such as a looking glass or a measurement project, wants
every route that a router has accepted, not only the best path for each
prefix. ADD-PATH (RFC 7911) lets one session carry several routes for a
prefix, each with a path identifier. To advertise every accepted route to
a neighbor, configure ADD-PATH in the sending direction and enable
`all_paths` in the neighbor's export policy:
```textproto
peers {
name: "collector"
endpoint { address: "203.0.113.10" }
remote_asn: 64496
role: PEER
add_path: ADD_PATH_SEND
export {
learned_routes: true
all_paths: true
}
export_limits {}
}
```
Ferrix advertises every route for a prefix only for the address families
for which the neighbor negotiated ADD-PATH receive. For other families it
advertises the best path, as without `all_paths`. Each route is advertised
with a path identifier that stays with the route for as long as it is
advertised; a route that is withdrawn is withdrawn by its identifier, and
a route whose attributes change is advertised again with the same
identifier. The export policy, the BGP role rules, the well-known
communities and the export limits apply to every route alike, and
`max_prefixes` in `export_limits` counts every route advertised.
A session with `all_paths` keeps a record of every route advertised and
its identifier, which takes about 50 bytes for each route. For a full
Internet table from several neighbors, this is some tens of megabytes.
## Limitations ## Limitations
- Per-client best path selection for route servers is not supported. Every - Per-client best path selection for route servers is not supported. Every
client receives the same best path. client receives the same best path.
- ADD-PATH is not supported for advertisement; only the best path is
advertised.
- Best path selection does not use ORIGINATOR_ID and CLUSTER_LIST as tie - Best path selection does not use ORIGINATOR_ID and CLUSTER_LIST as tie
breakers (RFC 4456 Section 9). breakers (RFC 4456 Section 9).
- UPDATE messages are generated separately for each session, even when - UPDATE messages are generated separately for each session, even when
docs/content/docs/guides/peers.md
+10 -4
View File
@@ -193,8 +193,11 @@ add_path: ADD_PATH_RECEIVE
``` ```
ADD-PATH is enabled for each address family and direction only if the ADD-PATH is enabled for each address family and direction only if the
neighbor also supports it. Ferrix negotiates `ADD_PATH_SEND` and neighbor also supports it. With `ADD_PATH_SEND` or `ADD_PATH_BOTH`,
`ADD_PATH_BOTH`, but it advertises only one path for each prefix. Ferrix advertises routes with path identifiers. By default it still
advertises one path for each prefix; to advertise every accepted route
for a prefix, enable `all_paths` in the neighbor's export policy. See
[Feeding a Route Collector](@/docs/guides/passing-routes-on.md#feeding-a-route-collector).
## Hold Timer ## Hold Timer
@@ -259,8 +262,11 @@ with this value. An established session with fewer than half the expected
routes is listed under **Needing attention**, with the expected number routes is listed under **Needing attention**, with the expected number
beside the actual one. The comparison starts one minute after the session beside the actual one. The comparison starts one minute after the session
is established, so that a neighbor that is still sending its table is not is established, so that a neighbor that is still sending its table is not
reported. The value does not affect route processing; `max_prefixes` is reported. The value does not affect which routes are accepted;
the limit that Ferrix enforces. `max_prefixes` is the limit that Ferrix enforces. Ferrix also sizes the
routing table at startup for the largest `expected_routes` value among
the neighbors, so that the table does not grow while the tables are
received.
## Verifying BGP Sessions ## Verifying BGP Sessions
docs/content/docs/operations/monitoring.md
+20 -2
View File
@@ -37,8 +37,15 @@ Below the status line, the page displays the following information:
- The configuration that was read at startup. - The configuration that was read at startup.
- Each neighbor, with its session state, time established, next hop, - Each neighbor, with its session state, time established, next hop,
session properties, number of accepted routes, rejected announcements, session properties, number of accepted routes, number of routes
and held routes, and its last error. Neighbors that need attention are advertised to it (**sent**), rejected announcements, and held routes,
and its last error. The **tcp** link on an established session expands
a row with the connection as the kernel reports it: round-trip time,
congestion window, the neighbor's receive window and Ferrix's, queues,
retransmissions, throughput, and the share of time the connection was
held up by the neighbor's window or by the local send buffer. This
shows whether a slow advertisement of a table is paced by the neighbor.
Linux only. Neighbors that need attention are
listed first; the others are grouped by role. The IPv4 and IPv6 sessions listed first; the others are grouped by role. The IPv4 and IPv6 sessions
of one neighbor, named `NAME-v4` and `NAME-v6`, are listed together, of one neighbor, named `NAME-v4` and `NAME-v6`, are listed together,
IPv6 first, and joined by a bar. The **problems only** option hides the neighbors that IPv6 first, and joined by a bar. The **problems only** option hides the neighbors that
@@ -152,6 +159,12 @@ advertise the same routes can therefore have different counts. The number
of routes currently held, because they are RPKI or ASPA invalid or their of routes currently held, because they are RPKI or ASPA invalid or their
next hop is unreachable, is displayed in the neighbor table as **held**. next hop is unreachable, is displayed in the neighbor table as **held**.
In the received routes table, the **attrs** column shows the attributes
that have no column of their own as badges. Clicking the cell expands a
row below the route that lists every attribute of the route by its name,
with the path identifier, the selection rank, and the RPKI and ASPA
states.
```json ```json
[{"peer":"transit-a","total":3,"by_reason":{"RPKI invalid":2,"bogon prefix":1}, [{"peer":"transit-a","total":3,"by_reason":{"RPKI invalid":2,"bogon prefix":1},
"recent":[{"prefix":"203.0.113.0/24","reason":"RPKI invalid","as_path":[64496,64511]}]}] "recent":[{"prefix":"203.0.113.0/24","reason":"RPKI invalid","as_path":[64496,64511]}]}]
@@ -238,6 +251,11 @@ Table 3 lists queries for a monitoring dashboard.
|---|---| |---|---|
| Established sessions | `sum(bgp_peer_up)`, of `count(bgp_peer_up)` | | Established sessions | `sum(bgp_peer_up)`, of `count(bgp_peer_up)` |
| Routes by neighbor | `bgp_peer_routes` | | Routes by neighbor | `bgp_peer_routes` |
| Routes advertised to each neighbor | `bgp_peer_advertised_routes` |
| Memory in use and resident | `bgp_memory_bytes{kind="allocated"}`, `bgp_memory_bytes{kind="resident"}` |
| Share of time a session is held up by the neighbor's receive window | `rate(bgp_peer_tcp_receive_window_limited_seconds[5m]) / rate(bgp_peer_tcp_busy_seconds[5m])` |
| Round-trip time to each neighbor | `bgp_peer_tcp_rtt_seconds` |
| Octets queued towards a neighbor | `bgp_peer_tcp_send_queue_bytes` |
| Prefixes with a best path | `bgp_rib_prefixes` | | Prefixes with a best path | `bgp_rib_prefixes` |
| Rejected routes by reason | `sum by (reason) (rate(bgp_routes_rejected_total[5m]))` | | Rejected routes by reason | `sum by (reason) (rate(bgp_routes_rejected_total[5m]))` |
| UPDATE messages received and sent | `rate(bgp_messages_received_total{type="update"}[5m])`, `rate(bgp_messages_sent_total{type="update"}[5m])` | | UPDATE messages received and sent | `rate(bgp_messages_received_total{type="update"}[5m])`, `rate(bgp_messages_sent_total{type="update"}[5m])` |
docs/content/docs/reference/configuration.md
+2 -1
View File
@@ -77,7 +77,7 @@ Statements for a BGP neighbor.
| `role` <span class="type">[Role](#role)</span> | None | Specifies the BGP role of the neighbor (RFC 9234). Not permitted for IBGP neighbors. Required for every EBGP neighbor if any `export` statement enables `learned_routes`. | | `role` <span class="type">[Role](#role)</span> | None | Specifies the BGP role of the neighbor (RFC 9234). Not permitted for IBGP neighbors. Required for every EBGP neighbor if any `export` statement enables `learned_routes`. |
| `require_role` <span class="type">bool</span> | `false` | Rejects the session if the neighbor does not advertise a BGP role. Requires `role`. | | `require_role` <span class="type">bool</span> | `false` | Rejects the session if the neighbor does not advertise a BGP role. Requires `role`. |
| `passive` <span class="type">bool</span> | `false` | Prevents Ferrix from initiating connections to the neighbor. Ferrix accepts connections from the address of `endpoint`; the port of `endpoint` is not used. See [Passive Neighbors](@/docs/guides/peers.md#passive-neighbors). | | `passive` <span class="type">bool</span> | `false` | Prevents Ferrix from initiating connections to the neighbor. Ferrix accepts connections from the address of `endpoint`; the port of `endpoint` is not used. See [Passive Neighbors](@/docs/guides/peers.md#passive-neighbors). |
| `expected_routes` <span class="type">uint32</span> | 0 | Specifies the approximate number of routes that the neighbor advertises in normal operation. The status page marks an established session with fewer than half this number of routes as needing attention. The value does not affect route processing. 0 disables the comparison. See [Expected Route Counts](@/docs/guides/peers.md#expected-route-counts). | | `expected_routes` <span class="type">uint32</span> | 0 | Specifies the approximate number of routes that the neighbor advertises in normal operation. The status page marks an established session with fewer than half this number of routes as needing attention. Ferrix also allocates the routing table with room for the largest `expected_routes` value among the neighbors at startup, which avoids the table growing, and briefly using twice its memory, while the tables are received. The value does not affect which routes are accepted. 0 disables the comparison. See [Expected Route Counts](@/docs/guides/peers.md#expected-route-counts). |
| `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Top-level `export` | Specifies the export policy for the neighbor. Replaces the top-level `export` statement. | | `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Top-level `export` | Specifies the export policy for the neighbor. Replaces the top-level `export` statement. |
| `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Top-level `graceful_restart` | Specifies graceful restart for the neighbor. Each statement that is set replaces the corresponding top-level statement. | | `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Top-level `graceful_restart` | Specifies graceful restart for the neighbor. Each statement that is set replaces the corresponding top-level statement. |
| `route_reflector_client` <span class="type">bool</span> | `false` | Makes the IBGP neighbor a route reflector client: Ferrix advertises to it the routes learned from other IBGP neighbors, and advertises its routes to them. Permitted only for IBGP neighbors. See [Route Reflection](@/docs/guides/passing-routes-on.md#route-reflection). | | `route_reflector_client` <span class="type">bool</span> | `false` | Makes the IBGP neighbor a route reflector client: Ferrix advertises to it the routes learned from other IBGP neighbors, and advertises its routes to them. Permitted only for IBGP neighbors. See [Route Reflection](@/docs/guides/passing-routes-on.md#route-reflection). |
@@ -154,6 +154,7 @@ Statements that determine which routes are advertised to a neighbor. See
| Statement | Default | Description | | Statement | Default | Description |
|---|---|---| |---|---|---|
| `learned_routes` <span class="type">bool</span> | `false` | Enables the advertisement of learned routes, subject to BGP roles and communities. | | `learned_routes` <span class="type">bool</span> | `false` | Enables the advertisement of learned routes, subject to BGP roles and communities. |
| `all_paths` <span class="type">bool</span> | `false` | Advertises every accepted route for a prefix, not only the best path, each with a path identifier (ADD-PATH, RFC 7911). Requires `learned_routes`, and takes effect for an address family only if the neighbor negotiated ADD-PATH receive for it; otherwise the best path is advertised. Export policy and export limits apply to every route. See [Feeding a Route Collector](@/docs/guides/passing-routes-on.md#feeding-a-route-collector). |
| `add_communities` <span class="type">string, repeated</span> | None | Specifies standard communities added to advertised routes. | | `add_communities` <span class="type">string, repeated</span> | None | Specifies standard communities added to advertised routes. |
| `add_large_communities` <span class="type">string, repeated</span> | None | Specifies large communities added to advertised routes. | | `add_large_communities` <span class="type">string, repeated</span> | None | Specifies large communities added to advertised routes. |
| `remove_communities` <span class="type">string, repeated</span> | None | Specifies patterns of communities removed from advertised routes. Applied before communities are added. | | `remove_communities` <span class="type">string, repeated</span> | None | Specifies patterns of communities removed from advertised routes. Applied before communities are added. |
docs/content/docs/reference/http-api.md
+74 -1
View File
@@ -29,7 +29,7 @@ server returns the following error statuses:
|---|---| |---|---|
| `/` | Redirects to `/statusz`. | | `/` | Redirects to `/statusz`. |
| `/statusz` | Status page. | | `/statusz` | Status page. |
| `/api/status` | Router ID, AS number, uptime, table sizes, and whether control communities are enabled (`control_communities`). | | `/api/status` | Router ID, AS number, uptime, table sizes, memory use (`memory`, as jemalloc accounts for it: `allocated`, `active`, `metadata`, `resident`, `mapped`, `retained` octets), and whether control communities are enabled (`control_communities`). |
| `/api/config` | The configuration that was read at startup. | | `/api/config` | The configuration that was read at startup. |
| `/api/peers` | Configured neighbors and their sessions. | | `/api/peers` | Configured neighbors and their sessions. |
| `/api/routes/local` | Originated routes. | | `/api/routes/local` | Originated routes. |
@@ -37,6 +37,7 @@ server returns the following error statuses:
| `/api/routes/rejected` | Rejected routes, by neighbor. | | `/api/routes/rejected` | Rejected routes, by neighbor. |
| `/api/rpki` | RPKI cache, data, and data age. | | `/api/rpki` | RPKI cache, data, and data age. |
| `/api/kernel` | Kernel route installation state and interfaces. | | `/api/kernel` | Kernel route installation state and interfaces. |
| `/api/tcp` | The TCP connection of one session, as the kernel reports it. |
| `/api/events` | Session and route changes, as server-sent events. | | `/api/events` | Session and route changes, as server-sent events. |
| `/metricz` | Prometheus metrics. See [Metrics Reference](@/docs/reference/metrics.md). | | `/metricz` | Prometheus metrics. See [Metrics Reference](@/docs/reference/metrics.md). |
@@ -61,6 +62,7 @@ neighbor:
"add_path_send": [], "add_path_send": [],
"routes": 1, "routes": 1,
"rejected_routes": 1, "rejected_routes": 1,
"advertised_routes": 2,
"held_routes": 0, "held_routes": 0,
"expected_routes": null, "expected_routes": null,
"role": null, "role": null,
@@ -84,6 +86,7 @@ neighbor:
| `direction` | `inbound` if the neighbor initiated the connection, `outbound` otherwise. | | `direction` | `inbound` if the neighbor initiated the connection, `outbound` otherwise. |
| `routes` | Number of the neighbor's routes in the routing table. | | `routes` | Number of the neighbor's routes in the routing table. |
| `rejected_routes` | Number of the neighbor's routes rejected by its import policy. | | `rejected_routes` | Number of the neighbor's routes rejected by its import policy. |
| `advertised_routes` | Number of routes advertised to the neighbor and not withdrawn since. With ADD-PATH and `all_paths`, every route advertised counts. |
| `held_routes` | Number of held routes: routes rejected for a reason that can change without a new UPDATE, such as RPKI validity or next hop resolution. | | `held_routes` | Number of held routes: routes rejected for a reason that can change without a new UPDATE, such as RPKI validity or next hop resolution. |
| `expected_routes` | The `expected_routes` value from the configuration, or `null` if it is not set. | | `expected_routes` | The `expected_routes` value from the configuration, or `null` if it is not set. |
| `role_confirmed` | Whether the neighbor advertised a BGP role that matches the configured role. | | `role_confirmed` | Whether the neighbor advertised a BGP role that matches the configured role. |
@@ -228,6 +231,76 @@ routes. To return the records for one neighbor, add `?peer=NAME`.
| `refused` | Number of prefixes whose active route the kernel rejected. | | `refused` | Number of prefixes whose active route the kernel rejected. |
| `errors` | Number of requests that the kernel rejected. | | `errors` | Number of requests that the kernel rejected. |
## TCP Connection
`/api/tcp?peer=PEER` returns what the kernel reports about the TCP
connection of the session with the neighbor `PEER`, named as for
`/api/routes/received`: its configured name, or its address, with or
without the port as `peer_addr` in `/api/peers` shows it. The report
covers the round-trip time, the congestion
and receive windows, the queues, the retransmissions, and how long the
connection has been held up by the neighbor's receive window or by the
local send buffer. Linux only; on other systems, and for a session the
kernel has no connection for, `available` is `false` and `reason` says
why.
```json
{
"available": true,
"reason": null,
"connection": {
"state": "established",
"congestion": "cubic",
"rtt_ms": 0.62,
"rtt_variance_ms": 0.11,
"min_rtt_ms": 0.4,
"rto_ms": 201,
"mss": 1428,
"pmtu": 1500,
"cwnd_segments": 10,
"ssthresh_segments": null,
"send_window": 1048576,
"receive_window": 3145728,
"receive_space": 65536,
"send_window_scale": 7,
"receive_window_scale": 10,
"send_queue": 0,
"not_sent": 0,
"recv_queue": 0,
"unacked_segments": 0,
"retrans_segments": 0,
"lost_segments": 0,
"total_retrans": 0,
"bytes_retrans": 0,
"pacing_rate": 2309920000,
"delivery_rate": 184860000,
"bytes_sent": 312993,
"bytes_acked": 312994,
"bytes_received": 190,
"segments_out": 230,
"segments_in": 118,
"busy_ms": 41,
"limited_by_receive_window_ms": 0,
"limited_by_send_buffer_ms": 0,
"limited_by_receive_window_share": 0,
"limited_by_send_buffer_share": 0,
"last_data_sent_ms": 1234,
"last_data_received_ms": 1300,
"last_ack_received_ms": 1234
}
}
```
Times are in milliseconds, sizes in octets, rates in octets per second.
`send_window` is the neighbor's receive window, which bounds what Ferrix
can send; `receive_window` is what Ferrix lets the neighbor send. The two
`_share` values are the fractions of the busy time, 0 to 1, in which the
connection was held up by the neighbor's window and by the local send
buffer. A session that advertises a table to a neighbor that reads it
slowly shows a high `limited_by_receive_window_share`. The status page
shows the same information under the **tcp** link of each established
session.
## Events ## Events
`/api/events` is a stream of `/api/events` is a stream of
docs/content/docs/reference/metrics.md
+15
View File
@@ -29,6 +29,21 @@ alerting rules based on these metrics, see
| Metric | Type | Description | | Metric | Type | Description |
|---|---|---| |---|---|---|
| `bgp_peer_routes{peer}` | Gauge | Number of the neighbor's routes in the routing table. | | `bgp_peer_routes{peer}` | Gauge | Number of the neighbor's routes in the routing table. |
| `bgp_peer_advertised_routes{peer}` | Gauge | Number of routes advertised to the neighbor and not withdrawn since. |
| `bgp_memory_bytes{kind}` | Gauge | The daemon's memory as jemalloc accounts for it. `kind` is `allocated` (octets in live allocations), `active` (the pages they occupy), `metadata` (the allocator's own), `resident` (active, metadata and freed pages not yet returned to the kernel), `mapped`, or `retained` (address space kept back, which costs nothing physical). |
| `bgp_peer_tcp_rtt_seconds{peer}` | Gauge | Smoothed round-trip time of the session's TCP connection. Also `bgp_peer_tcp_min_rtt_seconds` and `bgp_peer_tcp_rtt_variance_seconds`. Linux only; sampled from the kernel at each scrape, as are all `bgp_peer_tcp_*` metrics. |
| `bgp_peer_tcp_cwnd_segments{peer}` | Gauge | Congestion window, in segments. `bgp_peer_tcp_mss_bytes` is the segment size. |
| `bgp_peer_tcp_send_window_bytes{peer}` | Gauge | The neighbor's receive window: what it lets Ferrix send. Zero while the neighbor is not reading. |
| `bgp_peer_tcp_receive_window_bytes{peer}` | Gauge | Ferrix's receive window: what it lets the neighbor send. |
| `bgp_peer_tcp_send_queue_bytes{peer}` | Gauge | Octets written and not yet acknowledged by the neighbor; `bgp_peer_tcp_not_sent_bytes` is the part not yet sent. |
| `bgp_peer_tcp_receive_queue_bytes{peer}` | Gauge | Octets received and not yet read. |
| `bgp_peer_tcp_unacked_segments{peer}` | Gauge | Segments in flight. |
| `bgp_peer_tcp_retransmitted_segments{peer}` | Gauge | Segments retransmitted over the connection's life. `bgp_peer_tcp_lost_segments` are those thought lost now. |
| `bgp_peer_tcp_busy_seconds{peer}` | Gauge | Time the connection had data to send, over its life. |
| `bgp_peer_tcp_receive_window_limited_seconds{peer}` | Gauge | Of the busy time, time the connection was held up by the neighbor's receive window. |
| `bgp_peer_tcp_send_buffer_limited_seconds{peer}` | Gauge | Of the busy time, time the connection was held up by the local send buffer being full. |
| `bgp_peer_tcp_sent_bytes{peer}` | Gauge | Octets sent over the connection's life. `bgp_peer_tcp_received_bytes` likewise. |
| `bgp_peer_tcp_delivery_rate_bytes_per_second{peer}` | Gauge | Delivery rate the kernel measured most recently; `bgp_peer_tcp_pacing_rate_bytes_per_second` is the rate it paces the connection at. |
| `bgp_peer_max_prefixes{peer}` | Gauge | Prefix limit of the neighbor (`import.max_prefixes`), if configured. | | `bgp_peer_max_prefixes{peer}` | Gauge | Prefix limit of the neighbor (`import.max_prefixes`), if configured. |
| `bgp_prefix_limit_exceeded_total{peer}` | Counter | Number of sessions torn down because the neighbor exceeded its prefix limit. | | `bgp_prefix_limit_exceeded_total{peer}` | Counter | Number of sessions torn down because the neighbor exceeded its prefix limit. |
| `bgp_export_limit_exceeded_total{peer}` | Counter | Number of sessions torn down because the routes to be advertised to the neighbor exceeded its export limit (`export_limits.max_prefixes`). | | `bgp_export_limit_exceeded_total{peer}` | Counter | Number of sessions torn down because the routes to be advertised to the neighbor exceeded its export limit (`export_limits.max_prefixes`). |
docs/content/docs/reference/rfcs.md
+1 -1
View File
@@ -306,7 +306,7 @@ Version 2 is implemented from version 23 of the draft.
|---------|--------| |---------|--------|
| Capability negotiation per family and direction | ✅ | | Capability negotiation per family and direction | ✅ |
| Receiving several routes for a prefix | ✅ | | Receiving several routes for a prefix | ✅ |
| Sending several routes for a prefix | ❌ (path identifiers are sent; one route each) | | Sending several routes for a prefix | ✅ with `all_paths` in the export policy; the best path only otherwise |
### RFC 4271 §9.1.2 / RFC 4486 / operational practice — import policy ### RFC 4271 §9.1.2 / RFC 4486 / operational practice — import policy
scripts/interop/lab.sh
+60 -3
View File
@@ -29,7 +29,7 @@ cd "$(dirname "$0")/../.."
bgp="${CARGO_TARGET_DIR:-target}/release/bgp" bgp="${CARGO_TARGET_DIR:-target}/release/bgp"
work=/tmp/ferrix-lab work=/tmp/ferrix-lab
pathspace=ferrixlab pathspace=ferrixlab
all_phases="basics roles rpki refresh flaps restart graceful scale limits hijacks nexthops transit leaks v6only linklocal" all_phases="basics roles rpki refresh flaps restart graceful scale limits hijacks nexthops transit collector leaks v6only linklocal"
phases="${PHASES:-$all_phases}" phases="${PHASES:-$all_phases}"
scale_routes="${SCALE_ROUTES:-200000}" scale_routes="${SCALE_ROUTES:-200000}"
@@ -319,9 +319,9 @@ frr_down() {
# --- BIRD ----------------------------------------------------------------------- # --- BIRD -----------------------------------------------------------------------
# bird_config [extra-static-routes-file] # bird_config [extra-static-routes-file] [more-configuration]
bird_config() { bird_config() {
local extra=${1:-} local extra=${1:-} more=${2:-}
cat > "$work/bird.conf" <<END cat > "$work/bird.conf" <<END
log "$work/bird.log" all; log "$work/bird.log" all;
router id 10.9.0.3; router id 10.9.0.3;
@@ -355,6 +355,7 @@ protocol bgp dut6 from dut {
source address 2001:db8:9::3; source address 2001:db8:9::3;
ipv6 { import all; export where source = RTS_STATIC; }; ipv6 { import all; export where source = RTS_STATIC; };
} }
$more
END END
} }
@@ -488,6 +489,13 @@ phase_basics() {
in_ns dut ping -q -c 2 -W 2 -I 2001:db8:1::1 2001:db8:2::1 >/dev/null || fail "no IPv6 ping to FRR's lan" in_ns dut ping -q -c 2 -W 2 -I 2001:db8:1::1 2001:db8:2::1 >/dev/null || fail "no IPv6 ping to FRR's lan"
in_ns dut ping -q -c 2 -W 2 -I 2001:db8:1::1 2001:db8:3::1 >/dev/null || fail "no IPv6 ping to BIRD's lan" in_ns dut ping -q -c 2 -W 2 -I 2001:db8:1::1 2001:db8:3::1 >/dev/null || fail "no IPv6 ping to BIRD's lan"
ok "packets get through to both, over IPv4 and IPv6" ok "packets get through to both, over IPv4 and IPv6"
# The kernel's view of a session's connection, as the page asks for it.
frr4_addr=$(api /api/peers | jq -r '.[] | select(.name == "frr4") | .peer_addr')
api "/api/tcp?peer=$frr4_addr" | jq -e '.available and .connection.state == "established" and .connection.mss > 0' >/dev/null \
|| fail "no TCP details for frr4: $(api "/api/tcp?peer=$frr4_addr")"
api /metricz | grep -q "^bgp_peer_tcp_rtt_seconds{peer=\"frr4\"}" || fail "no TCP metrics for frr4"
ok "the kernel tells of a session's TCP connection"
} }
# RFC 9234: each end says what it is to the other, and routes are marked # RFC 9234: each end says what it is to the other, and routes are marked
@@ -959,6 +967,55 @@ control_communities: true'
ok "back to normal" ok "back to normal"
} }
# FRR is a route collector of ours, told every route with its path
# identifier (ADD-PATH). BIRD talks to us from a second address too, so
# that there are two routes for each of its prefixes; FRR is told both,
# and holds them apart. One of BIRD's sessions goes, and so does one of
# the two.
phase_collector() {
dut_down TERM
in_ns bird ip addr add 10.9.0.4/24 dev ix0
dut_config CUSTOMER 2000000 "" 'add_path: ADD_PATH_SEND
export { learned_routes: true all_paths: true }' "" 'peers {
name: "bird4b" endpoint { address: "10.9.0.4" } remote_asn: 4200000003
address_families: IPV4_UNICAST
role: PROVIDER
import { accept_bogons: true max_prefixes: 2000000 }
}'
bird_config "" 'protocol bgp dut4b from dut {
neighbor 10.9.0.1 as 64601;
source address 10.9.0.4;
ipv4 { import all; export where source = RTS_STATIC; };
}'
bird_ctl configure >/dev/null
frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role customer"
dut_up
wait_for "five sessions" established 5
wait_for "BIRD's route from both its sessions" bash -c "sudo ip netns exec lab-dut curl -sf 'http://127.0.0.1:8080/api/routes/received?prefix=10.3.0.0/24' | jq -e '.routes | length == 2'"
peer_is frr4 '.add_path_send | index("IPv4 unicast") != null' >/dev/null \
|| fail "ADD-PATH was not agreed with FRR: $(peer_is frr4 .add_path_send)"
wait_for "both routes at FRR, told apart" bash -c "sudo ip netns exec lab-frr vtysh -N $pathspace -c 'show bgp ipv4 unicast 10.3.0.0/24 json' 2>/dev/null | jq -e '.paths | length == 2 and all(.aspath.string == \"64601 4200000003\")'"
# Its two prefixes twice, and our own route: what the page and the
# metrics say it holds from us.
wait_for "the count of routes the collector holds" peer_is frr4 '.advertised_routes == 5'
ok "the collector is told both of BIRD's routes for a prefix, with path identifiers"
bird_ctl disable dut4b >/dev/null
wait_for "one route left at FRR" bash -c "sudo ip netns exec lab-frr vtysh -N $pathspace -c 'show bgp ipv4 unicast 10.3.0.0/24 json' 2>/dev/null | jq -e '.paths | length == 1'"
wait_for "the count of routes the collector holds" peer_is frr4 '.advertised_routes == 3'
ok "a route that goes is withdrawn from the collector by its identifier, the other stays"
frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role peer"
bird_config
bird_ctl configure >/dev/null
in_ns bird ip addr del 10.9.0.4/24 dev ix0
dut_down TERM
dut_config
dut_up
converged
ok "back to normal"
}
# A customer leaks, and the export limits hold. FRR, now our customer, # A customer leaks, and the export limits hold. FRR, now our customer,
# announces a route outside its address space and then more routes than # announces a route outside its address space and then more routes than
# BIRD, our provider, is to be told of. A customer's routes go to # BIRD, our provider, is to be told of. A customer's routes go to
scripts/profile/full-table.sh
Executable
+246
View File
@@ -0,0 +1,246 @@
#!/bin/bash
# Profiles the daemon as it takes in a full Internet routing table, on a
# Linux machine: a flame graph and a flat profile of where the CPU time
# goes, and how long the table takes to arrive and to reach the kernel.
#
# scripts/profile/full-table.sh one peer sends the table
# PEERS=4 scripts/profile/full-table.sh four peers send it at once
# KERNEL=0 scripts/profile/full-table.sh the routes stay in the table
# SCALE_DOWN=10 scripts/profile/full-table.sh a tenth of the table
# COLLECTOR=1 scripts/profile/full-table.sh a route collector is sent every path
# HEAP=1 scripts/profile/full-table.sh where the memory goes, not the time
#
# From a machine that is not Linux: scripts/linux-vm.sh scripts/profile/full-table.sh
#
# The daemon runs in a network namespace of its own, so that nothing it
# installs touches the machine's own routing, joined by a veth pair to a
# second namespace where the load generator (crates/bench/examples/feed.rs)
# plays the peers: neighbours on the link, as real peers would be. `perf`
# samples the daemon's threads meanwhile. Needs perf, inferno (cargo
# install inferno), curl and jq, and sudo for the namespaces and the
# kernel's routing table.
#
# Results go to $OUT (default ~/profile): flame.svg, top-self.txt (functions
# by their own time), top-children.txt (by the time of everything they
# call), perf.data, and the daemon's and the load generator's logs.
set -eu
peers=${PEERS:-1}
scale=${SCALE_DOWN:-1}
kernel=${KERNEL:-1}
collector=${COLLECTOR:-0}
heap=${HEAP:-0}
hold=${HOLD_SECS:-30}
level=${LOG_LEVEL:-warn}
freq=${FREQ:-1999}
out=${OUT:-$HOME/profile}
ns=ferrix-profile
ns_peer=ferrix-profile-peer
repository="$(cd "$(dirname "$0")/../.." && pwd)"
target="${CARGO_TARGET_DIR:-$repository/target}"
daemon_bin="$target/profiling/bgp"
feed_bin="$target/profiling/examples/feed"
fail() { echo "error: $*" >&2; exit 1; }
for tool in perf inferno-collapse-perf inferno-flamegraph curl jq; do
command -v "$tool" >/dev/null || fail "$tool not found"
done
# Release code with line tables, and frame pointers for perf to walk the
# stacks by (cheaper and more complete than DWARF unwinding here; the
# allocator is built with them too).
cd "$repository"
export RUSTFLAGS="-C force-frame-pointers=yes" CFLAGS="-fno-omit-frame-pointer"
# With HEAP=1 the daemon is built with jemalloc's heap profiler and told
# to sample allocations and to write a profile at exit; jeprof turns the
# profile into collapsed stacks for a flame graph of where the memory is.
features=()
[ "$heap" = 1 ] && features=(--features heap-profiling)
cargo build --profile profiling --locked -p bgp --bin bgp "${features[@]}"
cargo build --profile profiling --locked -p bgp-bench --example feed
mkdir -p "$out"
run() { sudo ip netns exec "$ns" "$@"; }
peer() { sudo ip netns exec "$ns_peer" "$@"; }
cleanup() {
sudo pkill -x bgp 2>/dev/null || true
sudo pkill -x feed 2>/dev/null || true
sudo ip netns del "$ns" 2>/dev/null || true
sudo ip netns del "$ns_peer" 2>/dev/null || true
}
trap cleanup EXIT
cleanup
sudo ip netns add "$ns"
sudo ip netns add "$ns_peer"
sudo ip link add bench0 netns "$ns" type veth peer name peer0 netns "$ns_peer"
run ip link set lo up
run ip addr add 192.0.2.254/24 dev bench0
run ip -6 addr add 2001:db8::ffff/64 dev bench0 nodad
run ip link set bench0 up
peer ip link set lo up
for n in $(seq "$((peers + collector))"); do
peer ip addr add "192.0.2.$n/24" dev peer0
peer ip -6 addr add "2001:db8::$n/64" dev peer0 nodad
done
peer ip link set peer0 up
peer ping -c 1 -W 2 192.0.2.254 > /dev/null || fail "the namespaces do not reach each other"
# An AS number that is in none of the synthetic paths: a route with the
# daemon's own AS in its path is refused as a loop.
prefixes=$(( 1000000 / scale + 230000 / scale ))
routes=$(( prefixes * peers ))
cat > "$out/daemon.textpb" <<CONFIG
router_id: "192.0.2.254"
local_asn: 4200000000
listen_endpoints { address: "192.0.2.254" port: 1790 }
listen_endpoints { address: "2001:db8::ffff" port: 1790 }
http_endpoint { address: "127.0.0.1" port: 8080 }
CONFIG
# The peers are configured as a router's would be, with how many routes
# to expect of each. They have private AS numbers and the synthetic table
# has a few bogon prefixes, so bogons are taken. With a collector, every
# peer has a role, since the collector is told what they send.
for n in $(seq "$peers"); do
cat >> "$out/daemon.textpb" <<CONFIG
peers {
name: "peer$n"
endpoint { address: "192.0.2.$n" port: 179 }
remote_asn: $(( 64512 + n ))
expected_routes: $prefixes
role: PEER
import { accept_bogons: true }
}
CONFIG
done
if [ "$collector" = 1 ]; then
cat >> "$out/daemon.textpb" <<CONFIG
peers {
name: "collector"
endpoint { address: "2001:db8::$((peers + 1))" port: 179 }
remote_asn: 212232
role: CUSTOMER
add_path: ADD_PATH_SEND
import { max_prefixes: 1 }
export { learned_routes: true all_paths: true }
}
CONFIG
fi
if [ "$kernel" = 1 ]; then
echo 'kernel { stale_seconds: 0 }' >> "$out/daemon.textpb"
fi
# `JEMALLOC_CONF`, if set, is jemalloc's MALLOC_CONF for the daemon:
# "thp:always", say. The symbols are prefixed, hence the variable's name.
malloc_conf=${JEMALLOC_CONF:-}
if [ "$heap" = 1 ]; then
# One sample every 512 KiB allocated, and a profile every 1 GiB
# allocated: the last of them is the nearest to the table being in.
rm -f "$out"/heap.*.heap
malloc_conf="prof:true,prof_active:true,lg_prof_sample:19,lg_prof_interval:30,prof_prefix:$out/heap${malloc_conf:+,$malloc_conf}"
fi
run env _RJEM_MALLOC_CONF="$malloc_conf" "$daemon_bin" "$out/daemon.textpb" --level "$level" > "$out/daemon.log" 2>&1 &
for _ in $(seq 100); do
run curl -sf http://127.0.0.1:8080/api/status > /dev/null 2>&1 && break
sleep 0.1
done
daemon_pid=$(pgrep -nx bgp) || fail "the daemon did not start; see $out/daemon.log"
status() { run curl -sf "http://127.0.0.1:8080/api/$1" | jq -r "$2"; }
now() { date +%s.%N; }
since() { echo "$(now) - $1" | bc; }
sudo perf record -F "$freq" -g -o "$out/perf.data" -p "$daemon_pid" > "$out/perf.log" 2>&1 &
perf_pid=$!
sleep 1
peer env FEED_TO=192.0.2.254:1790 FEED_PEERS="$peers" FEED_SCALE_DOWN="$scale" \
FEED_FROM=192.0.2 FEED_HOLD_SECS="$hold" FEED_COLLECTOR="$collector" \
FEED_COLLECTOR_TO="[2001:db8::ffff]:1790" FEED_COLLECTOR_FROM="2001:db8::$((peers + 1))" "$feed_bin" > "$out/feed.log" 2>&1 &
feed_pid=$!
feeding() { kill -0 "$feed_pid" 2>/dev/null || fail "the load generator stopped; see $out/feed.log"; }
# The clock starts at the first route in, not when the load generator
# starts: making the table takes it a while.
while [ "$(status status .received_routes)" = 0 ]; do feeding; sleep 0.02; done
start=$(now)
# Routes the policy refuses (the synthetic table has a few bogons in it)
# count as dealt with.
dealt_with() {
status status .received_routes
status peers '[.[] | .rejected_routes // 0] | add // 0'
}
while [ "$(dealt_with | paste -sd+ | bc)" -lt "$routes" ]; do feeding; sleep 0.05; done
arrived=$(since "$start")
received=$(status status .received_routes)
refused=$(status peers '[.[] | .rejected_routes // 0] | add // 0')
echo "$routes routes dealt with after $arrived s ($(echo "$routes / $arrived / 1000000" | bc -l | cut -c1-4) M routes/s): $received taken, $refused refused"
[ "$refused" = 0 ] || run curl -sf http://127.0.0.1:8080/api/rejections | jq -c '.' | cut -c1-400
if [ "$kernel" = 1 ]; then
prefixes=$(status status .received_prefixes)
while [ "$(status kernel '.synced and .installed >= '"$prefixes")" != true ]; do feeding; sleep 0.05; done
installed=$(since "$start")
echo "$prefixes routes in the kernel after $installed s ($(echo "$prefixes / $installed / 1000" | bc -l | cut -c1-3) k routes/s)"
fi
if [ "$collector" = 1 ]; then
# Nearly every taken route goes to the collector: not one with its AS
# on the path or a well-known community that says not to. The count
# has settled when it has not moved for a few seconds.
advertised() { status peers '[.[] | select(.name == "collector") | .advertised_routes] | add // 0'; }
last=-1
while :; do
feeding; sleep 3
now_held=$(advertised)
echo " collector holds $now_held of $received after $(since "$start") s"
[ "$now_held" -gt 0 ] && [ "$now_held" = "$last" ] && break
last=$now_held
done
echo "collector: $(advertised) routes advertised after $(since "$start") s"
collector_addr=$(status peers '.[] | select(.name == "collector") | .peer_addr')
run curl -sf "http://127.0.0.1:8080/api/tcp?peer=$collector_addr" \
| jq -r '.connection | "collector tcp: busy \(.busy_ms / 1000 | . * 10 | round / 10) s, rwnd-limited \(.limited_by_receive_window_share * 100 | round)%, rtt \(.rtt_ms) ms, cwnd \(.cwnd_segments), sent \(.bytes_sent / 1048576 | round) MiB in \(.segments_out) segments"'
fi
memory_report() {
rss=$(awk '/VmRSS/ { printf "%.0f", $2 / 1024 }' "/proc/$daemon_pid/status")
echo "daemon: $rss MiB resident $1"
status status '.memory | to_entries | map("\(.key) \(.value / 1048576 | floor) MiB") | join(", ")' | sed 's/^/jemalloc: /'
}
memory_report "with the table in"
# And once the allocator has had its decay time to give back what the
# load allocated and freed: the figure a router shows for the long run.
sleep 30
memory_report "30 s later"
run curl -sf http://127.0.0.1:8080/metricz | grep '^bgp_kernel_route_changes_total' | sed 's/^/kernel: /'
sleep 1
sudo kill -INT "$perf_pid"
wait "$perf_pid" || true
cat "$out/feed.log"
sudo chown "$(id -u)" "$out/perf.data"
if [ "$heap" = 1 ]; then
sudo chown "$(id -u)" "$out"/heap.*.heap 2>/dev/null || true
# jeprof comes with jemalloc's sources as a template; the prefix the
# allocator's symbols carry is filled in.
template=$(find "$HOME/.cargo/registry/src" -path "*tikv-jemalloc-sys*" -name jeprof.in | head -1)
[ -n "$template" ] || fail "jeprof.in not found in the cargo registry"
sed -e 's/@JEMALLOC_PREFIX@/_rjem_/g' -e 's/@jemalloc_version@/5.3/' "$template" > "$out/jeprof"
# The latest profile is the one nearest to the table being in.
profile=$(ls -t "$out"/heap.*.heap | head -1)
perl "$out/jeprof" --collapsed "$daemon_bin" "$profile" > "$out/heap.folded" 2> "$out/jeprof.log" \
|| fail "jeprof failed; see $out/jeprof.log"
inferno-flamegraph --title "ferrix heap: $peers peer(s), $prefixes prefixes each$([ "$collector" = 1 ] && echo ", a collector")" --countname bytes --colors mem --width 1600 "$out/heap.folded" > "$out/heap.svg"
perl "$out/jeprof" --text "$daemon_bin" "$profile" 2>/dev/null | head -40 > "$out/heap-top.txt"
echo "heap profile $profile: heap.svg heap-top.txt"
fi
# Without the period field, so that each sample counts one whatever event
# perf sampled by (cpu-clock on a machine without performance counters).
perf script -i "$out/perf.data" -F comm,pid,tid,time,event,ip,sym,dso 2> /dev/null \
| inferno-collapse-perf > "$out/stacks.folded"
title="ferrix: $peers peer(s) send $prefixes prefixes each"
[ "$kernel" = 1 ] && title="$title, into the kernel"
inferno-flamegraph --title "$title" --countname samples --width 1600 "$out/stacks.folded" > "$out/flame.svg"
perf report -i "$out/perf.data" --no-children --stdio --sort symbol --percent-limit 0.5 -g none 2> /dev/null \
| grep -v '^#' | grep -v '^$' | sed 's/ *$//' | head -80 > "$out/top-self.txt"
perf report -i "$out/perf.data" --children --stdio --sort symbol --percent-limit 1 -g none 2> /dev/null \
| grep -v '^#' | grep -v '^$' | sed 's/ *$//' | head -80 > "$out/top-children.txt"
echo "samples: $(awk '{ s += $NF } END { print s }' "$out/stacks.folded")"
echo "results in $out: flame.svg top-self.txt top-children.txt"