rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: rayhaan/ferrix:speaker-cleanup
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
...
pull from: rayhaan/ferrix:main
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
91 Commits
speaker-cleanup ... main
Author SHA1 Message Date
Rayhaan JaufeerallyandClaude Fable 5.1 db8edaae8e policy, config, web: the TCP gauges named once, one revalidator, one peer-naming rule
ci / interop (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A code-quality pass over policy, configuration and the status API.
The twenty TCP gauges were spelled out in the struct, the
constructor, the removal, the sampler in the web crate and the status
view; one macro names each once with its help text, `PeerTcp::set`
fills them from a connection, and the view is a `From`. Every metric
and the list that forces them at start come from one declaration,
with a test that every declared family is registered. The two
revalidators, for next hops and for RPKI, share the task body that
scans the table off the sessions' threads and fans the findings out
to every session; the RPKI scan judges a route with the same `Checks`
the import path uses. Which peer a query names is one rule for the
received, rejected and TCP endpoints, so `/api/tcp` takes a name or an
address like the others. The config parser says proto3's "zero is
unset" in three helpers instead of a dozen shapes, and `parse_peer`
is three named parts shorter. The policy's test-only next-hop flag is
named for what it does, `accept_own_host_next_hop`; the "listed, or
else ours" prefix rule and the mark-then-prefer step each exist once.
The status page derives its spanning rows from the table's header,
shares one key-value grid and one detail-row rule, and defines its
helpers before their use. The web crate no longer re-exports views
nothing uses, and the TCP endpoint sits above the test module.

Testing: the TCP view's arithmetic (a share of no busy time is zero,
not NaN), the peer-naming rule, the RPKI scan finding only what a
change can have made invalid, and `/api/tcp` by address, by name and
for an unknown peer. The server and web tests, the HTTP, roles, RPKI
and import-policy integration tests, the docs build and every lab
phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 4cddd6fb04 export: one rule each for admission, identifiers and what the peer had 
A code-quality pass over the exporter. The resync path (gather then
stream) had grown beside the steady-state path (judge then flush)
with its own copies of the admission predicate, the path identifier
rule, the reconciliation of a route the peer already had, the parser
contexts, the withdrawal loop and the advertised gauge. Each is one
function now: `prefix_may_go`, `lowest_free_id`, `Sent::renewed`,
`parser_contexts`, `send_withdrawals`, `within_limit` and
`note_held`. The 3,072-octet ceiling on a route's attributes, which
is what keeps an UPDATE within its size, is a named constant where it
is relied on. The UPDATEs are framed with the packet crate's helpers
instead of a bare type code and a length written by hand. The
positional tuples for a prefix's paths are structs with names. A
mismatched kind of bookkeeping, which nothing should produce, is
asserted in tests and logged in release instead of dropped silently.
Types used only inside the crate are `pub(crate)`. Five doc comments
that had drifted onto neighbours or named former designs are put
right; the exporter's reference encoder no longer breaks the CI doc
build with a link to an item that exists only under test.

Testing: ADD-PATH export has unit tests at last: two routes for a
prefix go with their own identifiers and keep them, and the
withdrawal of one carries its identifier; what is not heard of again
after a reset is withdrawn after the table. The byte-for-byte test of
the encoded UPDATEs covers IPv4 routes with an IPv6 next hop (RFC
8950) too. The test fixtures are shared helpers. The server tests and
the ADD-PATH, transit, limits, roles and deadlock integration tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 bef875e50b session: the way out to the peer as a type of its own 
A code-quality pass over the per-peer session. The outgoing side of a
session, eight fields served from six places in a two-thousand-line
file, is `Outgoing`: the write buffer, the batches waiting for the
writer task, the task and its word on why it ended. One real race
went with it: when the writer task failed with the queue full, the
session took whichever of two ready branches the select chose, and
half the time ended with an invented send hold timer expiry instead
of the connection's error, which decides whether the peer's routes
are kept over a restart; the writer's own word is taken now. The run
loop's handling of what the rest of the server has to send is a
method that goes by the state machine's state, so that pieces after a
NOTIFICATION in the same batch do not follow it out; `execute_all`
reads one line an action with the sending, delivering and
after-failure steps named. A peer's routes have one way onto the
table at a session's end, whether removed or kept for a restart. An
UPDATE whose attributes cannot be kept ends the session under an
error of its own, `Unstorable`, where a message of ours that cannot
be encoded is dropped. What is kept of a restarting peer is a value,
not seven arguments. The dialer no longer turns a session's end into
a Result and back. The server's screening of a connection is called
that, apart from the collision check's `admit`.

Testing: the back-pressure machinery has unit tests at last: batches
wait while the writer has no room, room goes to the oldest batch, a
piece is counted as the messages in it, and a lost connection is
reported as one while batches wait. The routes module gains tests of
holding at its bound, forgetting and revalidation. The server tests
and the whole integration suite pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 25597ab3ef packet, fib, bench: framing helpers, one netlink reader, a mesh with named phases 
A code-quality pass over the periphery. The packet crate frames a
message with `begin_message` and `finish_message`, which the
exporter's hand-written UPDATE header can use instead of a bare type
code, and keeps the NLRI reader and writer together as a pair; the
rule that the body's prefixes are IPv4 unicast lives once; a session
with two-octet AS numbers no longer reserves the attributes' wire
copy it never keeps. The TCP diagnostics read netlink with the
netlink module's own attribute walk and helpers instead of copies of
them. The kernel follower learns what the kernel has in one named
step used at start and on relearning, keeps its batch buffers between
wakes, and builds its task in a constructor. The mesh benchmark,
which had grown six phases in one main, is a `Mesh` with named phases
and a `Receiver`, with one report of the sessions for both deadlines;
the collector capabilities, the KEEPALIVE bytes, the draining of a
session and the printing of a table row are helpers shared by the
benches, the examples and the integration tests. The MRT reader uses
the packet crate's attribute splitter.

Testing: a message framed by hand is the one `to_wire` writes; a
short tcp_info from an old kernel leaves the later fields zero; the
synthetic table is the same every run and every announcement decodes
to itself. The packing example gives the same figures on the RIS dump
before and after. The crates' tests, the integration tests, the
kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 bcfd6e013f rib: the lock-free number in a slot of its own, and the reader in named steps 
A code-quality pass over the table. The latest sequence number of
each journal, which readers compare their cursors with without the
shard's lock, was an atomic inside the journal under the lock with
the table holding the other end of an Arc to it, stored on every
route; it is now in a slot beside each shard's lock, `ShardSlot`,
stored once a shard a change while the write guard is still held and
before the readers are woken. "One of each kind of journal" is one
type, `ByKind<T>`, where three hand-written dispatchers and an array
index were. `Reader::poll` is the loop skeleton over `resync_step`
and `follow_pass`; the journal's `tidy` is three named stages;
`live`'s nested Option is an enum that says what an entry says; the
place of a cursor in a journal is found by one function for both the
time chosen for a pass and the pass itself. The attribute set's four
byte-readers share one walk over the raw attributes, with the next
hop decoded once for both of its readers. `best` returns a `Route`
like the other lookups. Docs that described an earlier design (the
reader's `seen`, "every method takes &self", a `Routes::is_empty`
that could not be true) are put right.

Testing: new tests pin the lock-free skip, which nothing checked
directly before: the number equals the journal's after every kind of
change, with and without readers of each kind; a shard passed over is
read again once it has news, from a reader of the whole table and
from one that starts now. The OTC flag and the received-attributes
interning gain tests. The crate's tests, both proptests included, the
server tests, the integration tests, the kernel tests and every lab
phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 22:44:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c70a223d0d rib: a reader counts the table once a poll, not once a shard
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A router with many peers has a reader of the table's journal for each
session that is sent learned routes, seventy on ddln, and every change
to the table wakes them all. Each pass of a reader went round every
shard, and for each shard took a count on the table (an Arc clone) to
hold it while the shard was read, and gave it back after: two atomic
adds on one cache line, by seventy readers, 256 shards a pass. A
profile of ddln taking a table from the other router had the reader's
poll as the top function at two to three cores, and in the Linux VM
with 64 such sessions half of all CPU time was in the two atomic add
instructions. The sessions writing to the table waited behind it: the
table came in at 58,000 routes a second.

The count is now taken once a poll. And each journal keeps its latest
sequence number in an atomic the table holds the other end of, so that
a reader can tell a shard with nothing new for it from one with
something without the shard's lock, and pass over it: on a quiet
table, which is what the readers see most of the time, a pass is a
load a shard and no lock. An exporter that is sent little of a table,
as one to a peer at an exchange is, no longer makes a verdict of
nothing for a prefix the peer does not hold, which cost a map insert
and a lookup on the way out, and tells a route that came down or
across (RFC 9234) from its attributes' bytes before parsing them.

Testing: the mesh benchmark can have any number of collector sessions
of the first router (BENCH_COLLECTORS): peers in the sense of RFC 9234,
sent learned routes, with the feeders' routes marked as having come
across from a peer, so that each has a reader following the table
while it fills and passes nearly all of it over, as a router with many
peers at exchanges has. In the Linux VM with 64 of them, three rounds
alternating with the previous commit, the routers' exchange of their
tables takes 11 s instead of 20 s, at 86 s of CPU time instead of
155 s; without them it is unchanged. The table's tests, the server
tests, the integration tests, the kernel tests and every lab phase
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 17:12:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a15efaa26c export: give the leave to gather back once the table is gathered 
The last change held a session's leave to gather a table, of which
there are four, until its table had gone out. A peer that reads at the
pace of its receive window takes hours over a table, and two such
peers held two of the four for that long, keeping other sessions from
gathering theirs: they waited for a leave before their first UPDATE.
The leave is for the memory a gather takes; what is still held once
the table is gathered is let go of as it goes out, as it was before
the gather was reworked.

The mesh benchmark's new peer can also come once the first router has
its own peers' routes and the other router's are still arriving
(BENCH_RECEIVER_EARLY=2), as a router's sessions come up after a
restart while its peers' tables arrive; the walk of a full table while
the table fills costs the arriving routes nothing measurable in the
VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:43:48 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 624d20f965 Release 0.2.6 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:30:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b5ca42244d export: a lighter gather, and UPDATEs to the session by the piece 
Two things kept an exporter from its peer. A table being gathered to
go out anew kept a map of every prefix heard, to let a later word on a
prefix supersede an earlier one, and dressed every set of attributes
before the first UPDATE went: a map insert and a parse-and-encode a
route that the peer waited on. And the session's queue took UPDATEs
one by one, a thousand at most, so the exporter stood at the queue
once a hundred kilobytes were in it, behind a writer that takes a
megabyte: the two took turns instead of running together.

The journal gives a table out shard by shard, each prefix once, and a
change meanwhile comes after the table as news of its own, so there is
no second word to keep track of while gathering: the map is gone, and
a prefix the peer is not to have is simply not gathered, as what it
held of it is withdrawn once the table is through. Each set is dressed
as its turn to go comes, so the first UPDATE goes once the table is
gathered; only when the gathered routes alone would be over the
peer's limit are the sets dressed first, to know which cannot be and
take their routes with them, so that none goes if too many would. The
UPDATEs for a set now go to the session in pieces of 64 KiB or so,
each of whole messages, and the queue takes 64 of those: a few
megabytes ahead of the writer, not a hundred kilobytes. The session
counts the messages in a piece by their headers.

Testing: the byte-for-byte test of the encoded UPDATEs now encodes
thirty thousand routes and checks the pieces are of whole messages
and of the size meant. The server tests, the integration tests, the
kernel tests and every lab phase pass. In the Linux VM, three rounds
alternating with the previous commit, a new peer's first UPDATE comes
1.8 s after its session is up instead of 3.1 s, and it has its four
million routes in 3.8 s instead of 4.1 s; the smaller of the two
routers' tables crosses in 3.0 s instead of 4.2 s and the larger in
5.0 s instead of 5.2 s, at the same CPU time.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:28:40 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 7425b101ed export: gather a table by attributes, then stream it 
A table given out anew to a session, at its start or when the journal
had let changes go, was gathered as a verdict for each prefix, each
with the route dressed for the session, and sent once the table was
all there. Dressing a route parses its attributes and encodes them for
the session, so a table of five million routes took seconds before
the first UPDATE went, on one worker thread without a yield. And the
routes that share attributes went out in separate UPDATEs after all:
they were grouped by the dressed form, of which only so many are kept,
and the routes of a set are spread over the whole table, so by the
time the next was reached the form had been let go of and made anew.
The mesh benchmark's new peer got four million routes in 3.8 million
UPDATEs; on a RIPE RIS table, grouped whole, a full table is 160,000
UPDATEs and 21 MB, and grouped in slices of 4,096 routes it is 845,000
UPDATEs and 72 MB, so slices are not the answer either.

The table is now gathered by what dresses its routes: the attribute
set as the table has it, who the route is from, and its IP version.
Nothing is dressed while gathering, which is a map insert a route.
Once the table is all there, each set is dressed once, with the thread
lent to the other tasks every few thousand, and then the sets go out
one after the other, each as few UPDATEs as its routes fit in, with
the peer reading the first while the rest are sent. A prefix heard of
again while the table is read supersedes what was heard before: each
word on a prefix is numbered, and only the latest goes. Path
identifiers are given at gathering, in the order the table has a
prefix's routes, so that they are as before. What was announced before
and not heard of again is withdrawn after the table, as before. The
exporter never blocks the thread on a table: the steady state's batch
is small, and the gathered table is gone through with yields.

Testing: a new test gives a session a table of 300 routes sharing
three sets of attributes, spread over the shards, and checks they go
in three UPDATEs. The mesh benchmark has a new phase in which a peer
takes the first router's table once the mesh is complete, timing the
first UPDATE and the whole; its feeders now negotiate no hold timer,
since they say nothing after their tables, and `BENCH_LOG` prints the
routers' logs. A new example, packing, counts the UPDATEs and octets
a RIS table takes grouped whole and in slices. The server tests, the
integration tests, the kernel tests and every lab phase pass. In the
Linux VM, three rounds alternating with the previous commit, the new
peer's first UPDATE comes 2.9 s after its session is up instead of
5.3 s, and it has all four million routes in 4.0 s instead of 8.0 s,
in 667,000 UPDATEs instead of 3.76 million; the whole run takes
22.4 s of CPU time instead of 26.2 s. The routers' exchange of their
tables, which does not go through this path once their sessions are
up, is unchanged within its noise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 15:16:18 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 92b9bdd3c5 export: keep the peer reading while the next batch is dressed 
An exporter following the table took up to 65,536 changes at a time,
dressed them all, and only then sent them; and before each batch it
slept the 50 ms that lets a run of changes be gathered, whether or not
the table was already far ahead of it. The session's queue holds about
a thousand UPDATEs, so once the batch went out the exporter waited on
the peer reading it, and while the next batch was dressed the peer had
nothing to read. A profile of two routers exchanging their tables,
sampled in 25 ms windows, showed the two stages taking turns: the
exporters at two cores while the sessions were at none, then the other
way round, with a gap of 50 to 75 ms at the start of each cycle for the
sleep. Each task ran at about half a core.

A batch is now 4,096 changes, so that what one batch sends is being
read by the peer while the next is being dressed, and the gathering
sleep is skipped while the reader is behind the table: a run is in hand
already. A reader says whether its last poll caught up.

Testing: the server tests, the integration tests, the kernel tests and
every lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the larger table
(4.9 million routes) is across in 5.6 s instead of 6.7 s and the
smaller (2.5 million) in 4.4 s instead of 6.4 s; the whole exchange
takes 7.3 s of wall-clock time instead of 8.6 s, and 18.9 s of CPU time
instead of 20.5 s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 14:19:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 381eabd886 export: encode a route's attributes once a session 
The exporter built each UPDATE as a message: the attributes of the
route's set, parsed from the table, were cloned, had the next hop
added, were sorted, and were handed to the session, which encoded them
to the wire with the rest of the message. For a table whose attribute
sets each cover a few routes that is an encoding of the same octets for
nearly every route, and a copy of the message's bytes on the way to the
writer besides. A profile of a production router taking the other
router's table with ADD-PATH had the exporters at 55% of the CPU time,
their dressing of routes at 23%, and memcpy at 12%.

A route's attributes are now encoded once for a session, when it is
first dressed for it, and kept with where in the octets a NEXT_HOP or
an MP_REACH_NLRI belongs so that the attributes stay in order of type
code. An UPDATE is then its header, the encoded octets with the next
hop or the MP_REACH_NLRI spliced in, and the prefixes: written straight
into a buffer that goes to the session as it is, and from the session
to the writer without a copy. The session's queue takes either a
message or an encoded UPDATE; an encoded one for a session that is no
longer established is dropped, as the exporter made it for the session
as it was. What is kept for a route, now a few hundred octets instead
of a kilobyte, lets twice as many be kept. The exporter's maps hash
with foldhash instead of SipHash, and the parser context for an UPDATE
is made once a flush instead of once an UPDATE.

Testing: a new test encodes a thousand routes with each of the forms
an UPDATE takes (IPv4 with the next hop in its own attribute, IPv6 in
MP_REACH_NLRI, with and without path identifiers, to two-octet and
four-octet AS speakers, over several messages) both ways and checks the
octets are the same byte for byte; the message-building form is kept
for the test. The exporter tests decode what the session is handed.
The server tests, the integration tests, the kernel tests and every
lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the two routers'
exchange of 7.4 million routes takes 20.6 s of CPU time instead of
25.4 s (-19%) over 9.0 s of wall-clock time instead of 11.1 s, and the
larger of the two tables is across in 7.1 s instead of 9.3 s. The
smaller table's time varies too much from run to run (5-8 s) for a
difference to be read from it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:47:47 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 f1aca37ebe rib: a short row grows by doubling 
A prefix's row of paths is given room for as many paths as there are
peers, and from there grows by a quarter at a time, one path at a time
while it is short. A peer sending every path it has (ADD-PATH) gives a
prefix several paths from one peer, so a row goes past the peers' count
from the first few routes and was then reallocated for each path.

A row shorter than the first allocation's room doubles instead, and
grows by a quarter only from there. Measured on the mesh benchmark, the
reallocations in the table's insert path fall by half; the memory held
is as before, as the rows that reach the quarter-growth are the ones
that were already allocated for it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:45:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 cbc1dc7e44 bench: two routers exchanging their tables with ADD-PATH 
The full-table benchmark measures one router taking in tables from its
peers. What two routers of one AS do with each other, each sending the
other every path it has (ADD-PATH both ways with all_paths, as a mesh
of routers that each hold the network's whole view is configured), was
measured nowhere: the exporter following the table's journal while the
sessions fill the table, at the size of the Internet. In production
that exchange took 70 seconds for a table that arrives from the
Internet in 20.

A new benchmark, mesh, starts two servers in one AS with distinct
router IDs, peers them over IPv6 loopback with ADD-PATH both ways and
all_paths, and feeds the first from four synthetic peers and the second
from two over IPv4 loopback (BENCH_PEERS, BENCH_PEERS_B,
BENCH_SCALE_DOWN). It prints how long each router takes to hold its
own peers' routes and then the other's, with the rate of the latter.
BENCH_DEADLINE prints the sessions' state if a router is not there in
time.

The benchmarks now allocate with jemalloc, as the daemon does, except
when memory is being counted: the C library's allocator put its own
locks in the profiles. Two routers on one host have each other's
addresses as next hops, loopback addresses, which the import check
refuses as it refuses a next hop that is this machine; the permissive
import policy, which only tests use and no configuration can ask for,
now takes them.

Testing: the next-hop test covers both policies; the benchmark runs on
macOS and in the Linux VM under perf.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 13:45:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d0f092ff06 session: write to the peer from a task of its own 
A session read from and wrote to its socket in one loop: while it was
writing, it was not reading. Writing waits when the peer's receive
window is closed, and the peer's window closes when the peer is not
reading, which it is not while it is writing to us and waiting on our
window in turn. Two routers sending each other every path at once, as
they do with ADD-PATH both ways after a restart, each stopped reading
within seconds and waited on the other until the send hold timer ended
both sessions eight minutes later. The status page's TCP row showed it:
both windows zero, the send queue full, the receive queue unread. The
batching of writes in the previous change made a session spend far
longer writing and so made this certain; it was possible before.

Now each session has a writer task. The session encodes its messages
into batches and hands them to the task through a bounded queue without
ever waiting: a batch the queue has no room for is kept, and while any
is kept the session takes no more messages from the rest of the server
(which then waits, as before, in its own task) but goes on reading the
peer. The writer task's failure, or a write that makes no progress for
the send hold time, comes back to the session as a local failure and
ends it. When the session ends, what is still to go out, a NOTIFICATION
among it, is handed over and the writer shuts our side once it has
written it.

Testing: a new integration test has one peer's table in the server and
a second peer send a table of its own while reading nothing of what the
server sends it, and checks that the server takes the second table all
the same; against the previous commit the test hangs until its timeout.
The send hold timer test now reads the writer's failure. The session
tests, the integration tests, the kernel tests and every lab phase
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 12:22:06 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ce1b68ca59 bgp: jemalloc's background thread on Linux only 
The last change built jemalloc with its background thread on, so that
freed pages go back to the kernel on time. jemalloc has that thread on
Linux only: on macOS every start of the daemon, and every test that
starts one, printed "option background_thread currently supports
pthread only".

The feature now comes from a Linux-only dependency entry of the daemon
crate, so the macOS build asks for nothing jemalloc cannot do. The
Linux build is as before: measured in the VM with four peers and a
collector, resident memory falls after the load without any setting
in the environment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 12:06:39 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9276bc9f7a session: write messages in batches, with TCP_NODELAY 
A session wrote every message to its socket as it came: one write and,
with Nagle's algorithm, mostly one segment for each UPDATE, a hundred or
two octets. On the IPv6 cross-connect between the two routers, whose
round trip is 50 microseconds, a million routes took 42 seconds to go
out: the status page's new TCP row showed the connection busy for that
long with neither window nor buffer limiting it, a congestion window
stuck at ten segments, and a smoothed round-trip time of 18 ms. That is
the receiver's delayed acknowledgements, 40 ms at a time, on a stream
of small segments, holding the sender to a few segments a round trip.

Now a session gathers what is waiting for it, up to the outbound queue's
length, in one wake, encodes the messages into one buffer, and writes
the buffer when it reaches 64 KiB or when the actions of the wake are
done; the socket has TCP_NODELAY, so a batch's last, partial segment is
not held back for the acknowledgement of the one before. The
NOTIFICATION that ends a session, including one sent after a panic, is
flushed on its own.

Measured in the Linux VM with four peers' tables going to a collector
over a veth pair, 151 MiB: 121,850 segments instead of 313,773, the
connection busy 0.8 s instead of 2.0 s, and the congestion window grown
to 47 segments instead of 13. Over loopback the transit benchmark is
unchanged at 2.5 s for the second neighbor to have the table: there the
receiver's parsing sets the pace, not the segments.

Testing: the session tests, including the one that sends a NOTIFICATION
after a panic; the integration tests; the kernel tests and every lab
phase.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-10 11:14:08 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b0dd357986 export: a kilobyte less a route advertised with all_paths; memory accounted for
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A router feeding a collector every path (all_paths) was 4.8 GB resident
with a table that should take a third of that. A heap profile of the
daemon in the Linux VM, with four full tables and such a collector,
showed where it went: the exporter kept, for every route it had
advertised, the route's outgoing form, the parsed attribute list it was
dressed with, through an `Arc<Dressed>` in the per-route record. One
such form lived for every distinct attribute set advertised, about a
kilobyte each, for the life of the session, on top of the exporter's
cache of 262,144 of them. The record needs only to tell whether a route
changed, and the table's own attribute set, which is in memory anyway,
tells it that: the outgoing form is a function of the set, the peer the
route came from and the policy.

This change:

* `Sent` holds an `Arc<AttrSet>` instead of an `Arc<Dressed>`, and a
  route is sent again when its set differs. `Verdict::All` carries both
  while a flush is on.
* `DRESSED_KEPT` goes from 262,144 to 65,536: a kilobyte a set, so 64 MB
  rather than 256 MB a session at most; what a flush is sending holds
  its own.
* The status page's header shows the daemon's memory as jemalloc
  accounts for it (allocated and resident), `/api/status` has the six
  figures (`memory`: allocated, active, metadata, resident, mapped,
  retained), and `bgp_memory_bytes{kind}` exports them. jemalloc is built
  with statistics for this, and `bgp-web` reads them through
  `tikv-jemalloc-ctl`.
* `scripts/profile/full-table.sh` gains `COLLECTOR=1`, a collector
  session over IPv6 that takes both families with ADD-PATH, and `HEAP=1`,
  which builds the daemon with jemalloc's heap profiler (the new
  `heap-profiling` feature of the daemon crate, not for production) and
  turns the profile into a flame graph with jeprof; and it reports
  memory with the table in and again 30 s later. The feeder
  (`crates/bench/examples/feed.rs`) plays the collector with
  `FEED_COLLECTOR=1`.

Measured in the VM, four peers' full tables and a collector sent every
path (4.9 million routes advertised): 1,723 MiB resident before, 1,286
MiB after, with jemalloc's allocated figure from 1,500 MiB to 946 MiB.
Without the collector the same tables take 630 MiB, so the collector's
bookkeeping is now about 50 bytes a route advertised. Of the resident
figure, some 400 MiB was pages jemalloc had freed and not yet returned:
it returns them on allocator activity only, and a router that has its
tables in allocates little. jemalloc is now built with its background
thread on (the `background_threads` feature), which returns them on
time: 1,020 MiB resident 30 s after the load, against 1,286 MiB without
it.

Documentation: `memory` in the HTTP API reference, `bgp_memory_bytes`
in the metrics reference and the monitoring guide, the script's new
settings in the testing guide.

Testing: the ADD-PATH integration tests pass, including the collector
ones; the heap profile and the figures above are from
`PEERS=4 COLLECTOR=1 HEAP=1 scripts/profile/full-table.sh` in the VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 22:51:11 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d7d7d39bb9 metrics: export each session's TCP connection figures
ci / docs (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The kernel's view of a session's TCP connection, which the status page
shows on request since the last change, could not be looked at after
the fact: a feed that crawled overnight left no record of whether the
peer's window or our own side held it up.

This change samples it at every scrape of `/metricz`: for each
established session, one `NETLINK_SOCK_DIAG` round trip, and twenty
gauges labelled by peer, `bgp_peer_tcp_*`: smoothed, least and variance
of the round-trip time; the congestion window and segment size; the
peer's receive window and ours; the send queue, the part of it not yet
sent, the receive queue and the segments in flight; retransmitted and
lost segments; the busy time and the time held up by the peer's window
and by our send buffer, cumulative, so that their rates give the share
of time a session is held up; octets sent and received; and the
delivery and pacing rates. A session the kernel has no connection for
loses its series, as does one that ends.

`SessionHandle::metric_label` is the one place a peer's metric label is
made (its name, or its address), used by the advertiser's gauge and here.

Documentation: the metrics reference lists them; the monitoring guide
gives the query for the share of time a session is held up by the
neighbor's receive window, which is the question the overnight feed
raised.

Testing: a daemon in the Linux VM with one session shows the series and
drops them when the session ends; the lab's basics phase checks the
scrape for the FRR session's round-trip time; the workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 12:12:53 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 99a47da2af status: a session's TCP connection as the kernel sees it; a route in full 
When a session advertised a full table to a route collector at two
thousand routes a second, nothing on the status page said whether the
daemon or the peer set that pace. The answer is in the kernel: the
peer's receive window, the queues, and how long the connection was held
up by the window or by the send buffer, which `ss -ti` shows.

This change asks the kernel the same way `ss` does, over the socket
diagnostics netlink family (`NETLINK_SOCK_DIAG`, `SOCK_DIAG_BY_FAMILY`
with the connection's addresses and ports), and reads `tcp_info` and
the congestion control name from the answer. No external program is
run and no unsafe code is needed: the request and the answer are byte
layouts of the kernel's ABI, read field by field, with fields an older
kernel does not send left at zero. A socket on a link-local address is
found by its interface, which the request names.

* `bgp_fib::sockdiag`: the request, the parser, `TcpConnection`, and
  `tcp_connection(local, peer)`; `Ok(None)` where there is no such
  connection or the system is not Linux.
* `GET /api/tcp?peer=ADDRESS` in the HTTP API: the connection of the
  session with that peer, with times in milliseconds, the two windows
  named by what they allow, and the held-up times also as shares of the
  busy time.
* The status page: a "tcp" link on each established session, which
  expands a row below it with the figures, re-asked on every refresh
  while open. The group rows' span is corrected for the "sent" column.

Also on the status page: the attributes of a received route that have
no column of their own were badges with their value in a tooltip, which
is hard to read. Clicking the attrs cell now expands a row under the
route with every attribute by its name (ORIGIN, AS_PATH, NEXT_HOP, MED,
LOCAL_PREF, COMMUNITIES, LARGE_COMMUNITIES, AGGREGATOR and the rest),
the path identifier, the selection rank and the RPKI and ASPA states;
the badges stay. Both expansions use the terms of `ss` and the RFCs
rather than sentences.

Documentation: the endpoint in the HTTP API reference, and the link and
the expansion in the monitoring guide.

Testing: unit tests of the request and of the parser on a reply built
field by field, and of a "no such socket" and a refused answer; a
Linux-only test that asks the kernel about a loopback connection of the
test's own and about one that does not exist; the lab's basics phase
checks the endpoint for the FRR session; the page was checked by hand
against the daemon in the Linux VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 11:41:47 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 efde277fd8 Test: both families reach a collector over one IPv6 session 
A collector at an exchange is fed over one IPv6 session that carries
both families, with ADD-PATH for both and IPv4 routes with IPv6 next
hops (RFC 8950). The integration test for that has the collector on
IPv6 loopback with those capabilities and two feeders sending 20,000
IPv4 and 1,000 IPv6 prefixes each, and checks that every route of both
families arrives with its identifier. The test helper now reads the
prefixes that come and go in the multiprotocol attributes too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:37:57 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9b211b61e9 export: count routes as they go out, and say when the table is walked again 
The number of routes advertised to a peer was written once a flush was
over, after every UPDATE of it had been handed to the session. The first
flush to a route collector is the whole table, two million routes or
more to a peer that reads at the pace of the Internet, so the status
page showed the peer holding one route for as long as that took, which
read as nothing going out.

Now the count is written after each batch of a flush, so that the page
and `bgp_peer_advertised_routes` show the table going out.

Also, an exporter that had the whole table and is given it again,
because the table changed faster than the exporter read it and the
journal let the oldest changes go, now logs that it is going through the
table again, at info. Until now that was invisible, and a collector
session during a full-table load from several peers is where it can
happen: the exporter is held up sending while the shards' journals,
cut back to twice their rows, run past its place.

Testing: a new integration test has a collector connected before two
feeders send 40,000 prefixes each, and checks it is told all 80,000
routes with their identifiers, which takes the exporter's initial walk
of the table through many batches; the advertiser and ADD-PATH tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:31:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ec5c9be454 export: advertise every route for a prefix with ADD-PATH (all_paths) 
Before this change, a session on which ADD-PATH (RFC 7911) was
negotiated in the sending direction was sent path identifiers, but only
the best route for each prefix. A route collector such as bgp.tools
wants every route a router has accepted, from every neighbor, and from
one session: without this the collector saw one path per prefix, nearly
always the same neighbor's.

This change adds `all_paths` to the export policy. On a session with it,
and ADD-PATH send negotiated for a family, every accepted route for a
prefix of that family is advertised, each with a path identifier of its
own; for a family without ADD-PATH send the best route goes as before.
`all_paths` needs `learned_routes` (a configuration error otherwise) and
the peer's `add_path` to offer sending. The export policy, the role
rules, the well-known communities and the export limits apply to every
route alike, and the export limits count every route.

Implementation, in the advertiser:

* An exporter with `all_paths` follows the table's journal of all routes
  (`RibTable::reader_of_all_routes`, until now unused) instead of the
  best-route journal, and is given all the routes of a prefix whenever
  one of them changes (`News::Routes`).
* What the peer holds is `Announced`: the set of prefixes for a
  best-route session, as before; or for an all-paths session, for each
  prefix the routes announced, keyed by the peer they came from and that
  peer's own identifier, with the identifier they went out with and the
  form they went out in. A route keeps its identifier for as long as it
  is announced, goes out again only if it changed, and is withdrawn by
  its identifier; a new route gets the lowest identifier not in use for
  the prefix, from 1.
* Verdicts carry either kind (`Verdict::Best`, `Verdict::All`), and a
  reset of the table puts what was announced in doubt until heard of
  again, as before, with the same identifiers kept across the reset.
* The record for an all-paths session is about 50 bytes a route, some
  tens of megabytes for a full table from several neighbors; the
  best-route bookkeeping is unchanged in size.

Also, so that a collector session can be watched: the number of routes
advertised to each peer and not withdrawn since, every route counted
where several are sent for a prefix. The exporter keeps it in the
session's counters after every change; it is `advertised_routes` in
`/api/peers`, the "sent" column of the status page's neighbor table,
and the gauge `bgp_peer_advertised_routes{peer}`.

Documentation: a "Feeding a Route Collector" section in the guide to
passing routes on, with the configuration for a collector; `all_paths`
in the configuration reference; the ADD-PATH sections of the neighbor
guide and the standards reference; the limitation that was removed;
the new field, column and metric in the HTTP API, monitoring and
metrics references.

Testing: an integration test with two feeders and a collector that
negotiated ADD-PATH receive, which checks the two routes with their
identifiers, the withdrawal of one by its identifier, a changed route
keeping its identifier and a new route taking the free one; a
configuration test for `all_paths` without `learned_routes`; a new
`collector` phase of the interoperability lab in which FRR, as a route
collector, is told both of BIRD's routes for a prefix from BIRD's two
sessions and holds them apart, and loses one when its session is
disabled, with the advertised count checked at each step. The workspace tests, the kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:16:48 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 03f4f58ecb rib: give a row room for every peer's route at its first growth 
A row's routes are in a vector that grew by one route at a time, so with
four peers sending full tables every row's vector was reallocated three
times as their routes arrived, and the reallocations were 7% of the
daemon's CPU time in a profile of a four-peer load.

This change has the first growth of a row take it to as many routes as
the table has peers, at most 8 (`FIRST_ROOM`), since a prefix that one
full table has, the others nearly always have too; after that it grows
by a quarter as before. A table with one peer never grows a row and is
unchanged. A route server's clients do not all announce every prefix,
hence the cap: a row of a prefix that fewer peers announce has at most
seven spare slots of 16 bytes.

Measured in the Linux VM with four peers sending the synthetic table of
1.23 million prefixes: the kernel is in step 1.9 s after the first route
instead of 2.2 s, the daemon's CPU time over the load is 5% lower, and
it is 604 MiB resident instead of about 630, since there are no
reallocation leftovers. With one peer nothing changes. Memory per route
in the benchmarks, allocations counted: 77 bytes before and after with
the synthetic table from four peers; 64 before and 65 after with the
RIPE RIS dump's eight largest peers, whose load went from 2.8 s to
1.8 s on one thread, the allocation counting making every reallocation
dearer than it is in the daemon.

Also tried and not adopted, with the new `JEMALLOC_CONF` setting of the
profiling script: transparent huge pages for the allocator
(`thp:always,metadata_thp:always`). Over three pairs of runs with one
peer the table arrived in 0.66, 1.17 and 0.70 s with them against 0.87,
0.75 and 0.87 s without, the CPU time was within the noise, and the
daemon was 60 to 100 MiB larger. An arena for the routes would not help
either: jemalloc's size classes already give rows exact 16, 32, 48 and
64 byte allocations, and the cost was the number of reallocations, not
the allocator.

Testing: the table tests, including the one that grows a row past
65,535 routes, pass; the kernel tests and the interoperability lab pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:34:23 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 48da8cc7d2 rib: size the routing table for the routes the peers are expected to send 
The routing table's shards started empty and grew as routes came in,
each growth moving every row of the shard into a table twice the size.
In a profile of the daemon taking in a full table, the growth showed as
up to a seventh of the session's time, and the daemon's resident memory
had the old and the new table of each shard side by side for a moment.

This change adds `RibTable::expect_prefixes`, which reserves room for
that many prefixes spread over the shards, and has the server call it at
startup with the largest `expected_routes` among the configured peers:
the prefixes of several full tables are mostly the same ones. Without
`expected_routes` nothing changes. The `full_table` benchmark does the
same unless `BENCH_PRESIZE=0`.

Measured, the saving in time is within the run-to-run noise: in the
Linux VM one peer's table is in the routing table in 0.81 s instead of
0.89 s, and the kernel is in step at about the same time as before;
three pairs of runs of the benchmark on a Mac average 0.33 s against
0.35 s. The profile's attribution to the growth was mostly sampling skid.
The memory saving is real: 442 MiB resident with one peer's table in
instead of 483 MiB, and 626 MiB instead of 632 MiB with four.

Also: the profiling script printed nothing for the kernel's change
counts because it asked `/metrics` for them; the endpoint is `/metricz`.
With that fixed, a four-peer load shows 1.25 million installs for 1.23
million prefixes: the journal already gives the kernel task only the
latest change of a prefix when it lags, so no further coalescing of
installs is called for.

Documentation: `expected_routes` in the configuration reference, the
neighbor guide and the proto comment say that the table is sized by it;
the testing guide's VM results are brought up to date.

Testing: a table test checks that the room is there and is kept; the
workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:13:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9e9e4f90ef fib: take best routes from the table 16384 at a time 
The kernel task read changes of best route from the routing table 1024
at a time. A poll of the table does two binary searches and takes a read
lock in every shard, however few events the shard has; with the 128
shards of an 8-core machine, that overhead was paid for every 8 events a
shard, and the profile of a full-table load showed 8% to 9% of the
daemon's CPU time in the reader's searches and locks.

This change raises the batch to 16384. In the Linux VM, with four peers
sending the synthetic table of 1.23 million prefixes, the kernel is in
step 2.2 s after the first route instead of 2.9 s, and the daemon's CPU
time over the load falls 13%; the reader's share drops from 9% to under
1%. With one peer the kernel-side share falls the same way and the CPU
time 3%. A batch holds up to 16384 small records between polls, which is
about a megabyte.

Testing: the fib unit tests pass; measured with
scripts/profile/full-table.sh with one and four peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 5915c8e562 profile: configure the peers and report the kernel's change counts 
The full-table profiling script took its peers as unconfigured ones under
the permissive policy, which is not how a router is set up. Now it writes
a peer block for each of them, with `expected_routes` set to the table's
size, so that the default import policy and anything that sizes itself by
the expected routes are in the measurement. The few bogon prefixes in the
synthetic table are refused by that policy, as the output says.

After the table is in, the script also prints
`bgp_kernel_route_changes_total`, so that a run shows how many installs
and removals the kernel was asked for and not only how many routes it
ended up with.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1bc71aaac5 fib, bench: profile the daemon taking in a full table; count blackholes instead of scanning 
Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.

This change:

* Keeps counts of blackhole routes and of routes kept unconfirmed from
  an earlier run in `Plan`, in step with the map through `Plan::put`
  and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
  cost nothing. `kept()` had the same scan; it ran only while routes of
  an earlier run were being kept, which is exactly during a full-table
  load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
  new `profiling` Cargo profile (release with line tables) and frame
  pointers, runs it in a network namespace joined by a veth pair to a
  second namespace, feeds it the synthetic table from there as one or
  more neighbors, samples it with `perf`, and writes a flame graph
  (inferno), flat profiles by self and inclusive time, the time until
  the routes are in the table and in the kernel, and the daemon's
  resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
  and `OUT` control it. From macOS it runs in the Linux VM through
  scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
  from the peers' addresses, completes the handshakes, writes each
  neighbor's table, drains what the daemon sends back and holds the
  sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
  connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
  "Profiling" section, and the daemon's numbers in the VM.

Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.

Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:03:43 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 6d8f89e509 policy, fib: blackhole routing for routes with the BLACKHOLE community
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
Before this change, a route carrying the BLACKHOLE community (65535:666,
RFC 7999) from an EBGP neighbor was always rejected with the reason
"blackhole request", and nothing discarded traffic. A customer under a
denial-of-service attack had no way to ask this router to drop traffic
to an address.

This change adds `accept_blackholes` to the import policy. When it is
set, a route with the BLACKHOLE community is accepted if:

* its prefix is a single address, a /32 or a /128 (otherwise it is
  rejected with the new reason "blackhole for more than one address");
  the `max_ipv4_prefix_length` and `max_ipv6_prefix_length` limits do
  not apply, as RFC 7999 section 3.2 recommends;
* the address is inside a prefix the neighbor may announce, if
  `prefixes` is configured, regardless of the entry's `max_length`
  (new `PrefixSet::covers`); without a prefix list, an address inside
  the local AS's own prefixes is rejected as before ("prefix is our
  own");
* the AS path, origin and RPKI checks that apply to every route pass.

The next hop of an accepted blackhole route is not checked for
third-party addresses and is not resolved, because traffic is never
forwarded to it. The route takes part in best path selection as usual;
a host route is normally the active route for its address.

The kernel task installs the active route as a kernel blackhole route
(`RTN_BLACKHOLE`, shown by `ip route` as `blackhole PREFIX proto bgp`).
Blackhole routes left by an earlier run are adopted at startup like
other routes of ours. The route is advertised to other neighbors as the
export policy and export limits permit, with the community intact, so a
provider that honors BLACKHOLE can discard the traffic upstream.

Implementation:

* `plan::Target` becomes an enum: `Via { gateway, interface }` or
  `Discard`. `plan::resolve` takes a `blackhole` flag and returns
  `Discard` for such routes without looking at the next hop (the
  own-network check still applies). No nexthop object is made for a
  discard target.
* `AttrSet::is_blackhole` reads the community off the stored attribute
  bytes, like `next_hop` and `as_path`, so the kernel task, the
  revalidator and the RPKI revalidation can ask it of every route.
* `ImportPolicy::is_blackhole_request` and `check_blackhole_prefix`
  carry the import side; the import flow (`routes.rs`) computes once per
  UPDATE whether its routes are an accepted blackhole request and skips
  the next hop checks for them.
* `Status::blackholes` and the `blackholes` field of `/api/kernel` count
  the installed blackhole routes; the status page shows the count next
  to the installed routes.

Not covered: a prefix this router originates with the BLACKHOLE
community is advertised but not installed as a blackhole route. The
documentation says to add a kernel blackhole route by hand for that
case.

Documentation: the "Blackhole Routes" section of the import policy
guide replaces "Blackhole Requests", the kernel guide gains a "Blackhole
Routes" section, the configuration reference describes
`accept_blackholes`, the HTTP API reference describes `blackholes`, and
the RFC 7999 rows of the standards reference are updated.

Testing: unit tests for the attribute check, the prefix checks and the
plan; integration tests that a blackhole request is refused by default
and, with `accept_blackholes`, taken for one address inside the peer's
space and refused otherwise, with the recorded reasons; a kernel test
that a blackhole request becomes a kernel blackhole route whatever its
next hop, is removed on withdrawal and is adopted after a restart. The
existing kernel tests, the interoperability lab and the workspace tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 23:08:50 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 01ac3e2cd8 Test peer: no bail! in expression position 
Newer compilers warn of the macro's trailing semicolon there, and CI fails
on warnings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 22:23:52 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3608897245 fib: resolve next hops through the kernel's routing table 
Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.

Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:

* The route is looked up in one kernel table, the "resolution table"
  (`kernel { resolution_table: N }`, default 254 = main). The longest
  prefix that contains the next hop wins; of several routes for that
  prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
  route is installed with the gateway and interface of the kernel route
  that covers the next hop. A kernel route without a gateway (a route
  onto a point-to-point or tunnel interface) makes the next hop itself
  the gateway on that interface. A kernel route that names a nexthop
  object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
  own or another daemon's. A next hop that is reachable only through a
  BGP route depends on another BGP next hop, and such chains are not
  predictable.
* The default route does not resolve a next hop unless
  `kernel { resolve_via_default_route: true }` is set. On a router the
  default route points at a transit provider, and a next hop that is
  reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
  unresolvable, as does a route whose interface is down.

The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.

Implementation:

* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
  table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
  and the kernel's nexthop objects into one value. The kernel task
  publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
  and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
  alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
  kernels get a dump of every table, filtered in user space). It asks
  again, debounced, whenever a dedicated netlink socket that is a member
  of the route multicast groups receives anything. That socket is given a
  64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
  installs on it too, up to a million at a time, and an ENOBUFS overflow
  is treated as the same hint as a message. Its contents are never
  parsed. An earlier attempt to receive route notifications on the
  socket that writes routes failed, because the kernel only excludes the
  requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
  `Discarded`, which appear as the reasons in `/api/kernel` and on the
  status page.

Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.

Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 22:23:27 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 430addcf20 Status page: IPv6 first in a neighbour's pair of sessions 
Of <stem>-v6 and <stem>-v4, the IPv6 session is listed first and carries
the name in full; IPv4 is under it. The same order in the lookup's list of
peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 23:19:56 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 277b973f1a Status page: what moved, what is missing, and a dark room 
The attention strip's names lead to the peer's row, which lights up for a
moment. The IPv4 and IPv6 sessions of one neighbour (NAME-v4, NAME-v6) are
listed together, joined down the side, the name said once in full. A route
count that moved by more than 2% between refreshes carries ▲ or ▼ for two
minutes; a row whose state or count changed while the tab was in the
background carries ● until it is clicked. A compact mode drops the next
hop and session columns, and is remembered by the browser. Colours are
variables, with a second set for prefers-color-scheme: dark.

A new per-peer `expected_routes` says about how many routes a peer sends
when all is well. The daemon only passes it to the API; the page lists an
established peer with under half of it as needing attention, a minute
after the session came up, with the expected count beside the actual one.
An established session with next to no routes looked, until now, exactly
like one that was fine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 23:16:01 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 7b6547d3c9 Docs: say what the forge cannot render, and two headings by the style guide 
The documentation sources are Zola Markdown: front matter between +++
lines, shortcodes that insert the example configurations, and @/ internal
links. A code forge shows all three unrendered, which the README used to
deny. A docs/README.md now explains each, and the README points to it.

"Configuring BGP Neighbors" becomes "BGP Neighbors", as the other feature
pages are titled; "When a Neighbor Restarts" becomes "Neighbor Restart", a
noun phrase as the style guide asks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:18:18 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3a1e42e594 README for a public release
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details 
What Ferrix is and does, that it routes AS210036, how to build it and try
it without a router, the security points an operator must know, and a note
that the code is written with an AI coding agent under the maintainer's
review.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:09:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 35ce2e2e57 Status page: routes in order of selection, lookup by origin AS 
Received routes are listed from the best down: the RIB gains ranked_routes,
which runs selection again without the winners, so the second route is the
one that would be installed were the best withdrawn. The API gives each
route its rank, and the page shows it as a numbered badge after the star.

The lookup box moves to the received-routes table, which is what it filters,
and takes an AS number: a new `origin` query parameter finds every route
whose path ends in that AS, with an empty path counting as the local AS.

Long cells are clamped to a few lines with a control to open them, and an
AS repeated along a path (prepending) is shown once with a ×N badge. Peer
names are bold and dark, not underlined links, so a row can be found by
its name among the addresses and numbers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:09:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1ce7eef27e Status page: what needs attention first, glyphs for what repeats 
The page opened with every session in bold green, the normal state, and
the brightest numbers on it were the rejection counts, which RPKI doing
its job produces by the thousand. The two sessions that were actually
down were in the same red, thirty rows down. Now:

- A status line at the top says what needs attention, by name and with
  the reason: sessions down, peers restarting, the kernel behind or
  refusing routes, RPKI data missing or about to expire. When nothing
  does, it says so. A second line gives the normal state in grey.
- Colour means one thing each. Red: failed, needs a hand. Amber:
  degraded, worth watching. Green, set small: a system doing its job.
  Grey: inactive and expected. Normal is black; "Established" is no
  longer green, and a rejection count no longer red.
- Peers that need attention come first, then the others grouped by what
  they are to us, with a "problems only" switch. Numbers are right
  aligned with thousands separators; the established time and uptime
  are one cell; the session's properties are glyphs (v4, v6, GR, ENH,
  AP, 2B) with the meaning as a tooltip, as are the roles.
- Received routes: the best route leads, marked ★; attributes without a
  column are glyphs (OTC, AA, AGG, AS4, OID, CL); validity is ✓ ✗ –.
- Rejected routes: those alike (same prefix, reason and path) are one
  row with a count and the last time; each peer's reasons are a bar.
- Communities with a configured name (community_names, new) are badges
  with that name, the number as the tooltip: "comrack", "SwissIX",
  "interconnection 2011" rather than 210036:100:2 and so on. A name for
  a pattern shows the open part after it. The daemon does nothing with
  the names but pass them to the page.
- One lookup box at the top, which takes a peer's name too; a peer's
  name anywhere on the page leads to its routes; table headers and the
  status line stay in view while scrolling.
- The host name comes from gethostname(2), not from running `hostname`,
  which a static binary on a bare machine could not find: comrack said
  "unknown host".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 17:49:06 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 23e8b4aadb One word for a route that is not taken: rejected
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The API, the metrics, the status page and the documentation say a route
is "rejected" (/api/routes/rejected, rejected_routes,
bgp_routes_rejected_total, "Rejected routes"), and the reason type is
Rejection. The code that keeps the record said "refused": RefusalLog,
PeerRefusals, RefusedRoute, RefusalsView, RefusedRouteView, the refusals
module. Those are RejectionLog, PeerRejections, RejectedRoute,
RejectionsView, RejectedRouteView and rejections now, and the log line
says "Rejected a route". Nothing the API or the metrics expose changes.

Refusing stays what it was: a connection turned away at the door
(server::Refusal), a kernel that would not take a route, an OPEN we
would not have.

The two aliases of SendReceive, AddPathSendRecv and OrfSendRecv, are
gone; it is one type.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 12:10:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c218ea4d85 Integration tests: one copy of each helper 
The test files had grown their own copies of the same helpers: prefix()
twelve times, established() seven, an UPDATE builder six, a seventeen
field PeerConfig literal in fifteen places, a timeout constant in twenty
one. Each is in crates/integration-tests/src once now: peer_config() and
TestServer::with_peer for a configured peer, Announcement for building
UPDATEs, established(), the waits (wait_until, wait_for, wait_for_routes,
wait_for_sessions), get_json, and TIMEOUT. A new field on PeerConfig is
one line to add, not fifteen.

TestPeer::recv has a deadline, so a test that gets nothing fails instead
of hanging; is_closed, which read one byte and took a KEEPALIVE for an
open connection, is closed_within and closed_unanswered.

Net 788 lines fewer; every test checks what it did.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:54:49 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 673920d48b Names that said the same thing twice, or two things once 
- PathAttribute::UnknownPathAttribute and Capability::UnknownCapability
  are PathAttribute::Unknown and Capability::Unknown: the enum already
  says what kind of thing is unknown.
- The capability entry types are all named XxxEntry now
  (ExtendedNextHopEntry, LongLivedGracefulRestartEntry), rather than two
  of them sharing a name with the variant that holds a list of them; and
  MultiLabelCompat is MultiLabel, since nothing explained the "Compat".
- Two different things were both called Reflection: what a session needs
  to reflect routes (export.rs, still Reflection) and the identifiers a
  reflected route must not carry back to us (policy.rs, now
  ReflectorIds).
- well_known_allow's peer_is_internal is ibgp, as everywhere else.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:35:12 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 0829f873f2 Server: the long functions in named steps 
Five functions did several things each in one body of 120 to 190 lines.
Each is now a list of steps with a name and a doc comment of their own,
with nothing changed in what they do:

- run_session: hold_down_after, keep_for and release_routes.
- Session::establish: session_handle, validation and route_source. A
  session keeps the peer's settings as one PeerSettings rather than
  eight fields copied out of it one by one; the sweeps' two channel ends
  are named fields, not .0 and .1; whether the peer can be asked to send
  its routes again is a bool, not a copy of the families.
- Fsm::on_open: negotiate_role and negotiate_add_path.
- BgpServer::run: log_startup and spawn_tasks.
- Config::from_proto: parse_listen_endpoints, parse_peers, check_roles,
  parse_originate and Ours::tell_peers, with the three per-peer defaults
  in one PeerDefaults.
- Egress::outgoing: may_go, rewrite and next_hop_for.
- Exporter::follow: Learned::drain, judge and flush, with the three
  things called "announced" told apart (own_announced, learned.announced,
  batches) and the verdicts given a type.

The hold-downs are a module of their own, holddown.rs; policy.rs is
about policy again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:32:20 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 724bb99f11 Kernel: read what the table has before letting kept routes go 
Once every peer had sent all its routes, the kernel task let go of the
routes kept from the earlier run that the table did not have, at once.
But "the table has" and "the kernel task has read" are not the same
moment: a route a peer sent just before its End-of-RIB could still be
unread in the table's journal, in which case it was taken out of the
kernel and put back a moment later. The interop lab saw this now and
then as "N routes were taken out of the kernel over the restart".

The task now reads the table to the end before it decides what is
unconfirmed; follow() says whether the reader is caught up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:21:53 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 175a13d506 Hygiene: scripts, CI, docs and manifests 
- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
  it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
  in the namespace rebuilt what kernel-tests.sh had just built. Each is
  passed on only where set.
- The README's list of unsupported features still had route reflection
  and graceful restart, both of which exist; the command-line reference
  lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
  what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
  which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
  three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
  unused variable; small things in the web crate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e5341ea639 Packet: the wire format's shared pieces in one module 
What the parsers and writers of messages, attributes and capabilities
had each written for themselves is one module (wire.rs): the shape of a
parser's result and its error, reading a list to the end, checking that
nothing was left over, a non-empty list of fixed-size records (the
community attributes and CLUSTER_LIST, which now fail alike), an AS
number at the session's width, an AFI/SAFI pair, and a length written
before what it counts. open.rs loses its fail! macro and attributes.rs
its own copies of the same.

Tests for what had none: AS4_AGGREGATOR kept opaque and discarded when
malformed; IPv4 routes with IPv6 next hops (RFC 8950) in the property
tests; and what a part-level parser accepts re-encodes to what it read,
but for the fields the decoders normalise.

Nlri::wire_len had no callers; the mask of a prefix's last octet is one
function; a test's name is spelled right; a comment is on the test it
describes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b467b136e4 Kernel and RPKI: say what went wrong, and once 
The RPKI client had one Timeout error whose text said "did not answer
within 60s" whatever had run out: the connect timeout, the time for a
whole answer, or the data's lifetime. These are what /api/rpki shows as
the last error; each is its own now. The protocol version shown is the
one the cache answered in, not the one being tried. The VRP set, hashed
by the hundred thousand with every answer, uses the quick hash the ASPA
table already did.

In the kernel installer the writer task's error was worked out in three
places, and the next-hop resolution checked for our own networks twice
on every cache miss. A refusal was logged twice, once with the status
lock held. The interfaces are listed in order of index by the structure
that holds them, from a function that can be tested on any OS, and is.
EIO is with the other errnos; Plan has no installed_prefixes (no
callers) and clears its records in one place, with the order of
Unresolved::ALL asserted at compile time since indexing depends on it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3b22e318a2 Server: one way to work on the table off-thread, and smaller tidying 
Removing a peer's routes, sweeping its stale ones and sweeping the
attribute sets were each written out as their own spawn_blocking, and
each dealt with a failed task differently: the periodic sweep dropped a
panic on the floor, and a restarting peer whose kept routes could not be
removed was logged as "removed 0 routes". off_thread does it once, and
every caller logs a failure.

Also:
- the hard-reset wrapping of a NOTIFICATION is one function in the FSM;
- the two ways to close every session are one, and the registry has one
  leave() rather than leave() and shutdown();
- the dialer says why a session ended with SessionEnd::reason, and no
  longer turns a result into a result;
- as_path_prepend is parsed in one place, and the limits in error texts
  come from the constants rather than being written out;
- router_id and cluster_id are parsed by one function;
- the exporter builds its Egress in one place;
- the fields a session keeps only to hand to its handle are fewer:
  route_refresh is a bool, not a copy of the families;
- counting the stale routes removed cannot underflow;
- docs that had gone stale: the proto still said there was no route
  reflection, the advertiser's module doc said it announced only our own
  routes, from_server listed two of its five sources, and so on.

Tests for what had none: which session ends count as restarts, the Hard
Reset wrapping, the graceful restart settings, the dialer's jitter,
withdrawals in MP_UNREACH_NLRI, and AS_TRANS on a two-octet session.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c69aa38090 Routing table: a reader waits out a change under way instead of spinning 
A reader does not read past the earliest change still under way, which
may not have reached every shard. A change to every shard, as when a
peer's whole table goes, is under way for a while; changes others made
meanwhile are beyond it. A reader that found only those saw "more to
read" with nothing it could read, and asked again at once, round and
round, taking every shard's read lock each time, until the change ended:
every exporter and the kernel installer, spinning for as long as a full
table took to remove.

Now a pass that moves no cursor though there is more says it is caught
up, and waits. And a change wakes the readers when it ends if any other
change woke them while it was under way, since what they were woken for
is only readable now. The three per-shard change loops are one
(RibTable::change), which drops the change before waking, as only one of
them did.

Also: the add_peer doc was attached to peer(); some_routes had no
callers; count_stale is for tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 befc292215 Tidy the export limits, passive peers and status page code 
Readability, no change in behaviour, but for one log line:

- The routes a peer holds from us were counted twice over, differently,
  in the exporter: once per learned-route batch and once per originated
  route. Learned::held_by_peer counts the learned ones in one place, and
  ExportLimits::exceeded_by returns the limit that was passed instead of a
  bool, so that ending the session no longer has to dig it out again with
  a default that could not happen.
- The two hold-downs for a limit passed, ours or the peer's, were two
  near-identical blocks; they are one match.
- How a session ended is put into words in one place
  (SessionEnd::reason), for the dialer's backoff and the passive peer's
  state alike.
- parse_export_limits parses one block; what a missing block means (no
  limits at the top, the top's limits for a peer) is said where it
  differs, instead of Option in and Option out.
- The startup warnings about export limits are a function of their own,
  and the peers without limits are named in one line rather than one
  line each: a router with eighty sessions does not need eighty warnings.
  The warning about a peer without a role is gone: the configuration now
  refuses one.
- Hex formatting of attribute octets is one function; the text for an
  unknown attribute is built in three plain steps.
- The exporter's test fixture returns a named struct, not a tuple.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 10:05:31 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 2dfc00862b Release 0.2.5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 00:06:06 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 c9751883c4 Passive peers: configured, and left to connect 
A session could only be configured for a peer we would then dial, every
few seconds to two minutes, for as long as it was not up. A new peer
whose side was not ready yet filled the log with failed attempts, and a
peer that opens its sessions itself was dialled for nothing.

A peer with `passive: true` is not dialled. A connection from its address
is taken as for any configured peer, and the session comes up with all
that is configured for it: role, import policy, export limits,
hold-downs. Until then the status page and /api/peers show it as
`Passive`, waiting for the peer to connect; after a session, with how
that session ended, which nothing else would say since nobody dials it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 00:05:54 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 910dd258fb Release 0.2.4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:11:03 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 9cfd3f67f0 Export limits, and a role for every peer where routes are passed on 
What other networks were told of rested on one thing: the Only to
Customer mark that routes from providers, peers and route servers get as
they come in. A route without it, from a session without a role or by a
mistake anywhere along the way, went to every peer, from every router of
ours, and nothing of ours stood in the way.

export_limits are a second check, independent of the roles, the marks
and the export policy. For peers above or beside us (PROVIDER, PEER,
ROUTE_SERVER, and any in another AS without a role):

- prefixes: only routes for these go out, learned or our own. With our
  and our customers' space listed, a leak of anyone else's routes goes
  nowhere above us.
- max_prefixes: if more routes were to go out than this, none of them
  does. The session ends with a Cease / Administrative Shutdown whose
  Shutdown Communication says why (RFC 9003), and the peer is held down
  for its import.hold_down_seconds, so that it is told nothing rather
  than too much. The whole table is counted before any of it is sent.

They are set at the top and hold for every such peer that has none of its
own; unlike `export`, a peer's own export policy does not drop them. A
peer's own export_limits take their place, and an empty one lifts them.
Customers, route server clients and our own routers are not limited.

A configuration in which learned routes are passed on and a peer in
another AS has no role is now refused, as is accept_unconfigured_peers
with learned routes: without a role, a peer's routes go to everyone and
it is told of everything. This used to be a warning.

At startup, a warning for each peer the limits apply to that has none,
and for each originated route the limits leave out. A metric,
bgp_export_limit_exceeded_total, counts the sessions ended for it.

The interop lab has a phase in which a customer leaks: a route outside
its space never reaches BIRD, and when more were to go than the limit,
BIRD's session ends and its log has our reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:10:43 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 17045a3634 Status page: tables for the kernel, communities by AS, attributes as text 
The kernel section was one run-on line of interfaces and addresses. It is
now a small table of the installation state, and a table of interfaces
with their IPv4, IPv6 and link-local addresses one per line; ports that
are down and have no addresses share one line at the bottom.

Communities ran together into one block that wrapped anywhere. Each AS's
communities are now a line of their own, kept whole so that they copy,
with a hanging indent where a line wraps; pointing at a line names the AS.
Where control communities are on (now in /api/status), those of ours that
ask for something say what as a tooltip.

A rejected route's details were a fold-out that repeated the summary and
then dumped the decoded attributes as JSON. Rejected routes now have the
same columns as received ones, and both have an "other attributes" column
for what the summary had no field for, as text: aggregator, extended
communities (route targets and origins by name), AS4_PATH with its sets in
braces, originator ID, cluster list, and unknown attributes in hex. The
API has them as summary.other; the JSON stays in `attributes`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:09:02 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 89e0c3a3e2 Release 0.2.3 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:15:28 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 b16227c51f Route reflection, with ourselves as next hop 
A route learned from one router of our AS went to no other (RFC 4271
§9.2.1), so every router needed a session with every other. A router at
the end of a single link, with one session to its neighbour inside the AS,
heard only of what that neighbour learned itself, and nothing of what the
neighbour had from the routers behind it.

A peer in our own AS can now be made a route reflector client
(`route_reflector_client`, RFC 4456). What a client sends goes to all our
routers; what the others send goes to the clients. A reflected route gets
ORIGINATOR_ID (where it started, if it does not say) and our cluster ID at
the front of CLUSTER_LIST; it is not sent back to the router it started
at, nor round to a cluster it has been through, and one that comes back
anyway is refused as "route reflected back to us". The cluster ID is the
router ID unless `cluster_id` says otherwise. The two attributes are now
decoded as such rather than kept as bytes, and are never sent to other
networks.

Unlike a standard reflector, this one is the next hop of what it reflects,
as it is of everything it announces: there is no resolving a next hop
that is not a neighbour, so a client reaches what lies behind another
router through the reflector.

Only peers in our own AS can be clients; the configuration refuses the
setting for any other. Nothing changes in what other networks are told:
Only to Customer marks go along with reflected routes, and every
receiving session keeps its own max_prefixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:10:35 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 164b1099f1 Release 0.2.2 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:39:06 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 4e945b278f Import rules: preference and communities for some routes only 
The import policy did the same to every route of a session. Now `rules`
can do more to some of them: those with certain communities, those for
certain prefixes, or both. Each rule can set the local preference and put
communities on; each route takes the first rule that holds for it.

Rules see a route as the session's own statements leave it, with our
communities taken off and the session's marks put on. So a router can act
on where another router of ours learned a route, by the mark that router
put on it, and a customer's single prefix can be marked for a prepend.

Routes of one UPDATE that different rules hold for go into the table with
attribute sets of their own. An UPDATE that no rule touches is taken as
before, without its attributes being written out again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:34:17 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 4d13336874 Release 0.2.1 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:16:45 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 72444c2706 Control communities: session groups, prepends, and only ours to give 
Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:

  L:11:P .. L:13:P   our AS once to three times more towards AS P
                     (P 0: towards every peer in another AS)
  L:20:G / L:21:G    not to / only to the sessions of group G
  L:31:G .. L:33:G   our AS once to three times more towards group G

A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.

They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.

And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.

This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:12:05 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 3609451bdc Ignore our own prefixes from our own routers, and say why a route for a network of this machine is held 
Two routers of an AS that both originate a prefix (anycast) each
announce it to the other over their internal session. Each took the
other's route, could not install it in the kernel, since the prefix is
on one of its own interfaces, and so held it, counted as rejected with
"next hop unreachable": a permanent rejection on a healthy session, for
a next hop that was in fact fine.

A route from a peer in our own AS for a prefix this router originates
is now ignored: not taken, not held, not counted. Only that prefix
itself; what is inside it is judged as usual. From other networks such
routes are refused as our own prefixes, as before.

A route that is held because its prefix is a network of this machine,
or inside one (an exchange's LAN prefix, say), is now counted as "prefix
is a network of this router", apart from routes whose next hop cannot
be reached.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:11:53 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 52593bb2c2 RPKI: ask a cache without data yet again soon 
A validator that is still running its first validation answers a query
with No Data Available (error code 2). That ended the session, and the
next attempt came after the retry interval: 600 seconds when the cache
had not yet said otherwise, which it cannot have before it has data.
For those ten minutes after a validator started, nothing was validated.

No Data Available is now asked again after 30 seconds, or after
retry_seconds if that is shorter, and logged as information rather than
as a warning. Other failures wait as before, after trying the next
cache if there is one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:08:19 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 9362ba9e34 Release 0.2.0
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:48:08 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 65f136048e Graceful restart (RFC 4724, RFC 8538) 
A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:43:47 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 acecc22f48 Status page: count rejected announcements, and name AS numbers 
The "refused" column counted rejected announcements since the start, not
routes, which made two peers that send the same routes look different
when one of them re-sends more often. It is now "rejected announcements",
with the held routes, the number that compares, beside it as "held now".
The page says "rejected" throughout, as the documentation and metrics do.

Rows are highlighted under the pointer, and pointing at an AS number
shows its holder's name, which the browser asks RIPEstat for. The daemon
looks nothing up; the monitoring page says what the browser sends where.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 22:43:36 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 efb2d81a50 Reject blackhole requests with a reason of their own
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A route with the BLACKHOLE community (RFC 7999) asks for the traffic to
its prefix to be dropped. Exchange route servers pass these on as host
routes with the exchange's blackhole address as next hop, and they were
rejected as "prefix too specific", which hid what they are.

Routes from EBGP neighbors with BLACKHOLE are now rejected as "blackhole
request", whatever their length. One short enough to pass would have
been taken as an ordinary route, sending the traffic to the blackhole
next hop: the request honoured without anyone having chosen to. Acting
on blackhole requests is not supported; the import policy page and the
standards list say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:03:16 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 bd674002a2 Keep the path attributes of rejected routes, and show them 
A rejected route was recorded with its prefix, reason and AS path, which
is not enough to tell why a neighbor sent it: whether a /32 is a
blackhole request, which communities a leaked route carries, which
network marked it Only to Customer.

Each recorded rejection now keeps the attributes of the UPDATE it came
in, without the NLRI of MP_REACH_NLRI and MP_UNREACH_NLRI, and its next
hop. The attributes are copied once per UPDATE, only when a route of it
is rejected, and shared by its rejected routes. A route rejected later,
when the RPKI data changes, has the attributes it was stored with.

/api/routes/rejected gives them as `summary` and `attributes`, as for
received routes, and the status page shows them under "show" in each
row of the refused routes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:03:16 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 e1d819d629 Add a script that builds a static Linux executable 
scripts/build-static.sh builds the daemon for x86_64 Linux against
musl, as one executable without dependencies to copy to a router. It
finds a musl C compiler for jemalloc (musl-cross on macOS, musl-gcc on
Linux), and uses its ar and ranlib too: with the host's, a build from
macOS gets an empty libjemalloc.a and fails to link. It prints the
revision and the SHA-256 of the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:03:16 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 361c48f656 Use jemalloc as the daemon's memory allocator 
A full table from several neighbors means millions of small, long-lived
allocations that are replaced as routes change. jemalloc keeps the
resident size of such a workload closer to what is in use than the
system allocator does, glibc's in particular, and does not depend on
the C library the binary is linked with. Not on MSVC, which jemalloc
does not support.

Building now needs a C compiler and make; the installation page says
so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:03:16 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 9ce81a873e Docs: show the front page photograph in its own colours 
The blue two-colour treatment is replaced by the photograph's own
colours, slightly darkened, under a neutral shade that keeps the text
legible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-03 21:35:36 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 dc275f99d7 Add a documentation website, built with Zola
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The documentation moves from READMEs into a site in docs/: getting
started, a user guide, operations, a reference, and development pages,
written in the style of network vendors' documentation (docs/STYLE.md).
bin/bgp/README.md and docs/RFCs.md are folded into it.

- A theme of its own in the International Typographic Style, in blues,
  with self-hosted Inter and JetBrains Mono, search over Zola's index,
  dark mode, and no resources loaded from other hosts.
- Example configurations in docs/examples/, shown on the pages and
  checked with `bgp --check` in CI.
- A test that fails when a field of config.proto is missing from the
  configuration reference.
- A CI job that builds the site; Zola fails on broken links and anchors.
- A hardened systemd unit, tried under systemd with kernel routes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-03 15:12:43 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 9820a5d3bc Log without colour codes when the output is not a terminal 
Under systemd the log goes to the journal, where ANSI escape codes are
noise. Colours are kept for a person at a terminal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-03 15:12:43 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 b0c94f6643 Release 0.1.0 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-03 11:33:59 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1e2e8f2d86 Pass learned routes on to peers, with marks and control communities 
What was to be scripted is configuration, done natively:

- import { add_communities, add_large_communities, remove_communities,
  local_pref }: every route taken from a peer is marked (where it was
  learned, say), with nobody else's word for our marks taken, and given
  its preference, which selection goes by.
- export { add_communities, add_large_communities, remove_communities,
  as_path_prepend, learned_routes }, for a peer or for all that have none
  of their own.
- control_communities: the communities by which routes say which peers
  they are passed on to, as exchanges' route servers have them (0:P, L:P,
  0:L, and the large forms L:0:P, L:1:P, L:0:0), taken off as the route
  goes on.

And what the last of these is for: with export.learned_routes a peer is
told the best route of every prefix the others announced. The rules are
BGP's and cannot be turned off: not back where it came from, not to a
peer whose AS is on the path, not between two internal peers, not past
NO_EXPORT or NO_ADVERTISE; our AS on the path and ourselves as next hop
to another network, LOCAL_PREF to our own; to a route server's client
the path, MED and next hop as they came (RFC 7947). Which routes go where
follows from the roles of RFC 9234: a route that came down or across goes
on down only, so that with roles on every peer this makes us transit for
our customers and nothing else. A peer without a role is told everything,
and the daemon says so at startup.

Each session follows the table's best routes with a reader and keeps the
prefixes it has announced, so that it can withdraw them. Routes that go
out alike go in one UPDATE: changes are gathered for 50 ms first, and a
table that is given out anew is gathered whole (at most four sessions at
a time; it is tens of megabytes). For that to work the table's journal
had to give out together what was written together: every change now
carries a time on a table-wide clock, a reader's pass reads every shard
up to one time, no further than the changes that are complete in every
shard, and a resync takes its places in the journals as of one time.

One peer sending 1.23 M routes over a session, another told of them:
in our table after 0.47 s, all heard after 2.1 s, four routes an UPDATE.
The lab has a phase, transit: FRR as our customer and BIRD as our
provider, routes each way with the right paths and marks, one kept back
by a community, and pings between their networks through us.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 23:48:08 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9367c38907 Remove policy scripts 
Rhai made a decision about a route cost one to three microseconds,
some two hundred times what the same costs in Rust. That was bearable
where a function is asked once for an UPDATE, and is not where every
route is weighed against every peer, as it is when routes are passed
on. What is wanted of a policy (mark routes by where they came from,
mark what is announced, let communities say who is told what) needs no
language; it is to be configuration, done natively.

Kept from that work: route selection goes by a LOCAL_PREF that policy
set on a route from an external peer, and the lab's configuration can
be added to by a phase.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 22:47:57 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 cbf0df1eef Policy scripts: the operator's own rules for routes, in Rhai 
A new crate, bgp-policy, runs functions of a script on routes: on their
way in, after the general checks of a peer's import and before the
checks that can change (RPKI, next hop reachability); and on their way
out to a peer. A function is given the route and the peer, may refuse
the route, and may change its communities, large communities,
LOCAL_PREF and MED, and on the way out prepend our AS:

    fn from_peer(route, peer) {
        if route.has_community("65535:666") { return false; }
        route.add_community(LEARNED_FROM[peer.role]);
        route.local_pref = PREFERENCE[peer.role];
        true
    }

Configuration: policy { path | source, import_function, export_function }
and, for a peer, import { function } and export { function }. The script
is compiled at startup; `bgp --check` reads it and says which function
is whose. A function that fails on a route has refused it. Scripts have
no files, network or clock, and are stopped after 200,000 steps.

Route selection now goes by a LOCAL_PREF on a route from an external
peer: the decoder never lets a peer's own through, so one that is there
was set by policy.

What it costs was measured before it was built on (benches/policy.rs):
a call that tests a few communities and changes two takes 1 to 3
microseconds, some two hundred times what Rust would. So a function is
asked once for the routes of an UPDATE, which share their attributes,
unless it asks for the prefix. One peer sending 1.23 M routes over a
session: 0.37 s without a function, 0.80 s with one that marks and
ranks every route, 3.4 s if it also looks at the prefix.

Export applies to what is announced, which today is what the daemon
originates; learned routes are not passed on yet.

Tested by unit tests of the crate, sessions end to end, a fuzz target on
the example script (testdata/policy.rhai), and a lab phase in which a
script changes which of FRR's and BIRD's route is chosen and what each
of them is told.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 21:09:39 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 f1d80c66df Do not choose a route whose next hop cannot be reached (RFC 4271 §9.1.2.1) 
Where routes go into the kernel, a route the kernel could not be given
is no longer taken into the table: its next hop is no neighbor on any
interface, or its prefix is a network the machine is itself on. Such a
route used to be able to win selection and then leave its prefix without
a route in the kernel, though another peer had one that worked.

It is held aside the way an RPKI-invalid route is, and by the same
means: judged on arrival against the interfaces as the kernel last
described them, looked for in the table when they change, and taken
from what is held when its next hop can be reached. The fib publishes
the interfaces for this; the next hop is resolved once for an UPDATE
and not for each of its routes.

The lab has a phase, nexthops: FRR's better route with a next hop that
is nowhere is held and BIRD's is installed, and they change places as
an address comes and goes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 20:29:31 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3d946c32ce Code health after review: docs, metrics, structure, one more fuzz target 
What the code-health review found, and what a fuzz target written for it
found in its first seconds.

- A set of attributes with the same attribute twice is refused by the
  routing table. Its readers (the selection key, the path, the next hop)
  could each take a different one of the two. The decoder never lets one
  through; the new fuzz target `attrs` found that the table would.
- bgp_peer_up{peer}, to alert on a peer going away, and bgp_build_info.
  Every metric is there from the first scrape. Alerts for a peer down,
  hijack attempts and kernel errors are in the README.
- `bgp --check config` validates a configuration without starting, and
  `--version` says which build it is. CI runs the first on the example
  configuration, which a test now also parses with its commented fields
  put in; the example had to learn about the next hop check.
- Path attributes have a module of their own, packet::attributes,
  re-exported from packet::message as before.
- policy::From, which shadowed the trait, is RouteSource. The server's
  connection, dialer and fsm modules are private. Logging arguments live
  in the binary; the server library no longer depends on clap.
- One is_unicast_link_local (the standard library's) instead of two of
  our own; SelectionKey::origin(); one place where an Import is made.
- READMEs say what the daemon does today rather than what it did not do
  a year ago. Documentation links are checked in CI. Unused dependencies
  and bgp_web::serve are gone; the Docker build is --locked and does not
  ship the fuzz corpus.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 19:31:19 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9ca86ede2f Harden import, the kernel task and the RTR client after review 
Three independent reviews (hijack resistance, reliability, code health)
of the whole tree. This is what they found that could let a bad route in
or take the daemon down.

Hijacks:
- A route had two paths on a two-octet session, AS_PATH and AS4_PATH,
  read in two places by two rules. A peer could make the origin checked
  on import differ from the one checked at revalidation, and get an
  RPKI-invalid route into the table. AS4_PATH is now merged into AS_PATH
  once, on receipt (RFC 6793 §4.2.3), and nothing else knows of it.
- The next hop is judged on import: it must be usable, agreed on
  (RFC 8950), and by default the peer itself.
- Our own originated prefixes are not taken from other networks.
- A route for a network the machine is on is never put in the kernel.
- Paths with an AS_SET are refused (RFC 9774).
- A configured peer gets one session, not one per BGP Identifier.
- More IPv6 special-purpose prefixes are bogons.

Reliability:
- A batch of requests the kernel refused could overflow the netlink
  socket and end the daemon. The socket has room now, a lost answer or a
  silent kernel means asking what is there, and neither is fatal.
- Working the routes out again within the grace period after a restart
  no longer drops the routes being kept.
- That work can no longer be put off for ever by interface changes.
- Peers whose sessions keep failing are slowed down.
- Unused attribute sets are swept when they pile up, not only each
  minute.
- Log lines a peer can cause are said once in a while.
- An RTR answer is bounded in size and time; data cannot expire between
  two refreshes.
- The shard of a key is taken from a stirred hash: neighbouring bits of
  the hash proved alike for prefixes in a row. The test that showed it,
  failing one run in sixteen, is deterministic now.

The lab has a phase, hijacks, in which FRR tries the first three.
route::apply is a struct with methods instead of nested closures, and
the kernel task's loop is shorter.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 19:13:40 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 fd2248feb5 Peer at link-local addresses: fe80::1%eth0 
A peer's endpoint may be a link-local IPv6 address with the interface it
is on. Such a peer is the address and the interface together, wherever
peers are told apart: admission, collisions, hold-downs, the refusal log,
the status page. The interface is looked up by name each time it is
needed, so one that appears later or is made anew is found.

Our routes go out with our link-local address in both places of
MP_REACH_NLRI's next hop, as FRR sends its own. A peer's link-local next
hop is taken alone, twice, or after "::" (BIRD), and resolved in the
kernel on the interface of the session rather than by looking fe80::/64
up, which every interface has.

The lab has a new phase, linklocal: sessions with FRR (by interface, with
radvd beside the daemon) and BIRD over a LAN with link-local addresses
only, both families, pings across, the interface down and up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 18:24:13 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 92e013ca01 Carry IPv4 routes over IPv6 sessions with IPv6 next hops (RFC 8950) 
For a link that has only IPv6 addresses.

- The Extended Next Hop capability is offered on every session over IPv6
  whose families include IPv4 unicast, and used if the peer offers it too.
  Nothing to configure.
- On such a session our IPv4 routes go out in MP_REACH_NLRI with the
  session's IPv6 address as next hop, and are withdrawn in
  MP_UNREACH_NLRI. Without the agreement they are not sent, as before.
- IPv4 routes are read with an IPv6 next hop, global or global with
  link-local.
- The kernel installer resolves an IPv4 route's IPv6 next hop like any
  other and installs it: by way of an IPv6 nexthop object, or with RTA_VIA
  where routes carry their own gateway.

Tested against a real kernel in both modes, with a link-local next hop
too, and in the lab against FRR and BIRD: the new `v6only` phase takes
the IPv4 addresses off the exchange LAN, runs one IPv6 session with each
router for both families, and pings IPv4 across.

Sessions between link-local addresses are not supported yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 17:58:03 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 321af51e4b Add an interoperability lab with FRR, BIRD and Routinator 
scripts/interop/lab.sh runs the daemon against FRR, BIRD and Routinator,
each in a network namespace on one exchange LAN, and checks what every
side ends up with: sessions and routes over IPv4 and IPv6 with their
attributes, four-octet AS numbers, ADD-PATH, RFC 9234 roles and Only to
Customer, RTR version 2 and invalid routes, route refresh, session resets,
killed peers, a cut wire, the daemon stopped and killed, a table of a
million routes and a restart under it, role mismatch and the prefix limit.
All of it passes against FRR 10.5.1, BIRD 2.18 and Routinator 0.15.2.

scripts/interop/real-rpki.sh checks the RTR client against a Routinator
with the real RPKI: 713,258 IPv4 and 187,479 IPv6 VRPs and 3,379 ASPAs,
the same numbers on both sides.

What the lab found, fixed here:

- A daemon restarted with its routes in the kernel removed them all at
  once, its table being empty until the peers had sent theirs again, and
  then put them back: no forwarding meanwhile. Routes of an earlier run
  are now kept for `stale_seconds` (120 by default) and only those nobody
  announced again are removed. A restart under a million routes takes
  1.1 s to be in step, with nothing removed.
- FRR ends a session it loses to a collision with a state machine error
  without a subcode, which was shown as "error code 5 subcode 0".

CI gets a job that runs the lab.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 17:44:49 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1de75fc610 Install a full table in the kernel in about a second 
A million IPv4 routes took 1.6 s to install (release build); they take
1.1 s now, and 0.5 s to remove instead of 1.15 s. Most of what is left is
the kernel's own work on each route.

- Nexthop objects (Linux 5.3 and later): a gateway is put in the kernel
  once and routes name it, so the kernel checks it once and not for every
  route. The daemon makes its own objects, uses nobody else's, is told
  when one of its own is removed and puts it back, and removes them when
  it stops, which takes all the routes that name them along at once.
  `without_nexthop_objects` turns this off; a kernel without them is
  noticed and served as before.
- Writing to the kernel is a task of its own, fed through a short queue,
  so the next batch is worked out while the kernel is busy with the last.
  "In step" is said when the kernel has done the batch that completes the
  table, not when it was handed over.
- What a next hop resolves to is kept, by next hop: it was looked up among
  the machine's addresses for every route.
- A quicker hash for the plan's maps, and no lookups in the ones that are
  empty.

Fixes a hole found on the way: when an interface went down and came back
within the second it takes us to react, the kernel had dropped our routes
and we did not put them back, since nothing about them had changed for
us. After such a change the kernel is now asked what it actually has.

scripts/in-namespace.sh runs one command in a scratch namespace;
FERRIX_ROUTES sets the size of the timing test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 17:13:44 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1fc44798b7 Install best routes in the Linux kernel over netlink 
New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.

- netlink: the rtnetlink messages used (routes, addresses, links, errors),
  as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
  kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
  routes of ours found at startup are kept if still right and removed if
  not; a change of interfaces has every route worked out again; a clean
  stop removes our routes.

Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.

Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.

/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.

Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 15:15:43 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1f84235e87 Page through the received routes, on the status page and in the API 
/api/routes/received now returns one page: {routes, next_after,
prev_before}. Pages are in prefix order and begin at a prefix (after= or
before=), not at a count, so a table that changes between two pages does
not make them overlap or leave a gap. A page holds whole prefixes, about
`limit` routes' worth.

The table keeps prefixes in no order, so a page is found by looking at
every prefix once and keeping the nearest few; the routes of the others
are not touched (RibTable::for_each_prefix).

The status page shows about 100 routes with first / previous / next
links; the place is kept in the page's address with the lookup.

The response of /api/routes/received was an array before: this changes
its shape.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 14:33:38 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 f9ef781d51 Check roles in OPEN and prevent route leaks with Only to Customer (RFC 9234) 
A peer's configured role now does three more things:

- Our OPEN carries the BGP Role capability with what we are to the peer.
  A peer whose own says it is something that does not pair with that, or
  that sends two different roles, is refused with OPEN Message Error /
  Role Mismatch. A peer that says nothing is accepted unless require_role
  is set. A role on a peer in our own AS is a configuration error.
- Routes from a provider, a peer or a route server are marked Only to
  Customer on the way in if they are not already, and our routes are
  marked on the way out to a customer, a peer or a route server client.
- A marked route from a customer or route server client, or from a peer
  with someone else's mark, is a leak and is refused. No policy setting
  turns that off.

The packet crate gains the Only to Customer attribute (malformed:
treat-as-withdraw), and the Role Mismatch subcode is corrected from 8 to
11.

The status page shows whether the peer confirmed its role.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 14:27:36 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 8086aa994f Verify AS paths with ASPAs, over RTR version 2 
Origin validation says who may originate a prefix and nothing about the
rest of the path: it does not catch a route leak, nor a hijack that keeps
the right origin at the end of a made-up path. ASPAs, in which a network
lists its providers, do.

bgp-rpki:
- aspa: the ASPAs and the upstream and downstream verification procedures
  of draft-ietf-sidrops-aspa-verification-24, tested against the draft's
  wording as written.
- rtr: protocol version 2 (draft-ietf-sidrops-8210bis-23) with the ASPA
  PDU and the new error codes.
- client: asks for version 2 and falls back to whatever older version the
  cache speaks; keeps ASPAs with the VRPs, applied whole or not at all;
  publishes which customers' ASPAs changed; tells the cache why it is
  being left (Error Report); does not ask one version's serial of another.

Server:
- A peer has a role (provider, customer, peer, route server, route server
  client), which decides how its paths are verified. Without one they are
  not.
- A route whose path is invalid is refused ("ASPA invalid") and held, like
  one with an invalid origin; accept_aspa_invalid exempts a peer.
- When ASPAs change, routes in the table whose path has a changed customer
  on it are verified again, and held routes are released when they pass
  both checks.
- The revalidator finds a route's peer by index instead of searching a
  list for every route.

The status page and API show each peer's role, each route's path validity,
the number of ASPAs and the RTR version in use; bgp_rpki_aspas is on
/metricz.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 14:12:59 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 bdaf226a16 Validate route origins against the RPKI and refuse invalid routes 
With rpki { caches { ... } } in the configuration the server keeps the
validated ROA payloads of a validator and validates every route from a
peer outside its own AS (RFC 6811). An invalid route is refused, unless
its peer has accept_rpki_invalid. Valid and not-found routes are taken;
validity does not otherwise enter route selection.

Validity is not settled when a route arrives: the data changes every few
minutes. The routes follow, without the peer sending anything again.

- A route refused as invalid is held by its session, out of the table.
  When the data changes the session looks at what it holds, and what is
  no longer invalid goes into the table. A held route is the peer's to
  withdraw or replace like any other.
- A route in the table that becomes invalid is found by a task that
  looks through the table when the data changes, at the prefixes the
  changed VRPs cover, and is taken out and held by its session.
- If the data expires because no cache confirms it, nothing is invalid
  any more and everything held is taken.
- A session holds at most 200,000 routes; past that it asks the peer for
  a route refresh when the data changes.

So that the table can be looked through without parsing attributes, an
attribute set's key now has the route's origin AS. Memory per route is
unchanged at 52 bytes on the real table.

Shown: /api/rpki and a section of the status page with the cache, its
state and the age of the data; each route's validity; each peer's count
of held routes. Refusals are recorded under "RPKI invalid". Metrics:
bgp_rpki_vrps and bgp_rpki_updates_total.

Also: Negotiated says whether the peer supports route refresh, and
docs/RFCs.md is brought up to date.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 13:39:40 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1f89fb4b13 Add the bgp-rpki crate: route origin validation and an RTR client 
The first part of RPKI origin validation (RFC 6811): the data and how it
gets here. Nothing uses it yet.

VrpTable holds the validated ROA payloads and says whether a route is
valid, invalid or not found. It is a binary trie in one allocation with
a short cut past the first sixteen levels; a lookup against 600,000
entries takes about 40 ns for random addresses. A table is built once
and not changed: when the data changes another takes its place.

rtr is the RPKI-to-Router protocol's PDUs (RFC 8210, and version 0 of
RFC 6810), as bytes and back, without I/O. Lengths are checked from the
header, before the rest of a PDU is waited for.

The client keeps the table in step with a validator's cache. It asks
for everything once and then for changes, when the cache notifies and
at the refresh interval. An answer takes effect only when it is complete
and consistent. Each update says which VRPs came or went, so that only
the routes they cover need looking at again. A cache that fails is
retried and the next one tried; the data is used meanwhile, and dropped
if no cache confirms it within the expire interval. A cache that speaks
only version 0 is spoken to in version 0.

Tests: the cases of RFC 6811, a property test of the trie against the
plain definition, round-trip and hostile-input properties for the
codec, a fuzz target, and the client against a scripted cache.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 13:28:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 599cd0ad8b Look routes up, and see and alert on the ones that were refused 
Two things were hard to find out from the status page and the API: what
the table has for a given prefix or address, and what a peer announced
that was refused and why.

Looking up. /api/routes/received takes a query, and the status page has
a box for it:
- prefix: every route for that prefix, the best first;
- address: every prefix that covers the address, most specific first;
- within: a prefix and everything inside it;
- peer (by name or address) and best narrow any of these.
A query that makes no sense is answered 400 with the reason, which the
page shows. Results come in address order. Routes are shown with the
peer's configured name.

Refused routes. A refused route is not stored, so until now there was
only a count. For each peer the server now keeps how many routes were
refused for each reason, and the latest 200 with the AS path they came
with. The record is kept by the peer's address and outlives the
session, so that a peer that was cut off can still be looked into. It is
at /api/routes/rejected and in a new section of the status page; the
peers table links to it.

Alerting. bgp_routes_rejected_total is now by peer as well as reason.
New: bgp_peer_routes and bgp_peer_max_prefixes, to see a peer nearing
its limit, and bgp_prefix_limit_exceeded_total. The README has rules to
start from. The log has a line for a peer's first refused route and
then at most one a minute, with the count of the others.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 13:14:41 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e2b1979e38 Judge every route a peer announces: import policy, prefix limit, hold-down 
A configured peer could announce any prefix with any path, in any
number, and all of it was stored and could be chosen as best. Each
peer now has an import policy, and every route goes through it before
it reaches the routing table.

Checks that hold for any peer unless turned off for it:
- no prefixes that are not routed on the Internet (private, loopback,
  link-local, documentation, multicast, reserved; for IPv6, anything
  outside global unicast);
- nothing longer than /24 or /48, no default route, nothing shorter
  than /8 or /16;
- no reserved, private or documentation AS number in the path;
- a path that begins with the peer's own AS (turned off for a route
  server), and of at most 64 AS numbers.

Always: a route with our own AS number in its path is refused. RFC 4271
requires this loop check and it was missing.

Per peer, if configured: the prefixes it may announce, each with an
optional longest more-specific, and the AS numbers its routes may
originate from. This is the list generated from the routing registries.

max_prefixes limits how many routes a peer may have. One more ends the
session with Cease / Maximum Number of Prefixes Reached, and the peer
is kept down for hold_down_seconds (ten minutes unless set): its
connections are not taken and it is not dialled. Refused routes do not
count towards the limit.

A refused route is withdrawn rather than ignored, so that it replaces
whatever the peer had announced for it before.

On a session with two-octet AS numbers the path is judged with its
AS4_PATH merged in, so that the real AS numbers are what is checked.

Peers that are not configured (accept_unconfigured_peers, for test
networks) get a policy that takes everything but loops.

Refusals are counted in bgp_routes_rejected_total by reason. The status
page and /api/peers show, for each session, how many routes the peer
has and how many were refused. The server warns at startup about peers
with no limit.

The prefix check costs about 4 ns a route with the default policy; the
end-to-end benchmark is unchanged.

Also: the benchmark's server AS is one that is in none of its generated
paths, and a table test that depended on the hasher's seed is fixed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 12:59:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 630c175918 Rework the speaker: sans-I/O FSM, sharded table, RFC 7606, ADD-PATH 
The work of the speaker-cleanup branch as one commit; the branch keeps
the step-by-step history.

Sessions
- The state machine is a pure function from events to actions, driven by
  one task per connection. Idle, Connect and Active belong to a dialer
  that keeps configured peers up with backoff.
- Capabilities are negotiated: address families, four-octet AS numbers
  with AS_TRANS and AS4_PATH, route refresh, ADD-PATH per family and
  direction.
- Collision detection per RFC 4271 section 6.8, by address and
  identifier.
- Connections are taken only from configured peers, a few per address.
- A panic in a session is contained and answered with a Cease; a core
  task that stops shuts the server down with a Cease to every peer.
- Writes that make no progress are given up (RFC 9687).

Codec
- Malformed UPDATEs are handled as RFC 7606 says: treat-as-withdraw,
  attribute discard, or session reset, with the NOTIFICATION to send.
- Address families are open types; any can be named and ignored.
- Prefixes are fixed-size values, normalised, with path identifiers.
- Lengths are filled in after writing; one definition of each type code.
- Property tests and fuzz targets for every decoder.

Routing table
- Adj-RIBs-In and Loc-RIB in one sharded table that sessions write to
  directly. Routes are sixteen bytes; attribute sets are interned in
  wire form. Best-path selection per RFC 4271 section 9.1.2.2 with
  deterministic MED.
- A journal per shard lets followers read changes at their own pace;
  one that falls behind is resynchronised.
- On a RIPE RIS dump of 148 peers and 39.3M routes: loaded in 2.0 s on
  12 cores, about 52 bytes a route.

Announcing
- Originated routes come from the configuration. Each session has its
  own exporter with an Adj-RIB-Out, so a slow peer delays only itself.
- Routes to a peer in our own AS have an empty path and a LOCAL_PREF.

Operations
- HTTP status page, JSON API, event stream and Prometheus metrics.
- Benchmarks on synthetic and real tables.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 12:35:54 +02:00
268 changed files with 64073 additions and 3509 deletions
Expand all files Collapse all files

No files matched your search

.dockerignore
+7
View File
@@ -0,0 +1,7 @@
target
.git
.claude
fuzz/target
fuzz/corpus
fuzz/artifacts
crates/bench/data
.github/workflows/ci.yml
+99
View File
@@ -0,0 +1,99 @@
# Runs on every push and pull request. Gitea Actions reads this file too
# (it falls back to .github/workflows), so one definition serves both.
name: ci
on:
push:
pull_request:
env:
CARGO_TERM_COLOR: always
# Warnings fail the build here, and only here: locally they stay warnings
# so that work in progress compiles.
RUSTFLAGS: -D warnings
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- name: Format
run: cargo fmt --all -- --check
- name: Lint
run: cargo clippy --locked --workspace --all-targets -- -D warnings
# The memory-counting allocator of the benchmarks is behind a feature
# that the workspace build does not turn on.
- name: Lint the benchmarks with memory counting
run: cargo clippy --locked -p bgp-bench --all-targets --features count-memory -- -D warnings
- name: Test
run: cargo test --locked --workspace
# A link in the documentation to something that is not there.
- name: Documentation
env:
RUSTDOCFLAGS: -D warnings
run: cargo doc --locked --workspace --no-deps --document-private-items
# The examples operators start from, here and in the documentation,
# through the daemon itself.
- name: Example configurations
run: |
for config in testdata/server.textpb docs/examples/*.textpb; do
cargo run --locked -p bgp -- --check "$config"
done
# The tests that change a kernel's routes, in a network namespace of
# their own; and two daemons that route for each other.
- name: Kernel tests
run: scripts/kernel-tests.sh
# The documentation site. Zola fails the build on a link to a page or an
# anchor that is not there.
docs:
runs-on: ubuntu-latest
env:
ZOLA_VERSION: "0.22.1"
steps:
- uses: actions/checkout@v4
- name: Install Zola
run: |
curl -sSfL "https://github.com/getzola/zola/releases/download/v${ZOLA_VERSION}/zola-v${ZOLA_VERSION}-x86_64-unknown-linux-gnu.tar.gz" \
| sudo tar -xz -C /usr/local/bin zola
- name: Build
run: zola --root docs build
# The daemon against FRR, BIRD and Routinator; see
# docs/content/docs/development/testing.md.
interop:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- name: Install FRR, BIRD and Routinator
run: scripts/interop/install.sh
- name: Build
run: cargo build --release --locked -p bgp
- name: Interoperability lab
run: scripts/interop/lab.sh
# A short run of each fuzz target: enough to notice a parser change that
# breaks on input the seeds are close to. Long runs are done by hand; see
# fuzz/README.md.
fuzz:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
- name: Install cargo-fuzz
run: cargo install cargo-fuzz --locked
- name: Fuzz
working-directory: fuzz
# cargo-fuzz sets its own flags; the workspace's -D warnings would
# replace them.
env:
RUSTFLAGS: ""
run: |
for target in message message_body attrs prefix_text config rtr netlink; do
mkdir -p "corpus/$target"
cargo fuzz run "$target" "corpus/$target" "seeds/$target" -- -max_total_time=60
done
.gitignore
+4
View File
@@ -1 +1,5 @@
/target
.DS_Store
.claude/
/docs/public
/docs/static/giallo.css
Cargo.lock
Generated
+1168 -94
View File
File diff suppressed because it is too large. Load diff
Cargo.toml
+38 -3
View File
@@ -1,6 +1,8 @@
[workspace]
default-members = ["bin/bgp"]
members = ["bin/*", "crates/*"]
# Fuzz targets need a nightly toolchain; see fuzz/README.md.
exclude = ["fuzz"]
resolver = "3"
[workspace.package]
@@ -9,17 +11,31 @@ description = "A Border Gateway Protocol implementation"
edition = "2024"
homepage = "https://rayhaan.ch"
license = "Apache-2.0"
name = "bgp"
repository = "https://git.rayhaan.net/rayhaan/bgp"
version = "0.0.2"
repository = "https://git.rayhaan.net/rayhaan/ferrix"
version = "0.2.6"
# Every crate opts in with `[lints] workspace = true`. CI turns warnings into
# errors, so a lint added here is enforced.
[workspace.lints.rust]
unsafe_code = "forbid"
[workspace.lints.clippy]
all = { level = "warn", priority = -1 }
dbg_macro = "warn"
todo = "warn"
unimplemented = "warn"
[workspace.dependencies]
# --- Our crates ---
bgp-fib = { path = "crates/fib" }
bgp-packet = { path = "crates/packet" }
bgp-primitives = { path = "crates/primitives" }
bgp-rib = { path = "crates/rib" }
bgp-rpki = { path = "crates/rpki" }
bgp-server = { path = "crates/server" }
bgp-web = { path = "crates/web" }
bgp-integration-tests = { path = "crates/integration-tests" }
bgp-bench = { path = "crates/bench" }
# --- General ---
bitfield = "0.19.0"
@@ -27,6 +43,8 @@ bytes = "1.10.1"
eyre = "0.6.12"
nom = "8.0.0"
serde = { version = "1.0.219", features = ["derive"] }
nix = { version = "0.31", default-features = false, features = ["net"] }
socket2 = "0.6"
serde_json = "1.0.140"
serde_repr = "0.1.20"
thiserror = "2.0.12"
@@ -35,7 +53,24 @@ strum = { version = "0.27.1", features = ["derive"] }
tracing = "0.1.44"
tracing-subscriber = { version = "0.3.22", features = ["env-filter"] }
prometheus = "0.14.0"
proptest = "1.7"
protobuf = { package = "protobuf", version = "3.7.2" }
protobuf-codegen = { package = "protobuf-codegen", version = "3.7.2" }
clap = { version = "4.5.53", features = ["derive"] }
foldhash = "0.2"
futures = "0.3"
hashbrown = "0.17"
axum = "0.8.9"
tokio-stream = { version = "0.1.19", features = ["sync"] }
reqwest = { version = "0.13.5", default-features = false, features = ["json"] }
# The daemon's memory allocator; see bin/bgp/src/main.rs. With its
# statistics, which the status page and the metrics show.
tikv-jemallocator = { version = "0.7.0", features = ["stats"] }
tikv-jemalloc-ctl = { version = "0.7.0", features = ["stats"] }
# `cargo build --profile profiling`: the release build, with the line tables
# a profiler needs to name functions. See scripts/profile/.
[profile.profiling]
inherits = "release"
debug = "line-tables-only"
strip = "none"
Dockerfile
+11
View File
@@ -0,0 +1,11 @@
# Builds the daemon into a small image. Mount a configuration file at
# /etc/ferrix/server.textpb, or pass another path as the first argument.
FROM rust:1.92-bookworm AS build
WORKDIR /src
COPY . .
RUN cargo build --release --locked -p bgp
FROM debian:bookworm-slim
COPY --from=build /src/target/release/bgp /usr/local/bin/bgp
ENTRYPOINT ["/usr/local/bin/bgp"]
CMD ["/etc/ferrix/server.textpb", "--level", "info"]
LICENSE
+176
View File
@@ -0,0 +1,176 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
README.md
+185 -19
View File
@@ -1,27 +1,193 @@
# BGP
# Ferrix BGP
The `bgp` crate is an implementation of the Border Gateway Protocol, which is used to exchange routing and control plane information between Autonomous Systms.
Ferrix is a BGP daemon for Linux routers, written in Rust. It exchanges
routes with neighbors, filters them by policy and the RPKI, selects the
best path for each prefix, installs it in the kernel, and passes routes on
as BGP roles and communities permit. It aims to be small, safe, observable,
and configured from one file that can be read by someone who did not write
it.
The aim of this project is to provide a modern, high performance, secure and programatic and customizable software package that can be useful to network operators, researchers, and people interested in Internet.
## Status
## Current status.
Ferrix routes the author's network, AS210036: three routers carrying full
tables from transit providers, route servers at several Internet exchanges,
and bilateral peers, with routes reflected between the routers over iBGP.
It is young software all the same. Read the [limitations](#not-supported)
before relying on it, and treat version `0.x` as meaning that configuration
and APIs may still change between releases.
The project is currently being rewritten from scratch to be more modular and maintainable, as well as to fix deeper issues with the previous implementation.
What it does:
⚠️ Warning: This is currently a work in progress and not ready for use.
- BGP-4 over IPv4 and IPv6, with four-octet AS numbers, multiprotocol
reach for both families, IPv4 routes with IPv6 next hops (RFC 8950),
route refresh, graceful restart and ADD-PATH receive.
- Import policy per neighbor: prefix limits, prefix and origin AS lists,
bogon checks, AS path checks, and communities added on the way in.
- Route origin validation and ASPA path verification against an RTR cache.
- BGP roles (RFC 9234) and the Only to Customer attribute, with export
limits as a second line of defence against route leaks.
- Best path selection and installation of the result in the Linux kernel
over netlink, in a routing table of your choice.
- Route reflection between routers of the same AS, and passive neighbors.
- A status page, a JSON API, server-sent events, and Prometheus metrics.
- Graceful restart of the daemon itself for upgrades, with routes left in
the kernel while it is away.
### 🛣️ Roadmap to v1.0
### Not supported
We aim for the 1.0 release to be functional and stable to operate peering sessions, and to install routes in the Linux Kernel.
- Per-client best path selection for route servers, and ADD-PATH for
advertised routes
- TCP-AO, TCP MD5, and GTSM
- Long-Lived Graceful Restart, BFD, and configuration changes without a
restart
- Address families other than IPv4 and IPv6 unicast
- ☐ Implement basic parsers for the fundamental types required
- ✅ Path Attributes from RFC4271
- ✅ Open message
- ✅ Capabilities from RFC4271
- ✅ Update message
- ☐ Notification message
- ☐ Implement scaffolding required to start and configure a server
- ☐ Implement connection logic to establish sessions with peers
- ☐ Implement an efficient data structure for storing prefixes (e.g. Poptrie)
- ☐ Filtering for peer RIB in / out
- ☐ Main / auxillary RIBs for exchanging routes between peers
## Building
You need a stable Rust toolchain from [rustup.rs](https://rustup.rs), and a
C compiler and `make` for the jemalloc allocator the daemon uses (for
example `build-essential` on Debian and Ubuntu, `base-devel` on Arch Linux,
or the Xcode command line tools on macOS). The Protocol Buffers compiler is
not needed: the configuration schema is compiled in pure Rust.
```bash
git clone https://git.rayhaan.net/rayhaan/ferrix.git
cd ferrix
cargo build --release --locked -p bgp
```
The binary is `target/release/bgp`. `--locked` builds with the dependency
versions recorded in `Cargo.lock`, which are the versions CI tested.
To try it without touching any router, run the quick-start configuration.
It listens on loopback, needs no privileges, and serves the status page at
<http://127.0.0.1:8080>; the [quick start
guide](docs/content/docs/getting-started/quick-start.md) adds a test peer
that announces routes to it.
```bash
target/release/bgp --check docs/examples/quick-start.textpb
target/release/bgp docs/examples/quick-start.textpb
```
Other ways to build:
- **A static Linux executable** that runs on any x86_64 distribution with
nothing else installed: `scripts/build-static.sh`. It needs the
`x86_64-unknown-linux-musl` target and a musl C compiler; the script's
header says how to install them, including on macOS.
- **A container image** with the provided `Dockerfile`:
`docker build -t ferrix .`
Kernel route installation needs Linux and `CAP_NET_ADMIN`; everywhere else
Ferrix is a BGP speaker only. The [installation
guide](docs/content/docs/getting-started/installation.md) and the
[deployment guide](docs/content/docs/operations/deployment.md) cover
privileges, the systemd unit, and upgrading without withdrawing routes.
## Documentation
The documentation is a website built from [`docs/`](docs/) with
[Zola](https://www.getzola.org): `zola --root docs serve`, then open
<http://127.0.0.1:1111>. The sources are Markdown with Zola front matter,
shortcodes and internal links, so the forge shows them with those parts
unrendered; [`docs/README.md`](docs/README.md) explains what each is. The
pages:
- [Overview](docs/content/docs/getting-started/introduction.md),
[installation](docs/content/docs/getting-started/installation.md) and
[quick start](docs/content/docs/getting-started/quick-start.md)
- User guide: [BGP neighbors](docs/content/docs/guides/peers.md),
[import policy](docs/content/docs/guides/import-policy.md),
[BGP roles](docs/content/docs/guides/roles.md),
[RPKI](docs/content/docs/guides/rpki.md),
[kernel route installation](docs/content/docs/guides/kernel.md),
[route advertisement](docs/content/docs/guides/passing-routes-on.md),
[graceful restart](docs/content/docs/guides/graceful-restart.md),
[configuration examples](docs/content/docs/guides/examples.md)
- [Deployment](docs/content/docs/operations/deployment.md) and
[monitoring](docs/content/docs/operations/monitoring.md)
- Reference: [configuration](docs/content/docs/reference/configuration.md),
[command line](docs/content/docs/reference/command-line.md),
[HTTP API](docs/content/docs/reference/http-api.md),
[metrics](docs/content/docs/reference/metrics.md),
[supported standards](docs/content/docs/reference/rfcs.md)
- [Architecture](docs/content/docs/development/architecture.md),
[testing](docs/content/docs/development/testing.md) and
[contributing](docs/content/docs/development/contributing.md)
Complete configurations for an edge router, an exchange router and a
transit customer are in [`docs/examples/`](docs/examples/); CI checks that
each of them loads.
## Security
- The status page and HTTP API are not authenticated and show the whole
configuration and routing table. Bind `http_endpoint` to a loopback
address, or put it behind a proxy that authenticates.
- Ferrix does not implement TCP-MD5 or TCP-AO. Filter TCP port 179 in the
host firewall so that only configured neighbors can connect.
- Everything read from the network, the RPKI cache, and the kernel is
parsed without panicking; each parser has property tests and a fuzz
target. `unsafe` code is forbidden throughout.
If you find a vulnerability, please contact the maintainer directly rather
than opening a public issue.
## Layout
| Crate | What is in it |
| -------------------------- | -------------------------------------------------------------------------- |
| `crates/primitives` | Prefixes, address families, parsing helpers |
| `crates/packet` | The wire format: messages, attributes, capabilities |
| `crates/rib` | The routing table, best route selection, the journal of changes |
| `crates/rpki` | The RTR client; origin validation and ASPA path verification |
| `crates/fib` | The best routes in the Linux kernel, over netlink |
| `crates/server` | Sessions, the state machine, policy, configuration |
| `crates/web` | The status page, HTTP API and metrics |
| `crates/integration-tests` | A test server and test peers, and the tests that run the daemon end to end |
| `crates/bench` | Benchmarks at the scale of the Internet routing table |
| `bin/bgp` | The daemon |
## Development
CI runs these on every push, and fails on any warning:
```bash
cargo fmt --all -- --check
cargo clippy --locked --workspace --all-targets -- -D warnings
cargo test --locked --workspace
RUSTDOCFLAGS="-D warnings" cargo doc --locked --workspace --no-deps --document-private-items
zola --root docs build
```
It also runs the tests that need a real Linux kernel
(`scripts/kernel-tests.sh`) and a lab that peers the daemon with FRR, BIRD
and Routinator (`scripts/interop/lab.sh`); `scripts/linux-vm.sh` runs
either from macOS in a Lima VM.
`cargo test` includes property tests: generated messages must survive being
written and read back, and no input, however damaged, may make a parser
panic. They take about a second; for a longer hunt:
```bash
PROPTEST_CASES=1000000 cargo test --release -p bgp-packet --test properties
```
Coverage-guided fuzz targets for the same parsers are in `fuzz/`; see
[`fuzz/README.md`](fuzz/README.md). CI runs each of them for a minute.
### Development with AI agents
Much of Ferrix is written with an AI coding agent (Claude Code) working
alongside the maintainer, who directs the work, reviews every change, and
runs the result on real routers. Nothing is merged on the agent's say-so:
every change passes the checks above, the kernel tests and the
interoperability lab, and the changes that matter are tried on the network
first. Commits made with an agent carry a `Co-Authored-By` trailer that
says so. If you contribute, you are welcome to work the same way, or not;
what is reviewed is the change, not how it was produced.
## License
Apache License 2.0; see [`LICENSE`](LICENSE).
bin/bgp/Cargo.toml
+34 -7
View File
@@ -1,11 +1,38 @@
[package]
description = "A Border Gateway Protocol implementation"
edition.workspace = true
name = "bgp"
description = "A Border Gateway Protocol implementation"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
name = "bgp"
version.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
eyre.workspace = true
tokio.workspace = true
bgp-server.workspace = true
bgp-web.workspace = true
clap.workspace = true
eyre.workspace = true
protobuf.workspace = true
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
# jemalloc does not build with the MSVC toolchain; there the system
# allocator is used.
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator.workspace = true
# jemalloc's background thread returns freed memory to the kernel on time
# (see the memory figures on the status page); jemalloc has it on Linux
# only, and warns at every start elsewhere if asked for it.
[target.'cfg(target_os = "linux")'.dependencies]
tikv-jemallocator = { workspace = true, features = ["background_threads"] }
[features]
# jemalloc's heap profiler, for scripts/profile/full-table.sh with
# HEAP=1: not for a production build, where it costs memory and time.
heap-profiling = ["tikv-jemallocator/profiling"]
[lints]
workspace = true
bin/bgp/README.md
+13 -5
View File
@@ -1,9 +1,17 @@
# BGP
# bgp
This crate provides a binary which runs the Border Gateway Protocol (BGP).
The daemon: the BGP speaker, the RTR client, the kernel route installer and
the status page, in one binary.
Please refer to the other implementation crates part of this repository where most of the actual implemntation lives.
```
cargo run -p bgp -- --check docs/examples/quick-start.textpb
cargo run -p bgp -- docs/examples/quick-start.textpb
```
Warning: This is still very much a work in progress and is not yet ready for use.
How to configure, run and operate it is in the documentation, which is
built from `docs/` (`zola --root docs serve`):
The main ongoing work is happening currently in the BGP packet crate.
- [Quick start](../../docs/content/docs/getting-started/quick-start.md)
- [Configuration reference](../../docs/content/docs/reference/configuration.md)
- [Deployment](../../docs/content/docs/operations/deployment.md) and
[monitoring](../../docs/content/docs/operations/monitoring.md)
bin/bgp/src/logging.rs
+36
View File
@@ -0,0 +1,36 @@
//! What the daemon logs, and how much.
use std::io::IsTerminal;
use clap::Args;
use tracing::Level;
use tracing_subscriber::EnvFilter;
use tracing_subscriber::filter::ParseError;
/// What the level given on the command line is for: this program and the
/// crates it is made of, which all begin with `bgp`. Everything else (the
/// HTTP server, say) stays at what `RUST_LOG` says, or silent.
const OURS: &str = "bgp";
#[derive(Debug, Args)]
pub struct LoggingArgs {
/// Set the minimum log level. At `debug` and below there is a line for
/// every UPDATE and every route announced.
#[arg(short, long, global = true, default_value_t = Level::INFO, help_heading = "Logging")]
level: Level,
}
impl LoggingArgs {
pub fn install(&self) -> Result<(), ParseError> {
let filter = EnvFilter::builder()
.from_env_lossy()
.add_directive(format!("{OURS}={}", self.level).parse()?);
// Colours for a person at a terminal; plain text for the journal
// or a file, where escape codes are noise.
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_ansi(std::io::stdout().is_terminal())
.init();
Ok(())
}
}
bin/bgp/src/main.rs
+187 -3
View File
@@ -1,6 +1,190 @@
use eyre::Result;
use std::path::PathBuf;
use std::process::ExitCode;
use std::sync::Arc;
use bgp_server::config::Config;
use bgp_server::proto::config::BgpConfig;
use bgp_server::server::{BgpServer, leave_for_restart, shut_down};
use bgp_web::{AppState, CommunityName, RouterInfo};
use clap::Parser;
use eyre::{Context, Result};
use tracing::{error, info};
use crate::logging::LoggingArgs;
mod logging;
// jemalloc, rather than the C library's allocator. Taking in a full table
// allocates from many threads at once, which glibc's allocator handles
// less well and musl's, in the static build, much worse.
#[cfg(not(target_env = "msvc"))]
#[global_allocator]
static ALLOCATOR: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
/// The exit status after leaving to be started again (SIGUSR1):
/// EX_TEMPFAIL, on which the systemd unit starts the daemon again
/// (`RestartForceExitStatus=75`).
const RESTART_EXIT_STATUS: u8 = 75;
/// A BGP speaker.
#[derive(Parser, Debug)]
#[command(version)]
struct ServerArgs {
/// Path to the textproto configuration file.
config: PathBuf,
/// Read the configuration, say whether it is in order, and stop:
/// nothing is listened on and no route is touched.
#[arg(long)]
check: bool,
#[command(flatten)]
logging: LoggingArgs,
}
#[tokio::main]
pub async fn main() -> Result<()> {
todo!("Implement the server");
async fn main() -> Result<ExitCode> {
let args = ServerArgs::parse();
args.logging.install()?;
log_panics();
let config_string = std::fs::read_to_string(&args.config)
.wrap_err_with(|| format!("Failed to read {}", args.config.display()))?;
let proto_config = protobuf::text_format::parse_from_str::<BgpConfig>(&config_string)
.wrap_err("Failed to parse config file")?;
let config = Config::from_proto(proto_config).wrap_err("Invalid configuration")?;
if args.check {
println!(
"{} is in order: AS{}, peers: {}, routes originated: {}",
args.config.display(),
config.local_asn,
config.peers.len(),
config.originate.len()
);
return Ok(ExitCode::SUCCESS);
}
let server = BgpServer::bind(config).await?;
if let Some(http_endpoint) = server.config().http_endpoint {
// Bound here, so that an address that cannot be bound stops the
// daemon from starting, as it does for the BGP listeners.
let listener = tokio::net::TcpListener::bind(http_endpoint)
.await
.wrap_err_with(|| format!("Failed to bind HTTP listener on {http_endpoint}"))?;
let state = AppState {
info: Arc::new(
RouterInfo::new(
server.config().router_id,
server.config().local_asn,
env!("CARGO_PKG_VERSION"),
)
.with_config(args.config.to_string_lossy(), &config_string)
.with_control_communities(server.config().control.is_some())
.with_community_names(
server
.config()
.community_names
.iter()
.map(|(community, name)| CommunityName {
community: community.clone(),
name: name.clone(),
})
.collect(),
),
),
registry: server.registry.clone(),
local_rib: server.local_rib.clone(),
rib: Arc::clone(&server.rib),
rejections: server.rejections.clone(),
rpki: server.rpki.clone(),
kernel: server.kernel.clone(),
restarts: Arc::clone(&server.restarts),
};
tokio::spawn(async move {
if let Err(e) = bgp_web::serve_on(state, listener).await {
error!(err = %e, "HTTP server exited");
}
});
}
// Leave politely: tell every peer we are going (Cease / Administrative
// Shutdown) rather than letting them find a dead TCP connection, which
// makes them slower to take us back.
// And take our routes out of the kernel: with us gone nothing would
// keep them true.
// Unless we are to be back at once (SIGUSR1, for an upgrade): then the
// routes stay, and peers that agreed to graceful restart keep ours.
let (registry, kernel) = (server.registry.clone(), server.kernel.clone());
tokio::select! {
result = server.run() => {
result?;
Ok(ExitCode::SUCCESS)
}
signal = signalled() => match signal {
Signal::Stop(name) => {
info!(signal = name, "Shutting down, closing BGP sessions");
shut_down(&registry, kernel.as_ref()).await;
Ok(ExitCode::SUCCESS)
}
Signal::Restart => {
info!(
signal = "SIGUSR1",
"Leaving to be started again: routes stay in the kernel, and peers keep ours"
);
leave_for_restart(&registry, kernel.as_ref()).await;
Ok(ExitCode::from(RESTART_EXIT_STATUS))
}
}
}
}
/// Sends panics to the log, with where they happened. A panic is a bug, and
/// the default report goes to stderr in a format of its own, apart from
/// everything else the daemon says. The task that panicked still unwinds;
/// the server deals with what it leaves behind.
fn log_panics() {
std::panic::set_hook(Box::new(|info| {
let location = info
.location()
.map(|l| format!("{}:{}", l.file(), l.line()))
.unwrap_or_else(|| "an unknown location".into());
let message = info
.payload()
.downcast_ref::<&str>()
.map(|s| s.to_string())
.or_else(|| info.payload().downcast_ref::<String>().cloned())
.unwrap_or_else(|| "no message".into());
// Captured only if RUST_BACKTRACE is set.
let backtrace = std::backtrace::Backtrace::capture();
error!(%location, %message, %backtrace, "Panic: this is a bug");
}));
}
/// What a signal asks of the daemon.
enum Signal {
/// To stop: SIGINT or SIGTERM, by name.
Stop(&'static str),
/// To leave, to be started again at once: SIGUSR1.
Restart,
}
/// Resolves with the first signal received that ends the daemon.
async fn signalled() -> Signal {
#[cfg(unix)]
{
use tokio::signal::unix::{SignalKind, signal};
let mut term = signal(SignalKind::terminate()).expect("install SIGTERM handler");
let mut restart = signal(SignalKind::user_defined1()).expect("install SIGUSR1 handler");
tokio::select! {
_ = tokio::signal::ctrl_c() => Signal::Stop("SIGINT"),
_ = term.recv() => Signal::Stop("SIGTERM"),
_ = restart.recv() => Signal::Restart,
}
}
#[cfg(not(unix))]
{
let _ = tokio::signal::ctrl_c().await;
Signal::Stop("ctrl-c")
}
}
crates/bench/Cargo.toml
+52
View File
@@ -0,0 +1,52 @@
[package]
name = "bgp-bench"
description = "Benchmarks at the scale of the Internet routing table"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
publish = false
[dependencies]
bgp-packet.workspace = true
bgp-primitives.workspace = true
bytes.workspace = true
flate2 = "1"
stats_alloc = { version = "0.1.10", optional = true }
[target.'cfg(not(target_env = "msvc"))'.dependencies]
tikv-jemallocator.workspace = true
[dev-dependencies]
bgp-integration-tests.workspace = true
bgp-rib.workspace = true
bgp-server.workspace = true
tokio.workspace = true
tracing-subscriber.workspace = true
[features]
# Counts every allocation, to report how much memory a table holds. The
# counters are shared by all threads and slow them down, so timings are
# taken without this and memory is measured in a run of its own.
count-memory = ["dep:stats_alloc"]
[[bench]]
name = "full_table"
harness = false
[[bench]]
name = "real_table"
harness = false
[[bench]]
name = "transit"
harness = false
[lints]
workspace = true
[[bench]]
name = "mesh"
harness = false
crates/bench/benches/full_table.rs
+202
View File
@@ -0,0 +1,202 @@
//! What it costs to take in the Internet routing table, from the bytes on
//! the wire to a chosen best route.
//!
//! Run with `cargo bench -p bgp-bench --bench full_table`. Set
//! `BENCH_PEERS` (default 4) and `BENCH_SCALE_DOWN` (default 1; 10 makes the
//! table ten times smaller) to change the size. `BENCH_PRESIZE=0` leaves
//! the table to grow as it fills, as it does without `expected_routes`. The table is synthetic (see
//! the crate documentation); `real_table` runs on a real one.
//!
//! This is a stopwatch around whole operations at full size, not a
//! statistical micro-benchmark: the questions are "how long until a full
//! table is in" and "how much memory does it take", and their answers are
//! seconds and gigabytes.
use std::hint::black_box;
use std::net::Ipv4Addr;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::memory::{Region, describe};
use bgp_bench::report::{mib, per_second, row};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{Message, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
use bytes::BytesMut;
fn main() {
let peers = env("BENCH_PEERS", 4) as u32;
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
let threads = std::thread::available_parallelism().map_or(1, |n| n.get());
println!(
"== Table: {} IPv4 + {} IPv6 prefixes, from each of {peers} peers",
shape.ipv4_prefixes, shape.ipv6_prefixes
);
println!(
"(IpPrefix is {} bytes, PathAttribute {})",
size_of::<IpPrefix>(),
size_of::<PathAttribute>()
);
let prefixes = Prefixes::generate(shape);
let wire: Vec<Vec<BytesMut>> = (1..=peers)
.map(|peer| encoded(&announcements(&prefixes, shape, peer)))
.collect();
let updates: usize = wire.iter().map(Vec::len).sum();
let routes = shape.prefixes() * peers as usize;
let wire_bytes: usize = wire.iter().flatten().map(BytesMut::len).sum();
println!(
"{updates} UPDATE messages, {:.1} prefixes each, {:.1} MiB on the wire, {routes} routes",
routes as f64 / updates as f64,
mib(wire_bytes),
);
println!("\n== Reading the messages (one thread)");
let ctx = wire_context();
let started = Instant::now();
for message in wire.iter().flatten() {
black_box(Message::decode(&ctx, message).expect("what we wrote is well formed"));
}
let elapsed = started.elapsed();
row(
"decode UPDATEs",
elapsed,
format!(
"{} routes, {} messages, {:.0} MiB/s",
per_second(routes, elapsed),
per_second(updates, elapsed),
mib(wire_bytes) / elapsed.as_secs_f64()
),
);
println!("\n== From bytes to best routes, without the network");
for threads in [1, threads] {
into_table(&wire, routes, threads);
}
println!("\n== End to end: {peers} peers send the table over TCP at once");
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
// An AS number that is in none of the generated paths: a route
// with the server's own AS in its path is refused as a loop.
let server = TestServer::start(4_200_000_000).await.expect("a server");
let mut senders = Vec::new();
for (index, messages) in wire.iter().enumerate() {
let peer = index as u32 + 1;
let mut session = TestPeer::connect(server.addr).await.expect("a connection");
session
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 90)
.await
.expect("a session");
// The same bytes `table_stream` would make, from the messages
// already encoded for the decoding above.
let stream: Vec<u8> = messages.iter().flat_map(|m| m.iter().copied()).collect();
senders.push((session, stream));
}
let started = Instant::now();
let sending: Vec<_> = senders
.into_iter()
.map(|(mut session, stream)| {
tokio::spawn(async move {
session.send_raw(&stream).await.expect("the server reads");
// Held open until the table is in.
session
})
})
.collect();
while server.rib.len() < routes {
tokio::time::sleep(Duration::from_millis(2)).await;
}
let elapsed = started.elapsed();
row(
"sessions, state machine, table",
elapsed,
format!(
"{} routes; {} prefixes with a best route",
per_second(routes, elapsed),
server.rib.prefix_count()
),
);
for session in sending {
drop(session.await);
}
});
}
/// What a session does with each message, without the session: decode it,
/// find its attribute set, put its routes in the table, choose the best.
/// `threads` threads each take whole peers.
fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) {
let region = Region::begin();
let table = RibTable::new(threads * 16);
if env("BENCH_PRESIZE", 1) == 1 {
table.expect_prefixes(routes / wire.len());
}
let peers: Vec<PeerIndex> = (0..wire.len())
.map(|peer| {
table
.add_peer(PeerInfo {
asn: 64512 + peer as u32 + 1,
router_id: Ipv4Addr::from(peer as u32 + 1),
address: Ipv4Addr::new(192, 0, 2, peer as u8 + 1).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.expect("room for the peer")
})
.collect();
let ctx = wire_context();
let next = AtomicUsize::new(0);
let started = Instant::now();
std::thread::scope(|scope| {
for _ in 0..threads {
scope.spawn(|| {
loop {
let at = next.fetch_add(1, Ordering::Relaxed);
let Some(messages) = wire.get(at) else {
break;
};
for message in messages {
let decoded =
Message::decode_with_report(&ctx, message).expect("well formed");
let Message::Update(update) = decoded.message else {
continue;
};
let wire = decoded.attributes_wire.as_deref().expect("kept");
if !update.announced_routes.is_empty() {
let attrs = table.attrs().intern_received(wire, None).expect("valid");
table.announce(peers[at], &attrs, &update.announced_routes);
}
for attribute in &update.path_attributes {
if let PathAttribute::MpReachNlri(mp) = attribute {
let attrs = table
.attrs()
.intern_received(wire, Some(&mp.next_hop))
.expect("valid");
table.announce(peers[at], &attrs, &mp.prefixes);
}
}
}
}
});
}
});
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
row(
&format!("decode, intern, insert, choose: {threads} thread(s)"),
elapsed,
format!(
"{} routes; {}",
per_second(routes, elapsed),
describe(region.held(), routes, "route")
),
);
}
crates/bench/benches/mesh.rs
+659
View File
@@ -0,0 +1,659 @@
//! Two routers of one AS, each taking in full tables from its own peers
//! and sending the other every path (ADD-PATH both ways, `all_paths`), as
//! a mesh of routers that each hold the network's whole view do. How
//! long until each router has the other's routes is what this measures:
//! the exporter reading the table's journal while sessions fill the
//! table, which nothing else exercises at this size.
//!
//! Then a new peer takes the first router's table, as a collector or a
//! router coming back does: how soon its first UPDATE arrives and how
//! long the table takes are measured too. It is a peer of the routers'
//! AS over IPv4, so what it gets is the first router's IPv4 routes from
//! its own peers: the other router's are not reflected to it, and IPv6
//! routes need an IPv6 session.
//!
//! Run with `cargo bench -p bgp-bench --bench mesh`. `BENCH_PEERS`
//! (default 4) feed the first router and `BENCH_PEERS_B` (default 2) the
//! second; `BENCH_SCALE_DOWN` shrinks the table. The routers peer over
//! IPv6 loopback and the feeders come over IPv4 loopback; the new peer
//! comes from 127.0.0.2, which Linux has and macOS does not, so that
//! part is skipped on macOS. With `BENCH_RECEIVER_EARLY=1` the new peer
//! comes as the feeders start instead, so that its exporter goes
//! through the table while the table is being filled, as a router's
//! do after a restart. `BENCH_COLLECTORS` (default 0) more peers of the
//! first router, from 127.0.1.0/24, are peers in the sense of RFC 9234
//! told of learned routes, and the feeders' routes are marked as having
//! come across from a peer, so none of them goes: as many exporters
//! following the table while it fills and passing nearly all of it
//! over, as a router with many peers at exchanges has.
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::report::row;
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, table_stream};
use bgp_integration_tests::peer_config;
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::attributes::PathAttribute;
use bgp_packet::message::msg_type;
use bgp_packet::open::SendReceive;
use bgp_server::config::PeerConfig;
use bgp_server::family::SUPPORTED;
use bgp_server::policy::{ExportPolicy, ImportPolicy, PeerRole};
use bgp_server::session::SessionRegistry;
use tokio::io::{AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::tcp::OwnedWriteHalf;
use tokio::task::JoinHandle;
/// An AS number that is in none of the synthetic paths.
const ASN: u32 = 4_200_000_000;
/// Where the peer that takes the whole table connects from.
const RECEIVER: Ipv4Addr = Ipv4Addr::new(127, 0, 0, 2);
/// Where the collectors connect from: this and the addresses after it.
fn collector_address(n: u32) -> Ipv4Addr {
Ipv4Addr::new(127, 0, 1, n as u8)
}
/// The collectors' AS numbers start here.
const COLLECTOR_ASN: u32 = 65000;
/// How often the tables are looked at while they fill.
const POLL: Duration = Duration::from_millis(5);
/// What the routers send each other, and the receiver: every path.
fn every_path() -> Arc<ExportPolicy> {
Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
})
}
fn free_port() -> u16 {
std::net::TcpListener::bind((Ipv6Addr::LOCALHOST, 0))
.expect("a port")
.local_addr()
.expect("an address")
.port()
}
/// A router of the mesh, listening for feeders on IPv4 loopback and for
/// the other router on IPv6 loopback at `port`, which it dials at
/// `other_port`, knowing it as `other_name`.
async fn router(
other_name: &'static str,
id: u8,
port: u16,
other_port: u16,
with_receiver: bool,
collectors: u32,
) -> TestServer {
TestServer::start_with(ASN, |c| {
// Two routers of one AS must not share a BGP identifier.
c.router_id = Ipv4Addr::new(192, 0, 2, id);
c.cluster_id = c.router_id;
c.listen_endpoints = vec![
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 0),
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), port),
];
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Both),
import: ImportPolicy::permissive(),
export: every_path(),
..peer_config(
other_name,
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), other_port),
ASN,
)
}];
if with_receiver {
c.peers.push(PeerConfig {
add_path: Some(SendReceive::Send),
export: every_path(),
..peer_config("receiver", SocketAddr::new(RECEIVER.into(), 179), ASN)
});
}
for n in 1..=collectors {
c.peers.push(PeerConfig {
role: Some(PeerRole::Peer),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(collector_address(n).into(), 179),
COLLECTOR_ASN + n,
)
});
}
c.connect_retry = Duration::from_millis(200);
})
.await
.expect("a router")
}
/// Connects `count` feeders to `router`, numbered from `first`, and
/// returns their sessions with the bytes each is to send.
async fn feeders(
router: &TestServer,
prefixes: &Prefixes,
shape: Shape,
first: u32,
count: u32,
marked: bool,
) -> Vec<(TestPeer, Vec<u8>)> {
let mut sessions = Vec::new();
for peer in first..first + count {
let stream = if marked {
// As having come across from a peer (RFC 9234): not for the
// collectors, which are peers too.
let mut updates = announcements(prefixes, shape, peer);
for update in &mut updates {
update
.path_attributes
.push(PathAttribute::OnlyToCustomer(64512 + peer));
}
encoded(&updates)
.iter()
.flat_map(|m| m.iter().copied())
.collect()
} else {
table_stream(prefixes, shape, peer)
};
let mut session = TestPeer::connect(router.addr).await.expect("a connection");
// No hold timer: a feeder has nothing to say after its table.
session
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 0)
.await
.expect("a session");
sessions.push((session, stream));
}
sessions
}
/// Every session of `registry`, one line each, for the panic when a
/// deadline passes: where the routes have got to, and what each session
/// was to be sent.
fn sessions_report(registry: &SessionRegistry) -> Vec<String> {
registry
.list()
.iter()
.map(|s| {
format!(
"{} {:?} export {:?} add_path_send {:?} routes {} rejected {} advertised {}",
s.peer,
s.name,
s.export,
s.add_path_send,
s.counters.routes(),
s.counters.rejected(),
s.counters.advertised()
)
})
.collect()
}
/// How big a run is, from the environment.
struct Settings {
peers_a: u32,
peers_b: u32,
shape: Shape,
collectors: u32,
}
impl Settings {
fn from_env() -> Self {
Self {
peers_a: env("BENCH_PEERS", 4) as u32,
peers_b: env("BENCH_PEERS_B", 2) as u32,
shape: Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1)),
collectors: env("BENCH_COLLECTORS", 0) as u32,
}
}
/// How many routes A's own feeders bring, and B's.
fn own_routes(&self) -> (usize, usize) {
let per_peer = self.shape.prefixes();
(
per_peer * self.peers_a as usize,
per_peer * self.peers_b as usize,
)
}
/// How many routes the receiver is to be sent: A's IPv4 routes from
/// its own peers.
fn receiver_routes(&self) -> usize {
self.shape.ipv4_prefixes * self.peers_a as usize
}
}
/// The two routers, up and peered, with the collectors' sessions once
/// they are connected.
struct Mesh {
settings: Settings,
prefixes: Prefixes,
a: TestServer,
b: TestServer,
/// The collectors' connections, drained, held until the end.
collectors: Vec<(OwnedWriteHalf, JoinHandle<()>)>,
}
/// The feeders sending, from when they began.
struct Feeding {
started: Instant,
/// Each hands its session back once its table is written.
sending: Vec<JoinHandle<TestPeer>>,
}
/// When each router had what: its own peers' routes, and the other's too.
struct Tables {
a_own: Duration,
b_own: Duration,
a_all: Duration,
b_all: Duration,
}
impl Mesh {
/// Starts the routers, with the collectors and receiver configured at
/// A, and waits for them to peer.
async fn start(settings: Settings, prefixes: Prefixes) -> Self {
let (port_a, port_b) = (free_port(), free_port());
let a = router("b", 1, port_a, port_b, true, settings.collectors).await;
let b = router("a", 2, port_b, port_a, false, 0).await;
a.wait_for_sessions(1).await.expect("the mesh session at A");
b.wait_for_sessions(1).await.expect("the mesh session at B");
Self {
settings,
prefixes,
a,
b,
collectors: Vec::new(),
}
}
/// Connects the collectors to A. They are there before the feeders, as
/// a router's peers are before its tables arrive after a restart. An
/// address that is not this machine's ends the connecting, with a
/// word.
async fn connect_collectors(&mut self) {
for n in 1..=self.settings.collectors {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(collector_address(n).into(), 0)) {
println!(
"collectors: skipped, {} is not ours ({e})",
collector_address(n)
);
break;
}
let stream = socket.connect(self.a.addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(COLLECTOR_ASN + n, Ipv4Addr::new(192, 0, 3, n as u8), 0)
.await
.expect("A takes a collector's session");
self.collectors.push(session.drain());
}
if !self.collectors.is_empty() {
self.a
.wait_for_sessions(1 + self.collectors.len())
.await
.expect("the collectors' sessions at A");
}
}
/// Connects the feeders to both routers and starts them sending.
async fn feed(&self) -> Feeding {
let Settings {
peers_a,
peers_b,
shape,
..
} = self.settings;
let marked = !self.collectors.is_empty();
let feed_a = feeders(&self.a, &self.prefixes, shape, 1, peers_a, marked).await;
let feed_b = feeders(&self.b, &self.prefixes, shape, 1 + peers_a, peers_b, false).await;
let started = Instant::now();
let sending = feed_a
.into_iter()
.chain(feed_b)
.map(|(mut session, stream)| {
tokio::spawn(async move {
session.send_raw(&stream).await.expect("the router reads");
session
})
})
.collect();
Feeding { started, sending }
}
/// The receiver taking A's table while it fills, if `BENCH_RECEIVER_EARLY`
/// asks for that: as the feeders start (1), or once A has its own
/// peers' routes and B's are still coming (2), as a router's sessions
/// come up after a restart while its peers' tables arrive.
fn early_receiver(&self, deadline: Duration) -> Option<JoinHandle<()>> {
let when = env("BENCH_RECEIVER_EARLY", 0);
if when == 0 {
return None;
}
let (own_a, _) = self.settings.own_routes();
let routes = self.settings.receiver_routes();
let (addr, registry, rib) = (
self.a.addr,
self.a.registry.clone(),
Arc::clone(&self.a.rib),
);
Some(tokio::spawn(async move {
while when == 2 && rib.len() < own_a {
tokio::time::sleep(POLL).await;
}
resync(addr, registry, routes, deadline).await
}))
}
/// Waits until both routers have every route, noting when each had
/// what; or panics once `deadline` has passed since `started`.
async fn wait_for_tables(&self, started: Instant, deadline: Duration) -> Tables {
let (own_a, own_b) = self.settings.own_routes();
let mut a_own = None;
let mut b_own = None;
let mut a_all = None;
let mut b_all = None;
let (a_all, b_all) = loop {
tokio::time::sleep(POLL).await;
let (at_a, at_b) = (self.a.rib.len(), self.b.rib.len());
let now = started.elapsed();
if a_own.is_none() && at_a >= own_a {
a_own = Some(now);
}
if b_own.is_none() && at_b >= own_b {
b_own = Some(now);
}
if a_all.is_none() && at_a >= own_a + own_b {
a_all = Some(now);
}
if b_all.is_none() && at_b >= own_a + own_b {
b_all = Some(now);
}
if let (Some(a_all), Some(b_all)) = (a_all, b_all) {
break (a_all, b_all);
}
if now > deadline {
panic!(
"not all routes arrived: A has {at_a} {:?}, B has {at_b} {:?}",
sessions_report(&self.a.registry),
sessions_report(&self.b.registry)
);
}
};
Tables {
a_own: a_own.unwrap_or_default(),
b_own: b_own.unwrap_or_default(),
a_all,
b_all,
}
}
/// One row for each of the four moments of `tables`, with the rate the
/// routes that made the moment came at.
fn report(&self, tables: &Tables) {
let (own_a, own_b) = self.settings.own_routes();
let rate = |when: Duration, routes: usize| {
format!("{:.2} M routes/s", routes as f64 / when.as_secs_f64() / 1e6)
};
row(
"A has its own peers' routes",
tables.a_own,
rate(tables.a_own, own_a),
);
row(
"B has its own peers' routes",
tables.b_own,
rate(tables.b_own, own_b),
);
row(
"A has B's routes too (B's exporter, A's session)",
tables.a_all,
rate(tables.a_all, own_b),
);
row(
"B has A's routes too (A's exporter, B's session)",
tables.b_all,
rate(tables.b_all, own_a),
);
}
}
impl Feeding {
/// Waits for every feeder to have written its table, and returns the
/// sessions, which are to stay up until the end: a feeder going would
/// have its routes withdrawn from both tables meanwhile.
async fn finished(self) -> Vec<TestPeer> {
let mut feeders = Vec::new();
for session in self.sending {
feeders.push(session.await.expect("a feeder is done"));
}
feeders
}
}
fn main() {
// The routers' logs, with `BENCH_LOG` as a `RUST_LOG` filter.
if let Ok(filter) = std::env::var("BENCH_LOG") {
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.init();
}
let settings = Settings::from_env();
let deadline = Duration::from_secs(env("BENCH_DEADLINE", 600) as u64);
println!(
"== Two routers of AS{ASN}, every path both ways: A fed by {} peers, B by {}, {} prefixes each",
settings.peers_a,
settings.peers_b,
settings.shape.prefixes()
);
let prefixes = Prefixes::generate(settings.shape);
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
let mut mesh = Mesh::start(settings, prefixes).await;
mesh.connect_collectors().await;
let early_receiver = mesh.early_receiver(deadline);
let feeding = mesh.feed().await;
let tables = mesh.wait_for_tables(feeding.started, deadline).await;
mesh.report(&tables);
let feeders = feeding.finished().await;
match early_receiver {
Some(receiver) => receiver.await.expect("the receiver is done"),
None => {
resync(
mesh.a.addr,
mesh.a.registry.clone(),
mesh.settings.receiver_routes(),
deadline,
)
.await
}
}
drop(feeders);
drop(mesh);
});
}
/// The new peer taking a router's table: counting the UPDATEs it is sent
/// while keeping the session alive.
struct Receiver {
started: Instant,
/// When the first UPDATE arrived, in microseconds after `started`; 0
/// until it has.
first: Arc<AtomicU64>,
updates: Arc<AtomicU64>,
counting: JoinHandle<()>,
keepalives: JoinHandle<()>,
}
impl Receiver {
/// A session with the router at `addr` from [`RECEIVER`], as a
/// collector of every path; or `None`, with a word, where that address
/// is not this machine's to connect from.
async fn connect(addr: SocketAddr) -> Option<TestPeer> {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(RECEIVER.into(), 0)) {
println!("A new peer taking A's routes: skipped, {RECEIVER} is not ours ({e})");
return None;
}
let stream = socket.connect(addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake_with(
ASN,
Ipv4Addr::new(192, 0, 2, 9),
90,
collector_capabilities(ASN, &SUPPORTED, false),
)
.await
.expect("A takes the receiver's session");
Some(session)
}
/// Starts counting what `session` is sent, from now. Every millionth
/// UPDATE is noted, as is a NOTIFICATION or the connection closing:
/// what went wrong, if the table never all arrives.
fn count_updates(session: TestPeer) -> Self {
let started = Instant::now();
let (reader, mut writer) = session.into_stream().into_split();
// KEEPALIVEs, so that the session outlives the hold time.
let keepalives = tokio::spawn(async move {
let keepalive = TestPeer::keepalive_bytes();
loop {
tokio::time::sleep(Duration::from_secs(20)).await;
if writer.write_all(&keepalive).await.is_err() {
break;
}
}
});
let first = Arc::new(AtomicU64::new(0));
let updates = Arc::new(AtomicU64::new(0));
let counting = tokio::spawn({
let (first, updates) = (Arc::clone(&first), Arc::clone(&updates));
async move {
let mut reader = BufReader::with_capacity(1 << 20, reader);
let mut header = [0u8; 19];
let mut body = vec![0u8; 4096];
while reader.read_exact(&mut header).await.is_ok() {
let length = usize::from(u16::from_be_bytes([header[16], header[17]]));
if reader.read_exact(&mut body[..length - 19]).await.is_err() {
break;
}
match header[18] {
msg_type::UPDATE => {
let _ = first.compare_exchange(
0,
started.elapsed().as_micros() as u64,
Ordering::Relaxed,
Ordering::Relaxed,
);
let n = updates.fetch_add(1, Ordering::Relaxed) + 1;
if n % 1_000_000 == 0 {
println!(
" the receiver has read {n} UPDATEs after {:.2} s",
started.elapsed().as_secs_f64()
);
}
}
msg_type::NOTIFICATION => println!(
"the receiver was sent a NOTIFICATION, error {} subcode {}, after {} UPDATEs",
body[0],
body[1],
updates.load(Ordering::Relaxed)
),
_ => {}
}
}
println!(
"the receiver's connection closed after {} UPDATEs",
updates.load(Ordering::Relaxed)
);
}
});
Self {
started,
first,
updates,
counting,
keepalives,
}
}
/// Waits until the router whose sessions are in `registry` has
/// advertised `routes` routes to the receiver, and returns how many it
/// has; or panics once `deadline` has passed.
async fn wait_for_advertised(
&self,
registry: &SessionRegistry,
routes: usize,
deadline: Duration,
) -> u64 {
loop {
tokio::time::sleep(POLL).await;
let advertised = registry
.list()
.iter()
.find(|s| s.peer.addr.ip() == RECEIVER)
.map_or(0, |s| s.counters.advertised());
if advertised as usize >= routes {
return advertised;
}
if self.started.elapsed() > deadline {
panic!(
"the receiver has {advertised} of {routes} routes and read {} UPDATEs; sessions at A: {:#?}",
self.updates.load(Ordering::Relaxed),
sessions_report(registry)
);
}
}
}
/// Prints the row for the table being all there, `advertised` routes
/// of it, and stops counting.
fn report(self, advertised: u64) {
let took = self.started.elapsed();
self.counting.abort();
self.keepalives.abort();
let first = Duration::from_micros(self.first.load(Ordering::Relaxed));
row(
"A new peer has A's IPv4 routes (A's exporter)",
took,
format!(
"{:.2} M routes/s first UPDATE after {:.0} ms, {} UPDATEs",
advertised as f64 / took.as_secs_f64() / 1e6,
first.as_secs_f64() * 1e3,
self.updates.load(Ordering::Relaxed),
),
);
}
}
/// A new peer takes the table of the router at `addr`, whose sessions
/// are in `registry`: `routes` routes with every path. How long until
/// the first UPDATE, and until all of it is sent.
async fn resync(addr: SocketAddr, registry: SessionRegistry, routes: usize, deadline: Duration) {
let Some(session) = Receiver::connect(addr).await else {
return;
};
let receiver = Receiver::count_updates(session);
let advertised = receiver
.wait_for_advertised(&registry, routes, deadline)
.await;
receiver.report(advertised);
}
crates/bench/benches/real_table.rs
+423
View File
@@ -0,0 +1,423 @@
//! The same questions as `full_table`, asked of a real table dump: every
//! route that the peers of a RIPE RIS route collector had announced to it.
//!
//! Needs a dump in `crates/bench/data/` (see the README there) or at the
//! path in `BENCH_MRT`. Without one it says so and does nothing, so that
//! `cargo bench` works on a fresh checkout. `BENCH_PEERS` limits the run to
//! the peers with the most routes (default: all of them).
use std::collections::HashSet;
use std::net::Ipv4Addr;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::Instant;
use bgp_bench::memory::Region;
use bgp_bench::mrt::{Dump, parse_attributes};
use bgp_bench::report::{mib, per_second, row};
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{Event, PeerIndex, PeerInfo, RibTable};
/// Bytes allocated and not freed since `region` began; zero when memory
/// is not being counted (see `bgp_bench::memory`), and zero too in the
/// odd case of more freed than allocated, which is nothing held.
fn retained(region: &Region) -> usize {
usize::try_from(region.held().unwrap_or(0)).unwrap_or(0)
}
fn dump_path() -> Option<PathBuf> {
if let Ok(path) = std::env::var("BENCH_MRT") {
return Some(path.into());
}
let data = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("data");
let mut dumps: Vec<PathBuf> = std::fs::read_dir(data)
.ok()?
.filter_map(|entry| Some(entry.ok()?.path()))
.filter(|path| path.extension().is_some_and(|e| e == "gz"))
.collect();
dumps.sort();
dumps.pop()
}
type Attributes = (Arc<[PathAttribute]>, Option<NlriNextHop>);
fn main() {
if Region::begin().held().is_none() {
println!(
"Memory is not counted in this run: sizes print as 0. For sizes (and slower\ntimings) add --features count-memory.\n"
);
}
let Some(path) = dump_path() else {
println!("No table dump found; see crates/bench/data/README.md. Nothing to do.");
return;
};
println!("== Reading {}", path.display());
let started = Instant::now();
let dump = Dump::read(&path).expect("a readable dump");
println!(
"{} peers, {} routes, {} distinct attribute sets, read in {:.0} s",
dump.peers.len(),
dump.route_count(),
dump.attribute_sets.len(),
started.elapsed().as_secs_f64()
);
let limit = std::env::var("BENCH_PEERS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(dump.peers.len());
let peers = dump.largest_peers(limit);
let peers: Vec<usize> = peers
.into_iter()
.filter(|&peer| !dump.routes[peer].is_empty())
.collect();
describe(&dump, &peers);
// Parsed once, outside the timings: reading messages is measured by
// `full_table`, and here the attributes arrive already parsed.
let region = Region::begin();
let parsed: Vec<Option<Attributes>> = dump
.attribute_sets
.iter()
.map(|wire| parse_attributes(wire).map(|(a, next_hop)| (Arc::from(a), next_hop)))
.collect();
let parsed_bytes = retained(&region);
println!(
"\nattribute sets, parsed (Arc<[PathAttribute]>) {:.0} MiB = {:.0} bytes a set",
mib(parsed_bytes),
parsed_bytes as f64 / parsed.len() as f64
);
let unparsed = parsed.iter().filter(|a| a.is_none()).count();
if unparsed > 0 {
println!("{unparsed} attribute sets could not be parsed; their routes are left out");
}
let threads = std::thread::available_parallelism().map_or(1, |n| n.get());
sharded_table(&dump, &peers, &parsed, 1);
sharded_table(&dump, &peers, &parsed, threads);
followed_table(&dump, &peers, &parsed, threads);
}
/// What the table looks like, as far as it decides what a RIB costs.
fn describe(dump: &Dump, peers: &[usize]) {
let routes: usize = peers.iter().map(|&p| dump.routes[p].len()).sum();
let full = peers
.iter()
.filter(|&&p| dump.routes[p].len() > 800_000)
.count();
let prefixes: HashSet<IpPrefix> = peers
.iter()
.flat_map(|&p| dump.routes[p].iter().map(|(prefix, _)| *prefix))
.collect();
let mut sets_used = HashSet::new();
let mut sets_per_peer = 0usize;
for &peer in peers {
let own: HashSet<u32> = dump.routes[peer].iter().map(|(_, set)| *set).collect();
sets_per_peer += own.len();
sets_used.extend(own);
}
let attribute_bytes: usize = sets_used
.iter()
.map(|&set| dump.attribute_sets[set as usize].len())
.sum();
println!("\n== The table ({} peers used)", peers.len());
println!("routes {routes:>12}");
println!("peers with more than 800,000 routes {full:>12}");
println!(
"distinct prefixes {:>12}",
prefixes.len()
);
println!(
"routes per prefix {:>12.1}",
routes as f64 / prefixes.len() as f64
);
println!(
"distinct attribute sets {:>12}",
sets_used.len()
);
println!(
" if each peer's were kept apart {sets_per_peer:>12} ({:.2}x: sets shared between peers)",
sets_per_peer as f64 / sets_used.len() as f64
);
println!(
"routes per attribute set {:>12.1}",
routes as f64 / sets_used.len() as f64
);
println!(
"attribute set on the wire, mean {:>12.0} bytes",
attribute_bytes as f64 / sets_used.len() as f64
);
}
/// The sharded table, filled by `threads` threads. Each takes whole peers,
/// as sessions would, and for every UPDATE interns the attributes and then
/// announces the prefixes: everything between a decoded message and a
/// chosen best route.
fn sharded_table(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>], threads: usize) {
println!("\n== The sharded table, {threads} thread(s)");
let updates = grouped(dump, peers, parsed);
let routes: usize = updates
.iter()
.flatten()
.map(|(_, prefixes)| prefixes.len())
.sum();
let region = Region::begin();
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let started = Instant::now();
let best_changes = load(&table, dump, &updates, &indices, threads);
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
let held = retained(&region);
row(
"intern attributes, insert, choose best",
elapsed,
format!(
"{} routes; {:.0} MiB = {:.0} bytes a route, attribute sets included",
per_second(routes, elapsed),
mib(held),
held as f64 / routes as f64
),
);
println!(
" {} prefixes, {} attribute sets, best route changed {} times",
table.prefix_count(),
table.attrs().len(),
best_changes
);
let started = Instant::now();
let mut best = 0usize;
table.for_each_best(|prefix, _, _| {
best += usize::from(std::hint::black_box(prefix).length() > 0)
});
row(
"read every best route",
started.elapsed(),
format!("{best} of them, in place, table not stopped"),
);
// Reading has to look at each route for the timing to mean anything.
let started = Instant::now();
let mut all = 0usize;
let mut hops = 0u64;
table.for_each_route(|route| {
all += 1;
hops += u64::from(route.attrs.key().as_path_len);
});
row(
"read every route and its attributes",
started.elapsed(),
format!(
"{all} of them, mean AS path {:.1}",
hops as f64 / all as f64
),
);
let started = Instant::now();
let outcome = table.remove_peer(indices[0]);
let elapsed = started.elapsed();
row(
"remove one peer (session down)",
elapsed,
format!(
"{} routes; best route changed for {} prefixes",
per_second(outcome.removed, elapsed),
outcome.best_changed
),
);
let started = Instant::now();
let swept = table.attrs().sweep();
row(
"drop attribute sets no longer used",
started.elapsed(),
format!("{swept} dropped"),
);
}
/// A peer's routes grouped as UPDATEs: the attribute set and the prefixes
/// announced with it.
type Updates = Vec<Vec<(u32, Vec<IpPrefix>)>>;
fn grouped(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>]) -> Updates {
peers
.iter()
.map(|&peer| {
let mut routes = dump.routes[peer].clone();
routes.sort_by_key(|(_, set)| *set);
routes
.chunk_by(|a, b| a.1 == b.1)
.filter(|group| parsed[group[0].1 as usize].is_some())
.map(|group| {
(
group[0].1,
group.iter().map(|(prefix, _)| *prefix).collect(),
)
})
.collect()
})
.collect()
}
fn add_peers(table: &RibTable, dump: &Dump, peers: &[usize]) -> Vec<PeerIndex> {
peers
.iter()
.map(|&peer| {
table
.add_peer(PeerInfo {
asn: dump.peers[peer].asn,
router_id: Ipv4Addr::from(peer as u32 + 1),
address: dump.peers[peer].address,
interface: 0,
ibgp: false,
reflector_client: false,
})
.expect("room for the peer")
})
.collect()
}
/// Announces everything in `updates` on `threads` threads, each taking
/// whole peers. Returns how many times a best route changed.
fn load(
table: &RibTable,
dump: &Dump,
updates: &Updates,
indices: &[PeerIndex],
threads: usize,
) -> usize {
let next = AtomicUsize::new(0);
let best_changes = AtomicUsize::new(0);
std::thread::scope(|scope| {
for _ in 0..threads {
scope.spawn(|| {
let mut changes = 0;
loop {
let at = next.fetch_add(1, Ordering::Relaxed);
let Some(updates) = updates.get(at) else {
break;
};
for (set, prefixes) in updates {
// From the bytes, as a dump has them and as a
// session could hand them over without parsing.
let Some(attrs) = table
.attrs()
.intern_wire(&dump.attribute_sets[*set as usize])
else {
continue;
};
changes += table.announce(indices[at], &attrs, prefixes).best_changed;
}
}
best_changes.fetch_add(changes, Ordering::Relaxed);
});
}
});
best_changes.load(Ordering::Relaxed)
}
/// The same load with someone following the best routes through the
/// journal: once reading as fast as it can while the table fills, once not
/// reading at all until the table is full.
fn followed_table(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>], threads: usize) {
println!("\n== The sharded table with a follower, {threads} thread(s)");
let updates = grouped(dump, peers, parsed);
let routes: usize = updates
.iter()
.flatten()
.map(|(_, prefixes)| prefixes.len())
.sum();
let count = |counts: &mut (usize, usize, usize), event: Event<'_>| match event {
Event::Best { attrs, .. } => {
// Look at the route, as a follower would.
counts.0 += usize::from(attrs.key().as_path_len > 0);
}
Event::Withdrawn { .. } => counts.1 += 1,
Event::Reset | Event::Synced => counts.2 += 1,
// Not given to a follower of the best routes.
Event::Routes { .. } => {}
};
// Reading while the table fills.
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let done = AtomicBool::new(false);
let started = Instant::now();
let (best_changes, counts, longest, after_load) = std::thread::scope(|scope| {
let follower = scope.spawn(|| {
let mut reader = table.reader();
let mut counts = (0, 0, 0);
let mut longest = 0;
while !done.load(Ordering::Acquire) {
reader.poll(4096, |event| count(&mut counts, event));
longest = longest.max(table.journal_len());
}
let finishing = Instant::now();
while !reader
.poll(4096, |event| count(&mut counts, event))
.caught_up
{}
(counts, longest, finishing.elapsed())
});
let best_changes = load(&table, dump, &updates, &indices, threads);
done.store(true, Ordering::Release);
let (counts, longest, after_load) = follower.join().expect("the follower runs");
(best_changes, counts, longest, after_load)
});
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
row(
"load, follower reading throughout",
elapsed,
format!(
"{} routes; caught up {:.3} s after the last route",
per_second(routes, elapsed),
after_load.as_secs_f64()
),
);
println!(
" best route changed {best_changes} times; the follower was given {} events for it; \n journal at most {longest} entries for {} prefixes",
counts.0 + counts.1,
table.prefix_count()
);
drop(table);
// Not reading: the journal has to stay small on its own account.
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let mut reader = table.reader();
let mut counts = (0, 0, 0);
reader.poll(usize::MAX, |event| count(&mut counts, event));
let started = Instant::now();
load(&table, dump, &updates, &indices, threads);
let elapsed = started.elapsed();
row(
"load, follower not reading",
elapsed,
format!(
"{} routes; journal holds {} entries for {} prefixes",
per_second(routes, elapsed),
table.journal_len(),
table.prefix_count()
),
);
let started = Instant::now();
let mut counts = (0, 0, 0);
while !reader
.poll(4096, |event| count(&mut counts, event))
.caught_up
{}
let elapsed = started.elapsed();
row(
"the follower then catches up",
elapsed,
format!(
"{} events, {} of them a reset",
counts.0 + counts.1,
counts.2
),
);
}
crates/bench/benches/transit.rs
+146
View File
@@ -0,0 +1,146 @@
//! What passing routes on costs: one peer sends the table over a session,
//! and another is told of every route, over its own. The time is from the
//! first byte sent to the last route heard; alongside, the time for the
//! same table to be in our own table and no further.
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::PathAttribute;
use bgp_server::marks::{Control, Marks};
use bgp_server::policy::ExportPolicy;
use bytes::BytesMut;
/// How a server passes routes on in one run.
#[derive(Clone, Copy)]
enum Passing {
Not,
Plain,
/// With the communities that say who is told, and a mark put on every
/// route.
Dressed,
}
/// Sends the table from one peer and, where it is passed on, hears all of
/// it at another. Returns when the table was in, and when it was all
/// heard.
fn run(
wire: &[BytesMut],
routes: usize,
heard_routes: usize,
passing: Passing,
) -> (Duration, Option<Duration>) {
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
let server = TestServer::start_with(4_200_000_000, |config| {
config.export = Arc::new(ExportPolicy {
learned_routes: !matches!(passing, Passing::Not),
all_paths: false,
marks: match passing {
Passing::Dressed => Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
_ => Marks::default(),
},
prepend: 0,
});
config.control = matches!(passing, Passing::Dressed).then_some(Control {
local_asn: 4_200_000_000,
});
})
.await
.expect("a server");
let mut hearer = TestPeer::connect(server.addr).await.expect("a connection");
hearer
// An AS number that is on none of the generated paths: a route
// is not passed to a peer whose AS it has been through.
.handshake(4_100_000_000, Ipv4Addr::new(10, 0, 0, 100), 90)
.await
.expect("a session");
let mut sender = TestPeer::connect(server.addr).await.expect("a connection");
sender
.handshake(64513, Ipv4Addr::new(10, 0, 0, 1), 90)
.await
.expect("a session");
let stream: Vec<u8> = wire.iter().flat_map(|m| m.iter().copied()).collect();
let started = Instant::now();
let sending = tokio::spawn(async move {
sender.send_raw(&stream).await.expect("the server reads");
sender
});
let hearing = tokio::spawn(async move {
if matches!(passing, Passing::Not) {
return (hearer, None, 0);
}
let mut heard = 0usize;
let mut messages = 0usize;
while heard < heard_routes {
let update = hearer.recv_update().await.expect("an UPDATE");
messages += 1;
heard += update.announced_routes.len();
heard += update
.path_attributes
.iter()
.map(|attribute| match attribute {
PathAttribute::MpReachNlri(mp) => mp.prefixes.len(),
_ => 0,
})
.sum::<usize>();
}
(hearer, Some(started.elapsed()), messages)
});
while server.rib.len() < routes {
tokio::time::sleep(Duration::from_millis(2)).await;
}
let in_table = started.elapsed();
let (hearer, heard, messages) = hearing.await.expect("the hearer");
if messages > 0 {
println!(
" ({messages} UPDATEs, {:.1} routes each)",
heard_routes as f64 / messages as f64
);
}
drop(sending.await);
drop(hearer);
(in_table, heard)
})
}
fn main() {
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
let prefixes = Prefixes::generate(shape);
let wire = encoded(&announcements(&prefixes, shape, 1));
let routes = shape.prefixes();
// Over a session on IPv4 only the IPv4 routes go.
let heard_routes = shape.ipv4_prefixes;
println!(
"== One peer sends {routes} routes in {} UPDATEs over TCP; another, over IPv4, is told of the {heard_routes} IPv4 ones",
wire.len()
);
for (what, passing) in [
("not passed on", Passing::Not),
("passed on", Passing::Plain),
(
"passed on, with control communities and a mark",
Passing::Dressed,
),
] {
let (in_table, heard) = run(&wire, routes, heard_routes, passing);
let heard = heard.map_or(String::new(), |heard| {
format!(
" all heard by the other peer after {:>6.3} s ({:.2} M routes/s)",
heard.as_secs_f64(),
heard_routes as f64 / heard.as_secs_f64() / 1e6
)
});
println!(
"{what:<48} in the table after {:>6.3} s{heard}",
in_table.as_secs_f64()
);
}
}
crates/bench/data/.gitignore
+4
View File
@@ -0,0 +1,4 @@
# Table dumps downloaded for benchmarks; see README.md.
*
!.gitignore
!README.md
crates/bench/data/README.md
+18
View File
@@ -0,0 +1,18 @@
# Table dumps for the benchmarks
`cargo bench -p bgp-bench --bench real_table` runs on a real routing table:
a dump of every route that the peers of a route collector had announced to
it. The dumps are large and are not checked in. Fetch one into this
directory:
```bash
curl -O https://data.ris.ripe.net/rrc03/2026.09/bview.20260915.0000.gz
```
That is RIPE RIS collector `rrc03`, which peers at AMS-IX: about 340 MB,
3.4 GB uncompressed. Any `bview` file from <https://data.ris.ripe.net/> or
a RouteViews RIB works; the most recent `.gz` here is used, or the one named
by `BENCH_MRT`.
The benchmark needs memory in proportion to the dump. `BENCH_PEERS=8`
limits it to the eight peers with the most routes.
crates/bench/examples/feed.rs
+181
View File
@@ -0,0 +1,181 @@
//! Feeds the synthetic Internet routing table (see the crate documentation)
//! to a running daemon over TCP, as one or more peers would, and holds the
//! sessions open afterwards. This is the load for profiling the daemon as
//! it takes in a full table; `scripts/profile/full-table.sh` runs it under
//! `perf`.
//!
//! Settings, from the environment:
//!
//! - `FEED_TO`: where the daemon listens (default `127.0.0.1:1790`).
//! - `FEED_PEERS`: how many peers send the table (default 1).
//! - `FEED_SCALE_DOWN`: as `BENCH_SCALE_DOWN`; 10 makes the table ten
//! times smaller (default 1).
//! - `FEED_FROM`: the first three octets of an IPv4 /24, "192.0.2". Peer N
//! connects from .N, which is also the next hop of its IPv4 routes.
//! Without it, peers connect from wherever the system puts them.
//! - `FEED_ASN`: the first peer's AS number (default 64513); the others
//! follow on from it.
//! - `FEED_HOLD_SECS`: how long the sessions stay open after the table is
//! sent (default 30).
//! - `FEED_COLLECTOR=1`: before the peers, one more session, as a route
//! collector would have: over IPv6, to `FEED_COLLECTOR_TO` (default
//! `[::1]:1790`) from `FEED_COLLECTOR_FROM` if set, offering to receive
//! several routes for a prefix (ADD-PATH) for both families and IPv4
//! routes with IPv6 next hops, and reading and dropping whatever it is
//! sent. Its AS number is 212232.
//!
//! What is printed: when the sessions are up and how long the writes took.
//! The writes end when the kernel's socket buffers have taken the last
//! bytes, which is before the daemon has read them: when the table is in
//! is for the daemon's API to say.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::{Duration, Instant};
use bgp_bench::report::mib;
use bgp_bench::{Prefixes, Shape, env, table_stream};
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_primitives::constants::AfiSafi;
use tokio::io::AsyncWriteExt;
use tokio::net::TcpSocket;
use tokio::net::tcp::OwnedWriteHalf;
/// How often KEEPALIVEs go out while the sessions are held open: well
/// inside the 90 s hold time.
const KEEPALIVE_EVERY: Duration = Duration::from_secs(20);
#[tokio::main]
async fn main() {
let to: SocketAddr = std::env::var("FEED_TO")
.unwrap_or_else(|_| "127.0.0.1:1790".into())
.parse()
.expect("FEED_TO is an address and port");
let peers = env("FEED_PEERS", 1) as u32;
let shape = Shape::INTERNET.scaled_down(env("FEED_SCALE_DOWN", 1));
let from = std::env::var("FEED_FROM").ok();
let first_asn = env("FEED_ASN", 64513) as u32;
let hold = Duration::from_secs(env("FEED_HOLD_SECS", 30) as u64);
let started = Instant::now();
let prefixes = Prefixes::generate(shape);
let tables: Vec<Vec<u8>> = (1..=peers)
.map(|peer| table_stream(&prefixes, shape, peer))
.collect();
let routes = shape.prefixes() * peers as usize;
let bytes: usize = tables.iter().map(Vec::len).sum();
println!(
"table: {} IPv4 + {} IPv6 prefixes from each of {peers} peer(s): {routes} routes, {:.1} MiB on the wire, made in {:.2} s",
shape.ipv4_prefixes,
shape.ipv6_prefixes,
mib(bytes),
started.elapsed().as_secs_f64()
);
// The collector first, so that it is there as the table arrives, as a
// router's collector session is after a restart.
let collector = env("FEED_COLLECTOR", 0) == 1;
let mut draining = Vec::new();
let mut collector_writer = None;
if collector {
let to: SocketAddr = std::env::var("FEED_COLLECTOR_TO")
.unwrap_or_else(|_| "[::1]:1790".into())
.parse()
.expect("FEED_COLLECTOR_TO is an IPv6 address and port");
let socket = TcpSocket::new_v6().expect("a socket");
if let Ok(from) = std::env::var("FEED_COLLECTOR_FROM") {
let address: std::net::Ipv6Addr = from.parse().expect("an IPv6 address");
socket
.bind(SocketAddr::new(address.into(), 0))
.expect("the collector's address is ours to connect from");
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake_with(
212_232,
Ipv4Addr::new(10, 0, 0, 200),
90,
collector_capabilities(
212_232,
&[AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST],
true,
),
)
.await
.expect("the daemon takes the collector's session");
let (writer, sink) = session.drain();
draining.push(sink);
println!("collector session established");
collector_writer = Some(writer);
}
let mut sessions = Vec::new();
for peer in 1..=peers {
let socket = TcpSocket::new_v4().expect("a socket");
if let Some(from) = &from {
let address: Ipv4Addr = format!("{from}.{peer}")
.parse()
.expect("FEED_FROM is the first three octets of an IPv4 address");
socket
.bind(SocketAddr::new(address.into(), 0))
.expect("the peer's address is ours to connect from");
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(
first_asn + peer - 1,
Ipv4Addr::new(10, 0, 0, peer as u8),
90,
)
.await
.expect("the daemon takes the session");
// Whatever the daemon sends back, KEEPALIVEs as a rule, is read
// and dropped, so that the writer is never held up.
let (writer, sink) = session.drain();
draining.push(sink);
sessions.push(writer);
}
println!("{peers} session(s) established; sending");
let started = Instant::now();
let senders: Vec<_> = sessions
.into_iter()
.zip(tables)
.map(|(mut writer, table)| {
tokio::spawn(async move {
let started = Instant::now();
writer.write_all(&table).await.expect("the daemon reads");
(writer, started.elapsed())
})
})
.collect();
let mut writers: Vec<OwnedWriteHalf> = collector_writer.into_iter().collect();
for (index, sender) in senders.into_iter().enumerate() {
let (writer, took) = sender.await.expect("the sender runs");
println!(
"peer {}: table written in {:.3} s",
index + 1,
took.as_secs_f64()
);
writers.push(writer);
}
let elapsed = started.elapsed();
println!(
"all {routes} routes written in {:.3} s: {:.2} M routes/s into the sockets",
elapsed.as_secs_f64(),
routes as f64 / elapsed.as_secs_f64() / 1e6
);
let keepalive = TestPeer::keepalive_bytes();
let deadline = Instant::now() + hold;
while Instant::now() < deadline {
tokio::time::sleep(KEEPALIVE_EVERY.min(deadline - Instant::now())).await;
for writer in &mut writers {
// A session the daemon closed is left alone.
let _ = writer.write_all(&keepalive).await;
}
}
println!("sessions closed after {} s", hold.as_secs());
drop(draining);
}
crates/bench/examples/packing.rs
+71
View File
@@ -0,0 +1,71 @@
//! How many UPDATEs, and how many octets, a table takes when the routes
//! that share attributes are grouped only within slices of the table
//! taken in shard order, for a few slice sizes: what streaming a table
//! out in slices costs against grouping it whole.
use std::collections::HashMap;
use std::hash::{BuildHasher, RandomState};
use std::path::PathBuf;
use bgp_bench::mrt::Dump;
fn main() {
let path = std::env::var("BENCH_MRT")
.map(PathBuf::from)
.unwrap_or_else(|_| "crates/bench/data/rrc03-bview.20260915.0000.gz".into());
let dump = Dump::read(&path).expect("a readable dump");
let peers = dump.largest_peers(usize::MAX);
let full: Vec<usize> = peers
.iter()
.copied()
.filter(|&p| dump.routes[p].len() > 800_000)
.collect();
println!("{} peers with a full table", full.len());
let hasher = RandomState::new();
for (what, chosen) in [
("one full table", vec![full[0]]),
("every full table, every path", full.clone()),
] {
// (hash of prefix, attribute set, prefix octets) in shard order.
let mut routes: Vec<(u64, u32, usize)> = chosen
.iter()
.flat_map(|&p| dump.routes[p].iter())
.map(|(prefix, set)| {
(
hasher.hash_one(prefix),
*set,
1 + usize::from(prefix.length()).div_ceil(8) + 4,
)
})
.collect();
routes.sort_unstable_by_key(|r| r.0);
println!("== {what}: {} routes", routes.len());
for slice in [4_096usize, 16_384, 65_536, 262_144, usize::MAX] {
let mut updates = 0usize;
let mut octets = 0usize;
for chunk in routes.chunks(slice.min(routes.len())) {
let mut groups: HashMap<u32, usize> = HashMap::new();
for &(_, set, prefix_octets) in chunk {
*groups.entry(set).or_insert(0) += prefix_octets;
}
for (set, prefix_octets) in groups {
// One UPDATE per 4096 octets of prefixes, roughly.
let attrs = dump.attribute_sets[set as usize].len();
let messages = prefix_octets.div_ceil(4000 - attrs.min(3000)).max(1);
updates += messages;
octets += messages * (23 + attrs) + prefix_octets;
}
}
println!(
" slice {:>9}: {:>9} UPDATEs, {:>5.1} MB on the wire",
if slice == usize::MAX {
"whole".to_string()
} else {
slice.to_string()
},
updates,
octets as f64 / 1e6
);
}
}
}
crates/bench/src/lib.rs
+378
View File
@@ -0,0 +1,378 @@
//! A synthetic copy of the Internet routing table, as the UPDATE messages a
//! peer would send to announce it.
//!
//! It is made up, not captured, so that the benchmarks of a synthetic table
//! need no data files and give the same input every run; the one of a real
//! table reads an MRT dump instead, see [`mrt`]. It is shaped like the real
//! thing where that matters for cost:
//!
//! - about a million IPv4 prefixes, most of them /24s, and about a quarter
//! of a million IPv6 prefixes, most of them /48s;
//! - prefixes that share their attributes are announced together, a handful
//! to an UPDATE, as routers do;
//! - AS paths a few hops long, and communities on most routes;
//! - several peers announce the same prefixes, each with attributes of its
//! own.
pub mod memory;
pub mod mrt;
pub mod report;
// The benchmarks allocate as the daemon does (bin/bgp/src/main.rs), with
// jemalloc; the C library's allocator would put its own locks in the
// profile. Not when memory is counted: that needs the counting one.
#[cfg(all(not(feature = "count-memory"), not(target_env = "msvc")))]
#[global_allocator]
static ALLOCATOR: tikv_jemallocator::Jemalloc = tikv_jemallocator::Jemalloc;
use std::collections::HashSet;
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LargeCommunitiesPathAttribute,
LargeCommunity, Message, MpReachNlriPathAttribute, MultiExitDiscPathAttribute,
NextHopPathAttribute, NlriNextHop, OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::ParserContext;
use bytes::BytesMut;
/// How big a table to make.
#[derive(Debug, Clone, Copy)]
pub struct Shape {
pub ipv4_prefixes: usize,
pub ipv6_prefixes: usize,
/// How many prefixes share one set of attributes, on average.
pub prefixes_per_attribute_set: usize,
}
impl Shape {
/// Roughly the default-free zone in 2026.
pub const INTERNET: Self = Self {
ipv4_prefixes: 1_000_000,
ipv6_prefixes: 230_000,
prefixes_per_attribute_set: 6,
};
pub fn prefixes(&self) -> usize {
self.ipv4_prefixes + self.ipv6_prefixes
}
/// The same shape, `factor` times smaller.
pub fn scaled_down(&self, factor: usize) -> Self {
Self {
ipv4_prefixes: self.ipv4_prefixes / factor,
ipv6_prefixes: self.ipv6_prefixes / factor,
..*self
}
}
}
/// A small deterministic generator (SplitMix64); no need for a dependency.
pub struct Rng(u64);
impl Rng {
pub fn new(seed: u64) -> Self {
Self(seed)
}
pub fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
z ^ (z >> 31)
}
/// A number below `bound`.
pub fn below(&mut self, bound: u64) -> u64 {
self.next_u64() % bound
}
/// An entry of `weighted`, picked with probability proportional to its
/// weight.
fn pick<T: Copy>(&mut self, weighted: &[(T, u64)]) -> T {
let total: u64 = weighted.iter().map(|(_, w)| w).sum();
let mut at = self.below(total);
for (value, weight) in weighted {
if at < *weight {
return *value;
}
at -= weight;
}
weighted[weighted.len() - 1].0
}
}
/// The prefixes of the table. Every peer announces these same prefixes.
pub struct Prefixes {
pub ipv4: Vec<IpPrefix>,
pub ipv6: Vec<IpPrefix>,
}
impl Prefixes {
pub fn generate(shape: Shape) -> Self {
let mut rng = Rng::new(0x1234_5678);
// Shares of prefix lengths, close to what the table looks like.
let v4_lengths: [(u8, u64); 9] = [
(24, 60),
(23, 10),
(22, 11),
(21, 5),
(20, 4),
(19, 3),
(18, 2),
(16, 4),
(15, 1),
];
let v6_lengths: [(u8, u64); 7] = [
(48, 48),
(32, 14),
(40, 10),
(44, 10),
(36, 6),
(29, 6),
(47, 6),
];
let mut seen = HashSet::new();
let mut ipv4 = Vec::with_capacity(shape.ipv4_prefixes);
while ipv4.len() < shape.ipv4_prefixes {
let length = rng.pick(&v4_lengths);
// Unicast space, 1.0.0.0 to 223.255.255.255.
let address = 0x0100_0000 + (rng.next_u64() as u32) % 0xdf00_0000;
let prefix = IpPrefix::new(IpVersion::Ipv4, address.to_be_bytes(), length)
.expect("four bytes cover any IPv4 prefix");
if seen.insert(prefix) {
ipv4.push(prefix);
}
}
let mut ipv6 = Vec::with_capacity(shape.ipv6_prefixes);
while ipv6.len() < shape.ipv6_prefixes {
let length = rng.pick(&v6_lengths);
// Global unicast, 2000::/3.
let address = (0x2000_u128 << 112) | (u128::from(rng.next_u64()) << 61 >> 3);
let prefix = IpPrefix::new(IpVersion::Ipv6, address.to_be_bytes(), length)
.expect("sixteen bytes cover any IPv6 prefix");
if seen.insert(prefix) {
ipv6.push(prefix);
}
}
Self { ipv4, ipv6 }
}
}
/// The UPDATE messages with which the peer numbered `peer` announces the
/// whole table. The peer's AS is `64512 + peer`.
pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<UpdateMessage> {
let mut rng = Rng::new(0x9999 + u64::from(peer));
let peer_asn = 64512 + peer;
let attributes = |rng: &mut Rng| {
let hops = 2 + rng.below(4) as usize;
let mut path = vec![peer_asn];
path.extend((0..hops).map(|_| 1 + rng.below(400_000) as u32));
let attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
PathAttribute::AsPath(AsPathAttribute {
segments: vec![AsPathSegment {
ordered: true,
path,
}],
}),
];
let mut later = Vec::new();
if rng.below(4) == 0 {
later.push(PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(
rng.below(1000) as u32,
)));
}
let communities = rng.below(7) as usize;
if communities > 0 {
later.push(PathAttribute::Communities(CommunitiesPathAttribute(
(0..communities)
.map(|_| (rng.below(65000) as u16, rng.below(1000) as u16))
.collect(),
)));
}
let large = if rng.below(5) == 0 {
Some(PathAttribute::LargeCommunities(
LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: peer_asn,
data_1: rng.below(100) as u32,
data_2: rng.below(100) as u32,
}],
},
))
} else {
None
};
(attributes, later, large)
};
let mut updates = Vec::new();
// Prefixes that share attributes travel together. A set's prefixes are a
// run of the (randomly ordered) list, a few long.
let mut grouped = |list: &[IpPrefix], rng: &mut Rng, ipv6: bool| {
let mut rest = list;
while !rest.is_empty() {
let size = (1 + rng.below(2 * shape.prefixes_per_attribute_set as u64 - 1) as usize)
.min(rest.len());
let (group, after) = rest.split_at(size);
rest = after;
let (mut path_attributes, later, large) = attributes(rng);
let update = if ipv6 {
// In order of type code: MED and COMMUNITIES come before
// MP_REACH_NLRI.
path_attributes.extend(later);
path_attributes.push(PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(
0x2001,
0xdb8,
0,
0,
0,
0,
0,
peer as u16,
)),
prefixes: Nlri::plain(group.iter().copied()),
}));
path_attributes.extend(large);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: vec![],
}
} else {
path_attributes.push(PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(
192, 0, 2, peer as u8,
))));
path_attributes.extend(later);
path_attributes.extend(large);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: Nlri::plain(group.iter().copied()),
}
};
updates.push(update);
}
};
grouped(&prefixes.ipv4, &mut rng, false);
grouped(&prefixes.ipv6, &mut rng, true);
updates
}
/// The whole table as the peer numbered `peer` sends it, one stream of
/// bytes: [`announcements`], [`encoded`] and run together, for a session
/// that writes the table in one go.
pub fn table_stream(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<u8> {
encoded(&announcements(prefixes, shape, peer))
.iter()
.flat_map(|message| message.iter().copied())
.collect()
}
/// `updates` as they are on the wire, on a session with four-octet AS
/// numbers.
pub fn encoded(updates: &[UpdateMessage]) -> Vec<BytesMut> {
let ctx = wire_context();
updates
.iter()
.map(|update| {
let mut wire = BytesMut::new();
Message::Update(update.clone())
.to_wire(&ctx, &mut wire)
.expect("a few prefixes fit one message");
wire
})
.collect()
}
/// The context of the sessions the benchmarks pretend to have: an external
/// peer with four-octet AS numbers.
pub fn wire_context() -> ParserContext {
ParserContext {
four_octet_asn: true,
ibgp: false,
add_path: Vec::new(),
}
}
/// The value of the environment variable `name` as a number, or `default`
/// when it is not set or is not one: how the benchmarks take their size.
pub fn env(name: &str, default: usize) -> usize {
std::env::var(name)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
#[cfg(test)]
mod tests {
use super::*;
/// A small table, enough for every prefix length and attribute to
/// come up.
fn shape() -> Shape {
Shape::INTERNET.scaled_down(1000)
}
/// The benchmarks are compared from one run to the next, so the table
/// they are fed must not change between runs: the generator is
/// seeded, and nothing in it depends on where things are in memory.
#[test]
fn the_table_is_the_same_every_run() {
let shape = shape();
let once = Prefixes::generate(shape);
let again = Prefixes::generate(shape);
assert_eq!(once.ipv4, again.ipv4);
assert_eq!(once.ipv6, again.ipv6);
assert_eq!(once.ipv4.len(), shape.ipv4_prefixes);
assert_eq!(once.ipv6.len(), shape.ipv6_prefixes);
for peer in [1, 2] {
assert_eq!(
table_stream(&once, shape, peer),
table_stream(&again, shape, peer)
);
}
assert_ne!(
table_stream(&once, shape, 1),
table_stream(&once, shape, 2),
"each peer has attributes of its own"
);
}
/// What a peer of the benchmarks sends is well formed, and says what
/// it was meant to: every UPDATE reads back as it was made.
#[test]
fn every_announcement_decodes_to_itself() {
let shape = shape();
let prefixes = Prefixes::generate(shape);
let updates = announcements(&prefixes, shape, 1);
let ctx = wire_context();
let announced: usize = updates
.iter()
.map(|update| {
update.announced_routes.len()
+ update
.path_attributes
.iter()
.map(|attribute| match attribute {
PathAttribute::MpReachNlri(reach) => reach.prefixes.len(),
_ => 0,
})
.sum::<usize>()
})
.sum();
assert_eq!(announced, shape.prefixes());
for (update, wire) in updates.iter().zip(encoded(&updates)) {
let decoded = Message::decode(&ctx, &wire).expect("well formed");
assert_eq!(decoded, Message::Update(update.clone()));
}
}
}
crates/bench/src/memory.rs
+60
View File
@@ -0,0 +1,60 @@
//! Counting the memory a benchmark holds.
//!
//! With the `count-memory` feature every allocation is counted, and
//! [`Region::held`] says how many bytes were allocated and not freed since
//! the region began. Without it nothing is counted and the answer is
//! `None`: the counters are shared by all threads, and counting slows down
//! exactly the code whose speed is being measured.
#[cfg(feature = "count-memory")]
mod counting {
use std::alloc::System;
use stats_alloc::{INSTRUMENTED_SYSTEM, StatsAlloc};
#[global_allocator]
static GLOBAL: &StatsAlloc<System> = &INSTRUMENTED_SYSTEM;
pub struct Region(stats_alloc::Region<'static, System>);
impl Region {
pub fn begin() -> Self {
Self(stats_alloc::Region::new(GLOBAL))
}
pub fn held(&self) -> Option<i64> {
let stats = self.0.change();
Some(stats.bytes_allocated as i64 - stats.bytes_deallocated as i64)
}
}
}
#[cfg(not(feature = "count-memory"))]
mod counting {
pub struct Region;
impl Region {
pub fn begin() -> Self {
Self
}
pub fn held(&self) -> Option<i64> {
None
}
}
}
pub use counting::Region;
/// "`bytes` MiB = n bytes a `unit`", or a note that memory is not being
/// counted.
pub fn describe(held: Option<i64>, count: usize, unit: &str) -> String {
match held {
Some(bytes) => format!(
"{:.0} MiB = {:.0} bytes a {unit}",
bytes as f64 / (1024.0 * 1024.0),
bytes as f64 / count as f64
),
None => "memory not counted (run with --features count-memory)".into(),
}
}
crates/bench/src/mrt.rs
+227
View File
@@ -0,0 +1,227 @@
//! Reads a table dump in MRT format (RFC 6396, TABLE_DUMP_V2), as published
//! by RIPE RIS and RouteViews: the routes that every peer of a route
//! collector had announced to it at one moment.
//!
//! Only what the benchmarks need is read: the IPv4 and IPv6 unicast tables.
use std::collections::HashMap;
use std::fs::File;
use std::io::{self, BufReader, Read};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::path::Path;
use bgp_packet::message::{NlriNextHop, PathAttribute, attr_type, split_attribute};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use flate2::read::MultiGzDecoder;
use crate::wire_context;
const TABLE_DUMP_V2: u16 = 13;
const PEER_INDEX_TABLE: u16 = 1;
const RIB_IPV4_UNICAST: u16 = 2;
const RIB_IPV6_UNICAST: u16 = 4;
/// A peer of the collector.
#[derive(Debug, Clone)]
pub struct DumpPeer {
pub address: IpAddr,
pub asn: u32,
}
/// The contents of a table dump.
pub struct Dump {
/// When the dump was taken, in seconds since the Unix epoch.
pub timestamp: u32,
pub peers: Vec<DumpPeer>,
/// Every distinct list of path attributes in the dump, as it is on the
/// wire. Routes refer to these by index.
pub attribute_sets: Vec<Box<[u8]>>,
/// For each peer (by index into `peers`), its routes: a prefix and the
/// index of its attributes.
pub routes: Vec<Vec<(IpPrefix, u32)>>,
}
impl Dump {
/// Reads the dump at `path`, gzip-compressed as it is published.
pub fn read(path: &Path) -> io::Result<Self> {
let mut input = BufReader::with_capacity(
1 << 20,
MultiGzDecoder::new(BufReader::new(File::open(path)?)),
);
let mut dump = Dump {
timestamp: 0,
peers: Vec::new(),
attribute_sets: Vec::new(),
routes: Vec::new(),
};
let mut interned: HashMap<Box<[u8]>, u32> = HashMap::new();
let mut body = Vec::new();
loop {
let mut header = [0u8; 12];
match input.read_exact(&mut header) {
Ok(()) => {}
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e),
}
let timestamp = u32::from_be_bytes([header[0], header[1], header[2], header[3]]);
let kind = u16::from_be_bytes([header[4], header[5]]);
let subtype = u16::from_be_bytes([header[6], header[7]]);
let length = u32::from_be_bytes([header[8], header[9], header[10], header[11]]);
body.resize(length as usize, 0);
input.read_exact(&mut body)?;
if kind != TABLE_DUMP_V2 {
continue;
}
let mut record = Cursor(&body);
let malformed = || io::Error::new(io::ErrorKind::InvalidData, "truncated MRT record");
match subtype {
PEER_INDEX_TABLE => {
dump.timestamp = timestamp;
dump.peers = peer_index_table(&mut record).ok_or_else(malformed)?;
dump.routes = vec![Vec::new(); dump.peers.len()];
}
RIB_IPV4_UNICAST | RIB_IPV6_UNICAST => {
let version = if subtype == RIB_IPV4_UNICAST {
IpVersion::Ipv4
} else {
IpVersion::Ipv6
};
rib_entries(&mut record, version, &mut dump, &mut interned)
.ok_or_else(malformed)?;
}
_ => {}
}
}
Ok(dump)
}
pub fn route_count(&self) -> usize {
self.routes.iter().map(Vec::len).sum()
}
/// The peers with the most routes, most first: the ones that send a
/// full table.
pub fn largest_peers(&self, count: usize) -> Vec<usize> {
let mut peers: Vec<usize> = (0..self.peers.len()).collect();
peers.sort_by_key(|&peer| std::cmp::Reverse(self.routes[peer].len()));
peers.truncate(count);
peers
}
}
struct Cursor<'a>(&'a [u8]);
impl<'a> Cursor<'a> {
fn take(&mut self, n: usize) -> Option<&'a [u8]> {
if self.0.len() < n {
return None;
}
let (taken, rest) = self.0.split_at(n);
self.0 = rest;
Some(taken)
}
fn u8(&mut self) -> Option<u8> {
self.take(1).map(|b| b[0])
}
fn u16(&mut self) -> Option<u16> {
self.take(2).map(|b| u16::from_be_bytes([b[0], b[1]]))
}
fn u32(&mut self) -> Option<u32> {
self.take(4)
.map(|b| u32::from_be_bytes([b[0], b[1], b[2], b[3]]))
}
}
fn peer_index_table(record: &mut Cursor<'_>) -> Option<Vec<DumpPeer>> {
record.u32()?; // collector's BGP identifier
let name = record.u16()?;
record.take(name.into())?;
let count = record.u16()?;
let mut peers = Vec::with_capacity(count.into());
for _ in 0..count {
let kind = record.u8()?;
record.u32()?; // peer's BGP identifier
let address = if kind & 1 == 1 {
let bytes: [u8; 16] = record.take(16)?.try_into().ok()?;
IpAddr::from(Ipv6Addr::from(bytes))
} else {
let bytes: [u8; 4] = record.take(4)?.try_into().ok()?;
IpAddr::from(Ipv4Addr::from(bytes))
};
let asn = if kind & 2 == 2 {
record.u32()?
} else {
record.u16()?.into()
};
peers.push(DumpPeer { address, asn });
}
Some(peers)
}
fn rib_entries(
record: &mut Cursor<'_>,
version: IpVersion,
dump: &mut Dump,
interned: &mut HashMap<Box<[u8]>, u32>,
) -> Option<()> {
record.u32()?; // sequence number
let length = record.u8()?;
let bytes = record.take(usize::from(length.div_ceil(8)))?;
// A prefix the collector should not have had is not ours to fix.
let prefix = IpPrefix::new(version, bytes, length).ok();
let count = record.u16()?;
for _ in 0..count {
let peer = usize::from(record.u16()?);
record.u32()?; // when the route was received
let attributes_len = record.u16()?;
let attributes = record.take(attributes_len.into())?;
let (Some(prefix), Some(routes)) = (prefix, dump.routes.get_mut(peer)) else {
continue;
};
let id = match interned.get(attributes) {
Some(id) => *id,
None => {
let id = u32::try_from(dump.attribute_sets.len()).ok()?;
dump.attribute_sets.push(attributes.into());
interned.insert(attributes.into(), id);
id
}
};
routes.push((prefix, id));
}
Some(())
}
/// The path attributes of a route in a dump, parsed.
///
/// In a dump the AS numbers are always four octets wide, and
/// MP_REACH_NLRI holds only the next hop (RFC 6396 §4.3.4): the prefix is in
/// the record. The next hop is returned beside the other attributes, as the
/// RIB keeps it. `None` if the attributes cannot be parsed.
pub fn parse_attributes(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
let ctx = wire_context();
let mut attributes = Vec::new();
let mut next_hop = None;
while !wire.is_empty() {
let (raw, rest) = split_attribute(wire).ok()?;
if raw.type_code == attr_type::MP_REACH_NLRI {
// The value is the next hop's length and the next hop, and
// nothing of the family, reserved octet and prefixes that
// follow in an UPDATE, so the attribute's own parser would
// refuse it.
let (length, address) = raw.value.split_first()?;
if usize::from(*length) != address.len() {
return None;
}
next_hop = Some(NlriNextHop::from_bytes(address)?);
} else {
let ([], attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()? else {
return None;
};
attributes.push(attribute);
}
wire = rest;
}
Some((attributes, next_hop))
}
crates/bench/src/report.rs
+31
View File
@@ -0,0 +1,31 @@
//! How the benchmarks print what they measured: one row a measurement,
//! what was measured in a column wide enough for the longest of them, the
//! time, and whatever else there is to say about it. The rows are what a
//! run is compared against, so their shape is settled here once.
use std::time::Duration;
/// How wide the column that says what was measured is. The widest row is
/// the mesh benchmark's, so that is what every benchmark lines up to.
pub const WIDTH: usize = 52;
/// `bytes` in mebibytes.
pub fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
/// `count` things in `elapsed`, as a rate: millions a second where there
/// are that many, thousands otherwise.
pub fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
/// Prints one row: `what` took `elapsed`, and `detail` says more.
pub fn row(what: &str, elapsed: Duration, detail: impl std::fmt::Display) {
println!("{what:<WIDTH$} {:>8.3} s {detail}", elapsed.as_secs_f64());
}
crates/fib/Cargo.toml
+32
View File
@@ -0,0 +1,32 @@
[package]
name = "bgp-fib"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
bgp-packet.workspace = true
bgp-primitives.workspace = true
bgp-rib.workspace = true
bytes.workspace = true
foldhash.workspace = true
prometheus.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["full"] }
tracing.workspace = true
# The socket to the kernel. Only Linux has it; everything else in this
# crate builds, and is tested, anywhere.
[target.'cfg(target_os = "linux")'.dependencies]
netlink-sys = { version = "0.8", features = ["tokio_socket"] }
# For a socket option netlink-sys has no name for.
nix = { workspace = true, features = ["socket"] }
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/fib/src/interfaces.rs
+286
View File
@@ -0,0 +1,286 @@
//! The machine's interfaces and their addresses, as the kernel reports
//! them, and what they say about a next hop: whether it can be reached
//! directly, and from which interface.
use std::collections::{BTreeMap, HashSet};
use std::net::IpAddr;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::InterfaceStatus;
use crate::netlink::{Address, Link, Message};
/// The interface a neighbor is reached on.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Connected {
/// The interface's index.
pub interface: u32,
/// Our own address in the network the neighbor is in.
pub local: IpAddr,
}
/// The interfaces there are and the addresses they have.
#[derive(Debug, Clone, Default)]
pub struct Interfaces {
/// By index, which is the order they are given out in too.
links: BTreeMap<u32, Link>,
addresses: HashSet<Address>,
}
/// `address` as the prefix that is only it.
fn host(address: IpAddr) -> IpPrefix {
match address {
IpAddr::V4(v4) => IpPrefix::new(IpVersion::Ipv4, v4.octets(), 32),
IpAddr::V6(v6) => IpPrefix::new(IpVersion::Ipv6, v6.octets(), 128),
}
.expect("an address is a prefix of its own length")
}
impl Interfaces {
/// Takes in what a message from the kernel says about links and
/// addresses. Returns whether anything is different for it.
pub fn apply(&mut self, message: &Message) -> bool {
match message {
Message::Link {
present: true,
link,
} => self.links.insert(link.index, link.clone()).as_ref() != Some(link),
Message::Link {
present: false,
link,
} => {
self.addresses
.retain(|address| address.interface != link.index);
self.links.remove(&link.index).is_some()
}
Message::Address {
present: true,
address,
} => self.addresses.insert(*address),
Message::Address {
present: false,
address,
} => self.addresses.remove(address),
_ => false,
}
}
/// The interfaces, in order of index.
pub fn links(&self) -> impl Iterator<Item = &Link> {
self.links.values()
}
/// The interfaces and their addresses, for showing: in order of index,
/// each with its addresses in order.
pub fn status(&self) -> Vec<InterfaceStatus> {
self.links()
.map(|link| {
let mut addresses: Vec<(IpAddr, u8)> = self
.addresses_of(link.index)
.map(|address| (address.local, address.network.length()))
.collect();
addresses.sort();
InterfaceStatus {
index: link.index,
name: link.name.clone(),
up: link.up,
addresses,
}
})
.collect()
}
pub fn link(&self, index: u32) -> Option<&Link> {
self.links.get(&index)
}
/// The addresses of the interface with this index.
pub fn addresses_of(&self, interface: u32) -> impl Iterator<Item = &Address> {
self.addresses
.iter()
.filter(move |address| address.interface == interface)
}
/// Whether `address` is one of this machine's own.
pub fn is_local(&self, address: IpAddr) -> bool {
self.addresses.iter().any(|ours| ours.local == address)
}
/// Whether `prefix` is the network of one of this machine's addresses,
/// or inside one. Whether the interface is up does not matter: the
/// network is ours either way.
pub fn has_network_around(&self, prefix: &IpPrefix) -> bool {
self.addresses
.iter()
.any(|address| address.network.contains(prefix))
}
/// Where `neighbor` can be reached without a gateway: the interface
/// that is up and has an address in a network `neighbor` is in, the
/// most specific such network if there are several. `None` if there is
/// none, or if `neighbor` is this machine itself.
pub fn connected(&self, neighbor: IpAddr) -> Option<Connected> {
if self.is_local(neighbor) {
return None;
}
let neighbor = host(neighbor);
self.addresses
.iter()
.filter(|address| address.network.contains(&neighbor))
.filter(|address| {
self.links
.get(&address.interface)
.is_some_and(|link| link.up)
})
.max_by_key(|address| address.network.length())
.map(|address| Connected {
interface: address.interface,
local: address.local,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
fn link(index: u32, name: &str, up: bool) -> Message {
Message::Link {
present: true,
link: Link {
index,
name: name.into(),
up,
},
}
}
fn address(interface: u32, local: &str, network: &str) -> Address {
Address {
interface,
local: local.parse().unwrap(),
network: IpPrefix::try_from(network).unwrap(),
}
}
fn present(address: Address) -> Message {
Message::Address {
present: true,
address,
}
}
fn some() -> Interfaces {
let mut interfaces = Interfaces::default();
for message in [
link(1, "lo", true),
link(2, "eth0", true),
link(3, "eth1", true),
link(4, "eth2", false),
present(address(1, "127.0.0.1", "127.0.0.0/8")),
present(address(2, "192.0.2.1", "192.0.2.0/24")),
present(address(2, "2001:db8::1", "2001:db8::/64")),
// A more specific network inside eth0's, on eth1.
present(address(3, "192.0.2.129", "192.0.2.128/25")),
present(address(4, "198.51.100.1", "198.51.100.0/24")),
] {
assert!(interfaces.apply(&message));
}
interfaces
}
fn connected(interfaces: &Interfaces, neighbor: &str) -> Option<(u32, String)> {
interfaces
.connected(neighbor.parse().unwrap())
.map(|c| (c.interface, c.local.to_string()))
}
#[test]
fn a_neighbor_is_reached_from_the_interface_whose_network_it_is_in() {
let interfaces = some();
assert_eq!(
connected(&interfaces, "192.0.2.7"),
Some((2, "192.0.2.1".into()))
);
assert_eq!(
connected(&interfaces, "2001:db8::7"),
Some((2, "2001:db8::1".into()))
);
// The most specific network wins.
assert_eq!(
connected(&interfaces, "192.0.2.200"),
Some((3, "192.0.2.129".into()))
);
// Not in any network of ours.
assert_eq!(connected(&interfaces, "203.0.113.1"), None);
assert_eq!(connected(&interfaces, "2001:db8:1::1"), None);
// In the network of an interface that is down.
assert_eq!(connected(&interfaces, "198.51.100.7"), None);
// Ourselves.
assert_eq!(connected(&interfaces, "192.0.2.1"), None);
assert!(interfaces.is_local("192.0.2.1".parse().unwrap()));
assert!(!interfaces.is_local("192.0.2.7".parse().unwrap()));
}
#[test]
fn what_the_kernel_reports_changes_what_is_reachable() {
let mut interfaces = some();
// Told again what is known: nothing changes.
assert!(!interfaces.apply(&link(2, "eth0", true)));
assert!(!interfaces.apply(&present(address(2, "192.0.2.1", "192.0.2.0/24"))));
// An interface comes up.
assert!(interfaces.apply(&link(4, "eth2", true)));
assert_eq!(
connected(&interfaces, "198.51.100.7"),
Some((4, "198.51.100.1".into()))
);
// An address goes.
assert!(interfaces.apply(&Message::Address {
present: false,
address: address(3, "192.0.2.129", "192.0.2.128/25"),
}));
assert_eq!(
connected(&interfaces, "192.0.2.200"),
Some((2, "192.0.2.1".into()))
);
// An interface goes, and its addresses with it.
assert!(interfaces.apply(&Message::Link {
present: false,
link: Link {
index: 2,
name: "eth0".into(),
up: true
},
}));
assert_eq!(connected(&interfaces, "192.0.2.7"), None);
assert_eq!(interfaces.addresses_of(2).count(), 0);
// What is not about interfaces changes nothing.
assert!(!interfaces.apply(&Message::Done));
}
#[test]
fn the_interfaces_are_shown_in_order() {
let mut interfaces = some();
// Learned last, listed in its place.
assert!(interfaces.apply(&link(0, "dummy0", false)));
let shown = interfaces.status();
let indexes: Vec<u32> = shown.iter().map(|interface| interface.index).collect();
assert_eq!(indexes, [0, 1, 2, 3, 4]);
assert_eq!(
shown[2],
InterfaceStatus {
index: 2,
name: "eth0".into(),
up: true,
addresses: vec![
("192.0.2.1".parse().unwrap(), 24),
("2001:db8::1".parse().unwrap(), 64),
],
}
);
assert_eq!((shown[4].up, shown[4].addresses.len()), (false, 1));
assert!(shown[0].addresses.is_empty());
}
}
crates/fib/src/kernel.rs
+1349
View File
File diff suppressed because it is too large. Load diff
crates/fib/src/lib.rs
+331
View File
@@ -0,0 +1,331 @@
//! Putting the best routes into the kernel's routing table, so that the
//! machine forwards packets by them (Linux, over rtnetlink).
//!
//! The routing table chooses a best route for each prefix. This crate
//! follows those choices and keeps the kernel's table the same: one route
//! for each prefix, to the best route's next hop. It also learns from the
//! kernel what interfaces and addresses the machine has, which is what
//! says whether a next hop can be reached at all.
//!
//! - [`netlink`] is the kernel's messages, as bytes and back.
//! - [`interfaces`] is the machine's interfaces and what is reachable on
//! them; [`routes`] the kernel's own routes, which say the way to what
//! is not; [`topology`] the two together with the kernel's nexthop
//! objects, as one view.
//! - [`plan`] works out what to ask the kernel for.
//! - The rest, which only Linux has, asks it.
//!
//! # What is and is not touched
//!
//! Our routes are marked as BGP's ([`netlink::PROTOCOL_BGP`]) and carry
//! the configured metric, in the configured table. Only routes with that
//! mark, metric and table are ever replaced or removed; and all such
//! routes are taken to be ours, so two daemons must not share them.
//! Routes left behind by a run that ended badly are found at the next
//! start, and removed unless they are still wanted. A run that ends in
//! good order removes its routes.
//!
//! # What is installed
//!
//! A best route whose next hop can be reached: a neighbor, which is an
//! address in the network of one of the machine's interfaces, or an
//! address the kernel's own routes know the way to. In the second case
//! the route goes into the kernel with the gateway and interface of the
//! route that knows the way, since the kernel does not look a gateway up
//! in turn: a route whose next hop is a router of our own AS, reached over
//! a tunnel by a static route, is installed as a route into the tunnel.
//! Which of the kernel's routes count for this, and which never do, is the
//! business of [`routes`].
pub mod interfaces;
#[cfg(target_os = "linux")]
mod kernel;
pub mod netlink;
pub mod plan;
pub mod routes;
pub mod sockdiag;
pub mod topology;
use std::net::IpAddr;
use std::sync::{Arc, Mutex, PoisonError};
use std::time::Duration;
use bgp_rib::table::RibTable;
use tokio::sync::watch;
use tokio::task::JoinHandle;
/// The metric our routes get unless configured otherwise: the one other
/// routing daemons give routes from external BGP peers, so that ours rank
/// against static and connected routes as an operator expects.
pub const DEFAULT_METRIC: u32 = 20;
/// How long routes left by an earlier run are kept unless configured
/// otherwise: time for sessions to come up and tables to be sent.
pub const DEFAULT_STALE_TIME: Duration = Duration::from_secs(120);
/// Where our routes go in the kernel.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FibConfig {
/// The routing table: [`netlink::TABLE_MAIN`] to forward by them.
pub table: u32,
/// The metric, which with the table says which routes are ours.
pub metric: u32,
/// Have routes name nexthop objects where the kernel has them (Linux
/// 5.3 and later), which makes installing many routes a good deal
/// faster. Without, every route carries its own gateway.
pub nexthop_objects: bool,
/// How long routes of ours that an earlier run left in the kernel are
/// kept after a start without the routing table having them. The
/// table is empty until the peers have sent theirs again; removing
/// everything at once would stop the machine forwarding meanwhile.
/// After this long, what has not been announced again goes.
pub stale_time: Duration,
/// The kernel's table whose routes say the way to a next hop that is
/// not a neighbor: normally `main`, with the connected and static
/// routes. BGP's own routes in it never count; see [`routes`].
pub resolution_table: u32,
/// Whether the default route counts as the way to a next hop.
pub resolve_via_default: bool,
}
impl Default for FibConfig {
fn default() -> Self {
Self {
table: netlink::TABLE_MAIN,
metric: DEFAULT_METRIC,
nexthop_objects: true,
stale_time: DEFAULT_STALE_TIME,
resolution_table: netlink::TABLE_MAIN,
resolve_via_default: false,
}
}
}
/// Why routes cannot be kept in the kernel.
#[derive(Debug, thiserror::Error)]
pub enum FibError {
#[error("installing routes in the kernel is only supported on Linux")]
Unsupported,
#[error("the kernel's routing socket failed: {0}")]
Io(#[from] std::io::Error),
#[error("the kernel sent something that is not netlink: {0}")]
Malformed(#[from] netlink::NetlinkError),
#[error("the kernel refused to list {what}: error {errno}")]
Dump { what: &'static str, errno: i32 },
#[error(
"not permitted to change the kernel's routes: the daemon needs the CAP_NET_ADMIN \
capability (or root)"
)]
NotPermitted,
}
/// One of the machine's interfaces, for showing.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct InterfaceStatus {
pub index: u32,
pub name: String,
pub up: bool,
/// Its addresses, each with the length of the network it is in.
pub addresses: Vec<(IpAddr, u8)>,
}
/// How things stand with the kernel, for showing.
#[derive(Debug, Clone, Default)]
pub struct Status {
/// Whether the kernel has been brought in step with the routing table
/// at least once since the start.
pub synced: bool,
/// How many routes of ours the kernel has.
pub installed: usize,
/// How many of them are kept from an earlier run though the routing
/// table does not have them (yet); see [`FibConfig::stale_time`].
pub kept: usize,
/// How many of them are blackhole routes, which drop the packets for
/// their prefix.
pub blackholes: usize,
/// Prefixes whose best route is not in the kernel because its next
/// hop is not usable, by why.
pub unresolved: Vec<(&'static str, usize)>,
/// Prefixes whose best route the kernel refused.
pub refused: usize,
/// How many requests the kernel has refused in all.
pub errors: u64,
/// The last such refusal.
pub last_error: Option<String>,
pub interfaces: Vec<InterfaceStatus>,
}
/// The machine's interfaces, routes and nexthop objects as the kernel
/// last described them, as far as next hops are concerned; `None` until
/// it has. Whoever holds one is told when they change.
pub type TopologyWatch = watch::Receiver<Option<Arc<topology::Topology>>>;
/// How the task that keeps the kernel's routes is to end.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Stop {
/// Taking our routes out of the kernel.
Withdraw,
/// Leaving them there, for the next run to find.
Keep,
}
struct Shared {
config: FibConfig,
status: Mutex<Status>,
topology: watch::Sender<Option<Arc<topology::Topology>>>,
stop: watch::Sender<Option<Stop>>,
/// How many routes of an earlier run were found in the kernel at the
/// start, once that has been looked at.
adopted: watch::Sender<Option<usize>>,
/// Set to have what is kept of an earlier run go now.
release: watch::Sender<bool>,
/// Whether the task has ended.
ended: watch::Sender<bool>,
}
/// Handle to the task that keeps the kernel's routes. Cheap to clone.
#[derive(Clone)]
pub struct Fib {
shared: Arc<Shared>,
}
impl Fib {
/// Starts keeping the kernel's routes in step with the best routes of
/// `rib`, and returns a handle with the task that does it. The task
/// ends only by failing, or when [`Fib::stop`] or [`Fib::leave`] is
/// called.
///
/// Must be called from within a Tokio runtime. Fails at once where
/// there is no kernel to talk to in this way.
pub fn spawn(
rib: Arc<RibTable>,
config: FibConfig,
) -> Result<(Self, JoinHandle<Result<(), FibError>>), FibError> {
let (stop, stopping) = watch::channel(None);
let fib = Self {
shared: Arc::new(Shared {
config,
status: Mutex::default(),
topology: watch::channel(None).0,
stop,
adopted: watch::channel(None).0,
release: watch::channel(false).0,
ended: watch::channel(false).0,
}),
};
let task = start(rib, fib.clone(), stopping)?;
Ok((fib, task))
}
pub fn config(&self) -> FibConfig {
self.shared.config
}
/// The kernel's view of the machine, for those who want to know
/// whether a next hop can be reached before the kernel is asked to use
/// it.
pub fn topology(&self) -> TopologyWatch {
self.shared.topology.subscribe()
}
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
fn publish_topology(&self, topology: &topology::Topology) {
self.shared
.topology
.send_replace(Some(Arc::new(topology.clone())));
}
pub fn status(&self) -> Status {
self.shared
.status
.lock()
.unwrap_or_else(PoisonError::into_inner)
.clone()
}
/// Has the task take our routes out of the kernel and end, and waits
/// until it has. Returns at once if it has ended already, for whatever
/// reason; the task's own result says how.
pub async fn stop(&self) {
self.end(Stop::Withdraw).await;
}
/// Has the task stop following the routing table and end, leaving our
/// routes in the kernel for the next run to find and keep for a while
/// (see [`FibConfig::stale_time`]), and waits until it has. For a
/// restart that is to go unnoticed by the packets being forwarded.
pub async fn leave(&self) {
self.end(Stop::Keep).await;
}
async fn end(&self, how: Stop) {
let mut ended = self.shared.ended.subscribe();
self.shared.stop.send_if_modified(|stop| {
let first = stop.is_none();
if first {
*stop = Some(how);
}
first
});
// Fails only if the handle that holds the sender is gone, and this
// is one.
let _ = ended.wait_for(|ended| *ended).await;
}
/// How many routes of an earlier run the kernel had at the start, once
/// the task has looked; zero if it ended before it did.
pub async fn adopted(&self) -> usize {
let mut adopted = self.shared.adopted.subscribe();
let mut ended = self.shared.ended.subscribe();
tokio::select! {
found = adopted.wait_for(Option::is_some) => {
found.ok().and_then(|found| *found).unwrap_or(0)
}
_ = ended.wait_for(|ended| *ended) => 0,
}
}
/// Has the routes of an earlier run that the routing table does not
/// have go now, rather than when [`FibConfig::stale_time`] is up: the
/// peers have all sent their routes, and what is not among them is no
/// longer anybody's.
pub fn release_kept(&self) {
self.shared.release.send_replace(true);
}
/// Changes the status, and gives back what `change` does.
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
fn update<R>(&self, change: impl FnOnce(&mut Status) -> R) -> R {
change(
&mut self
.shared
.status
.lock()
.unwrap_or_else(PoisonError::into_inner),
)
}
}
#[cfg(target_os = "linux")]
fn start(
rib: Arc<RibTable>,
fib: Fib,
stopping: watch::Receiver<Option<Stop>>,
) -> Result<JoinHandle<Result<(), FibError>>, FibError> {
// Opened here, so that a machine that cannot is known of at once.
let sockets = kernel::Sockets::open()?;
Ok(tokio::spawn(async move {
let result = kernel::run(rib, fib.clone(), sockets, stopping).await;
fib.shared.ended.send_replace(true);
result
}))
}
#[cfg(not(target_os = "linux"))]
fn start(
_rib: Arc<RibTable>,
_fib: Fib,
_stopping: watch::Receiver<Option<Stop>>,
) -> Result<JoinHandle<Result<(), FibError>>, FibError> {
Err(FibError::Unsupported)
}
crates/fib/src/netlink.rs
+1189
View File
File diff suppressed because it is too large. Load diff
crates/fib/src/plan.rs
+1327
View File
File diff suppressed because it is too large. Load diff
crates/fib/src/routes.rs
+266
View File
@@ -0,0 +1,266 @@
//! The kernel's own routes, as far as next hops are concerned: the way to
//! a next hop that is not a neighbor on any of our interfaces.
//!
//! Only the routes of one table count, the resolution table, which is
//! normally `main`: the connected and static routes, and an IGP's if there
//! is one. Routes put there by BGP, ours or another daemon's, never count:
//! a next hop that is reached only by way of a BGP route is reached by way
//! of something that may itself depend on a next hop, and a table that
//! leans on itself like that is one that falls over in a way nobody can
//! predict. The default route counts only if asked for: on a router, the
//! default points at a transit provider, and a next hop that is "reached"
//! only by sending the packets to the Internet at large is not reached.
use std::net::IpAddr;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::netlink::{Message, PROTOCOL_BGP, Route};
use crate::plan::HashMap;
/// The routes of the resolution table that may say the way to a next hop.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct KernelRoutes {
table: u32,
/// By prefix, the routes there are for it: usually one, but the kernel
/// keeps one for each metric.
routes: HashMap<IpPrefix, Vec<Route>>,
/// How many prefixes there are of each length, by IP version: the
/// lengths a lookup has to try.
lengths: [[u32; 129]; 2],
}
/// `address` as the prefix that is only it.
pub(crate) fn host(address: IpAddr) -> IpPrefix {
match address {
IpAddr::V4(v4) => IpPrefix::new(IpVersion::Ipv4, v4.octets(), 32),
IpAddr::V6(v6) => IpPrefix::new(IpVersion::Ipv6, v6.octets(), 128),
}
.expect("an address is a prefix of its own length")
}
fn version_index(version: IpVersion) -> usize {
match version {
IpVersion::Ipv4 => 0,
IpVersion::Ipv6 => 1,
}
}
impl KernelRoutes {
pub fn new(table: u32) -> Self {
Self {
table,
routes: HashMap::default(),
lengths: [[0; 129]; 2],
}
}
/// The table the routes are of.
pub fn table(&self) -> u32 {
self.table
}
/// Whether `route` is one that may say the way to a next hop.
fn counts(&self, route: &Route) -> bool {
route.table == self.table && route.protocol != PROTOCOL_BGP
}
/// Takes in what a message from the kernel says about routes. Returns
/// whether anything is different for it.
pub fn apply(&mut self, message: &Message) -> bool {
let Message::Route { present, route } = message else {
return false;
};
if !self.counts(route) {
return false;
}
let version = version_index(route.prefix.address_family());
let length = usize::from(route.prefix.length());
if *present {
let routes = self.routes.entry(route.prefix).or_default();
// A route for a prefix with a metric there is one for already
// takes that one's place: the kernel keeps one per metric.
match routes.iter_mut().find(|known| known.metric == route.metric) {
Some(known) if known == route => false,
Some(known) => {
*known = *route;
true
}
None => {
if routes.is_empty() {
self.lengths[version][length] += 1;
}
// In order of metric, so that two tables with the
// same routes compare equal however they were told.
let at = routes.partition_point(|known| known.metric < route.metric);
routes.insert(at, *route);
true
}
}
} else {
let Some(routes) = self.routes.get_mut(&route.prefix) else {
return false;
};
let before = routes.len();
routes.retain(|known| known.metric != route.metric);
let (after, none_left) = (routes.len(), routes.is_empty());
if none_left {
self.routes.remove(&route.prefix);
self.lengths[version][length] -= 1;
}
before != after
}
}
/// How many prefixes have a route.
pub fn len(&self) -> usize {
self.routes.len()
}
pub fn is_empty(&self) -> bool {
self.routes.is_empty()
}
/// The route the kernel would send packets for `address` by, from
/// this table: the one for the longest prefix that covers it, and of
/// several for that prefix the one with the lowest metric. The
/// default route too, if there is one and nothing longer covers the
/// address.
pub fn lookup(&self, address: IpAddr) -> Option<&Route> {
let whole = host(address);
let lengths = &self.lengths[version_index(whole.address_family())];
(0..=whole.length())
.rev()
.filter(|length| lengths[usize::from(*length)] > 0)
.find_map(|length| self.routes.get(&whole.shortened(length)))
.and_then(|routes| routes.iter().min_by_key(|route| route.metric))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::netlink::{KIND_UNICAST, TABLE_MAIN};
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(prefix_text: &str, gateway: Option<&str>, interface: u32, metric: u32) -> Route {
Route {
prefix: prefix(prefix_text),
table: TABLE_MAIN,
protocol: 4, // static
metric,
gateway: gateway.map(|g| g.parse().unwrap()),
interface: Some(interface),
nexthop: None,
kind: KIND_UNICAST,
}
}
fn present(route: Route) -> Message {
Message::Route {
present: true,
route,
}
}
fn gone(route: Route) -> Message {
Message::Route {
present: false,
route,
}
}
fn lookup(routes: &KernelRoutes, address: &str) -> Option<(String, Option<String>, u32)> {
routes.lookup(address.parse().unwrap()).map(|route| {
(
route.prefix.to_string(),
route.gateway.map(|g| g.to_string()),
route.metric,
)
})
}
#[test]
fn the_longest_prefix_with_the_lowest_metric_is_the_way() {
let mut routes = KernelRoutes::new(TABLE_MAIN);
for r in [
route("0.0.0.0/0", Some("192.0.2.1"), 2, 100),
route("10.0.0.0/8", Some("192.0.2.9"), 2, 0),
route("10.1.0.0/16", None, 7, 0),
route("10.1.0.0/16", Some("198.51.100.1"), 3, 50),
route("2001:db8::/32", Some("2001:db8::1"), 2, 1024),
] {
assert!(routes.apply(&present(r)));
}
assert_eq!(routes.len(), 4);
assert_eq!(
lookup(&routes, "10.2.3.4"),
Some(("10.0.0.0/8".into(), Some("192.0.2.9".into()), 0))
);
// Two routes for the prefix: the lower metric wins.
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.1.0.0/16".into(), None, 0))
);
assert_eq!(
lookup(&routes, "203.0.113.1"),
Some(("0.0.0.0/0".into(), Some("192.0.2.1".into()), 100))
);
assert_eq!(
lookup(&routes, "2001:db8:1::1"),
Some(("2001:db8::/32".into(), Some("2001:db8::1".into()), 1024))
);
assert_eq!(lookup(&routes, "2001:db9::1"), None);
// Told again what is known: nothing changes.
assert!(!routes.apply(&present(route("10.0.0.0/8", Some("192.0.2.9"), 2, 0))));
// The same prefix and metric with another gateway: replaced.
assert!(routes.apply(&present(route("10.0.0.0/8", Some("192.0.2.8"), 2, 0))));
assert_eq!(
lookup(&routes, "10.2.3.4"),
Some(("10.0.0.0/8".into(), Some("192.0.2.8".into()), 0))
);
// The better of the two for 10.1/16 goes: the other is the way.
assert!(routes.apply(&gone(route("10.1.0.0/16", None, 7, 0))));
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.1.0.0/16".into(), Some("198.51.100.1".into()), 50))
);
assert!(routes.apply(&gone(route("10.1.0.0/16", Some("198.51.100.1"), 3, 50))));
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.0.0.0/8".into(), Some("192.0.2.8".into()), 0))
);
// Gone already: nothing changes.
assert!(!routes.apply(&gone(route("10.1.0.0/16", None, 7, 0))));
assert_eq!(routes.len(), 3);
}
#[test]
fn routes_of_other_tables_and_of_bgp_do_not_count() {
let mut routes = KernelRoutes::new(TABLE_MAIN);
let other_table = Route {
table: 200,
..route("10.0.0.0/8", Some("192.0.2.9"), 2, 0)
};
let bgp = Route {
protocol: PROTOCOL_BGP,
..route("10.0.0.0/8", Some("192.0.2.9"), 2, 20)
};
assert!(!routes.apply(&present(other_table)));
assert!(!routes.apply(&present(bgp)));
assert!(routes.is_empty());
assert_eq!(lookup(&routes, "10.2.3.4"), None);
// What is not about routes changes nothing either.
assert!(!routes.apply(&Message::Done));
let mut table_200 = KernelRoutes::new(200);
assert!(table_200.apply(&present(other_table)));
assert_eq!(table_200.table(), 200);
assert_eq!(table_200.len(), 1);
}
}
crates/fib/src/sockdiag.rs
+420
View File
@@ -0,0 +1,420 @@
//! What the kernel knows about one of our TCP connections: the round-trip
//! time, the windows, the queues and what has held the connection up,
//! asked over the socket diagnostics netlink family (`NETLINK_SOCK_DIAG`,
//! as `ss -ti` asks). For the status page, when someone wants to know
//! why a session is slow.
use std::io;
use std::net::{IpAddr, SocketAddr};
use crate::netlink::{
AF_INET, AF_INET6, HEADER_LEN, attributes, flag, message_type, ne_u16, ne_u32, ne_u64,
};
/// A TCP connection as the kernel sees it. Times are in microseconds,
/// sizes in octets, rates in octets per second, unless said otherwise.
/// Fields the running kernel does not report are zero.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct TcpConnection {
/// The TCP state, by its number (1 is ESTABLISHED); see
/// [`TcpConnection::state_name`].
pub state: u8,
/// The congestion control algorithm, if the kernel named it.
pub congestion: Option<String>,
/// Octets received and not yet read by us.
pub recv_queue: u32,
/// Octets written by us and not yet acknowledged by the peer.
pub send_queue: u32,
pub rtt: u32,
pub rtt_variance: u32,
pub min_rtt: u32,
pub rto: u32,
pub mss: u32,
pub pmtu: u32,
/// The congestion window, in segments.
pub cwnd: u32,
/// The slow start threshold, in segments; `None` if there is none.
pub ssthresh: Option<u32>,
/// What the peer lets us send: its receive window.
pub send_window: u32,
/// What we let the peer send.
pub receive_window: u32,
pub receive_space: u32,
pub receive_ssthresh: u32,
pub send_window_scale: u8,
pub receive_window_scale: u8,
/// Segments sent and not acknowledged.
pub unacked: u32,
/// Segments retransmitted and still out.
pub retrans: u32,
pub lost: u32,
pub total_retrans: u32,
pub bytes_retrans: u64,
/// Octets in the socket's buffer that are not sent yet.
pub not_sent: u32,
pub pacing_rate: u64,
pub delivery_rate: u64,
pub bytes_sent: u64,
pub bytes_acked: u64,
pub bytes_received: u64,
pub segments_out: u32,
pub segments_in: u32,
/// How long the connection had something to send.
pub busy: u64,
/// Of that, how long it was held up by the peer's receive window.
pub limited_by_receive_window: u64,
/// And how long by our own send buffer being full.
pub limited_by_send_buffer: u64,
pub delivered: u32,
pub reordering: u32,
/// Milliseconds since data was last sent, received, and since the
/// last acknowledgement came.
pub last_data_sent_ms: u32,
pub last_data_received_ms: u32,
pub last_ack_received_ms: u32,
}
impl TcpConnection {
pub fn state_name(&self) -> &'static str {
match self.state {
1 => "established",
2 => "syn-sent",
3 => "syn-received",
4 => "fin-wait-1",
5 => "fin-wait-2",
6 => "time-wait",
7 => "close",
8 => "close-wait",
9 => "last-ack",
10 => "listen",
11 => "closing",
_ => "unknown",
}
}
}
const SOCK_DIAG_BY_FAMILY: u16 = 20;
const IPPROTO_TCP: u8 = 6;
const INET_DIAG_INFO: u16 = 2;
const INET_DIAG_CONG: u16 = 4;
/// `idiag_ext` bits asking for those two.
const EXTENSIONS: u8 = (1 << (INET_DIAG_INFO - 1)) | (1 << (INET_DIAG_CONG - 1));
const NO_COOKIE: u32 = u32::MAX;
const REQUEST_LEN: usize = HEADER_LEN + 56;
const DIAG_MSG_LEN: usize = 72;
/// The request for the connection from `local` to `peer`, with `sequence`.
pub fn request(local: SocketAddr, peer: SocketAddr, sequence: u32) -> Vec<u8> {
let family = match local {
SocketAddr::V4(_) => AF_INET,
SocketAddr::V6(_) => AF_INET6,
};
let mut buf = Vec::with_capacity(REQUEST_LEN);
buf.extend_from_slice(&(REQUEST_LEN as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&flag::REQUEST.to_ne_bytes());
buf.extend_from_slice(&sequence.to_ne_bytes());
buf.extend_from_slice(&0u32.to_ne_bytes());
// inet_diag_req_v2
buf.extend_from_slice(&[family, IPPROTO_TCP, EXTENSIONS, 0]);
buf.extend_from_slice(&u32::MAX.to_ne_bytes()); // any state
// inet_diag_sockid: ports in network order, addresses as 16 octets.
buf.extend_from_slice(&local.port().to_be_bytes());
buf.extend_from_slice(&peer.port().to_be_bytes());
buf.extend_from_slice(&address_octets(local.ip()));
buf.extend_from_slice(&address_octets(peer.ip()));
// The interface, which a socket on a link-local address is bound to
// and is found by.
let interface = match local {
SocketAddr::V6(v6) => v6.scope_id(),
SocketAddr::V4(_) => 0,
};
buf.extend_from_slice(&interface.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
debug_assert_eq!(buf.len(), REQUEST_LEN);
buf
}
fn address_octets(address: IpAddr) -> [u8; 16] {
let mut octets = [0u8; 16];
match address {
IpAddr::V4(v4) => octets[..4].copy_from_slice(&v4.octets()),
IpAddr::V6(v6) => octets = v6.octets(),
}
octets
}
/// What the kernel answered.
#[derive(Debug, PartialEq, Eq)]
pub enum Answer {
Connection(Box<TcpConnection>),
/// No such connection.
None,
/// The kernel refused, with this error number.
Refused(i32),
}
/// Reads the kernel's answer to a [`request`].
pub fn parse(buf: &[u8]) -> io::Result<Answer> {
let header = buf
.get(..HEADER_LEN)
.ok_or_else(|| io::Error::other("short netlink answer"))?;
let length = ne_u32(&header[0..4]) as usize;
let kind = ne_u16(&header[4..6]);
let message = buf
.get(HEADER_LEN..length.min(buf.len()))
.ok_or_else(|| io::Error::other("short netlink answer"))?;
if kind == message_type::ERROR {
let errno = -(message
.first_chunk::<4>()
.map_or(0, |e| i32::from_ne_bytes(*e)));
// ENOENT: no such socket.
return Ok(if errno == 2 {
Answer::None
} else {
Answer::Refused(errno)
});
}
if kind != SOCK_DIAG_BY_FAMILY || message.len() < DIAG_MSG_LEN {
return Err(io::Error::other("not a socket diagnostic"));
}
let mut connection = TcpConnection {
state: message[1],
recv_queue: ne_u32(&message[56..60]),
send_queue: ne_u32(&message[60..64]),
..TcpConnection::default()
};
for (kind, value) in attributes(&message[DIAG_MSG_LEN..]) {
match kind {
INET_DIAG_INFO => fill_from_tcp_info(&mut connection, value),
INET_DIAG_CONG => {
let name = value.split(|b| *b == 0).next().unwrap_or(&[]);
connection.congestion = Some(String::from_utf8_lossy(name).into_owned());
}
_ => {}
}
}
Ok(Answer::Connection(Box::new(connection)))
}
/// Fills in what `tcp_info` (linux/tcp.h) has, as far as the kernel sent
/// it: the structure has grown over the years, and a field that is not
/// there is left at zero.
fn fill_from_tcp_info(c: &mut TcpConnection, info: &[u8]) {
let u32_at = |at: usize| info.get(at..at + 4).map_or(0, ne_u32);
let u64_at = |at: usize| info.get(at..at + 8).map_or(0, ne_u64);
c.send_window_scale = info.get(6).map_or(0, |b| b & 0x0f);
c.receive_window_scale = info.get(6).map_or(0, |b| b >> 4);
c.rto = u32_at(8);
c.mss = u32_at(16);
c.unacked = u32_at(24);
c.lost = u32_at(32);
c.retrans = u32_at(36);
c.last_data_sent_ms = u32_at(44);
c.last_data_received_ms = u32_at(52);
c.last_ack_received_ms = u32_at(56);
c.pmtu = u32_at(60);
c.receive_ssthresh = u32_at(64);
c.rtt = u32_at(68);
c.rtt_variance = u32_at(72);
c.ssthresh = Some(u32_at(76)).filter(|t| *t < 0x7fff_ffff);
c.cwnd = u32_at(80);
c.reordering = u32_at(88);
c.receive_space = u32_at(96);
c.total_retrans = u32_at(100);
c.pacing_rate = u64_at(104);
c.bytes_acked = u64_at(120);
c.bytes_received = u64_at(128);
c.segments_out = u32_at(136);
c.segments_in = u32_at(140);
c.not_sent = u32_at(144);
c.min_rtt = u32_at(148);
c.delivery_rate = u64_at(160);
c.busy = u64_at(168);
c.limited_by_receive_window = u64_at(176);
c.limited_by_send_buffer = u64_at(184);
c.delivered = u32_at(192);
c.bytes_sent = u64_at(200);
c.bytes_retrans = u64_at(208);
c.send_window = u32_at(228);
c.receive_window = u32_at(232);
}
/// Asks the kernel about the TCP connection from `local` to `peer`.
/// `Ok(None)` if there is no such connection, or on an operating system
/// that cannot be asked.
#[cfg(target_os = "linux")]
pub async fn tcp_connection(
local: SocketAddr,
peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
use netlink_sys::protocols::NETLINK_SOCK_DIAG;
use netlink_sys::{AsyncSocket, AsyncSocketExt, TokioSocket};
let mut socket = TokioSocket::new(NETLINK_SOCK_DIAG)?;
socket
.socket_mut()
.bind(&netlink_sys::SocketAddr::new(0, 0))?;
let sequence = 1;
socket.send(&request(local, peer, sequence)).await?;
let mut buf = bytes::BytesMut::with_capacity(4096);
tokio::time::timeout(std::time::Duration::from_secs(2), socket.recv(&mut buf))
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "the kernel did not answer"))??;
match parse(&buf)? {
Answer::Connection(connection) => Ok(Some(*connection)),
Answer::None => Ok(None),
Answer::Refused(errno) => Err(io::Error::from_raw_os_error(errno)),
}
}
#[cfg(not(target_os = "linux"))]
pub async fn tcp_connection(
_local: SocketAddr,
_peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
Ok(None)
}
#[cfg(test)]
mod tests {
use super::*;
/// A reply as the kernel writes it, for an established connection,
/// with the first `info_len` octets of a `tcp_info` of the full
/// modern length: an older kernel sends a shorter one.
fn reply(info_len: usize) -> Vec<u8> {
let mut info = vec![0u8; 248];
info[0] = 1; // established
info[6] = 0x7a; // snd_wscale 10, rcv_wscale 7
let put32 = |info: &mut Vec<u8>, at: usize, v: u32| {
info[at..at + 4].copy_from_slice(&v.to_ne_bytes())
};
let put64 = |info: &mut Vec<u8>, at: usize, v: u64| {
info[at..at + 8].copy_from_slice(&v.to_ne_bytes())
};
put32(&mut info, 16, 1448); // mss
put32(&mut info, 24, 12); // unacked
put32(&mut info, 68, 2300); // rtt
put32(&mut info, 76, 0x7fff_ffff); // no ssthresh
put32(&mut info, 80, 42); // cwnd
put32(&mut info, 144, 65536); // notsent
put64(&mut info, 176, 900_000); // rwnd limited
put64(&mut info, 168, 1_000_000); // busy
put32(&mut info, 228, 131072); // snd_wnd
put32(&mut info, 232, 262144); // rcv_wnd
info.truncate(info_len);
let cong = b"cubic\0";
let mut message = vec![0u8; DIAG_MSG_LEN];
message[1] = 1;
message[60..64].copy_from_slice(&80_000u32.to_ne_bytes()); // wqueue
// attributes: INFO, then CONG
let mut attrs = Vec::new();
attrs.extend_from_slice(&((4 + info.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_INFO.to_ne_bytes());
attrs.extend_from_slice(&info);
attrs.extend_from_slice(&((4 + cong.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_CONG.to_ne_bytes());
attrs.extend_from_slice(cong);
while attrs.len() % 4 != 0 {
attrs.push(0);
}
let length = HEADER_LEN + message.len() + attrs.len();
let mut buf = Vec::new();
buf.extend_from_slice(&(length as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&0u16.to_ne_bytes()); // flags
buf.extend_from_slice(&1u32.to_ne_bytes()); // sequence
buf.extend_from_slice(&0u32.to_ne_bytes()); // port
buf.extend_from_slice(&message);
buf.extend_from_slice(&attrs);
buf
}
#[test]
fn a_reply_is_read_field_by_field() {
let Answer::Connection(c) = parse(&reply(248)).unwrap() else {
panic!("a connection");
};
assert_eq!(c.state_name(), "established");
assert_eq!(c.congestion.as_deref(), Some("cubic"));
assert_eq!(c.send_queue, 80_000);
assert_eq!((c.send_window_scale, c.receive_window_scale), (10, 7));
assert_eq!((c.mss, c.unacked, c.rtt, c.cwnd), (1448, 12, 2300, 42));
assert_eq!(c.ssthresh, None);
assert_eq!(c.not_sent, 65536);
assert_eq!((c.busy, c.limited_by_receive_window), (1_000_000, 900_000));
assert_eq!((c.send_window, c.receive_window), (131072, 262144));
}
/// Linux 4.x sends 104 octets of `tcp_info`, up to `tcpi_total_retrans`:
/// what is there is read, and what came later is left at zero rather
/// than read from the attribute after it.
#[test]
fn a_short_tcp_info_from_an_old_kernel_leaves_later_fields_zero() {
let Answer::Connection(c) = parse(&reply(104)).unwrap() else {
panic!("a connection");
};
assert_eq!(c.congestion.as_deref(), Some("cubic"));
assert_eq!((c.mss, c.rtt, c.cwnd), (1448, 2300, 42));
assert_eq!((c.busy, c.send_window), (0, 0));
}
#[test]
fn no_such_connection_and_a_refusal_are_told_apart() {
let error = |errno: i32| {
let mut buf = vec![0u8; 20];
buf[0..4].copy_from_slice(&20u32.to_ne_bytes());
buf[4..6].copy_from_slice(&message_type::ERROR.to_ne_bytes());
buf[16..20].copy_from_slice(&(-errno).to_ne_bytes());
buf
};
assert_eq!(parse(&error(2)).unwrap(), Answer::None);
assert_eq!(parse(&error(13)).unwrap(), Answer::Refused(13));
}
#[test]
fn a_request_names_the_connection() {
let local: SocketAddr = "[2001:db8::1]:179".parse().unwrap();
let peer: SocketAddr = "[2001:db8::2]:40000".parse().unwrap();
let request = request(local, peer, 7);
assert_eq!(request.len(), REQUEST_LEN);
assert_eq!(request[16], AF_INET6);
assert_eq!(&request[24..26], &179u16.to_be_bytes());
assert_eq!(&request[26..28], &40000u16.to_be_bytes());
assert_eq!(
&request[28..44],
&"2001:db8::1"
.parse::<std::net::Ipv6Addr>()
.unwrap()
.octets()
);
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn the_kernel_tells_of_a_connection_of_ours() {
use tokio::io::AsyncWriteExt;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let mut client = tokio::net::TcpStream::connect(listener.local_addr().unwrap())
.await
.unwrap();
let (_server, _) = listener.accept().await.unwrap();
client.write_all(&[0u8; 1000]).await.unwrap();
let connection = tcp_connection(client.local_addr().unwrap(), client.peer_addr().unwrap())
.await
.unwrap()
.expect("the kernel knows it");
assert_eq!(connection.state_name(), "established");
assert!(connection.mss > 0 && connection.cwnd > 0);
assert!(connection.bytes_sent >= 1000 || connection.bytes_acked >= 1000);
let none = tcp_connection(
"127.0.0.1:1".parse().unwrap(),
"127.0.0.1:2".parse().unwrap(),
)
.await
.unwrap();
assert!(none.is_none());
}
}
crates/fib/src/topology.rs
+123
View File
@@ -0,0 +1,123 @@
//! What the kernel knows that says whether, and how, a next hop can be
//! reached: the machine's interfaces and addresses, the routes of the
//! resolution table, and the kernel's nexthop objects that those routes
//! may name. One value, replaced as a whole when any of it changes, so
//! that everyone judging next hops judges them by one moment's view.
use crate::interfaces::Interfaces;
use crate::netlink::{Message, Nexthop};
use crate::plan::HashMap;
use crate::routes::KernelRoutes;
/// The kernel's view of the machine, as far as next hops are concerned.
#[derive(Debug, Clone)]
pub struct Topology {
pub interfaces: Interfaces,
/// The routes a next hop that is not a neighbor is reached by.
pub routes: KernelRoutes,
/// Every nexthop object the kernel has, by number, whoever made it:
/// a route of the resolution table may name one instead of carrying
/// its own gateway.
pub nexthops: HashMap<u32, Nexthop>,
/// Whether the default route counts as the way to a next hop.
pub via_default: bool,
}
impl Topology {
/// Nothing known yet, with next hops to be resolved by the routes of
/// `resolution_table`.
pub fn new(resolution_table: u32, via_default: bool) -> Self {
Self {
interfaces: Interfaces::default(),
routes: KernelRoutes::new(resolution_table),
nexthops: HashMap::default(),
via_default,
}
}
/// Takes in what a message from the kernel says. Returns whether
/// anything is different for it: then what any next hop resolves to
/// may be different too.
pub fn apply(&mut self, message: &Message) -> bool {
match message {
Message::Nexthop {
present: true,
nexthop,
} => self.nexthops.insert(nexthop.id, *nexthop) != Some(*nexthop),
Message::Nexthop {
present: false,
nexthop,
} => self.nexthops.remove(&nexthop.id).is_some(),
Message::Route { .. } => self.routes.apply(message),
_ => self.interfaces.apply(message),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::netlink::{KIND_UNICAST, Link, PROTOCOL_BGP, Route, TABLE_MAIN};
use bgp_primitives::ip_prefix::IpPrefix;
#[test]
fn each_kind_of_news_goes_where_it_belongs() {
let mut topology = Topology::new(TABLE_MAIN, false);
let nexthop = Nexthop {
id: 7,
protocol: 4,
gateway: Some("192.0.2.1".parse().unwrap()),
interface: Some(2),
};
let present = |nexthop| Message::Nexthop {
present: true,
nexthop,
};
assert!(topology.apply(&present(nexthop)));
assert!(!topology.apply(&present(nexthop)));
assert!(topology.apply(&present(Nexthop {
interface: Some(3),
..nexthop
})));
assert_eq!(topology.nexthops[&7].interface, Some(3));
assert!(topology.apply(&Message::Nexthop {
present: false,
nexthop,
}));
assert!(topology.nexthops.is_empty());
let route = Route {
prefix: IpPrefix::try_from("10.0.0.0/8").unwrap(),
table: TABLE_MAIN,
protocol: 4,
metric: 0,
gateway: Some("192.0.2.1".parse().unwrap()),
interface: Some(2),
nexthop: None,
kind: KIND_UNICAST,
};
assert!(topology.apply(&Message::Route {
present: true,
route,
}));
assert_eq!(topology.routes.len(), 1);
// BGP's routes are not the way to anything.
assert!(!topology.apply(&Message::Route {
present: true,
route: Route {
protocol: PROTOCOL_BGP,
..route
},
}));
assert!(topology.apply(&Message::Link {
present: true,
link: Link {
index: 2,
name: "eth0".into(),
up: true,
},
}));
assert!(topology.interfaces.link(2).is_some());
assert!(!topology.apply(&Message::Done));
}
}
crates/fib/tests/kernel.rs
+910
View File
@@ -0,0 +1,910 @@
//! The kernel's routes against a real kernel: what `ip route` shows after
//! the routing table changes.
//!
//! These change the machine's interfaces and routes, so they are not run
//! by `cargo test`. `scripts/kernel-tests.sh` runs them in a network
//! namespace of their own, with the capability they need
//! (CAP_NET_ADMIN) and nothing more.
#![cfg(target_os = "linux")]
use std::collections::BTreeSet;
use std::net::{IpAddr, Ipv4Addr};
use std::process::Command;
use std::sync::Arc;
use std::time::{Duration, Instant};
use bgp_fib::{Fib, FibConfig, FibError};
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, MultiExitDiscPathAttribute, NextHopPathAttribute,
NlriNextHop, OriginPathAttribute, PathAttribute,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{AttrSet, PeerIndex, PeerInfo, RibTable};
use tokio::task::JoinHandle;
const TIMEOUT: Duration = Duration::from_secs(10);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// Runs `ip` and returns what it printed. Panics if it fails.
#[track_caller]
fn ip(arguments: &str) -> String {
let output = Command::new("ip")
.args(arguments.split_whitespace())
.output()
.expect("the ip command is there");
assert!(
output.status.success(),
"ip {arguments}: {}",
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8_lossy(&output.stdout).into_owned()
}
/// What a test has of the machine: an interface of its own with
/// 10.`n`.0.1/24 and 2001:db8:`n`::1/64 on it, and a routing table of its
/// own. Tests run side by side in one network namespace, and `n` keeps
/// them apart.
struct Machine {
n: u8,
rib: Arc<RibTable>,
/// Whether routes name nexthop objects, which is how it is unless a
/// test says otherwise.
nexthop_objects: bool,
/// How long routes of an earlier run are kept: not at all, unless a
/// test says otherwise.
stale_time: Duration,
}
impl Machine {
fn new(n: u8) -> Self {
let machine = Self {
n,
rib: Arc::new(RibTable::new(4)),
nexthop_objects: true,
stale_time: Duration::ZERO,
};
ip(&format!("link add {} type dummy", machine.interface()));
machine.add_addresses();
ip(&format!("link set {} up", machine.interface()));
machine
}
fn interface(&self) -> String {
format!("ferrix{}", self.n)
}
fn add_addresses(&self) {
let (n, interface) = (self.n, self.interface());
ip(&format!("addr add 10.{n}.0.1/24 dev {interface}"));
ip(&format!(
"-6 addr add 2001:db8:{n}::1/64 dev {interface} nodad"
));
}
fn add_addresses_v4(&self) {
ip(&format!(
"addr add 10.{}.0.1/24 dev {}",
self.n,
self.interface()
));
}
fn table(&self) -> u32 {
1000 + u32::from(self.n)
}
/// A neighbor on the test's interface.
fn neighbor(&self, host: u8) -> Ipv4Addr {
Ipv4Addr::new(10, self.n, 0, host)
}
fn start(&self) -> (Fib, JoinHandle<Result<(), FibError>>) {
Fib::spawn(
Arc::clone(&self.rib),
FibConfig {
table: self.table(),
metric: 20,
nexthop_objects: self.nexthop_objects,
stale_time: self.stale_time,
..FibConfig::default()
},
)
.expect("the kernel can be talked to")
}
fn peer(&self, host: u8) -> PeerIndex {
self.rib
.add_peer(PeerInfo {
asn: 65000 + u32::from(host),
router_id: self.neighbor(host),
address: self.neighbor(host).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap()
}
/// Attributes with the next hop `next_hop` and, to say which of two
/// routes is the better, `med`.
fn via(&self, next_hop: IpAddr, med: u32) -> Arc<AttrSet> {
let mut attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65001]),
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(med)),
];
let mp_next_hop = match next_hop {
IpAddr::V4(v4) => {
attributes.push(PathAttribute::NextHop(NextHopPathAttribute(v4)));
None
}
IpAddr::V6(v6) => Some(NlriNextHop::Ipv6(v6)),
};
self.rib
.attrs()
.intern(&attributes, mp_next_hop.as_ref())
.unwrap()
}
/// The routes in the test's table, as `ip route` prints them.
fn kernel_routes(&self) -> BTreeSet<String> {
let table = self.table();
let mut routes = BTreeSet::new();
for family in ["-4", "-6"] {
// A table nothing was ever put in does not exist yet.
let output = Command::new("ip")
.args([family, "route", "show", "table", &table.to_string()])
.output()
.unwrap();
routes.extend(
String::from_utf8_lossy(&output.stdout)
.lines()
.map(|line| line.split_whitespace().collect::<Vec<_>>().join(" "))
// IPv6 routes are shown with "pref medium".
.map(|line| line.replace(" pref medium", ""))
// A route that names a nexthop object is shown with
// its number, which is the kernel's to choose.
.map(|line| match line.split_once(" nhid ") {
Some((prefix, rest)) => {
let rest = rest.split_once(' ').map_or("", |(_, rest)| rest);
format!("{prefix} {rest}")
}
None => line,
})
.filter(|line| !line.is_empty()),
);
}
routes
}
/// Waits until the kernel's table holds exactly `expected`.
#[track_caller]
fn kernel_becomes(&self, expected: &[String]) {
let expected: BTreeSet<String> = expected.iter().cloned().collect();
let start = Instant::now();
while self.kernel_routes() != expected {
assert!(
start.elapsed() < TIMEOUT,
"the kernel has {:#?}; expected {expected:#?}",
self.kernel_routes()
);
std::thread::sleep(Duration::from_millis(20));
}
}
/// How `ip route` shows a route of ours.
fn shown(&self, prefix: &str, gateway: impl std::fmt::Display) -> String {
format!(
"{prefix} via {gateway} dev {} proto bgp metric 20",
self.interface()
)
}
}
impl Drop for Machine {
fn drop(&mut self) {
let _ = Command::new("ip")
.args(["link", "del", &self.interface()])
.output();
for family in ["-4", "-6"] {
let _ = Command::new("ip")
.args([family, "route", "flush", "table", &self.table().to_string()])
.output();
}
}
}
/// Waits for `condition` to hold of the status.
#[track_caller]
fn status_becomes(fib: &Fib, what: &str, condition: impl Fn(&bgp_fib::Status) -> bool) {
let start = Instant::now();
while !condition(&fib.status()) {
assert!(
start.elapsed() < TIMEOUT,
"{what}: the status is {:#?}",
fib.status()
);
std::thread::sleep(Duration::from_millis(20));
}
}
/// The best route of each prefix is in the kernel, and follows the table:
/// when the best route changes, when it goes, and when there is none whose
/// next hop can be reached.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn the_kernel_has_the_best_route_of_each_prefix() {
let machine = Machine::new(1);
let (near, far) = (machine.peer(2), machine.peer(3));
let via_near = machine.via(machine.neighbor(2).into(), 50);
// What was there before the daemon started is installed too.
machine
.rib
.announce(near, &via_near, &[prefix("203.0.113.0/24")]);
let (fib, task) = machine.start();
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
status_becomes(&fib, "in step", |status| {
status.synced && status.installed == 1
});
// More routes, IPv6 among them.
let v6_neighbor: IpAddr = "2001:db8:1::2".parse().unwrap();
machine.rib.announce(
near,
&machine.via(v6_neighbor, 50),
&[prefix("2001:db8:100::/48")],
);
machine
.rib
.announce(near, &via_near, &[prefix("198.18.0.0/15")]);
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
machine.shown("2001:db8:100::/48", v6_neighbor),
]);
// A better route from another peer takes the first one's place.
let via_far = machine.via(machine.neighbor(3).into(), 10);
machine
.rib
.announce(far, &via_far, &[prefix("203.0.113.0/24")]);
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", machine.neighbor(3)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
machine.shown("2001:db8:100::/48", v6_neighbor),
]);
// It goes: the first is the best again.
machine.rib.withdraw(far, &[prefix("203.0.113.0/24")]);
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
machine.shown("2001:db8:100::/48", v6_neighbor),
]);
// A best route whose next hop is not a neighbor is not installed, and
// the route it replaced does not stay.
let unreachable = machine.via("172.31.9.9".parse().unwrap(), 5);
machine
.rib
.announce(far, &unreachable, &[prefix("198.18.0.0/15")]);
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("2001:db8:100::/48", v6_neighbor),
]);
status_becomes(&fib, "one unresolved", |status| {
status.installed == 2 && status.unresolved == [("no route to the next hop", 1)]
});
// A session ends: its routes go.
machine.rib.remove_peer(near);
machine.kernel_becomes(&[]);
status_becomes(&fib, "nothing installed", |status| status.installed == 0);
fib.stop().await;
task.await.unwrap().unwrap();
}
/// Routes of ours left by a run that ended badly are removed at the next
/// start unless they are still wanted. Routes that are not ours are left
/// alone, whatever they are for.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_left_behind_are_cleared_and_others_are_not_touched() {
let machine = Machine::new(2);
let (table, interface) = (machine.table(), machine.interface());
let add = |route: &str| ip(&format!("route add {route} dev {interface} table {table}"));
// Ours, from before: one that is still right, one that has another
// next hop now, one that nobody announces any more.
add("203.0.113.0/24 via 10.2.0.2 proto bgp metric 20");
add("198.18.0.0/15 via 10.2.0.9 proto bgp metric 20");
add("192.88.99.0/24 via 10.2.0.2 proto bgp metric 20");
// Not ours: someone's static route, and a BGP route with another
// metric, which is how another daemon's would differ.
add("192.0.2.0/24 via 10.2.0.7 proto static metric 20");
add("192.88.99.0/24 via 10.2.0.7 proto bgp metric 30");
let others = [
format!("192.0.2.0/24 via 10.2.0.7 dev {interface} proto static metric 20"),
format!("192.88.99.0/24 via 10.2.0.7 dev {interface} proto bgp metric 30"),
];
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine.rib.announce(
peer,
&via,
&[prefix("203.0.113.0/24"), prefix("198.18.0.0/15")],
);
let (fib, task) = machine.start();
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
others[0].clone(),
others[1].clone(),
]);
// A run that ends in good order takes its routes with it, and only
// its own.
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&others);
}
/// What a next hop resolves to depends on the machine's interfaces. When
/// they change, the routes are worked out again.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_follow_the_machines_interfaces() {
let machine = Machine::new(3);
let interface = machine.interface();
// The interface has no addresses yet: nothing is a neighbor.
ip(&format!("addr flush dev {interface}"));
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine
.rib
.announce(peer, &via, &[prefix("203.0.113.0/24")]);
let (fib, task) = machine.start();
status_becomes(&fib, "unresolved", |status| {
status.synced && status.unresolved == [("no route to the next hop", 1)]
});
assert!(machine.kernel_routes().is_empty());
// It gets its address: the next hop is a neighbor now.
machine.add_addresses();
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
status_becomes(&fib, "the interface is shown", |status| {
status.unresolved.is_empty()
&& status.interfaces.iter().any(|shown| {
shown.name == interface
&& shown.up
&& shown
.addresses
.contains(&(IpAddr::from(machine.neighbor(1)), 24))
})
});
// It loses it. The kernel drops the route by itself; what matters is
// that we know it has.
ip(&format!("addr del 10.3.0.1/24 dev {interface}"));
machine.kernel_becomes(&[]);
status_becomes(&fib, "unresolved again", |status| {
status.installed == 0 && status.unresolved.len() == 1
});
// The interface goes down and comes straight back, faster than we
// work the routes out again. The kernel dropped the route when it went
// down, though nothing about the route has changed for us.
machine.add_addresses_v4();
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
ip(&format!("link set {interface} down"));
ip(&format!("link set {interface} up"));
assert!(machine.kernel_routes().is_empty(), "the kernel dropped it");
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
ip(&format!("addr del 10.3.0.1/24 dev {interface}"));
machine.kernel_becomes(&[]);
// And gets it back, with the interface having been down meanwhile.
ip(&format!("link set {interface} down"));
ip(&format!("addr add 10.3.0.1/24 dev {interface}"));
ip(&format!("link set {interface} up"));
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
/// A next hop that is no neighbor goes the way the kernel's own routes
/// say: a static route's gateway, a tunnel's interface, a nexthop object.
/// When those routes change, so do ours; a route that discards packets is
/// the way to nowhere.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn a_far_next_hop_goes_the_way_the_kernels_routes_say() {
let machine = Machine::new(10);
let interface = machine.interface();
let peer = machine.peer(2);
// Next hops that are not in any network of ours.
let far = machine.via("172.31.10.9".parse().unwrap(), 0);
let farther = machine.via("172.31.11.9".parse().unwrap(), 0);
machine
.rib
.announce(peer, &far, &[prefix("203.0.113.0/24")]);
machine
.rib
.announce(peer, &farther, &[prefix("198.18.0.0/15")]);
let (fib, task) = machine.start();
status_becomes(&fib, "no way to either", |status| {
status.synced && status.unresolved == [("no route to the next hop", 2)]
});
assert!(machine.kernel_routes().is_empty());
// A static route with a gateway says the way to one: the route goes
// in with that gateway.
ip(&format!(
"route add 172.31.10.0/24 via 10.10.0.9 dev {interface}"
));
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", "10.10.0.9")]);
// A route onto the interface, as a tunnel's is, says the way to the
// other: the next hop itself is the neighbor there.
ip(&format!("route add 172.31.11.0/24 dev {interface}"));
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.9"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
status_becomes(&fib, "both resolved", |status| {
status.unresolved.is_empty() && status.installed == 2
});
// The way changes: so does the route. Then it goes, and so does the
// route.
ip(&format!(
"route replace 172.31.10.0/24 via 10.10.0.8 dev {interface}"
));
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.8"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
ip("route del 172.31.10.0/24");
machine.kernel_becomes(&[machine.shown("198.18.0.0/15", "172.31.11.9")]);
status_becomes(&fib, "one unresolved again", |status| {
status.unresolved == [("no route to the next hop", 1)]
});
// A route that names a nexthop object goes where the object says.
ip(&format!(
"nexthop add id 1010 via 10.10.0.7 dev {interface}"
));
ip("route add 172.31.10.0/24 nhid 1010");
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.7"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
// The tunnel's route becomes a blackhole: the way to nowhere.
ip("route replace blackhole 172.31.11.0/24");
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", "10.10.0.7")]);
status_becomes(&fib, "discarded", |status| {
status.unresolved == [("the route to the next hop discards packets", 1)]
});
ip("route del 172.31.10.0/24");
ip("route del blackhole 172.31.11.0/24");
ip("nexthop del id 1010");
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
/// A route that asks for its packets to be dropped goes into the kernel
/// as a blackhole route, whatever its next hop, and comes out again with
/// the request.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn a_blackhole_request_becomes_a_blackhole_route() {
let machine = Machine::new(13);
let peer = machine.peer(2);
// Asked for from a next hop nothing is a neighbor of, and with one
// ordinary route beside it to show the two apart.
let mut attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65002]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(172, 31, 11, 9))),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65002, 1), (0xFFFF, 0x029A)])),
];
let blackhole = machine.rib.attrs().intern(&attributes, None).unwrap();
attributes.pop();
attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65002, 1),
])));
let plain = machine.rib.attrs().intern(&attributes, None).unwrap();
machine.rib.announce(
peer,
&blackhole,
&[prefix("203.0.113.9/32"), prefix("2001:db8:100::9/128")],
);
machine
.rib
.announce(peer, &plain, &[prefix("198.18.0.0/15")]);
let (fib, task) = machine.start();
// The kernel puts an IPv6 blackhole route on lo, and says so.
machine.kernel_becomes(&[
"blackhole 203.0.113.9 proto bgp metric 20".into(),
"blackhole 2001:db8:100::9 dev lo proto bgp metric 20".into(),
]);
status_becomes(&fib, "two blackholes, one unresolved", |status| {
status.synced
&& status.installed == 2
&& status.blackholes == 2
&& status.unresolved == [("no route to the next hop", 1)]
});
// The request is withdrawn: the blackhole goes.
machine.rib.withdraw(peer, &[prefix("203.0.113.9/32")]);
machine.kernel_becomes(&["blackhole 2001:db8:100::9 dev lo proto bgp metric 20".into()]);
// A blackhole left by an earlier run is found and kept like any other.
fib.leave().await;
task.await.unwrap().unwrap();
let (fib, task) = machine.start();
status_becomes(&fib, "the blackhole is adopted", |status| {
status.synced && status.installed == 1 && status.blackholes == 1
});
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
/// A table's worth of routes goes in, and comes out, in reasonable time.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn many_routes_are_installed_and_removed() {
// FERRIX_ROUTES=1000000 for the size of the Internet's IPv4 table.
let routes: usize = std::env::var("FERRIX_ROUTES")
.ok()
.and_then(|routes| routes.parse().ok())
.unwrap_or(200_000);
let machine = Machine::new(4);
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
let prefixes: Vec<IpPrefix> = (0..routes as u32)
.map(|n| {
let [_, a, b, c] = n.to_be_bytes();
IpPrefix::new(
bgp_primitives::constants::IpVersion::Ipv4,
[100 + a, b, c, 0],
24,
)
.unwrap()
})
.collect();
machine.rib.announce(peer, &via, &prefixes);
let started = Instant::now();
let (fib, task) = machine.start();
status_becomes(&fib, "all installed", |status| {
status.synced && status.installed == routes
});
let installed_in = started.elapsed();
assert_eq!(machine.kernel_routes().len(), routes);
assert_eq!(fib.status().errors, 0, "{:?}", fib.status().last_error);
let started = Instant::now();
fib.stop().await;
task.await.unwrap().unwrap();
let removed_in = started.elapsed();
assert!(machine.kernel_routes().is_empty());
println!(
"{routes} routes: installed in {installed_in:?} ({:.0}/s), removed in {removed_in:?}",
routes as f64 / installed_in.as_secs_f64()
);
}
/// Someone empties the table behind our back (`ip route flush`), and then
/// the peers withdraw everything: the kernel refuses every removal, a
/// table's worth of rejections at once. That is more than the socket holds
/// answers for, and must be no more than a reason to look again.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn a_flood_of_refusals_does_not_stop_the_daemon() {
const ROUTES: u32 = 20_000;
let mut machine = Machine::new(11);
machine.nexthop_objects = false;
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
let prefixes: Vec<IpPrefix> = (0..ROUTES)
.map(|n| {
let [_, a, b, c] = n.to_be_bytes();
IpPrefix::new(
bgp_primitives::constants::IpVersion::Ipv4,
[100 + a, b, c, 0],
24,
)
.unwrap()
})
.collect();
machine.rib.announce(peer, &via, &prefixes);
let (fib, task) = machine.start();
status_becomes(&fib, "all installed", |status| {
status.synced && status.installed == ROUTES as usize
});
ip(&format!("route flush table {}", machine.table()));
assert!(machine.kernel_routes().is_empty());
let routes: Vec<bgp_primitives::ip_prefix::Nlri> =
prefixes.iter().map(|prefix| (*prefix).into()).collect();
machine.rib.withdraw(peer, &routes);
status_becomes(&fib, "nothing installed", |status| {
status.synced && status.installed == 0
});
assert!(
!task.is_finished(),
"the task that talks to the kernel ended"
);
// And it goes on working.
machine.rib.announce(peer, &via, &prefixes[..3]);
status_becomes(&fib, "three installed", |status| {
status.synced && status.installed == 3
});
assert_eq!(machine.kernel_routes().len(), 3);
assert!(!task.is_finished());
fib.stop().await;
task.await.unwrap().unwrap();
}
/// Routes name a nexthop object for their gateway where the kernel has
/// such objects. The object is ours: made when first needed, put back if
/// someone removes it, and removed when we stop. Other objects are not
/// touched.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_name_a_nexthop_object_that_is_ours_to_keep() {
let machine = Machine::new(5);
let interface = machine.interface();
// Someone else's object, to the same gateway even.
ip(&format!(
"nexthop add id 50005 via 10.5.0.2 dev {interface} protocol bgp"
));
let nexthops = || -> Vec<String> {
ip("nexthop show")
.lines()
.filter(|line| line.contains(&format!("dev {interface} ")))
.map(|line| line.trim().to_string())
.collect()
};
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine.rib.announce(
peer,
&via,
&[prefix("203.0.113.0/24"), prefix("198.18.0.0/15")],
);
let (fib, task) = machine.start();
let both = [
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
];
machine.kernel_becomes(&both);
// One object of ours beside the other, and both routes name it.
let ours: Vec<String> = nexthops()
.into_iter()
.filter(|line| !line.starts_with("id 50005 "))
.collect();
assert_eq!(ours.len(), 1, "{:?}", nexthops());
assert!(ours[0].contains("via 10.5.0.2") && ours[0].contains("proto bgp"));
let id = ours[0].split_whitespace().nth(1).unwrap().to_string();
let raw = ip(&format!("route show table {}", machine.table()));
assert_eq!(raw.matches(&format!("nhid {id} ")).count(), 2, "{raw}");
// Someone removes our object, and the kernel our routes with it. We
// are told, and put both back.
ip(&format!("nexthop del id {id}"));
machine.kernel_becomes(&both);
// When we stop, our object goes with our routes; the other stays.
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
assert_eq!(nexthops().len(), 1, "{:?}", nexthops());
assert!(nexthops()[0].starts_with("id 50005 "));
ip("nexthop del id 50005");
}
/// On a kernel without nexthop objects, or where the configuration says
/// so, every route carries its own gateway.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_can_carry_their_own_gateway() {
let mut machine = Machine::new(6);
machine.nexthop_objects = false;
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine
.rib
.announce(peer, &via, &[prefix("203.0.113.0/24")]);
let (fib, task) = machine.start();
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
let raw = ip(&format!("route show table {}", machine.table()));
assert!(!raw.contains("nhid"), "{raw}");
assert!(
!ip("nexthop show").contains(&format!("dev {} ", machine.interface())),
"no nexthop object was made"
);
// The next hop changes, and the route goes.
let other = machine.via(machine.neighbor(3).into(), 0);
machine
.rib
.announce(peer, &other, &[prefix("203.0.113.0/24")]);
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(3))]);
machine.rib.withdraw(peer, &[prefix("203.0.113.0/24")]);
machine.kernel_becomes(&[]);
fib.stop().await;
task.await.unwrap().unwrap();
}
/// When the daemon starts, its table is empty until its peers have sent
/// theirs. The routes an earlier run left are kept for a while, so that
/// the machine goes on forwarding; then those nobody announced again go.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_of_an_earlier_run_are_kept_while_peers_come_back() {
let mut machine = Machine::new(7);
machine.stale_time = Duration::from_secs(3);
let (table, interface) = (machine.table(), machine.interface());
let add = |route: &str| ip(&format!("route add {route} dev {interface} table {table}"));
add("203.0.113.0/24 via 10.7.0.2 proto bgp metric 20");
add("198.18.0.0/15 via 10.7.0.2 proto bgp metric 20");
let left = [
machine.shown("203.0.113.0/24", machine.neighbor(2)),
machine.shown("198.18.0.0/15", machine.neighbor(2)),
];
// The daemon starts with nothing in its table: both stay.
let (fib, task) = machine.start();
status_becomes(&fib, "in step, keeping two", |status| {
status.synced && status.installed == 2 && status.kept == 2
});
assert_eq!(machine.kernel_routes(), left.iter().cloned().collect());
// A peer comes back with one of them.
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine
.rib
.announce(peer, &via, &[prefix("203.0.113.0/24")]);
status_becomes(&fib, "one confirmed", |status| status.kept == 1);
assert_eq!(machine.kernel_routes(), left.iter().cloned().collect());
// Time is up: the one nobody announced goes.
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
status_becomes(&fib, "none kept", |status| {
status.installed == 1 && status.kept == 0
});
fib.stop().await;
task.await.unwrap().unwrap();
}
/// A restart that is to go unnoticed (graceful restart): the routes are
/// left in the kernel, by way of their nexthop objects; the next run
/// finds them, says how many, and lets those nobody announced again go
/// as soon as it is told the peers have all sent theirs.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn routes_stay_in_the_kernel_over_a_restart() {
let mut machine = Machine::new(12);
machine.stale_time = Duration::from_secs(60);
let (fib, task) = machine.start();
assert_eq!(fib.adopted().await, 0);
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine.rib.announce(
peer,
&via,
&[prefix("203.0.113.0/24"), prefix("198.18.0.0/15")],
);
let both = [
machine.shown("198.18.0.0/15", machine.neighbor(2)),
machine.shown("203.0.113.0/24", machine.neighbor(2)),
];
machine.kernel_becomes(&both);
fib.leave().await;
task.await.unwrap().unwrap();
assert_eq!(machine.kernel_routes(), both.iter().cloned().collect());
// The next run, with a table that is empty until the peers are back.
machine.rib = Arc::new(RibTable::new(4));
let (fib, task) = machine.start();
assert_eq!(fib.adopted().await, 2);
status_becomes(&fib, "keeping both", |status| status.kept == 2);
let peer = machine.peer(2);
let via = machine.via(machine.neighbor(2).into(), 0);
machine
.rib
.announce(peer, &via, &[prefix("203.0.113.0/24")]);
status_becomes(&fib, "one confirmed", |status| status.kept == 1);
// Every peer has sent its routes: what was not among them goes now,
// and not a minute later.
fib.release_kept();
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", machine.neighbor(2))]);
fib.stop().await;
task.await.unwrap().unwrap();
}
/// An IPv4 route whose next hop is an IPv6 neighbor: routes over a link
/// that has only IPv6 addresses (RFC 8950). The kernel takes such a route,
/// by way of a nexthop object or with its gateway spelled out.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn an_ipv4_route_goes_by_way_of_an_ipv6_neighbor() {
for (n, nexthop_objects) in [(8, true), (9, false)] {
let mut machine = Machine::new(n);
machine.nexthop_objects = nexthop_objects;
// Only IPv6 on the link.
ip(&format!(
"addr del 10.{n}.0.1/24 dev {}",
machine.interface()
));
let neighbor: IpAddr = format!("2001:db8:{n}::2").parse().unwrap();
let peer = machine
.rib
.add_peer(PeerInfo {
asn: 65002,
router_id: machine.neighbor(2),
address: neighbor,
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap();
// Both families by way of the same neighbor.
let via = machine.via(neighbor, 0);
machine.rib.announce(
peer,
&via,
&[prefix("203.0.113.0/24"), prefix("2001:db8:100::/48")],
);
// And an IPv4 route whose next hop has a link-local address too,
// with a global one that is not on any link of ours.
let link_local = machine
.rib
.attrs()
.intern(
&[
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65002]),
],
Some(&NlriNextHop::Ipv6WithLl {
global: "2001:db8:ffff::2".parse().unwrap(),
link_local: "fe80::2".parse().unwrap(),
}),
)
.unwrap();
machine
.rib
.announce(peer, &link_local, &[prefix("198.18.0.0/15")]);
let (fib, task) = machine.start();
let interface = machine.interface();
machine.kernel_becomes(&[
format!("203.0.113.0/24 via inet6 {neighbor} dev {interface} proto bgp metric 20"),
format!("198.18.0.0/15 via inet6 fe80::2 dev {interface} proto bgp metric 20"),
machine.shown("2001:db8:100::/48", neighbor),
]);
assert_eq!(fib.status().errors, 0, "{:?}", fib.status().last_error);
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
}
crates/integration-tests/Cargo.toml
+17
View File
@@ -1,13 +1,30 @@
[package]
name = "bgp-integration-tests"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
publish = false
[dependencies]
bgp-fib.workspace = true
bgp-packet.workspace = true
bgp-primitives.workspace = true
bgp-rib.workspace = true
bgp-rpki.workspace = true
bgp-server.workspace = true
bgp-web.workspace = true
bytes.workspace = true
eyre.workspace = true
reqwest.workspace = true
serde_json.workspace = true
tokio.workspace = true
# Used by the examples only.
[dev-dependencies]
clap.workspace = true
[lints]
workspace = true
crates/integration-tests/examples/fake_peer.rs
+163
View File
@@ -0,0 +1,163 @@
//! A minimal stand-in for a neighbouring router: connects to the daemon,
//! completes the handshake, optionally announces some prefixes, and prints
//! every UPDATE it receives. Useful for trying things out when no real
//! router is around.
//!
//! cargo run -p bgp-integration-tests --example fake_peer -- \
//! --server 127.0.0.1:1790 --asn 65001 --announce 203.0.113.0/24
//!
//! With `--echo`, every prefix received is re-announced back to the server
//! with this peer's ASN prepended, which lets a single daemon show its own
//! routes "coming back" in the received table.
//!
//! With `--listen ADDR` it instead waits for the server to dial it (one
//! connection at a time), which exercises the server's active side.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_packet::message::{Message, PathAttribute, UpdateMessage};
use bgp_primitives::ip_prefix::IpPrefix;
use clap::Parser;
use eyre::Result;
use tokio::net::TcpListener;
#[derive(Parser, Debug)]
struct Args {
/// Address of the daemon's BGP listener to connect to.
#[arg(long, default_value = "127.0.0.1:1790", conflicts_with = "listen")]
server: String,
/// Instead of connecting, listen here and accept connections from the
/// server, one at a time, forever.
#[arg(long)]
listen: Option<String>,
/// ASN this peer speaks as.
#[arg(long, default_value_t = 65001)]
asn: u32,
/// Router ID to send in OPEN.
#[arg(long, default_value = "10.0.0.2")]
router_id: Ipv4Addr,
/// Hold time to negotiate.
#[arg(long, default_value_t = 30)]
hold_time: u16,
/// Prefixes to announce after the session is established.
#[arg(long)]
announce: Vec<String>,
/// Re-announce every received prefix back to the server, with our ASN
/// prepended and NO_EXPORT-tagged routes dropped.
#[arg(long)]
echo: bool,
}
const NO_EXPORT: (u16, u16) = (0xFFFF, 0xFF01);
#[tokio::main]
async fn main() -> Result<()> {
let args = Args::parse();
match &args.listen {
Some(listen) => {
let listener = TcpListener::bind(listen).await?;
println!("listening on {}", listener.local_addr()?);
loop {
let (stream, from) = listener.accept().await?;
println!("accepted connection from {from}");
if let Err(e) = run(&args, TestPeer::from_stream(stream), from).await {
println!("session ended: {e}");
}
}
}
None => {
let addr = args.server.parse()?;
run(&args, TestPeer::connect(addr).await?, addr).await
}
}
}
async fn run(args: &Args, mut peer: TestPeer, addr: std::net::SocketAddr) -> Result<()> {
let open = peer
.handshake(args.asn, args.router_id, args.hold_time)
.await?;
println!(
"established with {} (AS{} router-id {})",
addr, open.asn, open.identifier
);
for p in &args.announce {
let prefix = IpPrefix::try_from(p.as_str())?;
peer.announce(announce(prefix, vec![args.asn], vec![(args.asn as u16, 1)]))
.await?;
println!("announced {prefix}");
}
let keepalive = Duration::from_secs((args.hold_time / 3).max(1) as u64);
let mut ticker = tokio::time::interval(keepalive);
loop {
tokio::select! {
_ = ticker.tick() => peer.send(&Message::KeepAlive).await?,
// However long the server is quiet: this peer stays on the line.
msg = peer.recv_untimed() => match msg? {
Message::KeepAlive => {}
Message::Update(u) => {
for w in &u.withdrawn_routes {
println!("WITHDRAW {w}");
}
for a in &u.announced_routes {
println!("UPDATE {a} attrs={:?}", u.path_attributes);
}
if args.echo {
echo_back(&mut peer, &u, args.asn).await?;
}
}
Message::Notification(n) => {
println!("NOTIFICATION {}; session over", n.error);
return Ok(());
}
other => println!("unexpected {other:?}"),
},
}
}
}
fn announce(prefix: IpPrefix, as_path: Vec<u32>, communities: Vec<(u16, u16)>) -> Announcement {
Announcement::of([prefix.into()], &as_path)
.with_next_hop(Ipv4Addr::new(10, 0, 0, 2))
.with_communities(&communities)
}
async fn echo_back(peer: &mut TestPeer, u: &UpdateMessage, asn: u32) -> Result<()> {
let mut as_path = vec![asn];
let mut communities = vec![];
for attr in &u.path_attributes {
match attr {
PathAttribute::AsPath(p) => {
as_path.extend(p.segments.iter().flat_map(|s| s.path.iter().copied()))
}
PathAttribute::Communities(c) => communities = c.0.clone(),
_ => {}
}
}
if communities.contains(&NO_EXPORT) {
println!(" not echoing: NO_EXPORT");
return Ok(());
}
for nlri in &u.announced_routes {
let prefix = nlri.prefix;
peer.announce(announce(prefix, as_path.clone(), communities.clone()))
.await?;
println!(" echoed {prefix} back with path {as_path:?}");
}
if !u.withdrawn_routes.is_empty() {
peer.announce(withdrawal(u.withdrawn_routes.iter().cloned()))
.await?;
println!(
" echoed withdraw of {:?}",
u.withdrawn_routes
.iter()
.map(|p| p.to_string())
.collect::<Vec<_>>()
);
}
Ok(())
}
crates/integration-tests/src/announcement.rs
+110
View File
@@ -0,0 +1,110 @@
//! The UPDATEs a test peer sends, built from what a test has to say about
//! them and nothing more.
use std::net::Ipv4Addr;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::Nlri;
use crate::prefix;
/// An UPDATE announcing IPv4 routes as the peers of these tests do: with
/// ORIGIN IGP, the AS path given, and 192.0.2.1 as the next hop unless
/// another is set. Attributes are written in order of type code.
#[derive(Debug, Clone)]
pub struct Announcement {
routes: Vec<Nlri>,
path: Vec<u32>,
next_hop: Ipv4Addr,
only_to_customer: Option<u32>,
communities: Vec<(u16, u16)>,
more: Vec<PathAttribute>,
}
/// Announces `prefixes`, written as text, with the AS path `path`.
pub fn announcement(prefixes: &[&str], path: &[u32]) -> Announcement {
Announcement::of(prefixes.iter().map(|p| prefix(p).into()), path)
}
/// Withdraws `routes`.
pub fn withdrawal(routes: impl IntoIterator<Item = Nlri>) -> UpdateMessage {
UpdateMessage {
withdrawn_routes: routes.into_iter().collect(),
path_attributes: vec![],
announced_routes: vec![],
}
}
impl Announcement {
/// Announces `routes`, prefixes with or without path identifiers, with
/// the AS path `path`.
pub fn of(routes: impl IntoIterator<Item = Nlri>, path: &[u32]) -> Self {
Self {
routes: routes.into_iter().collect(),
path: path.to_vec(),
next_hop: Ipv4Addr::new(192, 0, 2, 1),
only_to_customer: None,
communities: Vec::new(),
more: Vec::new(),
}
}
pub fn with_next_hop(mut self, next_hop: Ipv4Addr) -> Self {
self.next_hop = next_hop;
self
}
/// With a COMMUNITIES attribute carrying `communities`, if there are
/// any.
pub fn with_communities(mut self, communities: &[(u16, u16)]) -> Self {
self.communities = communities.to_vec();
self
}
/// Marked as only to go to customers, by `asn` (RFC 9234).
pub fn with_only_to_customer(mut self, asn: u32) -> Self {
self.only_to_customer = Some(asn);
self
}
/// With `attributes` after the mandatory ones, as given.
pub fn with_attributes(mut self, attributes: impl IntoIterator<Item = PathAttribute>) -> Self {
self.more.extend(attributes);
self
}
}
impl From<Announcement> for UpdateMessage {
fn from(announcement: Announcement) -> Self {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(announcement.path),
PathAttribute::NextHop(NextHopPathAttribute(announcement.next_hop)),
];
if !announcement.communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
announcement.communities,
)));
}
path_attributes.extend(
announcement
.only_to_customer
.map(PathAttribute::OnlyToCustomer),
);
path_attributes.extend(announcement.more);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: announcement.routes,
}
}
}
impl From<Announcement> for Message {
fn from(announcement: Announcement) -> Self {
Message::Update(announcement.into())
}
}
crates/integration-tests/src/lib.rs
+99
View File
@@ -1,2 +1,101 @@
pub mod announcement;
pub mod test_cache;
pub mod test_peer;
pub mod test_server;
use std::time::Duration;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::address::Endpoint;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use eyre::{Result, bail};
use serde_json::Value;
/// How long a test waits for something the server does on its own: a
/// route reaching the table, a session coming up, a message arriving.
/// Everything happens on the loopback, so it is generous.
pub const TIMEOUT: Duration = Duration::from_secs(5);
/// The policy a peer has with nothing configured, but for one thing: the
/// peers of these tests are all at 127.0.0.1, which no route can have as
/// its next hop, so their routes name another.
pub fn default_policy() -> bgp_server::policy::ImportPolicy {
bgp_server::policy::ImportPolicy {
accept_third_party_next_hop: true,
accept_blackholes: false,
..Default::default()
}
}
/// A prefix written as text. The tests' prefixes are literals, so a bad
/// one is a mistake in the test.
pub fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A configured peer with nothing said about it but where it is and what
/// AS it is in: the default hold time, every family, the default policy of
/// these tests, and no role, limits, groups or ADD-PATH. A test says what
/// else with struct update syntax: `PeerConfig { role: Some(..),
/// ..peer_config(..) }`.
pub fn peer_config(name: &str, endpoint: impl Into<Endpoint>, remote_asn: u32) -> PeerConfig {
PeerConfig {
name: name.into(),
endpoint: endpoint.into(),
remote_asn,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: default_policy(),
role: None,
require_role: false,
export: Default::default(),
export_limits: None,
graceful_restart: Default::default(),
groups: Vec::new(),
reflector_client: false,
passive: false,
expected_routes: None,
}
}
/// Waits until `done` holds, looking every few milliseconds, or fails
/// with `what` once [`TIMEOUT`] has passed.
pub async fn wait_until(what: &str, mut done: impl FnMut() -> bool) -> Result<()> {
let waited = tokio::time::timeout(TIMEOUT, async {
while !done() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await;
if waited.is_err() {
bail!("timed out waiting until {what}");
}
Ok(())
}
/// Waits until `get` has something, and returns it; fails with `what`
/// once [`TIMEOUT`] has passed.
pub async fn wait_for<T>(what: &str, mut get: impl FnMut() -> Option<T>) -> Result<T> {
let mut got = None;
wait_until(what, || {
got = get();
got.is_some()
})
.await?;
Ok(got.expect("the wait ended with something"))
}
/// `GET`s `url` and reads the JSON it answers with. An error status is an
/// error.
pub async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
}
crates/integration-tests/src/test_cache.rs
+156
View File
@@ -0,0 +1,156 @@
//! An RPKI cache for tests: an RTR server that hands out whatever VRPs and
//! ASPAs the test says there are.
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::{Arc, Mutex};
use bgp_rpki::Vrp;
use bgp_rpki::rtr::{Pdu, Timers, VERSION_2};
use bytes::BytesMut;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::{TcpListener, TcpStream};
use tokio::sync::watch;
use tokio::task::JoinHandle;
const SESSION: u16 = 0x1234;
/// What the cache has to give.
#[derive(Clone, Default)]
struct Data {
vrps: Vec<Vrp>,
/// Customer ASes and their providers.
aspas: Vec<(u32, Vec<u32>)>,
serial: u32,
}
/// The cache. Stops, closing its connections, when dropped.
pub struct TestCache {
address: SocketAddr,
data: Arc<Mutex<Data>>,
/// The serial, for telling connected routers there is something new.
serial: watch::Sender<u32>,
task: JoinHandle<()>,
}
impl TestCache {
pub async fn start(vrps: Vec<Vrp>) -> Self {
Self::start_with(vrps, vec![]).await
}
/// A cache with ASPAs too: customer ASes and their providers.
pub async fn start_with(vrps: Vec<Vrp>, aspas: Vec<(u32, Vec<u32>)>) -> Self {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await.unwrap();
let address = listener.local_addr().unwrap();
let data = Arc::new(Mutex::new(Data {
vrps,
aspas,
serial: 1,
}));
let (serial, _) = watch::channel(1);
let task = tokio::spawn({
let (data, serial) = (Arc::clone(&data), serial.clone());
async move {
// Connections end with this task: they are part of it.
let mut connections = tokio::task::JoinSet::new();
loop {
let Ok((stream, _)) = listener.accept().await else {
return;
};
connections.spawn(serve(stream, Arc::clone(&data), serial.subscribe()));
}
}
});
Self {
address,
data,
serial,
task,
}
}
pub fn address(&self) -> SocketAddr {
self.address
}
/// Replaces the VRPs and tells connected routers.
pub fn set(&self, vrps: Vec<Vrp>) {
self.change(|data| data.vrps = vrps);
}
/// Replaces the ASPAs and tells connected routers.
pub fn set_aspas(&self, aspas: Vec<(u32, Vec<u32>)>) {
self.change(|data| data.aspas = aspas);
}
fn change(&self, change: impl FnOnce(&mut Data)) {
let serial = {
let mut data = self.data.lock().unwrap();
change(&mut data);
data.serial += 1;
data.serial
};
self.serial.send_replace(serial);
}
}
impl Drop for TestCache {
fn drop(&mut self) {
self.task.abort();
}
}
/// One router's connection. Asked for changes, this cache always says to
/// ask for everything: the router works out what changed.
async fn serve(mut stream: TcpStream, data: Arc<Mutex<Data>>, mut serial: watch::Receiver<u32>) {
let mut buf = BytesMut::new();
// The version the router speaks, once it has said something.
let mut spoken = VERSION_2;
loop {
let mut out = BytesMut::new();
tokio::select! {
read = stream.read_buf(&mut buf) => {
if !matches!(read, Ok(read) if read > 0) {
return;
}
while let Ok(Some((version, pdu, used))) = Pdu::decode(&buf) {
let _ = buf.split_to(used);
spoken = version;
match pdu {
Pdu::SerialQuery { .. } => Pdu::CacheReset.encode(version, &mut out),
Pdu::ResetQuery => {
let data = data.lock().unwrap().clone();
Pdu::CacheResponse { session_id: SESSION }.encode(version, &mut out);
for vrp in data.vrps {
Pdu::Prefix { announce: true, vrp }.encode(version, &mut out);
}
// Only version 2 has ASPAs.
for (customer, providers) in data.aspas {
if version >= VERSION_2 {
Pdu::Aspa { announce: true, customer, providers }
.encode(version, &mut out);
}
}
Pdu::EndOfData {
session_id: SESSION,
serial: data.serial,
timers: Some(Timers::default()),
}
.encode(version, &mut out);
}
_ => return,
}
}
}
changed = serial.changed() => {
if changed.is_err() {
return;
}
let serial = *serial.borrow_and_update();
Pdu::SerialNotify { session_id: SESSION, serial }.encode(spoken, &mut out);
}
}
if stream.write_all(&out).await.is_err() {
return;
}
}
}
crates/integration-tests/src/test_peer.rs
+265 -35
View File
@@ -1,37 +1,170 @@
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_packet::message::{BGP_HEADER_LEN, Message};
use bgp_packet::open::{Capability, OpenMessage, OpenOption};
use bgp_primitives::constants::{AddressFamilyId, SubsequentAfi};
use bgp_packet::message::{BGP_HEADER_LEN, Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{
AddPathEntry, Capability, ExtendedNextHopEntry, OpenMessage, OpenOption, SendReceive,
};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::ParserContext;
use bytes::BytesMut;
use eyre::{Result, bail, eyre};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio::net::tcp::OwnedWriteHalf;
use tokio::net::{TcpListener, TcpStream};
use tokio::task::JoinHandle;
use crate::TIMEOUT;
/// The capabilities a route collector offers in its OPEN: four-octet AS
/// numbers as `asn`, each of `families`, and to be sent several routes
/// for a prefix (ADD-PATH, RFC 7911) in each of them; with
/// `ipv4_over_ipv6`, IPv4 routes with IPv6 next hops too (RFC 8950), as a
/// collector on an IPv6-only session wants. For the tests and benchmarks
/// that play a collector taking a router's whole table.
pub fn collector_capabilities(
asn: u32,
families: &[AfiSafi],
ipv4_over_ipv6: bool,
) -> Vec<Capability> {
let mut capabilities = vec![Capability::FourByteAsn { asn }];
capabilities.extend(families.iter().copied().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: families
.iter()
.map(|family| AddPathEntry {
afi: family.afi,
safi: family.safi,
send_recv: SendReceive::Receive,
})
.collect(),
});
if ipv4_over_ipv6 {
capabilities.push(Capability::ExtendedNextHop {
entries: vec![ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
}],
});
}
capabilities
}
/// A minimal BGP peer for use in tests. Connects over TCP and can exchange
/// framed BGP messages using the `bgp-packet` wire format.
pub struct TestPeer {
stream: TcpStream,
pub ctx: ParserContext,
/// The address families for which this peer sends path identifiers
/// with its prefixes (ADD-PATH, RFC 7911). What it expects to be sent
/// is `ctx.add_path`. A test sets both to match the capabilities it
/// exchanged.
pub send_add_path: Vec<AfiSafi>,
/// Whether the OPEN we sent offered four-octet AS numbers. They are in
/// use only if the server's did too.
offered_four_octet_asn: bool,
}
impl TestPeer {
pub async fn connect(addr: SocketAddr) -> Result<Self> {
let stream = TcpStream::connect(addr).await?;
Ok(Self { stream, ctx: ParserContext::default() })
Ok(Self::from_stream(stream))
}
/// The next connection to `listener`, as the server dials it. It is
/// waited for at most [`TIMEOUT`].
pub async fn accept(listener: &TcpListener) -> Result<Self> {
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
Ok(Self::from_stream(stream))
}
/// Wraps an already-connected stream, e.g. one accepted from a server
/// that dialled us.
pub fn from_stream(stream: TcpStream) -> Self {
Self {
stream,
ctx: ParserContext::default(),
send_add_path: Vec::new(),
offered_four_octet_asn: true,
}
}
/// The connection itself, for a test or a tool that goes on without
/// the framing, once the handshake is done.
pub fn into_stream(self) -> TcpStream {
self.stream
}
/// The connection with a task reading and dropping whatever the server
/// sends over it, and the half that is left for writing: for a peer
/// that has nothing to hear, as one feeding a table or taking one by
/// the megabyte has not, and whose writer must never be held up by
/// what it has not read. The task ends with the connection, or when
/// its handle is dropped.
pub fn drain(self) -> (OwnedWriteHalf, JoinHandle<()>) {
let (mut reader, writer) = self.stream.into_split();
let sink = tokio::spawn(async move {
let mut sink = vec![0u8; 1 << 16];
while matches!(reader.read(&mut sink).await, Ok(n) if n > 0) {}
});
(writer, sink)
}
/// A KEEPALIVE as it goes on the wire, for whoever writes to the
/// connection without the framing and has to keep the session alive
/// past its hold time.
pub fn keepalive_bytes() -> Vec<u8> {
let mut wire = BytesMut::new();
Message::KeepAlive
.to_wire(&ParserContext::default(), &mut wire)
.expect("a KEEPALIVE is nineteen octets");
wire.to_vec()
}
/// Our end of the connection: what the server sees the peer as.
pub fn local_addr(&self) -> Result<SocketAddr> {
Ok(self.stream.local_addr()?)
}
/// Send a BGP message.
pub async fn send(&mut self, msg: &Message) -> Result<()> {
let send_ctx = ParserContext {
add_path: self.send_add_path.clone(),
..self.ctx.clone()
};
let mut buf = BytesMut::new();
msg.to_wire(&self.ctx, &mut buf)?;
msg.to_wire(&send_ctx, &mut buf)?;
self.stream.write_all(&buf).await?;
Ok(())
}
/// Read exactly one BGP message.
/// Send an UPDATE: an [`Announcement`](crate::announcement::Announcement),
/// a [`withdrawal`](crate::announcement::withdrawal), or one built by
/// hand.
pub async fn announce(&mut self, update: impl Into<UpdateMessage>) -> Result<()> {
self.send(&Message::Update(update.into())).await
}
/// Read exactly one BGP message, waiting at most [`TIMEOUT`] for it. A
/// test that expects nothing to arrive says for how long with
/// [`TestPeer::recv_within`].
pub async fn recv(&mut self) -> Result<Message> {
self.recv_within(TIMEOUT).await
}
/// Read exactly one BGP message, waiting at most `wait` for it.
pub async fn recv_within(&mut self, wait: Duration) -> Result<Message> {
match tokio::time::timeout(wait, self.recv_untimed()).await {
Ok(message) => message,
Err(_) => Err(eyre::eyre!("no message from the server within {wait:?}")),
}
}
/// Read exactly one BGP message, however long it takes to come. For a
/// peer that is to stay on the line, like the `fake_peer` example;
/// tests want [`TestPeer::recv`].
pub async fn recv_untimed(&mut self) -> Result<Message> {
let mut header = [0u8; BGP_HEADER_LEN as usize];
self.stream.read_exact(&mut header).await?;
@@ -42,50 +175,109 @@ impl TestPeer {
let mut buf = vec![0u8; total_len];
buf[..BGP_HEADER_LEN as usize].copy_from_slice(&header);
self.stream.read_exact(&mut buf[BGP_HEADER_LEN as usize..]).await?;
self.stream
.read_exact(&mut buf[BGP_HEADER_LEN as usize..])
.await?;
let (_, msg) = Message::from_wire(&self.ctx, &buf)
.map_err(|e| eyre!("Failed to parse BGP message: {e:?}"))?;
Ok(msg)
Message::decode(&self.ctx, &buf).map_err(|e| eyre!("Failed to parse BGP message: {e}"))
}
/// Read messages until an UPDATE arrives, skipping KEEPALIVEs. Any other
/// message type is an error.
pub async fn recv_update(&mut self) -> Result<UpdateMessage> {
loop {
match self.recv().await? {
Message::Update(u) => return Ok(u),
Message::KeepAlive => {}
other => return Err(eyre::eyre!("Expected UPDATE, got {:?}", other)),
}
}
}
/// Read messages until a NOTIFICATION arrives, skipping KEEPALIVEs. Any
/// other message type is an error.
pub async fn expect_notification(&mut self) -> Result<NotificationMessage> {
loop {
match self.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => return Err(eyre::eyre!("expected NOTIFICATION, got {other:?}")),
}
}
}
/// Receive and unwrap as `OpenMessage`, failing if a different type arrives.
pub async fn recv_open(&mut self) -> Result<OpenMessage> {
match self.recv().await? {
Message::Open(o) => {
// Enable 4-octet ASN parsing if the peer advertised it.
// AS numbers are four octets wide only if both sides
// offered it (RFC 6793 §4.1): the server's offer is read
// here, ours was noted when the OPEN went out.
let four_byte = o.options.iter().any(|opt| {
matches!(opt, OpenOption::Capabilities(caps)
if caps.iter().any(|c| matches!(c, Capability::FourByteAsn { .. })))
});
self.ctx.four_octet_asn = Some(four_byte);
self.ctx.four_octet_asn = four_byte && self.offered_four_octet_asn;
Ok(o)
}
other => bail!("Expected OPEN, got {:?}", other),
other => Err(eyre::eyre!("Expected OPEN, got {:?}", other)),
}
}
/// Send an OPEN message with the given parameters and standard capabilities.
/// Send an OPEN message with the given parameters and the capabilities
/// of a typical dual-stack speaker: four-octet AS numbers, IPv4 and IPv6
/// unicast, route refresh.
pub async fn send_open(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
) -> Result<()> {
let asn_field = if local_asn > u16::MAX as u32 { 23456 } else { local_asn as u16 };
let capabilities = vec![
Capability::FourByteAsn { asn: local_asn },
Capability::MultiProtocol {
afi: Afi::IPV4,
safi: Safi::UNICAST,
},
Capability::MultiProtocol {
afi: Afi::IPV6,
safi: Safi::UNICAST,
},
Capability::RouteRefresh,
];
self.send_open_with(local_asn, router_id, hold_time, capabilities)
.await
}
/// Send an OPEN message carrying exactly `capabilities`.
pub async fn send_open_with(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
capabilities: Vec<Capability>,
) -> Result<()> {
self.offered_four_octet_asn = capabilities
.iter()
.any(|c| matches!(c, Capability::FourByteAsn { .. }));
// If the server's OPEN has been read already, what was settled
// then still depends on what is offered now.
self.ctx.four_octet_asn &= self.offered_four_octet_asn;
let asn_field = if local_asn > u16::MAX as u32 {
23456
} else {
local_asn as u16
};
let open = Message::Open(OpenMessage {
version: 4,
asn: asn_field,
hold_time,
identifier: router_id,
options: vec![OpenOption::Capabilities(vec![
Capability::FourByteAsn { asn: local_asn },
Capability::MultiProtocol {
afi: AddressFamilyId::Ipv4,
safi: SubsequentAfi::Unicast,
},
Capability::RouteRefresh,
])],
options: if capabilities.is_empty() {
vec![]
} else {
vec![OpenOption::Capabilities(capabilities)]
},
});
self.send(&open).await
}
@@ -99,29 +291,67 @@ impl TestPeer {
hold_time: u16,
) -> Result<OpenMessage> {
self.send_open(local_asn, router_id, hold_time).await?;
self.finish_handshake().await
}
/// As [`TestPeer::handshake`], with an OPEN carrying exactly
/// `capabilities`.
pub async fn handshake_with(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
capabilities: Vec<Capability>,
) -> Result<OpenMessage> {
self.send_open_with(local_asn, router_id, hold_time, capabilities)
.await?;
self.finish_handshake().await
}
async fn finish_handshake(&mut self) -> Result<OpenMessage> {
let server_open = self.recv_open().await?;
// Expect server KEEPALIVE
// The server's KEEPALIVE says it accepted our OPEN; anything else
// is its refusal, or a server out of step.
match self.recv().await? {
Message::KeepAlive => {}
other => bail!("Expected KEEPALIVE from server, got {:?}", other),
other => {
return Err(eyre::eyre!(
"Expected KEEPALIVE from server, got {:?}",
other
));
}
}
// Send our KEEPALIVE
// Ours says the same of its OPEN, and the session is established.
self.send(&Message::KeepAlive).await?;
Ok(server_open)
}
/// Send raw bytes without framing — useful for testing invalid input.
/// Writes `bytes` as they are, with no framing of its own: for a test
/// that sends what no message would be, and for the benchmarks, which
/// send a table encoded once as one stream rather than message by
/// message.
pub async fn send_raw(&mut self, bytes: &[u8]) -> Result<()> {
self.stream.write_all(bytes).await?;
Ok(())
}
/// Returns true if the remote end has closed the connection.
pub async fn is_closed(&mut self) -> bool {
let mut buf = [0u8; 1];
match self.stream.read(&mut buf).await {
Ok(0) | Err(_) => true,
Ok(_) => false,
}
/// Whether the server closes the connection within `wait`. Whatever it
/// still sends first, KEEPALIVEs as a rule, is read past: the end of
/// the stream, or anything unreadable, is what counts.
pub async fn closed_within(&mut self, wait: Duration) -> bool {
tokio::time::timeout(wait, async { while self.recv_untimed().await.is_ok() {} })
.await
.is_ok()
}
/// Whether the server closes the connection within [`TIMEOUT`] without
/// having sent anything: what a peer that is not let in sees. A server
/// that sends its OPEN first, or that leaves the connection open, is
/// answering.
pub async fn closed_unanswered(&mut self) -> bool {
matches!(
tokio::time::timeout(TIMEOUT, self.recv_untimed()).await,
Ok(Err(_))
)
}
}
crates/integration-tests/src/test_server.rs
+341 -19
View File
@@ -1,19 +1,52 @@
use std::collections::BTreeSet;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use bgp_rib::AdjRibIn;
use bgp_server::config::Config;
use bgp_server::server::BgpServerBuilder;
use eyre::Result;
use bgp_fib::Fib;
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_packet::open::Capability;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::LocalRib;
use bgp_rib::table::{Event, PeerIndex, Reader, RibTable};
use bgp_rpki::Rpki;
use bgp_server::config::{Config, PeerConfig};
use bgp_server::error::ServerError;
use bgp_server::rejections::RejectionLog;
use bgp_server::restart::Restarts;
use bgp_server::server::BgpServer;
use bgp_server::session::{SessionHandle, SessionRegistry};
use bgp_web::{AppState, RouterInfo};
use eyre::{Result, eyre};
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
use crate::test_peer::TestPeer;
use crate::{peer_config, wait_for, wait_until};
/// A BGP server started on an ephemeral port for testing.
/// Aborts the server task when dropped.
pub struct TestServer {
handle: JoinHandle<Result<()>>,
handle: JoinHandle<Result<(), ServerError>>,
/// The single IPv4 loopback address the server is bound to.
pub addr: SocketAddr,
/// Handle to the server's Adj-RIB-In for test introspection.
pub adj_rib_in: AdjRibIn,
/// Every address it is bound to, in the order configured: `addr`
/// first.
pub addrs: Vec<SocketAddr>,
/// The server's routing table.
pub rib: Arc<RibTable>,
/// Handle to the server's locally originated routes.
pub local_rib: LocalRib,
/// Live view of the server's established sessions.
pub registry: SessionRegistry,
/// What peers announced that was not taken.
pub rejections: RejectionLog,
/// The data for route origin validation, if the server validates.
pub rpki: Option<Rpki>,
/// What keeps the best routes in the kernel, if the server does.
pub kernel: Option<Fib>,
/// The routes kept of restarting peers (graceful restart).
pub restarts: Arc<Restarts>,
pub local_asn: u32,
}
impl TestServer {
@@ -24,28 +57,317 @@ impl TestServer {
/// Start a server with an explicit hold time (useful for timer tests).
pub async fn start_with_hold_time(local_asn: u32, hold_time_secs: u16) -> Result<Self> {
Self::start_inner(local_asn, hold_time_secs).await
Self::start_with(local_asn, |c| c.hold_time_secs = hold_time_secs).await
}
async fn start_inner(local_asn: u32, hold_time_secs: u16) -> Result<Self> {
let config = Config {
/// Start a server after letting `customize` edit the default test config
/// (e.g. to add configured peers).
pub async fn start_with(local_asn: u32, customize: impl FnOnce(&mut Config)) -> Result<Self> {
let mut config = Config {
router_id: Ipv4Addr::new(127, 0, 0, 1),
cluster_id: Ipv4Addr::new(127, 0, 0, 1),
local_asn,
hold_time_secs,
hold_time_secs: 90,
listen_endpoints: vec![SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0)],
peers: vec![],
connect_retry: std::time::Duration::from_secs(5),
collision_detect_established_state: false,
// Most tests play a peer that connects from wherever the
// system puts it; the ones about who may connect turn this off.
accept_unconfigured_peers: true,
originate: vec![],
http_endpoint: None,
rpki: None,
kernel: None,
export: Default::default(),
export_limits: None,
community_names: Vec::new(),
control: None,
graceful_restart: Default::default(),
};
customize(&mut config);
let bound = BgpServerBuilder::new(config).bind().await?;
let addr = bound
.local_addrs()
.into_iter()
.next()
.expect("at least one listener");
let adj_rib_in = bound.adj_rib_in.clone();
let bound = BgpServer::bind(config).await?;
let addrs = bound.local_addrs();
let addr = *addrs.first().expect("at least one listener");
let rib = Arc::clone(&bound.rib);
let local_rib = bound.local_rib.clone();
let registry = bound.registry.clone();
let rejections = bound.rejections.clone();
let rpki = bound.rpki.clone();
let kernel = bound.kernel.clone();
let restarts = Arc::clone(&bound.restarts);
let handle = tokio::spawn(bound.run());
Ok(Self { handle, addr, adj_rib_in })
Ok(Self {
handle,
addr,
addrs,
rib,
local_rib,
registry,
rejections,
rpki,
kernel,
restarts,
local_asn,
})
}
/// A server in `local_asn` that takes connections from one configured
/// peer only: "r1", in `remote_asn`, at the loopback address the test
/// connects from. `customize` edits the configuration and the peer's
/// before the server starts. The server dials the peer too; where that
/// goes is returned with it, for the test to answer or to leave alone.
pub async fn with_peer(
local_asn: u32,
remote_asn: u32,
customize: impl FnOnce(&mut Config, &mut PeerConfig),
) -> Result<ServerWithPeer> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = Self::start_with(local_asn, |config| {
config.accept_unconfigured_peers = false;
let mut peer = peer_config("r1", endpoint, remote_asn);
customize(config, &mut peer);
config.peers = vec![peer];
})
.await?;
Ok(ServerWithPeer { server, listener })
}
}
/// A server with one configured peer, and the listener at the peer's
/// address, which the server's dialling goes to.
pub struct ServerWithPeer {
pub server: TestServer,
pub listener: TcpListener,
}
impl TestServer {
/// A session with the server from a peer in `asn` with `router_id`,
/// offering the capabilities of a typical dual-stack speaker. Returns
/// once the server lists the session as established. Distinct peers
/// need distinct identifiers, or the second connection is (correctly)
/// treated as a collision with the first.
pub async fn established(&self, asn: u32, router_id: Ipv4Addr) -> Result<TestPeer> {
let mut peer = TestPeer::connect(self.addr).await?;
peer.handshake(asn, router_id, 90).await?;
self.listed(&peer).await?;
Ok(peer)
}
/// As [`TestServer::established`], with an OPEN carrying exactly
/// `capabilities`.
pub async fn established_with(
&self,
asn: u32,
router_id: Ipv4Addr,
capabilities: Vec<Capability>,
) -> Result<TestPeer> {
let mut peer = TestPeer::connect(self.addr).await?;
peer.handshake_with(asn, router_id, 90, capabilities)
.await?;
self.listed(&peer).await?;
Ok(peer)
}
/// Waits until the registry lists `peer`'s session.
async fn listed(&self, peer: &TestPeer) -> Result<()> {
let from = peer.local_addr()?;
wait_until("the session is established", || {
self.registry
.list()
.iter()
.any(|session| session.peer.addr == from)
})
.await
}
/// Waits until the registry lists `n` established sessions.
pub async fn wait_for_sessions(&self, n: usize) -> Result<()> {
wait_until(&format!("{n} sessions are established"), || {
self.registry.list().len() == n
})
.await
}
/// Waits until a session is established, and returns it.
pub async fn wait_for_session(&self) -> Result<SessionHandle> {
wait_for("a session is established", || {
self.registry.list().into_iter().next()
})
.await
}
/// Waits until the table holds `routes` routes.
pub async fn wait_for_routes(&self, routes: usize) -> Result<()> {
wait_until(&format!("the table holds {routes} routes"), || {
self.rib.len() == routes
})
.await
}
/// Waits until the table holds exactly the prefixes `expected` and the
/// peer has `held` routes held aside.
pub async fn table_becomes(&self, expected: &[&str], held: u64) -> Result<()> {
let expected: BTreeSet<String> = expected.iter().map(|p| p.to_string()).collect();
let table = || -> BTreeSet<String> {
self.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect()
};
let held_now = || {
self.registry
.list()
.first()
.map_or(0, |session| session.counters.held())
};
wait_until("the table is as expected", || {
table() == expected && held_now() == held
})
.await
.map_err(|_| {
eyre!(
"the table has {:?} with {} held; expected {expected:?} with {held} held",
table(),
held_now()
)
})
}
}
/// A route in the server's table, as a test wants to look at it.
#[derive(Debug, Clone)]
pub struct ReceivedRoute {
pub prefix: IpPrefix,
pub peer: PeerIndex,
/// The peer's identifier for the route; zero without ADD-PATH.
pub path_id: u32,
/// The route's attributes, without MP_REACH_NLRI.
pub attributes: Vec<PathAttribute>,
/// The next hop, if the route came in MP_REACH_NLRI.
pub next_hop: Option<NlriNextHop>,
/// Whether it is the best route for its prefix.
pub best: bool,
}
/// A change of a prefix's best route.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RouteChange {
/// The prefix has a best route, or another one than before.
Announced { prefix: IpPrefix, peer: PeerIndex },
/// The prefix has no route any more.
Withdrawn { prefix: IpPrefix },
}
/// The changes of best route in a server's table from the moment this was
/// made, read from the table's journal.
pub struct RouteChanges {
reader: Reader,
pending: std::collections::VecDeque<RouteChange>,
}
impl RouteChanges {
/// The next change, waiting for one if need be. Changes to one prefix
/// that happen before they are read arrive as one.
pub async fn recv(&mut self) -> Result<RouteChange> {
loop {
if let Some(change) = self.pending.pop_front() {
return Ok(change);
}
self.reader.changed().await;
self.read();
}
}
fn read(&mut self) {
let pending = &mut self.pending;
let mut read = |event: Event<'_>| match event {
Event::Best { prefix, peer, .. } => {
pending.push_back(RouteChange::Announced { prefix, peer })
}
Event::Withdrawn { prefix } => pending.push_back(RouteChange::Withdrawn { prefix }),
// Not given to a follower of the best routes.
Event::Routes { .. } => {}
Event::Reset | Event::Synced => {}
};
while !self.reader.poll(1024, &mut read).caught_up {}
}
}
impl TestServer {
/// Every route in the server's table.
pub fn received_routes(&self) -> Vec<ReceivedRoute> {
let mut routes = Vec::new();
self.rib.for_each_route(|route| {
let (attributes, next_hop) = route.attrs.attributes().expect("stored attributes parse");
routes.push(ReceivedRoute {
prefix: route.prefix,
peer: route.peer,
path_id: route.path_id,
attributes,
next_hop,
best: route.best,
});
});
routes
}
/// Follows the changes of best route from now on.
pub fn route_changes(&self) -> RouteChanges {
let mut changes = RouteChanges {
reader: self.rib.reader(),
pending: Default::default(),
};
// A new reader is first given the table as it is. That is not a
// change.
changes.read();
changes.pending.clear();
changes
}
/// Waits for the server to stop by itself and returns why. A healthy
/// server never does.
pub async fn stopped(&mut self) -> Result<(), ServerError> {
(&mut self.handle)
.await
.expect("the server task does not panic")
}
/// Starts the HTTP API on an ephemeral loopback port and returns its
/// base URL (e.g. `http://127.0.0.1:12345`). The API task is aborted
/// when the returned guard drops.
pub async fn serve_http(&self) -> Result<(String, HttpGuard)> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let base_url = format!("http://{}", listener.local_addr()?);
let state = AppState {
info: std::sync::Arc::new(
RouterInfo::new(Ipv4Addr::new(127, 0, 0, 1), self.local_asn, "test")
.with_config("test.textpb", format!("local_asn: {}\n", self.local_asn)),
),
registry: self.registry.clone(),
local_rib: self.local_rib.clone(),
rib: Arc::clone(&self.rib),
rejections: self.rejections.clone(),
rpki: self.rpki.clone(),
kernel: self.kernel.clone(),
restarts: Arc::clone(&self.restarts),
};
let handle = tokio::spawn(bgp_web::serve_on(state, listener));
Ok((base_url, HttpGuard { handle }))
}
}
/// Aborts the HTTP API task when dropped.
pub struct HttpGuard {
handle: JoinHandle<std::io::Result<()>>,
}
impl Drop for HttpGuard {
fn drop(&mut self) {
self.handle.abort();
}
}
crates/integration-tests/tests/add_path.rs
+569
View File
@@ -0,0 +1,569 @@
//! End-to-end tests of ADD-PATH (RFC 7911): a peer sending several routes
//! for one prefix, told apart by path identifiers.
use std::collections::BTreeMap;
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal};
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{peer_config, prefix, wait_until};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::open::{AddPathEntry, Capability, OpenMessage, SendReceive};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::Nlri;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::SUPPORTED;
use bgp_server::policy::{ExportPolicy, ImportPolicy};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
/// A server on which connections from this machine are those of a
/// configured peer, with ADD-PATH offered to it as `add_path` says.
async fn server_offering(add_path: Option<SendReceive>, originate: &[&str]) -> Result<TestServer> {
let originate: Vec<LocalRoute> = originate
.iter()
.map(|p| LocalRoute::new(prefix(p), LocalRouteAttrs::default()))
.collect();
TestServer::start_with(LOCAL_ASN, |c| {
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
add_path,
import: ImportPolicy::permissive(),
..peer_config("route-server-client", at, PEER_ASN)
}];
c.originate = originate;
// Keep the dialer out of the way.
c.connect_retry = Duration::from_secs(3600);
})
.await
}
/// A session from a peer whose OPEN offers ADD-PATH as `offer` says, for
/// IPv4 unicast. Returns the peer and the server's OPEN.
async fn peer_offering(
server: &TestServer,
offer: Option<SendReceive>,
) -> Result<(TestPeer, OpenMessage)> {
let mut capabilities = vec![Capability::FourByteAsn { asn: PEER_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
if let Some(send_recv) = offer {
capabilities.push(Capability::AddPath {
entries: vec![AddPathEntry {
afi: AfiSafi::IPV4_UNICAST.afi,
safi: AfiSafi::IPV4_UNICAST.safi,
send_recv,
}],
});
}
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90, capabilities)
.await?;
server.wait_for_session().await?;
Ok((peer, open))
}
fn add_path_offered(open: &OpenMessage) -> Vec<(AfiSafi, SendReceive)> {
open.capabilities()
.filter_map(|capability| match capability {
Capability::AddPath { entries } => Some(entries),
_ => None,
})
.flatten()
.map(|entry| (entry.family(), entry.send_recv))
.collect()
}
fn announcement(nlri: Nlri, as_path: &[u32]) -> Announcement {
Announcement::of([nlri], as_path)
}
/// The routes the server holds, as (path identifier, AS path, best).
fn held(server: &TestServer) -> Vec<(u32, Vec<u32>, bool)> {
let mut held: Vec<(u32, Vec<u32>, bool)> = server
.received_routes()
.into_iter()
.map(|route| {
let path = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.unwrap_or_default();
(route.path_id, path, route.best)
})
.collect();
held.sort();
held
}
/// A peer that sends several routes for one prefix: each is kept, the best
/// of them is chosen, and each is withdrawn on its own.
#[tokio::test]
async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
let server = server_offering(Some(SendReceive::Receive), &[]).await?;
let (base, _http) = server.serve_http().await?;
let (mut peer, open) = peer_offering(&server, Some(SendReceive::Send)).await?;
// The server offered to receive, for every family it offers.
assert_eq!(
add_path_offered(&open),
SUPPORTED.map(|family| (family, SendReceive::Receive))
);
// In use for the one family the peer sends for, and in that direction.
let session = &server.registry.list()[0];
assert_eq!(
session.add_path_receive,
[AfiSafi::IPV4_UNICAST].into_iter().collect()
);
assert!(session.add_path_send.is_empty());
peer.send_add_path = vec![AfiSafi::IPV4_UNICAST];
let p = prefix("203.0.113.0/24");
let path = |path_id| Nlri { prefix: p, path_id };
peer.announce(announcement(path(1), &[PEER_ASN, 64999, 64998]))
.await?;
peer.announce(announcement(path(2), &[PEER_ASN, 64998]))
.await?;
server.wait_for_routes(2).await?;
assert_eq!(
held(&server),
[
(1, vec![PEER_ASN, 64999, 64998], false),
(2, vec![PEER_ASN, 64998], true),
]
);
assert_eq!(server.rib.prefix_count(), 1);
// The API tells them apart too.
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
let mut ids: Vec<u64> = routes["routes"]
.as_array()
.unwrap()
.iter()
.map(|route| route["path_id"].as_u64().unwrap())
.collect();
ids.sort_unstable();
assert_eq!(ids, [1, 2]);
let peers: Value = reqwest::get(format!("{base}/api/peers"))
.await?
.json()
.await?;
assert_eq!(
peers[0]["add_path_receive"],
serde_json::json!(["IPv4 unicast"])
);
// Announcing a path again replaces that path only.
peer.announce(announcement(path(1), &[PEER_ASN])).await?;
wait_until("the path is replaced", || held(&server)[0].1.len() == 1).await?;
assert_eq!(
held(&server),
[(1, vec![PEER_ASN], true), (2, vec![PEER_ASN, 64998], false),]
);
// And withdrawing one leaves the other.
peer.announce(withdrawal([path(1)])).await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(2, vec![PEER_ASN, 64998], true)]);
// When the session ends, all of its routes go.
peer.announce(announcement(path(3), &[PEER_ASN, 1, 2, 3]))
.await?;
server.wait_for_routes(2).await?;
drop(peer);
server.wait_for_routes(0).await?;
Ok(())
}
/// ADD-PATH changes how prefixes are written. It must be in use only when
/// both sides said so; a peer that offers it to a server that does not is
/// spoken to, and listened to, in plain BGP.
#[tokio::test]
async fn test_add_path_is_not_used_unless_both_sides_offer_it() -> Result<()> {
let server = server_offering(None, &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, Some(SendReceive::Both)).await?;
assert!(add_path_offered(&open).is_empty());
let session = &server.registry.list()[0];
assert!(session.add_path_receive.is_empty() && session.add_path_send.is_empty());
// Plain prefixes both ways.
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
peer.announce(announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
// And the other way round: offered by the server, not by the peer.
let server = server_offering(Some(SendReceive::Both), &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, None).await?;
assert_eq!(add_path_offered(&open).len(), SUPPORTED.len());
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
Ok(())
}
/// Where the peer offered to receive and the server to send, the server's
/// prefixes carry a path identifier, and a peer that reads them so
/// understands them.
#[tokio::test]
async fn test_prefixes_are_sent_with_path_identifiers_when_agreed() -> Result<()> {
let server = server_offering(Some(SendReceive::Both), &["10.100.0.0/16"]).await?;
let (mut peer, _) = peer_offering(&server, Some(SendReceive::Receive)).await?;
let session = &server.registry.list()[0];
assert_eq!(
session.add_path_send,
[AfiSafi::IPV4_UNICAST].into_iter().collect()
);
assert!(session.add_path_receive.is_empty());
// Read as agreed: with identifiers.
peer.ctx.add_path = vec![AfiSafi::IPV4_UNICAST];
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes.len(), 1);
assert_eq!(update.announced_routes[0].prefix, prefix("10.100.0.0/16"));
// The peer did not offer to send, so the server reads its prefixes
// plain.
peer.announce(announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
Ok(())
}
const COLLECTOR_ASN: u32 = 65100;
/// What a collector holds: for each prefix and path identifier, the
/// route's AS path.
type Held = BTreeMap<(String, u32), Vec<u32>>;
/// Reads what the collector is told, into `held`, until `done` says so.
async fn told(
collector: &mut TestPeer,
held: &mut Held,
done: impl Fn(&Held) -> bool,
) -> Result<()> {
while !done(held) {
let update = collector.recv_update().await?;
let path = update
.path_attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.unwrap_or_default();
for nlri in &update.withdrawn_routes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
for nlri in &update.announced_routes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
// And what came and went in the multiprotocol attributes.
for attribute in &update.path_attributes {
match attribute {
PathAttribute::MpReachNlri(reach) => {
for nlri in &reach.prefixes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
}
PathAttribute::MpUnreachNlri(unreach) => {
for nlri in &unreach.prefixes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
}
_ => {}
}
}
}
Ok(())
}
/// The routes held for 203.0.113.0/24, by path identifier.
fn paths(held: &Held) -> Vec<(u32, Vec<u32>)> {
held.iter()
.map(|((prefix, id), path)| {
assert_eq!(prefix, "203.0.113.0/24");
(*id, path.clone())
})
.collect()
}
/// A collector that agreed to be sent several routes for a prefix, and
/// whose export policy says to send it every route, is told of every
/// peer's route for a prefix, each with a path identifier of its own that
/// stays with the route; a route that goes is withdrawn by its identifier.
#[tokio::test]
async fn test_every_route_goes_to_a_collector_told_apart_by_path_identifiers() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// The collector is the configured peer at the IPv4 loopback
// address; the feeders come over IPv6 loopback as peers of no
// account.
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let route = |path: &[u32]| Announcement::of([prefix("203.0.113.0/24").into()], path);
let feeder = async |n: u32| -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
Ok(peer)
};
let mut a = feeder(0).await?;
let mut b = feeder(1).await?;
server.wait_for_sessions(2).await?;
a.announce(route(&[PEER_ASN, 64510])).await?;
b.announce(route(&[PEER_ASN + 1, 64510])).await?;
server.wait_for_routes(2).await?;
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, false),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
// Both routes, told apart.
let mut held = Held::new();
told(&mut collector, &mut held, |held| held.len() == 2).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 1, 64510]),
]
);
// The second peer's route goes: withdrawn by its identifier, the
// first's untouched.
b.announce(withdrawal([prefix("203.0.113.0/24").into()]))
.await?;
told(&mut collector, &mut held, |held| held.len() == 1).await?;
assert_eq!(paths(&held), [(1, vec![LOCAL_ASN, PEER_ASN, 64510])]);
// The first peer's route changes: it goes out again with the same
// identifier. A third peer's route gets the lowest free one.
a.announce(route(&[PEER_ASN, 64511, 64510])).await?;
let mut c = feeder(2).await?;
server.wait_for_sessions(4).await?;
c.announce(route(&[PEER_ASN + 2, 64510])).await?;
let changed = |held: &Held| {
held.len() == 2
&& held.get(&("203.0.113.0/24".to_string(), 1))
== Some(&vec![LOCAL_ASN, PEER_ASN, 64511, 64510])
};
told(&mut collector, &mut held, changed).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64511, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 2, 64510]),
]
);
Ok(())
}
/// A collector that is connected before the table arrives is told the
/// whole table, however large: the walk of the table that a new follower
/// starts with takes many batches with a table of some size, and goes on
/// while the feeders keep sending.
#[tokio::test]
async fn test_a_large_table_reaches_a_collector_that_was_there_first() -> Result<()> {
const PREFIXES: u32 = 40_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, false),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
server.wait_for_sessions(1).await?;
// Two feeders, each sending the same prefixes a hundred to an UPDATE.
let nth = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..PREFIXES).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| nth(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * PREFIXES as usize
})
.await?;
let ids: std::collections::BTreeSet<u32> = held.keys().map(|(_, id)| *id).collect();
assert_eq!(ids.into_iter().collect::<Vec<_>>(), [1, 2]);
Ok(())
}
/// As a router at an exchange feeds a collector: one IPv6 session that
/// carries both families, with ADD-PATH for both and IPv4 routes with
/// IPv6 next hops (RFC 8950), and a table of some size from two feeders.
#[tokio::test]
async fn test_both_families_reach_a_collector_over_one_ipv6_session() -> Result<()> {
const V4: u32 = 20_000;
const V6: u32 = 1_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut collector = TestPeer::connect(server.addrs[1]).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
collector_capabilities(COLLECTOR_ASN, &SUPPORTED, true),
)
.await?;
collector.ctx.add_path = SUPPORTED.to_vec();
server.wait_for_sessions(1).await?;
let session = &server.registry.list()[0];
assert!(session.extended_next_hop, "RFC 8950 agreed");
assert_eq!(session.add_path_send.iter().count(), 2);
let v4 = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let v6 = |n: u32| prefix(&format!("2001:db8:{:x}::/48", 0x1000 + n));
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..V4).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| v4(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
for start in (0..V6).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
let update = UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN + n as u32, 64510]),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::9".parse().unwrap()),
prefixes: (start..start + 100).map(|i| v6(i).into()).collect(),
}),
],
announced_routes: vec![],
};
feeder.send(&Message::Update(update)).await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * (V4 + V6) as usize
})
.await?;
let v6_routes = held
.keys()
.filter(|(prefix, _)| prefix.contains(':'))
.count();
assert_eq!(v6_routes, 2 * V6 as usize);
Ok(())
}
crates/integration-tests/tests/admission.rs
+89
View File
@@ -0,0 +1,89 @@
//! Who may connect: only the addresses of configured peers, and each of
//! them only so many times at once.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, peer_config};
use bgp_server::config::PeerConfig;
use bgp_server::policy::ImportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
/// A host that is not a configured peer gets nothing: no OPEN to answer,
/// no session, no routes taken from it. Any host used to be able to open a
/// session as any AS it named.
#[tokio::test]
async fn test_a_host_that_is_not_a_configured_peer_is_refused() -> Result<()> {
// The only peer configured is somewhere else (documentation space).
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
let elsewhere: SocketAddr = "192.0.2.9:179".parse().unwrap();
c.peers = vec![PeerConfig {
import: ImportPolicy::permissive(),
..peer_config("r1", elsewhere, PEER_ASN)
}];
})
.await?;
let mut stranger = TestPeer::connect(server.addr).await?;
assert!(stranger.closed_unanswered().await);
assert!(server.registry.list().is_empty());
Ok(())
}
/// The address of a configured peer is let in, and held to what is
/// configured for that peer.
#[tokio::test]
async fn test_a_configured_peer_is_let_in() -> Result<()> {
// The server's own dialling goes to the listener; it is never answered.
let ServerWithPeer {
server,
listener: _listener,
} = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |_, peer| {
peer.import = ImportPolicy::permissive();
})
.await?;
let _peer = server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2))
.await?;
assert_eq!(server.registry.list()[0].name.as_deref(), Some("r1"));
Ok(())
}
/// One address may hold only a few connections at a time, so that one host
/// cannot use up the server's sockets with connections that never get as
/// far as a session.
#[tokio::test]
async fn test_one_address_may_open_only_a_few_connections() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
// Four that say nothing: each is sent the server's OPEN and waited on.
let mut held = Vec::new();
for _ in 0..4 {
let mut peer = TestPeer::connect(server.addr).await?;
peer.recv_open().await?;
held.push(peer);
}
let mut one_too_many = TestPeer::connect(server.addr).await?;
assert!(one_too_many.closed_unanswered().await);
// When one of them goes, there is room again.
held.pop();
tokio::time::timeout(TIMEOUT, async {
loop {
let mut peer = TestPeer::connect(server.addr).await?;
if peer.recv_open().await.is_ok() {
return Ok::<_, eyre::Report>(());
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await??;
Ok(())
}
crates/integration-tests/tests/announce.rs
+464
View File
@@ -0,0 +1,464 @@
//! End-to-end tests for locally originated routes: announcing via the
//! `LocalRib` handle must produce UPDATEs on every established session.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
LocalPrefPathAttribute, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::session::SessionEvent;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
fn route(p: &str, attrs: LocalRouteAttrs) -> LocalRoute {
LocalRoute::new(prefix(p), attrs)
}
async fn established_peer(server: &TestServer) -> Result<TestPeer> {
established_peer_with_id(server, Ipv4Addr::new(10, 0, 0, 1)).await
}
async fn established_peer_with_id(server: &TestServer, router_id: Ipv4Addr) -> Result<TestPeer> {
server.established(PEER_ASN, router_id).await
}
async fn next_update(peer: &mut TestPeer) -> Result<UpdateMessage> {
peer.recv_update().await
}
// ---------------------------------------------------------------------------
// Session registry
// ---------------------------------------------------------------------------
#[tokio::test]
async fn test_registry_tracks_session_lifecycle() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut events = server.registry.subscribe();
assert!(server.registry.list().is_empty());
let peer = established_peer(&server).await?;
let up = tokio::time::timeout(TIMEOUT, events.recv()).await??;
let SessionEvent::Up(handle) = up else {
panic!("expected Up, got {up:?}")
};
assert_eq!(handle.peer.asn, PEER_ASN);
assert_eq!(handle.local_addr, server.addr);
assert!(handle.four_octet_asn);
assert_eq!(server.registry.list().len(), 1);
drop(peer);
let down = tokio::time::timeout(TIMEOUT, events.recv()).await??;
assert!(
matches!(down, SessionEvent::Down(p) if p == handle.peer),
"got {down:?}"
);
assert!(server.registry.list().is_empty());
Ok(())
}
// ---------------------------------------------------------------------------
// Announce / withdraw
// ---------------------------------------------------------------------------
/// A route announced while the session is up is sent immediately, with the
/// attributes the announcer chose and our address as the next hop.
#[tokio::test]
async fn test_announce_after_session_up() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
origin: OriginPathAttribute::IGP,
prepend: 0,
med: None,
communities: vec![(65000, 100), (65535, 65281)],
large_communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 7,
data_2: 8,
}],
},
))
.await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(update.withdrawn_routes.is_empty());
assert_eq!(
update.path_attributes,
vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![LOCAL_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::LOCALHOST)),
PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65000, 100),
(65535, 65281)
])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 7,
data_2: 8
}],
}),
]
);
Ok(())
}
/// Routes announced before any session exists are replayed when a session
/// reaches Established.
#[tokio::test]
async fn test_routes_replayed_on_session_up() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
server
.local_rib
.announce(route("10.100.2.0/24", LocalRouteAttrs::default()))
.await?;
let mut peer = established_peer(&server).await?;
let mut seen = vec![next_update(&mut peer).await?, next_update(&mut peer).await?]
.into_iter()
.flat_map(|u| u.announced_routes)
.map(|p| p.to_string())
.collect::<Vec<_>>();
seen.sort();
assert_eq!(seen, vec!["10.100.1.0/24", "10.100.2.0/24"]);
Ok(())
}
/// An announcement is fanned out to every established session.
#[tokio::test]
async fn test_announce_reaches_all_sessions() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
server.wait_for_sessions(2).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
for peer in [&mut a, &mut b] {
let update = next_update(peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
}
Ok(())
}
#[tokio::test]
async fn test_withdraw_sends_withdrawn_route() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let _ = next_update(&mut peer).await?;
server.local_rib.withdraw(prefix("10.100.1.0/24")).await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.withdrawn_routes, vec![prefix("10.100.1.0/24")]);
assert!(update.announced_routes.is_empty());
assert!(update.path_attributes.is_empty());
Ok(())
}
/// Re-announcing with new attributes sends a fresh UPDATE (implicit replace).
#[tokio::test]
async fn test_reannounce_with_prepend_and_med() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let first = next_update(&mut peer).await?;
assert_eq!(
first.path_attributes[1],
AsPathAttribute::from_asns(vec![LOCAL_ASN])
);
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
prepend: 3,
med: Some(42),
..LocalRouteAttrs::default()
},
))
.await?;
let second = next_update(&mut peer).await?;
assert_eq!(second.announced_routes, vec![prefix("10.100.1.0/24")]);
assert_eq!(
second.path_attributes[1],
AsPathAttribute::from_asns(vec![LOCAL_ASN; 4])
);
assert!(
second
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::MultiExitDisc(m) if m.0 == 42))
);
Ok(())
}
/// The session stays healthy (keepalives keep flowing) while UPDATEs are sent.
#[tokio::test]
async fn test_keepalives_continue_alongside_updates() -> Result<()> {
let server = TestServer::start_with_hold_time(LOCAL_ASN, 3).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 3)
.await?;
server.wait_for_sessions(1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let mut saw_update = false;
let mut saw_keepalive = false;
tokio::time::timeout(TIMEOUT, async {
while !(saw_update && saw_keepalive) {
match peer.recv().await? {
bgp_packet::message::Message::Update(_) => saw_update = true,
bgp_packet::message::Message::KeepAlive => saw_keepalive = true,
other => eyre::bail!("unexpected {other:?}"),
}
}
Ok::<_, eyre::Error>(())
})
.await??;
Ok(())
}
/// Routes learned from a peer disappear from the Adj-RIB-In when its
/// session ends, with a Withdrawn event for each.
#[tokio::test]
async fn test_received_routes_removed_when_session_closes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut rib_events = server.route_changes();
let mut session_events = server.registry.subscribe();
let mut peer = established_peer(&server).await?;
server.wait_for_sessions(1).await?;
peer.announce(
announcement(&["203.0.113.0/24", "198.51.100.0/24"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 1)),
)
.await?;
for _ in 0..2 {
let ev = tokio::time::timeout(TIMEOUT, rib_events.recv()).await??;
assert!(matches!(ev, RouteChange::Announced { .. }), "{ev:?}");
}
assert_eq!(server.received_routes().len(), 2);
drop(peer);
let mut withdrawn = vec![];
for _ in 0..2 {
match tokio::time::timeout(TIMEOUT, rib_events.recv()).await?? {
RouteChange::Withdrawn { prefix, .. } => withdrawn.push(prefix.to_string()),
other => panic!("expected Withdrawn, got {other:?}"),
}
}
withdrawn.sort();
assert_eq!(withdrawn, vec!["198.51.100.0/24", "203.0.113.0/24"]);
assert!(server.received_routes().is_empty());
// Withdrawals precede the Down event.
let _up = tokio::time::timeout(TIMEOUT, session_events.recv()).await??;
let down = tokio::time::timeout(TIMEOUT, session_events.recv()).await??;
assert!(matches!(down, SessionEvent::Down(_)), "{down:?}");
Ok(())
}
// ---------------------------------------------------------------------------
// Route refresh and shutdown
// ---------------------------------------------------------------------------
/// ROUTE-REFRESH for IPv4 unicast exactly as a router sends it (RFC 2918):
/// 19-byte header plus AFI, reserved, SAFI.
#[rustfmt::skip]
const ROUTE_REFRESH_IPV4: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x17, 0x05,
0x00, 0x01, 0x00, 0x01,
];
/// A peer whose import policy changed asks for our routes again; it must get
/// every local route re-sent, and the session must stay up.
#[tokio::test]
async fn test_route_refresh_resends_local_routes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
server
.local_rib
.announce(route("10.100.2.0/24", LocalRouteAttrs::default()))
.await?;
let _ = next_update(&mut peer).await?;
let _ = next_update(&mut peer).await?;
peer.send_raw(ROUTE_REFRESH_IPV4).await?;
let mut resent = vec![next_update(&mut peer).await?, next_update(&mut peer).await?]
.into_iter()
.flat_map(|u| u.announced_routes)
.map(|p| p.to_string())
.collect::<Vec<_>>();
resent.sort();
assert_eq!(resent, vec!["10.100.1.0/24", "10.100.2.0/24"]);
assert_eq!(
server.registry.list().len(),
1,
"session must survive a route refresh"
);
Ok(())
}
/// Shutting down tells each peer why (Cease / Administrative Shutdown) and
/// closes the sessions, instead of leaving dead TCP connections behind.
#[tokio::test]
async fn test_shutdown_sends_cease_to_every_peer() -> Result<()> {
use bgp_packet::errors::{BgpError, CeaseSubcode};
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
server.wait_for_sessions(2).await?;
let left = server
.registry
.leave(bgp_server::restart::Leave::Stop, Duration::from_secs(2))
.await;
assert_eq!(left, 0, "sessions still open after shutdown");
assert!(server.registry.is_shutting_down());
for peer in [&mut a, &mut b] {
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::Cease(CeaseSubcode::AdminShutdown)),
"{notif:?}"
);
assert!(peer.closed_within(TIMEOUT).await);
}
Ok(())
}
// ---------------------------------------------------------------------------
// Routes from the configuration
// ---------------------------------------------------------------------------
/// Routes listed under `originate` are in the table before any peer
/// connects, and each session is sent them when it comes up.
#[tokio::test]
async fn test_configured_routes_are_announced_to_every_new_session() -> Result<()> {
let attrs = LocalRouteAttrs {
med: Some(7),
communities: vec![(65000, 100)],
..Default::default()
};
let configured = vec![
route("10.100.1.0/24", attrs),
route("10.100.2.0/24", LocalRouteAttrs::default()),
];
let server = TestServer::start_with(LOCAL_ASN, |c| c.originate = configured).await?;
assert_eq!(server.local_rib.snapshot().await.unwrap().len(), 2);
for router_id in [Ipv4Addr::new(10, 0, 0, 1), Ipv4Addr::new(10, 0, 0, 2)] {
let mut peer = established_peer_with_id(&server, router_id).await?;
let mut seen = [next_update(&mut peer).await?, next_update(&mut peer).await?];
seen.sort_by_key(|u| u.announced_routes[0].to_string());
assert_eq!(seen[0].announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(
seen[0]
.path_attributes
.contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65000, 100)
])))
);
assert_eq!(seen[1].announced_routes, vec![prefix("10.100.2.0/24")]);
}
Ok(())
}
/// A peer in our own AS is sent a route we originate with an empty AS path
/// and a LOCAL_PREF, whatever prepending is configured: our AS number goes
/// on the path only as the route leaves the AS. It used to be sent as to
/// an external peer, which an internal one refuses.
#[tokio::test]
async fn test_a_peer_in_our_own_as_gets_an_empty_path_and_a_local_pref() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(LOCAL_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
// The peer reads as a router in the server's AS does: one outside it
// would drop the LOCAL_PREF.
peer.ctx.ibgp = true;
server.wait_for_sessions(1).await?;
assert!(server.registry.list()[0].ibgp);
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
prepend: 2,
..LocalRouteAttrs::default()
},
))
.await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(
update
.path_attributes
.contains(&PathAttribute::AsPath(AsPathAttribute { segments: vec![] })),
"{:?}",
update.path_attributes
);
assert!(
update
.path_attributes
.contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(100))),
"{:?}",
update.path_attributes
);
Ok(())
}
crates/integration-tests/tests/aspa.rs
+275
View File
@@ -0,0 +1,275 @@
//! AS path verification with ASPAs, end to end: a server that gets ASPAs
//! from a cache, a peer that announces routes, and the ASPAs changing
//! under both.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_cache::TestCache;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_packet::message::PathAttribute;
use bgp_rpki::{CacheState, RpkiConfig};
use bgp_server::policy::{ImportPolicy, PeerRole};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server that validates against `cache`, with one configured peer at
/// the address the test connects from, which is `role` to it. Returns once
/// the server has the cache's data.
async fn server_with(
cache: &TestCache,
role: Option<PeerRole>,
import: ImportPolicy,
) -> Result<ServerWithPeer> {
let cache = cache.address();
let setup = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.rpki = Some(RpkiConfig {
caches: vec![cache],
retry: Some(Duration::from_millis(100)),
..RpkiConfig::default()
});
peer.import = import;
peer.role = role;
})
.await?;
let rpki = setup.server.rpki.clone().expect("validation is configured");
wait_until("the server has the cache's data", || {
rpki.status().state == CacheState::Synchronised
})
.await?;
Ok(setup)
}
async fn established(server: &TestServer) -> Result<TestPeer> {
server.established(PEER_ASN, PEER_ID).await
}
/// Announces `prefixes` with the AS path `path`.
async fn announce(peer: &mut TestPeer, prefixes: &[&str], path: &[u32]) -> Result<()> {
peer.announce(announcement(prefixes, path)).await
}
/// AS 15169 buys from 3356 and says so; 3356 and the peer, 3333, have no
/// providers.
fn aspas() -> Vec<(u32, Vec<u32>)> {
vec![(15169, vec![3356]), (3356, vec![0]), (PEER_ASN, vec![0])]
}
/// A peer that hands us a route it got from one of its own peers is
/// leaking it. The route is refused; routes whose paths are attested, or
/// that nothing is known about, are taken.
#[tokio::test]
async fn test_a_leaked_route_is_refused_and_the_rest_are_taken() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
// The peer's own route; one of a customer of its that has no ASPA;
// and 15169's route by way of 3356, which is not for 3333 to pass on
// to a peer: 3356 is not its customer.
announce(&mut peer, &["193.0.0.0/21"], &[PEER_ASN]).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121]).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
server
.table_becomes(&["193.0.0.0/21", "194.0.0.0/24"], 1)
.await?;
let client = reqwest::Client::new();
let get = |path: &str| {
let request = client.get(format!("{base}{path}"));
async move { Ok::<Value, eyre::Report>(request.send().await?.error_for_status()?.json().await?) }
};
let routes = get("/api/routes/received").await?["routes"].take();
let validity: Vec<(&str, &str)> = routes
.as_array()
.unwrap()
.iter()
.map(|r| (r["prefix"].as_str().unwrap(), r["aspa"].as_str().unwrap()))
.collect();
assert_eq!(
validity,
[("193.0.0.0/21", "valid"), ("194.0.0.0/24", "unknown")]
);
let refused = get("/api/routes/rejected").await?;
assert_eq!(
refused[0]["by_reason"],
serde_json::json!({"ASPA invalid": 1})
);
let peers = get("/api/peers").await?;
assert_eq!(peers[0]["role"], "peer");
assert_eq!(peers[0]["held_routes"], 1);
let rpki = get("/api/rpki").await?;
assert_eq!(rpki["aspas"], 3);
assert_eq!(rpki["protocol_version"], 2);
Ok(())
}
/// What a path may look like depends on who it comes from. The same route
/// that a peer may not pass on is in order from a provider, which may hand
/// us anything it has. Where it is not said what the peer is, paths are
/// not verified.
#[tokio::test]
async fn test_a_route_that_has_crossed_a_peering_is_taken_from_a_provider_only() -> Result<()> {
for (role, taken) in [
(Some(PeerRole::Provider), true),
(Some(PeerRole::Customer), false),
(Some(PeerRole::Peer), false),
(Some(PeerRole::RouteServerClient), false),
(None, true),
] {
let cache = TestCache::start_with(vec![], aspas()).await;
let ServerWithPeer { server, .. } = server_with(&cache, role, default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
let expected = if taken {
server.table_becomes(&["8.8.8.0/24"], 0).await
} else {
server.table_becomes(&[], 1).await
};
expected.map_err(|e| eyre::eyre!("from a peer that is {role:?}: {e}"))?;
}
Ok(())
}
/// An Internet exchange's route server does not put its AS on the path, so
/// the path need not begin with the route server's AS. A leak is refused
/// across it all the same.
#[tokio::test]
async fn test_paths_from_a_route_server_do_not_begin_with_its_as() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let import = ImportPolicy {
accept_any_first_as: true,
..default_policy()
};
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::RouteServer), import).await?;
let mut peer = established(&server).await?;
// 3356's customer route, as 3356 gave it to the route server.
announce(&mut peer, &["8.8.8.0/24"], &[3356, 15169]).await?;
// The same route leaked to the route server by another member.
announce(&mut peer, &["8.8.4.0/24"], &[2121, 3356, 15169]).await?;
server.table_becomes(&["8.8.8.0/24"], 1).await
}
/// Validity changes with the ASPAs, and the routes follow: one that was
/// taken goes when its path becomes invalid, and one that was refused is
/// taken when it no longer is. The peer sends nothing more meanwhile.
#[tokio::test]
async fn test_routes_follow_the_aspas() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121]).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
server.table_becomes(&["194.0.0.0/24"], 1).await?;
// 2121 says that its only provider is 1101: its route by way of 3333
// is now invalid, and goes.
let mut changed = aspas();
changed.push((2121, vec![1101]));
cache.set_aspas(changed);
server.table_becomes(&[], 2).await?;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 2);
assert_eq!(refused.recent()[0].prefix, prefix("194.0.0.0/24"));
assert_eq!(&refused.recent()[0].as_path[..], [PEER_ASN, 2121]);
// 3356 becomes a customer of 3333, and 2121 too: both routes come
// back, with the paths they were announced with.
cache.set_aspas(vec![
(15169, vec![3356]),
(3356, vec![PEER_ASN]),
(PEER_ASN, vec![0]),
(2121, vec![1101, PEER_ASN]),
]);
server
.table_becomes(&["194.0.0.0/24", "8.8.8.0/24"], 0)
.await?;
let paths: BTreeSet<Vec<u32>> = server
.received_routes()
.iter()
.flat_map(|route| &route.attributes)
.filter_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.collect();
assert_eq!(
paths,
[vec![PEER_ASN, 2121], vec![PEER_ASN, 3356, 15169]].into()
);
// The validator loses its ASPAs altogether: nothing is known about
// any path, and nothing is refused.
cache.set_aspas(vec![
(15169, vec![3356]),
(3356, vec![0]),
(PEER_ASN, vec![0]),
]);
server.table_becomes(&["194.0.0.0/24"], 1).await?;
cache.set_aspas(vec![]);
server
.table_becomes(&["194.0.0.0/24", "8.8.8.0/24"], 0)
.await
}
/// A peer can be exempt: its routes with invalid paths are taken, and
/// shown for what they are.
#[tokio::test]
async fn test_a_peer_can_be_exempt_from_path_verification() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let import = ImportPolicy {
accept_aspa_invalid: true,
..default_policy()
};
let ServerWithPeer { server, .. } = server_with(&cache, Some(PeerRole::Peer), import).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
server.table_becomes(&["8.8.8.0/24"], 0).await?;
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
assert_eq!(routes["routes"][0]["aspa"], "invalid");
assert!(server.rejections.peers().is_empty());
Ok(())
}
/// A route can fail both checks. It is taken only when it passes both.
#[tokio::test]
async fn test_a_route_is_held_until_origin_and_path_are_both_in_order() -> Result<()> {
let google = |asn| bgp_rpki::Vrp {
prefix: prefix("8.8.8.0/24"),
max_length: 24,
asn,
};
let cache = TestCache::start_with(vec![google(13335)], aspas()).await;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
server.table_becomes(&[], 1).await?;
// The origin is put right; the path is still a leak.
cache.set(vec![google(15169)]);
tokio::time::sleep(Duration::from_millis(500)).await;
server.table_becomes(&[], 1).await?;
cache.set_aspas(vec![(15169, vec![3356]), (3356, vec![PEER_ASN])]);
server.table_becomes(&["8.8.8.0/24"], 0).await
}
crates/integration-tests/tests/basic_connection.rs
+210 -67
View File
@@ -2,17 +2,40 @@ use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::RouteChange;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_integration_tests::{TIMEOUT, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode, FsmSubcode, MsgHeaderSubcode, UpdateMsgSubcode};
use bgp_packet::message::{
Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute, NlriNextHop,
NotificationMessage, PathAttribute, UpdateMessage,
AsPathAttribute, Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute,
NextHopPathAttribute, NlriNextHop, NotificationMessage, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::constants::{AddressFamilyId, SubsequentAfi};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::RibEvent;
use eyre::Result;
/// The attributes every announcement of IPv4 routes must carry. An UPDATE
/// without them has its routes withdrawn instead (RFC 7606).
fn mandatory_attributes() -> Vec<PathAttribute> {
let mut attributes = with_origin_and_as_path(PathAttribute::NextHop(NextHopPathAttribute(
Ipv4Addr::new(192, 0, 2, 1),
)));
// In order of type code: ORIGIN, AS_PATH, NEXT_HOP.
attributes.rotate_left(1);
attributes
}
/// `first` followed by the attributes every announcement must carry.
/// Routes in MP_REACH_NLRI have their next hop there.
fn with_origin_and_as_path(first: PathAttribute) -> Vec<PathAttribute> {
vec![
first,
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65001]),
]
}
// ---------------------------------------------------------------------------
// Connectivity
// ---------------------------------------------------------------------------
@@ -74,6 +97,130 @@ async fn test_bad_marker_does_not_crash_server() -> Result<()> {
Ok(())
}
/// A header with the given length and type, followed by `body`.
fn framed(length: u16, msg_type: u8, body: &[u8]) -> Vec<u8> {
let mut buf = vec![0xFF; 16];
buf.extend_from_slice(&length.to_be_bytes());
buf.push(msg_type);
buf.extend_from_slice(body);
buf
}
/// What the peer sent decides the NOTIFICATION it gets back (RFC 4271 §6),
/// in every state, and the session ends.
#[tokio::test]
async fn test_malformed_messages_are_answered_with_the_matching_notification() -> Result<()> {
let server = TestServer::start(65000).await?;
let header = |subcode| BgpError::MsgHeader(subcode);
let mut bad_marker = framed(19, 4, &[]);
bad_marker[0] = 0;
let cases: Vec<(&str, Vec<u8>, BgpError, Vec<u8>)> = vec![
(
"length beyond the maximum",
framed(5000, 2, &[]),
header(MsgHeaderSubcode::BadMessageLength),
5000u16.to_be_bytes().to_vec(),
),
(
"length below a header",
framed(5, 4, &[]),
header(MsgHeaderSubcode::BadMessageLength),
vec![0, 5],
),
(
"unknown message type",
framed(19, 9, &[]),
header(MsgHeaderSubcode::BadMessageType),
vec![9],
),
(
"marker not all ones",
bad_marker,
header(MsgHeaderSubcode::ConnNotSynchronized),
vec![],
),
(
"UPDATE with attributes running past its end",
framed(23, 2, &[0, 0, 0, 40]),
BgpError::UpdateMsg(UpdateMsgSubcode::MalformedAttrs),
vec![],
),
];
for (i, (what, bytes, error, data)) in cases.into_iter().enumerate() {
// Established: a distinct router ID each time, so that a session
// still closing does not collide with the next one.
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 1, i as u8 + 1), 90)
.await?;
peer.send_raw(&bytes).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
(notification.error, notification.data),
(error.clone(), data.clone()),
"{what}, Established"
);
// OpenSent: the same bytes instead of an OPEN.
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send_raw(&bytes).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
(notification.error, notification.data),
(error, data),
"{what}, OpenSent"
);
}
// A well-formed message at the wrong time is an FSM error, not a
// message error.
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send(&Message::KeepAlive).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::FsmError(FsmSubcode::UnexpectedOpenSent)
);
Ok(())
}
/// A message may arrive in pieces, with the server's own timers firing in
/// between. The server once threw away the piece it had already read when
/// that happened, so the message was silently lost (and the stream was only
/// back in step if the remainder happened to be thrown away as well).
#[tokio::test]
async fn test_message_split_across_a_keepalive_tick_is_still_one_message() -> Result<()> {
// Hold time 3 s: the server sends a KEEPALIVE every second.
let server = TestServer::start_with_hold_time(65000, 3).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 3).await?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
let mut update = bytes::BytesMut::new();
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
})
.to_wire(&peer.ctx, &mut update)?;
// The first half, a pause long enough for the server's keepalive timer
// to fire, then the rest.
peer.send_raw(&update[..10]).await?;
tokio::time::sleep(Duration::from_millis(1300)).await;
peer.send_raw(&update[10..]).await?;
wait_until("the split UPDATE reaches the RIB", || {
server.received_routes().iter().any(|e| e.prefix == prefix)
})
.await?;
assert_eq!(server.registry.list().len(), 1, "session was dropped");
Ok(())
}
// ---------------------------------------------------------------------------
// OpenConfirm → Established
// ---------------------------------------------------------------------------
@@ -101,7 +248,7 @@ async fn test_server_sends_keepalives() -> Result<()> {
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 3).await?;
// We should receive a KEEPALIVE within ~1 second (hold/3).
let msg = tokio::time::timeout(Duration::from_secs(3), peer.recv()).await??;
let msg = peer.recv_within(Duration::from_secs(3)).await?;
assert!(
matches!(msg, Message::KeepAlive),
"Expected KEEPALIVE, got {:?}",
@@ -121,16 +268,7 @@ async fn test_hold_timer_expires() -> Result<()> {
// Don't send any keepalives. The server will send keepalives until its own
// hold timer fires and it sends a NOTIFICATION. Drain messages until we
// see it (or the outer timeout fires).
let notif = tokio::time::timeout(Duration::from_secs(6), async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {} // server keepalive — keep waiting
other => eyre::bail!("Unexpected message: {:?}", other),
}
}
})
.await??;
let notif = tokio::time::timeout(Duration::from_secs(6), peer.expect_notification()).await??;
assert!(
matches!(notif.error, BgpError::HoldTimer),
"Expected HoldTimer NOTIFICATION, got {:?}",
@@ -154,8 +292,7 @@ async fn test_peer_sends_notification() -> Result<()> {
.await?;
// Server should close the connection cleanly.
tokio::time::sleep(Duration::from_millis(100)).await;
assert!(peer.is_closed().await);
assert!(peer.closed_within(TIMEOUT).await);
Ok(())
}
@@ -173,9 +310,9 @@ async fn test_server_accepts_update() -> Result<()> {
let update = Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
path_attributes: mandatory_attributes(),
// 10.0.0.0/8 — one byte of prefix for a /8.
announced_routes: vec![IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?],
announced_routes: vec![IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?.into()],
});
peer.send(&update).await?;
@@ -193,15 +330,16 @@ async fn test_server_accepts_ipv6_update() -> Result<()> {
.await?;
// 2001:db8::/32
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let update = Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
});
peer.send(&update).await?;
@@ -219,26 +357,27 @@ async fn test_server_accepts_ipv6_update() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv4_announce() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
announced_routes: vec![prefix.clone()],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
}))
.await?;
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Announced { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Announced { prefix: p, .. } if p == &prefix),
"Expected Announced for 10.0.0.0/8, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
snapshot.iter().any(|e| e.prefix == prefix),
"10.0.0.0/8 missing from RIB snapshot",
@@ -250,25 +389,26 @@ async fn test_rib_records_ipv4_announce() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv4_withdraw() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
// Announce first.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
announced_routes: vec![prefix.clone()],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
}))
.await?;
tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
// Now withdraw.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![prefix.clone()],
withdrawn_routes: vec![prefix.into()],
path_attributes: vec![],
announced_routes: vec![],
}))
@@ -276,11 +416,11 @@ async fn test_rib_records_ipv4_withdraw() -> Result<()> {
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Withdrawn { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Withdrawn { prefix: p, .. } if p == &prefix),
"Expected Withdrawn for 10.0.0.0/8, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
!snapshot.iter().any(|e| e.prefix == prefix),
"10.0.0.0/8 should have been removed from RIB snapshot",
@@ -292,31 +432,33 @@ async fn test_rib_records_ipv4_withdraw() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv6_announce() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.clone()],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
}))
.await?;
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Announced { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Announced { prefix: p, .. } if p == &prefix),
"Expected Announced for 2001:db8::/32, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
snapshot.iter().any(|e| e.prefix == prefix),
"2001:db8::/32 missing from RIB snapshot",
@@ -328,22 +470,24 @@ async fn test_rib_records_ipv6_announce() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv6_withdraw() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
// Announce first.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.clone()],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
}))
.await?;
@@ -353,9 +497,8 @@ async fn test_rib_records_ipv6_withdraw() -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpUnreachNlri(MpUnreachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
prefixes: vec![prefix.clone()],
family: AfiSafi::IPV6_UNICAST,
prefixes: vec![prefix.into()],
})],
announced_routes: vec![],
}))
@@ -363,11 +506,11 @@ async fn test_rib_records_ipv6_withdraw() -> Result<()> {
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Withdrawn { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Withdrawn { prefix: p, .. } if p == &prefix),
"Expected Withdrawn for 2001:db8::/32, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
!snapshot.iter().any(|e| e.prefix == prefix),
"2001:db8::/32 should have been removed from RIB snapshot",
crates/integration-tests/tests/best_route.rs
+210
View File
@@ -0,0 +1,210 @@
//! End-to-end tests of route selection: routes for one prefix from several
//! peers, which of them the server chooses, and how that is shown.
use std::net::Ipv4Addr;
use bgp_integration_tests::announcement::{Announcement, withdrawal as withdrawal_of};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_integration_tests::{TIMEOUT, get_json, prefix};
use bgp_packet::message::{PathAttribute, UpdateMessage};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
/// A session from the peer numbered `number`, in AS 65000 + `number`.
async fn peer(server: &TestServer, number: u8) -> Result<TestPeer> {
server
.established(
LOCAL_ASN + u32::from(number),
Ipv4Addr::new(10, 0, 0, number),
)
.await
}
fn announcement(prefixes: &[IpPrefix], as_path: &[u32]) -> Announcement {
Announcement::of(Nlri::plain(prefixes.iter().copied()), as_path)
}
fn withdrawal(prefixes: &[IpPrefix]) -> UpdateMessage {
withdrawal_of(Nlri::plain(prefixes.iter().copied()))
}
/// The AS path of the route the server holds as best for `prefix`.
fn best_path(server: &TestServer, prefix: &IpPrefix) -> Option<Vec<u32>> {
let best = server
.received_routes()
.into_iter()
.find(|route| route.prefix == *prefix && route.best)?;
best.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
}
/// Two peers offer the same prefix. The route with the shorter AS path is
/// chosen; when it goes, the other is.
#[tokio::test]
async fn test_the_best_route_is_chosen_and_replaced_when_it_goes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut changes = server.route_changes();
let p = prefix("203.0.113.0/24");
let mut far = peer(&server, 1).await?;
far.announce(announcement(&[p], &[65001, 64999, 64998]))
.await?;
server.wait_for_routes(1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
let mut near = peer(&server, 2).await?;
near.announce(announcement(&[p], &[65002, 64998])).await?;
server.wait_for_routes(2).await?;
assert_eq!(best_path(&server, &p), Some(vec![65002, 64998]));
// Both routes are kept and shown; one is marked.
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["received_routes"], 2);
assert_eq!(status["received_prefixes"], 1);
let routes = get_json(&client, &format!("{base}/api/routes/received")).await?["routes"].take();
let best: Vec<&Value> = routes
.as_array()
.unwrap()
.iter()
.filter(|route| route["best"] == true)
.collect();
assert_eq!(routes.as_array().unwrap().len(), 2);
assert_eq!(best.len(), 1);
assert_eq!(best[0]["peer_asn"], 65002);
// Listed from the best down, each with its place in selection.
let ranks: Vec<(u64, u64)> = routes
.as_array()
.unwrap()
.iter()
.map(|route| {
(
route["rank"].as_u64().unwrap(),
route["peer_asn"].as_u64().unwrap(),
)
})
.collect();
assert_eq!(ranks, vec![(1, 65002), (2, 65001)]);
// The session with the best route ends: the other route takes over.
drop(near);
server.wait_for_routes(1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
far.announce(withdrawal(&[p])).await?;
server.wait_for_routes(0).await?;
assert_eq!(best_path(&server, &p), None);
// A follower of the best routes is told of each change: the first
// route, the better one, back to the first, none.
let mut seen = Vec::new();
while seen.last() != Some(&RouteChange::Withdrawn { prefix: p }) {
seen.push(tokio::time::timeout(TIMEOUT, changes.recv()).await??);
}
assert!(
seen.iter()
.all(|change| matches!(change, RouteChange::Announced { prefix, .. } | RouteChange::Withdrawn { prefix } if *prefix == p)),
"{seen:?}"
);
Ok(())
}
/// The event stream reports changes of best route, and only those.
#[tokio::test]
async fn test_changes_of_best_route_are_on_the_event_stream() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let p = prefix("203.0.113.0/24");
// A route that is there before the client connects is not an event.
let mut first = peer(&server, 1).await?;
first
.announce(announcement(&[prefix("198.51.100.0/24")], &[65001]))
.await?;
server.wait_for_routes(1).await?;
let mut stream = reqwest::Client::new()
.get(format!("{base}/api/events"))
.send()
.await?
.error_for_status()?;
first.announce(announcement(&[p], &[65001, 64998])).await?;
server.wait_for_routes(2).await?;
first.announce(withdrawal(&[p])).await?;
server.wait_for_routes(1).await?;
let mut text = String::new();
let mut events: Vec<Value> = Vec::new();
tokio::time::timeout(TIMEOUT, async {
while !events.iter().any(|event| event["type"] == "best_withdrawn") {
let chunk = stream
.chunk()
.await?
.ok_or_else(|| eyre::eyre!("stream ended"))?;
text.push_str(std::str::from_utf8(&chunk)?);
events = text
.lines()
.filter_map(|line| line.strip_prefix("data:"))
.filter_map(|data| serde_json::from_str(data.trim()).ok())
.collect();
}
Ok::<_, eyre::Error>(())
})
.await??;
let about_routes: Vec<&Value> = events
.iter()
.filter(|event| {
event["type"]
.as_str()
.is_some_and(|t| t.starts_with("best_"))
})
.collect();
// The announcement and the withdrawal may have been read as one change
// or as two; either way nothing is said of the route that was there
// from the start, and the last word is that the prefix is gone.
assert!(about_routes.len() <= 2, "{about_routes:?}");
for event in &about_routes {
let named = event["prefix"]
.as_str()
.or(event["route"]["prefix"].as_str());
assert_eq!(named, Some("203.0.113.0/24"), "{event}");
}
assert_eq!(about_routes.last().unwrap()["type"], "best_withdrawn");
Ok(())
}
/// The table can be far too large to send whole: the API gives a part of
/// it, and says how much there is.
#[tokio::test]
async fn test_received_routes_are_returned_up_to_a_limit() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let prefixes: Vec<IpPrefix> = (0..40).map(|n| prefix(&format!("10.{n}.0.0/16"))).collect();
let mut peer = peer(&server, 1).await?;
peer.announce(announcement(&prefixes, &[65001])).await?;
server.wait_for_routes(40).await?;
let count = |page: Value| page["routes"].as_array().unwrap().len();
let url = format!("{base}/api/routes/received");
assert_eq!(count(get_json(&client, &url).await?), 40);
assert_eq!(
count(get_json(&client, &format!("{url}?limit=7")).await?),
7
);
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["received_routes"], 40);
Ok(())
}
crates/integration-tests/tests/blackhole.rs
+141
View File
@@ -0,0 +1,141 @@
//! Blackhole requests (RFC 7999): refused by default, and with
//! `accept_blackholes` taken for one address inside what the peer may
//! announce, whatever their next hop.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_server::policy::{ImportPolicy, PrefixSet};
use eyre::Result;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
const BLACKHOLE: (u16, u16) = (0xFFFF, 0x029A);
async fn server_with(import: ImportPolicy) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(50);
peer.import = import;
})
.await
}
/// Waits until the server has taken `taken` routes of the session's and
/// refused `refused`, and returns the prefixes in its table.
async fn table_after(server: &TestServer, taken: u64, refused: u64) -> Result<BTreeSet<String>> {
let counters = server.registry.list()[0].counters.clone();
wait_until("the routes are dealt with", || {
(counters.routes(), counters.rejected()) == (taken, refused)
})
.await
.map_err(|_| {
eyre::eyre!(
"the server took {} routes and refused {}; expected {taken} and {refused}",
counters.routes(),
counters.rejected()
)
})?;
Ok(server
.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect())
}
/// The policy that lets the peer announce 193.0.0.0/21 and nothing
/// longer than a /24 inside it, as a customer's would.
fn customer_policy(accept_blackholes: bool) -> ImportPolicy {
let mut prefixes = PrefixSet::default();
prefixes.allow(prefix("193.0.0.0/21"), 24);
ImportPolicy {
prefixes: Some(prefixes),
accept_blackholes,
..default_policy()
}
}
#[tokio::test]
async fn test_blackhole_requests_are_refused_unless_accepted() -> Result<()> {
let with_peer = server_with(customer_policy(false)).await?;
let server = &with_peer.server;
let mut peer: TestPeer = server.established(PEER_ASN, PEER_ID).await?;
peer.announce(announcement(&["193.0.0.9/32"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
peer.announce(announcement(&["193.0.0.0/24"], &[PEER_ASN]))
.await?;
assert_eq!(table_after(server, 1, 1).await?, set(&["193.0.0.0/24"]));
let refused = server.rejections.peers();
assert_eq!(refused.len(), 1);
assert_eq!(
refused[0]
.1
.by_reason()
.iter()
.map(|(reason, n)| (reason.as_str(), *n))
.collect::<Vec<_>>(),
[("blackhole request", 1)]
);
Ok(())
}
#[tokio::test]
async fn test_a_blackhole_request_is_taken_for_one_address_in_the_peers_space() -> Result<()> {
let with_peer = server_with(customer_policy(true)).await?;
let server = &with_peer.server;
let mut peer: TestPeer = server.established(PEER_ASN, PEER_ID).await?;
// One address inside the peer's prefix, from a next hop that is no
// neighbor and would not pass as an ordinary route's.
peer.announce(
announcement(&["193.0.0.9/32"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(172, 31, 0, 9))
.with_communities(&[BLACKHOLE]),
)
.await?;
// More than one address: not a blackhole request that is taken.
peer.announce(announcement(&["193.0.0.0/25"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
// One address outside the peer's space.
peer.announce(announcement(&["193.0.8.9/32"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
// And an ordinary route, taken as ever.
peer.announce(announcement(&["193.0.0.0/24"], &[PEER_ASN]))
.await?;
assert_eq!(
table_after(server, 2, 2).await?,
set(&["193.0.0.0/24", "193.0.0.9/32"])
);
let refused = server.rejections.peers();
let mut reasons: Vec<(&str, u64)> = refused[0]
.1
.by_reason()
.iter()
.map(|(reason, n)| (reason.as_str(), *n))
.collect();
reasons.sort();
assert_eq!(
reasons,
[
("blackhole for more than one address", 1),
("prefix not allowed", 1)
]
);
// The route is in the table with its request on it, for the kernel
// and for whoever it goes on to.
let routes = server.received_routes();
let blackhole = routes
.iter()
.find(|route| route.prefix == prefix("193.0.0.9/32"))
.expect("the blackhole route is in the table");
assert!(ImportPolicy::is_blackhole_request(&blackhole.attributes));
Ok(())
}
fn set(prefixes: &[&str]) -> BTreeSet<String> {
prefixes.iter().map(|p| p.to_string()).collect()
}
crates/integration-tests/tests/collision.rs
+257
View File
@@ -0,0 +1,257 @@
//! Connection collision detection (RFC 4271 §6.8) against a live server.
//!
//! The test server's BGP Identifier is 127.0.0.1. A test peer picks an
//! identifier above or below that to steer which connection must survive.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode, OpenMsgSubcode};
use bgp_packet::message::{Message, NotificationMessage};
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::collision::Direction;
use bgp_server::policy::ImportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
/// Above the server's 127.0.0.1.
const HIGH_ID: Ipv4Addr = Ipv4Addr::new(192, 0, 2, 1);
/// Below the server's 127.0.0.1.
const LOW_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
/// A server that dials its peer "r1" every 200 ms until it has a session.
async fn server_dialing() -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(200);
peer.import = ImportPolicy::permissive();
})
.await
}
/// Exchanges OPENs only, leaving the server's side in OpenConfirm (it has
/// sent its KEEPALIVE and is waiting for ours).
async fn open_only(peer: &mut TestPeer, router_id: Ipv4Addr) -> Result<()> {
peer.send_open(PEER_ASN, router_id, 90).await?;
peer.recv_open().await?;
Ok(())
}
/// Finishes the handshake begun by [`open_only`].
async fn confirm(peer: &mut TestPeer) -> Result<()> {
match peer.recv().await? {
Message::KeepAlive => {}
other => eyre::bail!("expected KEEPALIVE, got {other:?}"),
}
peer.send(&Message::KeepAlive).await
}
fn is_collision_cease(n: &NotificationMessage) -> bool {
matches!(
n.error,
BgpError::Cease(CeaseSubcode::ConnCollisionResolution)
)
}
/// Peer has the higher identifier, so the connection the *peer* initiated
/// must survive: the server closes its own outbound connection.
#[tokio::test]
async fn test_simultaneous_open_peer_higher_keeps_inbound() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing().await?;
// Server's outbound connection reaches OpenConfirm.
let mut dialled = TestPeer::accept(&listener).await?;
open_only(&mut dialled, HIGH_ID).await?;
// The peer now also dials the server.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, HIGH_ID).await?;
let notif = dialled.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(dialled.closed_within(TIMEOUT).await);
confirm(&mut inbound).await?;
server.wait_for_sessions(1).await?;
let session = server.registry.list().remove(0);
assert_eq!(session.direction, Direction::Inbound);
assert_eq!(session.name.as_deref(), Some("r1"));
// The survivor is fully functional.
server
.local_rib
.announce(LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = inbound.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
// The dialer notices the surviving session and does not open another.
let extra = tokio::time::timeout(Duration::from_millis(700), listener.accept()).await;
assert!(
extra.is_err(),
"dialer opened a second connection despite an established session"
);
Ok(())
}
/// Peer has the lower identifier, so the connection the *server* initiated
/// must survive: the peer's inbound connection is refused.
#[tokio::test]
async fn test_simultaneous_open_peer_lower_keeps_outbound() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing().await?;
let mut dialled = TestPeer::accept(&listener).await?;
open_only(&mut dialled, LOW_ID).await?;
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
let notif = inbound.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(inbound.closed_within(TIMEOUT).await);
confirm(&mut dialled).await?;
server.wait_for_sessions(1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
/// The order in which the two OPENs are processed must not change the
/// outcome: here the inbound connection reaches OpenConfirm first.
#[tokio::test]
async fn test_outcome_is_independent_of_arrival_order() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing().await?;
let mut dialled = TestPeer::accept(&listener).await?;
// Inbound first this time; peer is lower so outbound must still win.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
open_only(&mut dialled, LOW_ID).await?;
let notif = inbound.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
confirm(&mut dialled).await?;
server.wait_for_sessions(1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
/// RFC default: a collision with an Established session closes the new
/// connection, whatever the identifiers say.
#[tokio::test]
async fn test_established_session_wins_by_default() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
server.wait_for_sessions(1).await?;
let mut second = TestPeer::connect(server.addr).await?;
open_only(&mut second, HIGH_ID).await?;
let notif = second.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
// The original is untouched.
assert_eq!(server.registry.list().len(), 1);
server
.local_rib
.announce(LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = first.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
/// With collision_detect_established_state, a peer that restarts and dials
/// again replaces the stale session at once, and the routes learned over the
/// old session are flushed.
#[tokio::test]
async fn test_restarted_peer_replaces_stale_session_when_enabled() -> Result<()> {
let server =
TestServer::start_with(LOCAL_ASN, |c| c.collision_detect_established_state = true).await?;
let mut stale = TestPeer::connect(server.addr).await?;
stale.handshake(PEER_ASN, HIGH_ID, 90).await?;
server.wait_for_sessions(1).await?;
stale
.announce(
announcement(&["203.0.113.0/24"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 9)),
)
.await?;
server.wait_for_routes(1).await?;
let stale_addr = server.registry.list()[0].peer.addr;
// "Restart": a fresh connection from the same router.
let mut fresh = TestPeer::connect(server.addr).await?;
fresh.handshake(PEER_ASN, HIGH_ID, 90).await?;
let notif = stale.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
wait_until("the fresh session has replaced the stale one", || {
let sessions = server.registry.list();
sessions.len() == 1 && sessions[0].peer.addr != stale_addr
})
.await?;
assert!(
server.received_routes().is_empty(),
"stale routes were not flushed"
);
Ok(())
}
/// Different routers (different identifiers) never collide.
#[tokio::test]
async fn test_distinct_identifiers_do_not_collide() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = TestPeer::connect(server.addr).await?;
a.handshake(PEER_ASN, HIGH_ID, 90).await?;
let mut b = TestPeer::connect(server.addr).await?;
b.handshake(PEER_ASN, LOW_ID, 90).await?;
server.wait_for_sessions(2).await?;
Ok(())
}
/// A failed connection frees its slot: the same router can come back.
#[tokio::test]
async fn test_identifier_is_released_when_connection_ends() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
server.wait_for_sessions(1).await?;
drop(first);
server.wait_for_sessions(0).await?;
let mut second = TestPeer::connect(server.addr).await?;
second.handshake(PEER_ASN, HIGH_ID, 90).await?;
server.wait_for_sessions(1).await?;
Ok(())
}
/// A peer claiming our own identifier is refused outright.
#[tokio::test]
async fn test_peer_with_our_identifier_is_refused() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
open_only(&mut peer, Ipv4Addr::new(127, 0, 0, 1)).await?;
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadBgpId)),
"{notif:?}"
);
Ok(())
}
crates/integration-tests/tests/deadlock.rs
+92
View File
@@ -0,0 +1,92 @@
//! A peer that sends us a table while not reading what we send it must
//! not stop us reading it: two routers sending each other their tables at
//! once would otherwise each wait on the other for ever.
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::Announcement;
use bgp_integration_tests::prefix;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_server::policy::ExportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 64601;
const ROUTES: u32 = 100_000;
fn nth(n: u32) -> bgp_primitives::ip_prefix::IpPrefix {
let [a, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("{a}.{b}.{c}.0/24"))
}
#[tokio::test]
async fn a_peer_that_does_not_read_is_still_read() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
});
})
.await?;
// One peer's table, for the server to pass on.
let mut first = server
.established(64701, Ipv4Addr::new(10, 0, 0, 1))
.await?;
// A route to an UPDATE, each with attributes of its own, as a real
// table has them: passing it on is then megabytes, not a few hundred
// UPDATEs that the socket buffers would take whole.
for i in 0..ROUTES {
first
.announce(
Announcement::of([nth(i).into()], &[64701, 64510])
.with_communities(&[((i >> 16) as u16 + 1, i as u16)]),
)
.await?;
}
server.wait_for_routes(ROUTES as usize).await?;
// A second peer, which the server starts sending that table to as
// soon as it is up, sends a table of its own and reads nothing back:
// the server's writes to it fill its socket and stop. The server has
// to go on reading it all the same. The peer's receive buffer is
// small, as a real link's window would be, so that this happens with
// the first few hundred kilobytes and not after the loopback's
// megabytes.
let socket = tokio::net::TcpSocket::new_v4()?;
socket.set_recv_buffer_size(16 * 1024)?;
let stream = socket.connect(server.addr).await?;
let mut second = TestPeer::from_stream(stream);
second
.handshake(64702, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
server.wait_for_sessions(2).await?;
// Sent over a couple of seconds, so that the server's export to the
// peer is well under way while the peer is still sending.
let sending = async {
for start in (0..ROUTES).step_by(100) {
second
.announce(Announcement::of(
(start..start + 100).map(|i| nth(i).into()),
&[64702, 64510],
))
.await?;
if start % 2000 == 0 {
tokio::time::sleep(Duration::from_millis(40)).await;
}
}
Ok::<_, eyre::Report>(())
};
tokio::time::timeout(Duration::from_secs(30), sending)
.await
.expect("the server reads the peer's table though the peer reads nothing")?;
tokio::time::timeout(
Duration::from_secs(30),
server.wait_for_routes(2 * ROUTES as usize),
)
.await
.expect("every route is in the table")?;
Ok(())
}
crates/integration-tests/tests/debugging.rs
+261
View File
@@ -0,0 +1,261 @@
//! Finding out what the server has and why: looking routes up, and seeing
//! what was refused.
use std::net::Ipv4Addr;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, wait_until};
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use serde_json::Value;
/// The routes a query returns, as "prefix from AS, best or not".
async fn routes(client: &reqwest::Client, base: &str, query: &str) -> Result<Vec<String>> {
let reply = client
.get(format!("{base}/api/routes/received?{query}"))
.send()
.await?
.error_for_status()?;
let page: Value = reply.json().await?;
Ok(page["routes"]
.as_array()
.unwrap()
.iter()
.map(|r| {
format!(
"{} AS{}{}",
r["prefix"].as_str().unwrap(),
r["peer_asn"],
if r["best"] == true { " best" } else { "" }
)
})
.collect())
}
/// The questions asked when something is wrong with a prefix: who sends it
/// and which route won, what covers an address, what is inside a block.
#[tokio::test]
async fn test_routes_can_be_looked_up() -> Result<()> {
let server = TestServer::start(65000).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut one = server
.established(65001, Ipv4Addr::new(10, 0, 0, 1))
.await?;
let mut two = server
.established(65002, Ipv4Addr::new(10, 0, 0, 2))
.await?;
one.announce(announcement(
&["10.0.0.0/8", "10.1.0.0/16", "10.1.2.0/24", "192.0.2.0/24"],
&[65001],
))
.await?;
// A longer path for one of the same prefixes: it loses.
two.announce(announcement(&["10.1.0.0/16"], &[65002, 65003]))
.await?;
server.wait_for_routes(5).await?;
// One prefix: every route for it, the best first.
assert_eq!(
routes(&client, &base, "prefix=10.1.0.0/16").await?,
["10.1.0.0/16 AS65001 best", "10.1.0.0/16 AS65002"]
);
assert!(
routes(&client, &base, "prefix=10.2.0.0/16")
.await?
.is_empty()
);
// An address: every prefix that covers it, the most specific first.
assert_eq!(
routes(&client, &base, "address=10.1.2.3").await?,
[
"10.1.2.0/24 AS65001 best",
"10.1.0.0/16 AS65001 best",
"10.1.0.0/16 AS65002",
"10.0.0.0/8 AS65001 best"
]
);
assert_eq!(
routes(&client, &base, "address=10.200.0.1").await?,
["10.0.0.0/8 AS65001 best"]
);
// A block: itself and everything inside it.
assert_eq!(
routes(&client, &base, "within=10.1.0.0/16").await?,
[
"10.1.0.0/16 AS65001 best",
"10.1.0.0/16 AS65002",
"10.1.2.0/24 AS65001 best"
]
);
// Only what was chosen; and everything, in address order.
assert_eq!(
routes(&client, &base, "within=10.0.0.0/8&best=true").await?,
[
"10.0.0.0/8 AS65001 best",
"10.1.0.0/16 AS65001 best",
"10.1.2.0/24 AS65001 best"
]
);
assert_eq!(routes(&client, &base, "").await?.len(), 5);
// A page holds whole prefixes: at most the limit, unless one prefix
// alone has more.
assert!((1..=2).contains(&routes(&client, &base, "limit=2").await?.len()));
// One peer's routes, by its address; a peer that is not there has none.
assert_eq!(routes(&client, &base, "peer=127.0.0.1").await?.len(), 5);
assert!(routes(&client, &base, "peer=192.0.2.9").await?.is_empty());
// A query that makes no sense says so.
for (query, complaint) in [
("prefix=10.1.0.0", "is not a prefix"),
("address=example", "is not an IP address"),
("prefix=10.0.0.0/8&address=10.0.0.1", "at most one of"),
] {
let reply = client
.get(format!("{base}/api/routes/received?{query}"))
.send()
.await?;
assert_eq!(reply.status(), 400, "{query}");
let text = reply.text().await?;
assert!(text.contains(complaint), "{query}: {text}");
}
Ok(())
}
/// A refused route is not in the table, so it has to be findable somewhere
/// else: which peer, which prefix, with what path, and why. And there is
/// something to alert on.
#[tokio::test]
async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
let ServerWithPeer {
server,
listener: _listener,
} = TestServer::with_peer(1299, 3333, |_, peer| {
peer.name = "refused-demo".into();
peer.import = ImportPolicy {
max_prefixes: Some(100),
..default_policy()
};
})
.await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let get = |path: &str| {
let request = client.get(format!("{base}{path}"));
async move { Ok::<_, eyre::Report>(request.send().await?.error_for_status()?) }
};
let refused: Value = get("/api/routes/rejected").await?.json().await?;
assert_eq!(refused, serde_json::json!([]), "nothing refused yet");
let mut peer = server.established(3333, Ipv4Addr::new(10, 0, 0, 2)).await?;
peer.announce(announcement(&["193.0.0.0/21", "10.0.0.0/8"], &[3333]))
.await?;
peer.announce(announcement(
&["194.0.1.0/24", "194.0.2.0/24"],
&[3333, 64512],
))
.await?;
peer.announce(announcement(&["193.0.9.0/25"], &[3333, 2121]))
.await?;
wait_until("four routes are refused", || {
server
.registry
.list()
.first()
.map(|s| s.counters.rejected())
== Some(4)
})
.await?;
let refused: Value = get("/api/routes/rejected").await?.json().await?;
let record = &refused[0];
assert_eq!(record["peer"], "refused-demo");
assert_eq!(record["peer_addr"], "127.0.0.1");
assert_eq!(record["total"], 4);
assert_eq!(
record["by_reason"],
serde_json::json!({"bogon AS in path": 2, "bogon prefix": 1, "prefix too specific": 1})
);
// The latest first, each with the path it came with.
let recent: Vec<(String, String, String)> = record["recent"]
.as_array()
.unwrap()
.iter()
.map(|r| {
(
r["prefix"].as_str().unwrap().to_string(),
r["reason"].as_str().unwrap().to_string(),
r["as_path"].to_string(),
)
})
.collect();
let row = |p: &str, why: &str, path: &str| (p.to_string(), why.to_string(), path.to_string());
assert_eq!(
recent,
[
row("193.0.9.0/25", "prefix too specific", "[3333,2121]"),
row("194.0.2.0/24", "bogon AS in path", "[3333,64512]"),
row("194.0.1.0/24", "bogon AS in path", "[3333,64512]"),
row("10.0.0.0/8", "bogon prefix", "[3333]"),
]
);
// And with all its attributes, as for a route that was taken, to see
// what came along with it.
let latest = &record["recent"][0];
assert_eq!(latest["summary"]["next_hop"], "192.0.2.1");
assert_eq!(latest["summary"]["origin"], "IGP");
assert_eq!(
latest["summary"]["as_path"],
serde_json::json!([3333, 2121])
);
assert_eq!(
latest["attributes"].as_array().map(Vec::len),
Some(3),
"ORIGIN, AS_PATH and NEXT_HOP: {latest}"
);
// Asked for by peer, by name or by address.
for peer in ["refused-demo", "127.0.0.1"] {
let one: Value = get(&format!("/api/routes/rejected?peer={peer}"))
.await?
.json()
.await?;
assert_eq!(one.as_array().unwrap().len(), 1, "{peer}");
}
let none: Value = get("/api/routes/rejected?peer=someone-else")
.await?
.json()
.await?;
assert_eq!(none, serde_json::json!([]));
// The peer's line in the list of peers has the numbers too.
let peers: Value = get("/api/peers").await?.json().await?;
assert_eq!(peers[0]["routes"], 1);
assert_eq!(peers[0]["rejected_routes"], 4);
// What an alert would be made from.
let metrics = get("/metricz").await?.text().await?;
for line in [
r#"bgp_routes_rejected_total{peer="refused-demo",reason="bogon AS in path"} 2"#,
r#"bgp_routes_rejected_total{peer="refused-demo",reason="bogon prefix"} 1"#,
r#"bgp_peer_routes{peer="refused-demo"} 1"#,
r#"bgp_peer_max_prefixes{peer="refused-demo"} 100"#,
] {
assert!(metrics.contains(line), "{line} is not in:\n{metrics}");
}
// The record outlives the session; the count of routes does not.
drop(peer);
server.wait_for_sessions(0).await?;
let refused: Value = get("/api/routes/rejected").await?.json().await?;
assert_eq!(refused[0]["total"], 4);
let metrics = get("/metricz").await?.text().await?;
assert!(!metrics.contains(r#"bgp_peer_routes{peer="refused-demo"}"#));
Ok(())
}
crates/integration-tests/tests/dialer.rs
+284
View File
@@ -0,0 +1,284 @@
//! The active side: the server dials its configured peers and keeps
//! retrying.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, peer_config, prefix, wait_for, wait_until};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::Message;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::policy::ImportPolicy;
use bgp_server::session::DialState;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server that dials its peer "r1", in `remote_asn`, every 200 ms.
async fn server_dialing(remote_asn: u32) -> Result<ServerWithPeer> {
server_with_peer(remote_asn, false).await
}
/// A server with peer "r1" in `remote_asn`, which it dials every 200 ms
/// unless the peer is `passive`.
async fn server_with_peer(remote_asn: u32, passive: bool) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, remote_asn, |c, peer| {
c.connect_retry = Duration::from_millis(200);
peer.import = ImportPolicy::permissive();
peer.passive = passive;
})
.await
}
#[tokio::test]
async fn test_server_dials_configured_peer_and_establishes() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut peer = TestPeer::accept(&listener).await?;
// Both sides send OPEN on connect; the test peer's handshake order works
// whichever side opened TCP.
let open = peer.handshake(PEER_ASN, PEER_ID, 90).await?;
assert_eq!(open.asn, LOCAL_ASN as u16);
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
assert_eq!(listed.peer.asn, PEER_ASN);
assert_eq!(listed.peer.addr, listener.local_addr()?);
// Configured view reflects it; the dialer is not in backoff.
let (cfg, state) = server
.registry
.configured_peers()
.into_iter()
.next()
.unwrap();
assert_eq!(cfg.name, "r1");
assert!(
matches!(state, DialState::Connecting | DialState::Idle),
"{state:?}"
);
// Local routes flow over the dialled session too.
server
.local_rib
.announce(LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
#[tokio::test]
async fn test_server_reconnects_after_peer_closes() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut first = TestPeer::accept(&listener).await?;
first.handshake(PEER_ASN, PEER_ID, 90).await?;
server.wait_for_session().await?;
drop(first);
// Session goes away, then the dialer comes back.
server.wait_for_sessions(0).await?;
let (_, state) = server
.registry
.configured_peers()
.into_iter()
.next()
.unwrap();
assert!(matches!(state, DialState::Backoff { .. }), "{state:?}");
let mut second = TestPeer::accept(&listener).await?;
second.handshake(PEER_ASN, PEER_ID, 90).await?;
server.wait_for_session().await?;
Ok(())
}
#[tokio::test]
async fn test_wrong_asn_is_refused_with_bad_peer_as() -> Result<()> {
let ServerWithPeer {
server: _server,
listener,
} = server_dialing(PEER_ASN).await?;
let mut peer = TestPeer::accept(&listener).await?;
peer.send_open(65099, PEER_ID, 90).await?;
let _ = peer.recv_open().await?;
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{notif:?}"
);
// The dialer retries rather than giving up.
let _again = TestPeer::accept(&listener).await?;
Ok(())
}
/// An inbound connection from a configured peer's address is also checked
/// against the configured ASN and gets the configured name.
#[tokio::test]
async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
// Configure a peer at the loopback address on a port nobody listens on,
// so the dialer never succeeds; then connect inbound from loopback.
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_secs(60);
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
import: ImportPolicy::permissive(),
..peer_config("r1", at, PEER_ASN)
}];
})
.await?;
// Wrong ASN inbound → refused.
let mut bad = TestPeer::connect(server.addr).await?;
bad.send_open(65099, PEER_ID, 90).await?;
let _ = bad.recv_open().await?;
let msg = bad.recv().await?;
assert!(
matches!(&msg, Message::Notification(n) if matches!(n.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs))),
"{msg:?}"
);
// Right ASN inbound → established, named.
let mut good = TestPeer::connect(server.addr).await?;
good.handshake(PEER_ASN, PEER_ID, 90).await?;
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
Ok(())
}
/// With `local_address` set the connection must come from that address (the
/// peer may identify us by it), and an address the host does not have is
/// reported as the reason for backing off rather than silently ignored.
#[tokio::test]
async fn test_local_address_is_used_as_the_source() -> Result<()> {
let ServerWithPeer {
server: _server,
listener,
} = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(200);
peer.import = ImportPolicy::permissive();
peer.local_address = Some(Ipv4Addr::LOCALHOST.into());
})
.await?;
let (_stream, from) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
assert_eq!(from.ip(), std::net::IpAddr::from(Ipv4Addr::LOCALHOST));
// 192.0.2.1 is documentation space; no host has it.
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
local_address: Some(Ipv4Addr::new(192, 0, 2, 1).into()),
import: ImportPolicy::permissive(),
..peer_config("r1", endpoint, PEER_ASN)
}];
})
.await?;
let reason = wait_for("the dialer is backing off", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { last_error, .. })) => Some(last_error),
_ => None,
}
})
.await?;
assert!(reason.contains("local_address 192.0.2.1"), "{reason}");
Ok(())
}
/// A peer that takes the connection and then refuses the session, here
/// because it is not the AS we are configured to expect, will refuse it
/// again. It is dialled ever more slowly, like a peer that cannot be
/// reached: connecting used to count as success and reset the wait, so
/// such a peer was dialled every few seconds for ever.
#[tokio::test]
async fn test_a_peer_that_refuses_the_session_is_dialled_ever_more_slowly() -> Result<()> {
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut waits = Vec::new();
for _ in 0..3 {
let mut peer = TestPeer::accept(&listener).await?;
peer.send_open(65099, PEER_ID, 90).await?;
let _ = peer.recv_open().await?;
let refusal = peer.recv().await?;
assert!(
matches!(&refusal, Message::Notification(n) if n.error == BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{refusal:?}"
);
drop(peer);
// How long the dialer says it will wait before trying again.
let wait = wait_for("the dialer is backing off", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { until, last_error })) => {
assert!(last_error.contains("AS65099"), "{last_error}");
Some(
until
.duration_since(std::time::SystemTime::now())
.unwrap_or_default(),
)
}
_ => None,
}
})
.await?;
waits.push(wait);
}
// 200 ms, then 400, then 800, less what passed before each was read.
assert!(
waits[2] > Duration::from_millis(500),
"the waits were {waits:?}"
);
Ok(())
}
/// A passive peer is not dialled: its session comes up when it connects,
/// and until then, and after, it is shown as waiting, with how its last
/// session ended.
#[tokio::test]
async fn test_a_passive_peer_is_not_dialled_and_comes_up_when_it_connects() -> Result<()> {
let ServerWithPeer { server, listener } = server_with_peer(PEER_ASN, true).await?;
let state = || server.registry.configured_peers()[0].1.clone();
// Many retry intervals pass, and nothing connects.
assert!(
tokio::time::timeout(Duration::from_secs(1), listener.accept())
.await
.is_err(),
"a passive peer was dialled"
);
assert_eq!(state(), DialState::Passive { last_error: None });
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, PEER_ID, 90).await?;
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
drop(peer);
wait_until("the peer is shown as waiting, with how it ended", || {
matches!(
state(),
DialState::Passive {
last_error: Some(_)
}
)
})
.await?;
assert!(server.registry.list().is_empty());
assert!(
tokio::time::timeout(Duration::from_millis(600), listener.accept())
.await
.is_err(),
"a passive peer was dialled after its session ended"
);
Ok(())
}
crates/integration-tests/tests/export_limits.rs
+147
View File
@@ -0,0 +1,147 @@
//! Export limits, end to end: what a peer above or beside us is told is
//! held to the prefixes they list and to as many routes as they allow,
//! whatever else would send it more.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
use bgp_packet::open::{BgpRole, Capability};
use bgp_primitives::constants::AfiSafi;
use bgp_rib::LocalRoute;
use bgp_server::policy::{ExportLimits, PeerRole, PrefixSet};
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server that originates `originated`, with one configured peer at the
/// address the test connects from, which is a peer of ours held to
/// `limits`.
async fn server_with(originated: &[&str], limits: ExportLimits) -> Result<ServerWithPeer> {
let originate = originated
.iter()
.map(|p| LocalRoute::new(prefix(p), Default::default()))
.collect();
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.originate = originate;
peer.export_limits = Some(Arc::new(limits));
peer.role = Some(PeerRole::Peer);
})
.await
}
/// A session with the server, as its peer.
async fn established(server: &TestServer) -> Result<TestPeer> {
let capabilities = vec![
Capability::FourByteAsn { asn: PEER_ASN },
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
Capability::BgpRole {
role: BgpRole::Peer,
},
];
server
.established_with(PEER_ASN, PEER_ID, capabilities)
.await
}
/// What the peer has been told once it has been quiet for a while: the
/// prefixes it holds, and the NOTIFICATION that ended the session, if one
/// did.
async fn told(peer: &mut TestPeer) -> (BTreeSet<String>, Option<(BgpError, Vec<u8>)>) {
let mut routes = BTreeSet::new();
while let Ok(message) = peer.recv_within(Duration::from_millis(500)).await {
match message {
Message::Update(update) => {
for route in &update.withdrawn_routes {
routes.remove(&route.prefix.to_string());
}
for route in &update.announced_routes {
routes.insert(route.prefix.to_string());
}
}
Message::Notification(notification) => {
return (routes, Some((notification.error, notification.data)));
}
_ => {}
}
}
(routes, None)
}
fn allowing(prefixes: &[&str]) -> Option<PrefixSet> {
let mut set = PrefixSet::default();
for p in prefixes {
let p = prefix(p);
set.allow(p, p.length());
}
Some(set)
}
/// A route of ours whose prefix is not in the peer's export limits is not
/// announced to it; the one that is, is.
#[tokio::test]
async fn test_a_route_outside_the_export_limits_is_not_announced() -> Result<()> {
let limits = ExportLimits {
prefixes: allowing(&["193.0.0.0/21"]),
max_prefixes: None,
};
let ServerWithPeer { server, .. } =
server_with(&["193.0.0.0/21", "198.51.100.0/24"], limits).await?;
let mut peer = established(&server).await?;
let (routes, ended) = told(&mut peer).await;
assert_eq!(routes, BTreeSet::from(["193.0.0.0/21".to_string()]));
assert_eq!(ended, None);
Ok(())
}
/// One route more than the export limit allows is not announced: the
/// session ends with an Administrative Shutdown that says why (RFC 9003),
/// and the peer is kept down, so that it is told nothing rather than too
/// much.
#[tokio::test]
async fn test_going_over_the_export_limit_ends_the_session_and_keeps_the_peer_down() -> Result<()> {
let limits = ExportLimits {
prefixes: None,
max_prefixes: Some(2),
};
let ServerWithPeer { server, .. } =
server_with(&["193.0.0.0/21", "193.0.8.0/21"], limits).await?;
let mut peer = established(&server).await?;
let (routes, ended) = told(&mut peer).await;
assert_eq!(routes.len(), 2);
assert_eq!(ended, None);
server
.local_rib
.announce(LocalRoute::new(prefix("193.0.16.0/21"), Default::default()))
.await?;
let (more, ended) = told(&mut peer).await;
assert_eq!(more, BTreeSet::new(), "nothing more is announced");
let Some((error, data)) = ended else {
bail!("the session was not ended");
};
assert_eq!(error, BgpError::Cease(CeaseSubcode::AdminShutdown));
let text = String::from_utf8(data[1..].to_vec())?;
assert!(
text.starts_with("export limit of 2 routes reached"),
"{text}"
);
assert!(peer.closed_within(TIMEOUT).await);
// The peer is kept down: a new connection is closed unanswered.
let mut again = TestPeer::connect(server.addr).await?;
assert!(
again.closed_unanswered().await,
"a held-down peer gets no session"
);
Ok(())
}
crates/integration-tests/tests/failure.rs
+59
View File
@@ -0,0 +1,59 @@
//! What the server does when a part of it that it cannot do without fails.
//!
//! The local RIB's actor stops only by panicking. Before, nothing noticed:
//! sessions stayed up, nothing was announced any more, and the status page
//! showed an empty table. Now the server tells its peers it is going and
//! stops with an error, for whatever supervises it to restart it.
use std::net::Ipv4Addr;
use bgp_integration_tests::TIMEOUT;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_rib::RibStopped;
use bgp_server::error::ServerError;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
#[tokio::test]
async fn test_losing_the_local_rib_stops_the_server_and_tells_the_peers() -> Result<()> {
let mut server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1))
.await?;
server.local_rib.stop_for_testing();
let outcome = tokio::time::timeout(TIMEOUT, server.stopped()).await?;
assert!(
matches!(outcome, Err(ServerError::RibStopped(RibStopped { rib })) if rib == "local RIB"),
"{outcome:?}"
);
assert_eq!(
peer.expect_notification().await?.error,
BgpError::Cease(CeaseSubcode::AdminShutdown)
);
// The status interface says the table cannot be read, where it used to
// show an empty one.
let client = reqwest::Client::new();
for path in ["/api/status", "/api/routes/local", "/metricz"] {
let response = client.get(format!("{base}{path}")).send().await?;
assert_eq!(response.status(), 503, "{path}");
assert_eq!(
response.text().await?,
"the local RIB has stopped",
"{path}"
);
}
// What does not need that table still answers.
let response = client
.get(format!("{base}/api/routes/received"))
.send()
.await?;
assert_eq!(response.status(), 200);
Ok(())
}
crates/integration-tests/tests/graceful_restart.rs
+332
View File
@@ -0,0 +1,332 @@
//! Graceful restart (RFC 4724, RFC 8538): a peer that restarts keeps its
//! routes with us while it does, and we keep ours with it while we do.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, announcement};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, wait_until as until};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::{Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{Capability, GracefulRestartEntry};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_server::restart::Leave;
use eyre::{Result, bail};
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
const PREFIXES: [&str; 3] = ["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"];
/// The graceful restart capability of a peer that speaks IPv4 and IPv6.
fn restart(restarted: bool, notification: bool, forwarding: bool, time: u16) -> Capability {
Capability::GracefulRestart {
restart_state: restarted,
restart_time: time,
notification,
entries: [Afi::IPV4, Afi::IPV6]
.into_iter()
.map(|afi| GracefulRestartEntry {
afi,
safi: Safi::UNICAST,
preserve_forwarding: forwarding,
})
.collect(),
}
}
/// What a dual-stack peer offers, with `more` besides.
fn capabilities(more: impl IntoIterator<Item = Capability>) -> Vec<Capability> {
let mut capabilities = vec![
Capability::FourByteAsn { asn: PEER_ASN },
Capability::MultiProtocol {
afi: Afi::IPV4,
safi: Safi::UNICAST,
},
Capability::MultiProtocol {
afi: Afi::IPV6,
safi: Safi::UNICAST,
},
Capability::RouteRefresh,
];
capabilities.extend(more);
capabilities
}
/// The peer's own routes.
fn own(prefixes: &[&str]) -> Announcement {
announcement(prefixes, &[PEER_ASN])
}
fn end_of_rib(family: AfiSafi) -> UpdateMessage {
UpdateMessage::end_of_rib(family)
}
/// The prefixes in the server's table, and how many of its routes are
/// stale.
fn table(server: &TestServer) -> (Vec<String>, usize) {
let mut prefixes = Vec::new();
let mut stale = 0;
server.rib.for_each_route(|route| {
prefixes.push(route.prefix.to_string());
stale += usize::from(route.stale);
});
prefixes.sort();
(prefixes, stale)
}
/// A peer that has announced `PREFIXES` with graceful restart as `offer`
/// says, and is Established.
async fn peer_with_routes(server: &TestServer, offer: Capability) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities([offer]))
.await?;
peer.announce(own(&PREFIXES)).await?;
until("the routes are in", || server.rib.len() == 3).await?;
Ok(peer)
}
/// The peer's connection fails; its routes are kept, stale, and used as
/// before. It comes back, sends some of them again, and its End-of-RIB:
/// the others go.
#[tokio::test]
async fn test_a_restarting_peer_keeps_its_routes_until_it_has_sent_them_again() -> Result<()> {
let server = TestServer::start(65000).await?;
let peer = peer_with_routes(&server, restart(false, true, true, 90)).await?;
drop(peer);
until("the session is down", || server.registry.list().is_empty()).await?;
until("the routes are stale", || table(&server).1 == 3).await?;
let kept = server.restarts.kept();
assert_eq!(kept.len(), 1);
assert_eq!(kept[0].1.routes, 3);
// Still chosen: the RFC has stale routes used like any others.
assert!(server.received_routes().iter().all(|route| route.best));
// And shown, as the peer's, stale.
let (base, _http) = server.serve_http().await?;
let received: serde_json::Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
let routes = received["routes"].as_array().unwrap();
assert_eq!(routes.len(), 3, "{received}");
for route in routes {
assert_eq!(route["stale"], true, "{route}");
assert_eq!(route["peer_asn"], PEER_ASN, "{route}");
}
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(
PEER_ASN,
PEER_ID,
90,
capabilities([restart(true, true, true, 90)]),
)
.await?;
until("the routes are taken over", || {
server.registry.list().len() == 1 && server.restarts.kept().is_empty()
})
.await?;
peer.announce(own(&PREFIXES[..2])).await?;
until("two are fresh", || table(&server).1 == 1).await?;
// The peer has them all, as far as its limit goes.
assert_eq!(server.registry.list()[0].counters.routes(), 3);
peer.announce(end_of_rib(AfiSafi::IPV4_UNICAST)).await?;
until("the one not sent again is gone", || {
table(&server) == (vec!["10.1.0.0/16".into(), "10.2.0.0/16".into()], 0)
})
.await?;
until("it is counted out", || {
server
.registry
.list()
.first()
.is_some_and(|session| session.counters.routes() == 2)
})
.await?;
Ok(())
}
/// What ends a session decides whether the peer is restarting: a Hard
/// Reset never is, and a NOTIFICATION is only where both sides said so
/// (RFC 8538). A peer that does not offer graceful restart never is.
#[tokio::test]
async fn test_only_a_restart_keeps_the_routes() -> Result<()> {
let cease = |subcode| NotificationMessage {
error: BgpError::Cease(subcode),
data: vec![],
};
let hard_reset = NotificationMessage {
error: BgpError::Cease(CeaseSubcode::HardReset),
data: vec![6, 2],
};
for (offer, goodbye, kept) in [
(
Some(restart(false, true, true, 90)),
Some(cease(CeaseSubcode::AdminReset)),
true,
),
(
Some(restart(false, true, true, 90)),
Some(hard_reset.clone()),
false,
),
(
Some(restart(false, false, true, 90)),
Some(cease(CeaseSubcode::AdminReset)),
false,
),
(Some(restart(false, false, true, 90)), None, true),
(Some(restart(false, true, true, 0)), None, false),
(None, None, false),
] {
let server = TestServer::start(65000).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(offer))
.await?;
peer.announce(own(&PREFIXES)).await?;
until("the routes are in", || server.rib.len() == 3).await?;
if let Some(goodbye) = &goodbye {
peer.send(&Message::Notification(goodbye.clone())).await?;
}
drop(peer);
until("the session is down", || server.registry.list().is_empty()).await?;
if kept {
until("the routes are stale", || table(&server).1 == 3).await?;
} else {
until("the routes are gone", || server.rib.is_empty()).await?;
assert!(server.restarts.kept().is_empty());
}
}
Ok(())
}
/// A peer that does not come back within its restart time has its routes
/// removed then.
#[tokio::test]
async fn test_the_routes_go_when_the_peer_does_not_come_back_in_time() -> Result<()> {
let server = TestServer::start(65000).await?;
let peer = peer_with_routes(&server, restart(false, true, true, 1)).await?;
drop(peer);
until("the routes are stale", || table(&server).1 == 3).await?;
until("the routes are gone", || server.rib.is_empty()).await?;
assert!(server.restarts.kept().is_empty());
Ok(())
}
/// A peer that comes back without having kept its forwarding (F bit), or
/// without graceful restart at all, has its stale routes removed at once.
#[tokio::test]
async fn test_routes_whose_forwarding_was_lost_go_when_the_peer_is_back() -> Result<()> {
for back_with in [Some(restart(true, true, false, 90)), None] {
let server = TestServer::start(65000).await?;
let peer = peer_with_routes(&server, restart(false, true, true, 90)).await?;
drop(peer);
until("the routes are stale", || table(&server).1 == 3).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(back_with.clone()))
.await?;
until("the stale routes are gone", || server.rib.is_empty()).await?;
until("they are counted out", || {
server
.registry
.list()
.first()
.is_some_and(|session| session.counters.routes() == 0)
})
.await?;
drop(peer);
}
Ok(())
}
/// Our OPEN offers graceful restart, as just restarted, and a peer that
/// agreed is sent End-of-RIB markers once it has our routes.
#[tokio::test]
async fn test_we_offer_graceful_restart_and_say_when_our_routes_are_all_sent() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(
PEER_ASN,
PEER_ID,
90,
capabilities([restart(false, true, true, 90)]),
)
.await?;
let offered = open.capabilities().find_map(|capability| match capability {
Capability::GracefulRestart {
restart_state,
restart_time,
notification,
entries,
} => Some((
*restart_state,
*restart_time,
*notification,
entries.clone(),
)),
_ => None,
});
let Some((restarted, time, notification, entries)) = offered else {
bail!("no graceful restart in {open:?}");
};
assert!(restarted, "started just now");
assert_eq!(time, 120);
assert!(notification);
// Without kernel routes, our forwarding is not ours to lose.
assert!(entries.iter().all(|entry| entry.preserve_forwarding));
// Nothing is originated: the markers are all there is.
let mut markers = Vec::new();
while markers.len() < 2 {
let update = peer.recv_update().await?;
match update.end_of_rib_family() {
Some(family) => markers.push(family),
None => bail!("expected End-of-RIB, got {update:?}"),
}
}
markers.sort();
assert_eq!(markers, [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST]);
Ok(())
}
/// Leaving to restart closes the connection without a NOTIFICATION where
/// graceful restart was agreed, so that the peer keeps our routes; leaving
/// for good is a Hard Reset there, and a Cease elsewhere.
#[tokio::test]
async fn test_leaving_tells_each_peer_as_graceful_restart_has_it() -> Result<()> {
let hard_reset = BgpError::Cease(CeaseSubcode::HardReset);
let shutdown = BgpError::Cease(CeaseSubcode::AdminShutdown);
for (leave, offer, told) in [
(Leave::Restart, Some(restart(false, true, true, 90)), None),
(Leave::Restart, None, Some(shutdown.clone())),
(
Leave::Stop,
Some(restart(false, true, true, 90)),
Some(hard_reset),
),
(Leave::Stop, None, Some(shutdown)),
] {
let server = TestServer::start(65000).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(offer.clone()))
.await?;
until("the session is up", || server.registry.list().len() == 1).await?;
server.registry.leave(leave, Duration::from_secs(2)).await;
let mut notification = None;
// Whatever came before (End-of-RIB markers) is passed over.
while let Ok(message) = peer.recv().await {
if let Message::Notification(n) = message {
notification = Some(n.error);
break;
}
}
assert_eq!(notification, told, "{leave:?} with {offer:?}");
assert!(peer.closed_within(TIMEOUT).await);
}
Ok(())
}
crates/integration-tests/tests/http_api.rs
+296
View File
@@ -0,0 +1,296 @@
//! Drives the HTTP API against a real server with a real BGP peer attached.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, get_json, prefix};
use bgp_packet::message::LargeCommunity;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
use serde_json::{Value, json};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
async fn established_peer(server: &TestServer) -> Result<TestPeer> {
server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1))
.await
}
#[tokio::test]
async fn test_status_and_peers() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let config = get_json(&client, &format!("{base}/api/config")).await?;
assert_eq!(config["path"], "test.textpb");
assert_eq!(config["text"], format!("local_asn: {LOCAL_ASN}\n"));
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["local_asn"], LOCAL_ASN);
assert_eq!(status["router_id"], "127.0.0.1");
assert_eq!(status["sessions"], 0);
assert!(status["started_at_unix"].as_u64().unwrap() > 1_700_000_000);
assert!(!status["hostname"].as_str().unwrap().is_empty());
let _peer = established_peer(&server).await?;
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["sessions"], 1);
let peers = get_json(&client, &format!("{base}/api/peers")).await?;
assert_eq!(peers.as_array().unwrap().len(), 1);
assert_eq!(peers[0]["peer_asn"], PEER_ASN);
assert_eq!(peers[0]["local_addr"], server.addr.to_string());
assert_eq!(peers[0]["four_octet_asn"], true);
assert_eq!(
peers[0]["address_families"],
serde_json::json!(["IPv4 unicast", "IPv6 unicast"])
);
// The session's TCP connection, by the address the peer list gives.
// What the kernel says of it depends on the platform; that it was
// asked, and why it had nothing to say if it had not, does not.
let peer_addr = peers[0]["peer_addr"].as_str().unwrap();
let tcp = get_json(&client, &format!("{base}/api/tcp?peer={peer_addr}")).await?;
let available = tcp["available"].as_bool().expect("available is a bool");
if available {
assert!(tcp["connection"].is_object(), "{tcp}");
} else {
assert!(tcp["reason"].is_string(), "{tcp}");
}
// The address alone names the peer too, as it does elsewhere.
let by_address = get_json(&client, &format!("{base}/api/tcp?peer=127.0.0.1")).await?;
assert_eq!(by_address["available"], tcp["available"]);
// A peer there is no session with is a bad request, not an answer.
let nobody = client
.get(format!("{base}/api/tcp?peer=nobody"))
.send()
.await?;
assert_eq!(nobody.status(), 400);
Ok(())
}
/// Originated routes are listed with their attributes as text.
#[tokio::test]
async fn test_originated_routes_are_listed() -> Result<()> {
let attrs = LocalRouteAttrs {
prepend: 1,
med: Some(10),
communities: vec![(65000, 100), (65535, 65281)],
large_communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2,
}],
..Default::default()
};
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.originate = vec![LocalRoute::new(prefix("10.100.1.0/24"), attrs)];
})
.await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let local = get_json(&client, &format!("{base}/api/routes/local")).await?;
assert_eq!(local.as_array().unwrap().len(), 1);
assert_eq!(local[0]["prefix"], "10.100.1.0/24");
assert_eq!(local[0]["origin"], "IGP");
assert_eq!(local[0]["prepend"], 1);
assert_eq!(local[0]["med"], 10);
assert_eq!(local[0]["communities"], json!(["65000:100", "65535:65281"]));
assert_eq!(local[0]["large_communities"], json!(["65000:1:2"]));
assert_eq!(
get_json(&client, &format!("{base}/api/status")).await?["local_routes"],
1
);
Ok(())
}
/// The API only reports. Nothing reachable over HTTP changes what the
/// router announces.
#[tokio::test]
async fn test_api_is_read_only() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let body = json!({ "prefix": "10.100.1.0/24" });
for path in [
"/api/routes/local",
"/api/routes/received",
"/api/peers",
"/api/status",
] {
let post = client
.post(format!("{base}{path}"))
.json(&body)
.send()
.await?;
assert_eq!(post.status(), 405, "POST {path}");
let delete = client
.delete(format!("{base}{path}"))
.json(&body)
.send()
.await?;
assert_eq!(delete.status(), 405, "DELETE {path}");
}
assert!(server.local_rib.snapshot().await.unwrap().is_empty());
Ok(())
}
/// Routes the peer sends us show up in the received table with a summary.
#[tokio::test]
async fn test_received_routes_summary() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut peer = established_peer(&server).await?;
peer.announce(
announcement(&["203.0.113.0/24"], &[PEER_ASN, 65002])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 1))
.with_communities(&[(65001, 7)]),
)
.await?;
let received = tokio::time::timeout(TIMEOUT, async {
loop {
let v =
get_json(&client, &format!("{base}/api/routes/received")).await?["routes"].take();
if !v.as_array().unwrap().is_empty() {
return Ok::<_, eyre::Error>(v);
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await??;
assert_eq!(received[0]["prefix"], "203.0.113.0/24");
assert_eq!(received[0]["peer_asn"], PEER_ASN);
assert_eq!(received[0]["summary"]["as_path"], json!([PEER_ASN, 65002]));
assert_eq!(received[0]["summary"]["next_hop"], "10.0.0.1");
assert_eq!(received[0]["summary"]["communities"], json!(["65001:7"]));
Ok(())
}
/// The SSE stream reports session and route changes as they happen.
#[tokio::test]
async fn test_event_stream() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut resp = client
.get(format!("{base}/api/events"))
.send()
.await?
.error_for_status()?;
assert!(
resp.headers()["content-type"]
.to_str()?
.starts_with("text/event-stream")
);
let _peer = established_peer(&server).await?;
server
.local_rib
.announce(LocalRoute::new(prefix("10.100.1.0/24"), Default::default()))
.await?;
// Collect SSE frames until we have seen both events.
let mut buf = String::new();
let mut seen_up = false;
let mut seen_announce = false;
tokio::time::timeout(TIMEOUT, async {
while !(seen_up && seen_announce) {
let chunk = resp
.chunk()
.await?
.ok_or_else(|| eyre::eyre!("stream ended"))?;
buf.push_str(std::str::from_utf8(&chunk)?);
for line in buf.lines().filter_map(|l| l.strip_prefix("data:")) {
let ev: Value = serde_json::from_str(line.trim())?;
match ev["type"].as_str() {
Some("session_up") if ev["peer"]["peer_asn"] == PEER_ASN => seen_up = true,
Some("local_announced") if ev["route"]["prefix"] == "10.100.1.0/24" => {
seen_announce = true
}
_ => {}
}
}
}
Ok::<_, eyre::Error>(())
})
.await??;
Ok(())
}
#[tokio::test]
async fn test_statusz_page_is_served() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let resp = client.get(format!("{base}/statusz")).send().await?;
assert_eq!(resp.status(), 200);
assert!(
resp.headers()["content-type"]
.to_str()?
.starts_with("text/html")
);
let body = resp.text().await?;
assert!(body.contains("<title>Status for ferrix_bgp</title>"));
assert!(body.contains("/api/events"));
// The root redirects to the page.
let no_redirect = reqwest::Client::builder()
.redirect(reqwest::redirect::Policy::none())
.build()?;
let resp = no_redirect.get(format!("{base}/")).send().await?;
assert_eq!(resp.status(), 307);
assert_eq!(resp.headers()["location"], "/statusz");
Ok(())
}
#[tokio::test]
async fn test_metricz() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let _peer = established_peer(&server).await?;
let text = client
.get(format!("{base}/metricz"))
.send()
.await?
.error_for_status()?
.text()
.await?;
// Metrics are process-global, so other tests in this binary contribute
// too; assert on presence and a lower bound rather than exact values.
let metric = |name: &str| -> Option<i64> {
text.lines()
.find(|l| l.starts_with(name) && l[name.len()..].starts_with(' '))
.and_then(|l| l.rsplit(' ').next()?.parse().ok())
};
assert!(
metric("bgp_sessions_established").unwrap_or(0) >= 1,
"{text}"
);
assert!(
metric("bgp_messages_sent_total{type=\"open\"}").unwrap_or(0) >= 1,
"{text}"
);
assert!(
metric("bgp_messages_received_total{type=\"keepalive\"}").unwrap_or(0) >= 1,
"{text}"
);
assert!(metric("bgp_local_routes").is_some(), "{text}");
assert!(metric("bgp_rib_routes").is_some(), "{text}");
Ok(())
}
crates/integration-tests/tests/import_policy.rs
+456
View File
@@ -0,0 +1,456 @@
//! What is taken from a peer: the checks every route goes through, and the
//! limit on how many a peer may have.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_for, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::{
CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, PathAttribute,
};
use bgp_server::marks::{Marks, Patterns, Scrub};
use bgp_server::policy::{ImportPolicy, ImportRule, PrefixSet};
use bgp_server::session::DialState;
use eyre::Result;
/// AS numbers as they are on the Internet: the checks refuse the private
/// ones the other tests use.
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server with one configured peer, at the address the test connects
/// from, whose routes are judged by `import`. The listener that keeps the
/// server's own dialling occupied comes with it.
async fn server_with(import: ImportPolicy) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(50);
peer.import = import;
})
.await
}
async fn established(server: &TestServer) -> Result<TestPeer> {
server.established(PEER_ASN, PEER_ID).await
}
/// Announces IPv4 `prefixes` with the AS path `path`.
async fn announce(peer: &mut TestPeer, prefixes: &[&str], path: &[u32]) -> Result<()> {
peer.announce(announcement(prefixes, path)).await
}
/// Waits until the server has dealt with `count` announced routes of the
/// session's, taken or refused, and returns the prefixes in its table.
async fn table_after(server: &TestServer, taken: u64, refused: u64) -> Result<BTreeSet<String>> {
let counters = server.registry.list()[0].counters.clone();
wait_until("the routes are dealt with", || {
(counters.routes(), counters.rejected()) == (taken, refused)
})
.await
.map_err(|_| {
eyre::eyre!(
"the server took {} routes and refused {}; expected {taken} and {refused}",
counters.routes(),
counters.rejected()
)
})?;
Ok(server
.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect())
}
fn set(prefixes: &[&str]) -> BTreeSet<String> {
prefixes.iter().map(|p| p.to_string()).collect()
}
/// With nothing configured for a peer but its address and AS, its routes
/// still go through the checks that hold for anyone: only space that is
/// routed on the Internet, nothing too specific, a path that begins with
/// the peer and has no reserved AS in it, and never one that has been
/// through us. Everything a peer sent used to be taken.
#[tokio::test]
async fn test_routes_that_no_peer_should_announce_are_refused() -> Result<()> {
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
// One UPDATE with good and bad prefixes: the good ones are taken.
announce(
&mut peer,
&[
"193.0.0.0/21",
"10.0.0.0/8", // private
"193.0.10.0/25", // more specific than is routed
"0.0.0.0/0", // default
"193.0.16.0/24",
],
&[PEER_ASN],
)
.await?;
assert_eq!(
table_after(&server, 2, 3).await?,
set(&["193.0.0.0/21", "193.0.16.0/24"])
);
// Paths that are refused whatever the prefix: not the peer's own,
// with a private AS in it, and one that has been through us.
announce(&mut peer, &["194.0.1.0/24"], &[174, PEER_ASN]).await?;
announce(&mut peer, &["194.0.2.0/24"], &[PEER_ASN, 64512]).await?;
announce(&mut peer, &["194.0.3.0/24"], &[PEER_ASN, LOCAL_ASN, 174]).await?;
assert_eq!(
table_after(&server, 2, 6).await?,
set(&["193.0.0.0/21", "193.0.16.0/24"])
);
Ok(())
}
/// A route that is refused takes the place of the one the peer had
/// announced before, which goes: the peer no longer has the old route,
/// and the new one is not acceptable.
#[tokio::test]
async fn test_a_refused_route_replaces_the_one_that_was_taken() -> Result<()> {
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.0.0/21", "193.0.16.0/24"], &[PEER_ASN]).await?;
assert_eq!(table_after(&server, 2, 0).await?.len(), 2);
announce(&mut peer, &["193.0.0.0/21"], &[PEER_ASN, 64512]).await?;
assert_eq!(table_after(&server, 1, 1).await?, set(&["193.0.16.0/24"]));
Ok(())
}
/// A peer with a list of what it may announce is held to it: its own
/// prefixes, originated by the AS numbers given.
#[tokio::test]
async fn test_a_peer_is_held_to_its_prefix_list_and_origins() -> Result<()> {
let mut prefixes = PrefixSet::default();
prefixes.allow(prefix("193.0.0.0/21"), 24);
let ServerWithPeer {
server,
listener: _listener,
} = server_with(ImportPolicy {
prefixes: Some(prefixes),
origin_asns: Some([PEER_ASN, 2121].into()),
..default_policy()
})
.await?;
let mut peer = established(&server).await?;
// Its own, and a more specific that the list allows.
announce(&mut peer, &["193.0.0.0/21", "193.0.4.0/24"], &[PEER_ASN]).await?;
// A customer of its that is listed as an origin.
announce(&mut peer, &["193.0.6.0/24"], &[PEER_ASN, 2121]).await?;
// Someone else's prefix, as a hijack or a leak would be.
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN]).await?;
// Its own prefix, but passed on from an origin that is not listed.
announce(&mut peer, &["193.0.7.0/24"], &[PEER_ASN, 15169]).await?;
assert_eq!(
table_after(&server, 3, 2).await?,
set(&["193.0.0.0/21", "193.0.4.0/24", "193.0.6.0/24"])
);
Ok(())
}
/// A peer that sends more routes than it is allowed is cut off and told
/// why, and kept down for a while: neither its connections are taken nor
/// is it dialled. A session that came straight back would only fill up
/// again.
#[tokio::test]
async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
let ServerWithPeer { server, listener } = server_with(ImportPolicy {
max_prefixes: Some(3),
hold_down: Duration::from_millis(1500),
..default_policy()
})
.await?;
// The server dials the peer as well; that connection is left unanswered
// until the end.
let dialled = TestPeer::accept(&listener).await?;
let mut peer = established(&server).await?;
// Refused routes do not count towards the limit, and replacing a
// route does not either.
announce(
&mut peer,
&["193.0.0.0/24", "193.0.1.0/24", "10.0.0.0/8"],
&[PEER_ASN],
)
.await?;
announce(&mut peer, &["193.0.1.0/24", "193.0.2.0/24"], &[PEER_ASN]).await?;
assert_eq!(table_after(&server, 3, 1).await?.len(), 3);
announce(&mut peer, &["193.0.3.0/24"], &[PEER_ASN]).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::Cease(CeaseSubcode::MaxPrefixes)
);
drop(peer);
wait_until("the peer's routes are gone", || {
server.received_routes().is_empty()
})
.await?;
// Held down: a new connection gets nothing.
let mut early = TestPeer::connect(server.addr).await?;
assert!(
early.closed_unanswered().await,
"a peer that is held down is not answered"
);
// Nor is it dialled: once the connection the server had open to it is
// gone, the dialer waits out the hold-down and says so.
drop(dialled);
let reason = wait_for("the dialer says the peer is held down", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { last_error, .. }))
if last_error.contains("held down") =>
{
Some(last_error)
}
_ => None,
}
})
.await?;
assert!(reason.contains("limit of routes"), "{reason}");
// When the time is up the peer is dialled again, and taken back.
TestPeer::accept(&listener).await?;
let mut later = TestPeer::connect(server.addr).await?;
later.handshake(PEER_ASN, PEER_ID, 90).await?;
Ok(())
}
/// Routes from a peer are marked with where they were learned and given
/// a preference, as the peer's `import` says; and marks that mean
/// something to us are not taken on the peer's word.
#[tokio::test]
async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()> {
use bgp_packet::message::{CommunitiesPathAttribute, LocalPrefPathAttribute};
use bgp_server::marks::Marks;
let ServerWithPeer {
server,
listener: _listener,
} = server_with(ImportPolicy {
marks: Marks::parse(
&["1299:*".to_string()],
&["1299:1001".to_string()],
&["1299:9:7".to_string()],
)
.unwrap(),
local_pref: Some(250),
..default_policy()
})
.await?;
let mut peer = established(&server).await?;
peer.announce(
announcement(&["193.0.0.0/21", "193.0.8.0/22"], &[PEER_ASN])
.with_communities(&[(3333, 5), (1299, 2002)]),
)
.await?;
table_after(&server, 2, 0).await?;
let routes = server.received_routes();
for route in &routes {
assert!(
route
.attributes
.contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
(3333, 5),
(1299, 1001)
]))),
"{route:?}"
);
assert!(
route
.attributes
.contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(250))),
"{route:?}"
);
assert!(
route.attributes.iter().any(|attribute| matches!(
attribute,
PathAttribute::LargeCommunities(large) if large.communities.len() == 1
)),
"{route:?}"
);
}
assert_eq!(server.rib.attrs().len(), 1, "they share their attributes");
Ok(())
}
/// The communities of our own AS are taken off what another network
/// sends, before the route is kept, and nothing else is; from a peer that
/// is to steer its routes with them (a customer), they are kept.
#[tokio::test]
async fn test_our_communities_are_not_taken_from_other_networks() -> Result<()> {
let ours_and_theirs = || {
vec![
PathAttribute::Communities(CommunitiesPathAttribute(vec![
(0, 13335),
(3356, 3),
(1299, 5),
])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![large(1299, 0, 0), large(1299, 31, 2), large(3356, 1, 2)],
}),
]
};
let kept = |server: &TestServer| -> Vec<PathAttribute> {
let routes = server.received_routes();
routes[0]
.attributes
.iter()
.filter(|attribute| {
matches!(
attribute,
PathAttribute::Communities(_) | PathAttribute::LargeCommunities(_)
)
})
.cloned()
.collect()
};
let scrubbed = ImportPolicy {
scrub: Some(Scrub {
local_asn: LOCAL_ASN,
zero: true,
}),
..default_policy()
};
let ServerWithPeer {
server,
listener: _listener,
} = server_with(scrubbed).await?;
let mut peer = established(&server).await?;
announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
table_after(&server, 1, 0).await?;
assert_eq!(
kept(&server),
[
PathAttribute::Communities(CommunitiesPathAttribute(vec![(3356, 3)])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![large(3356, 1, 2)],
}),
]
);
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
table_after(&server, 1, 0).await?;
assert_eq!(kept(&server), ours_and_theirs());
Ok(())
}
fn large(global_admin: u32, data_1: u32, data_2: u32) -> LargeCommunity {
LargeCommunity {
global_admin,
data_1,
data_2,
}
}
/// Announces IPv4 `prefixes` from the peer itself, with `more` attributes.
async fn announce_with(
peer: &mut TestPeer,
prefixes: &[&str],
more: Vec<PathAttribute>,
) -> Result<()> {
peer.announce(announcement(prefixes, &[PEER_ASN]).with_attributes(more))
.await
}
/// Rules do to some routes what the rest of the policy does to all: by
/// their communities, as the session has marked them, or by their prefix,
/// even when they came in one UPDATE with others. The first rule that
/// holds for a route is the one applied.
#[tokio::test]
async fn test_rules_apply_to_the_routes_they_hold_for() -> Result<()> {
let mut within = PrefixSet::default();
within.allow(prefix("193.0.0.0/21"), 24);
let rule = |communities: &[&str], prefixes: Option<PrefixSet>, local_pref, add: &[&str]| {
let strings = |texts: &[&str]| texts.iter().map(|t| t.to_string()).collect::<Vec<_>>();
ImportRule {
communities: Patterns::parse(&strings(communities)).unwrap(),
prefixes,
local_pref,
marks: Marks::parse(&[], &[], &strings(add)).unwrap(),
}
};
let policy = ImportPolicy {
// The session marks where its routes were learned.
marks: Marks::parse(&[], &[], &["1299:101:1".to_string()]).unwrap(),
rules: vec![
rule(&["3356:3"], None, Some(50), &[]),
rule(&[], Some(within), None, &["1299:31:10"]),
rule(&["1299:101:*"], None, Some(120), &[]),
],
..default_policy()
};
let ServerWithPeer {
server,
listener: _listener,
} = server_with(policy).await?;
let mut peer = established(&server).await?;
announce_with(
&mut peer,
&["194.0.0.0/16"],
vec![PathAttribute::Communities(CommunitiesPathAttribute(vec![
(3356, 3),
]))],
)
.await?;
announce_with(&mut peer, &["193.0.0.0/22", "195.0.0.0/16"], vec![]).await?;
table_after(&server, 3, 0).await?;
let kept = |wanted: &str| {
let routes = server.received_routes();
let route = routes
.iter()
.find(|route| route.prefix.to_string() == wanted)
.unwrap_or_else(|| panic!("{wanted} is in the table"));
let local_pref = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::LocalPref(local_pref) => Some(local_pref.0),
_ => None,
});
let large = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::LargeCommunities(large) => Some(large.communities.clone()),
_ => None,
})
.unwrap_or_default();
(local_pref, large)
};
let learned = large(1299, 101, 1);
// By its community: the first rule, and only that one.
assert_eq!(kept("194.0.0.0/16"), (Some(50), vec![learned.clone()]));
// By its prefix, apart from the route it came with.
assert_eq!(
kept("193.0.0.0/22"),
(None, vec![learned.clone(), large(1299, 31, 10)])
);
// By the mark the session put on.
assert_eq!(kept("195.0.0.0/16"), (Some(120), vec![learned]));
Ok(())
}
crates/integration-tests/tests/interop.rs
+142
View File
@@ -0,0 +1,142 @@
//! Interoperability with what real routers put on the wire.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{Message, PathAttribute};
use bgp_packet::open::{Capability, OpenOption};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
/// The OPEN a BIRD 2 router sends for an IPv6-only session from AS210036:
/// one optional parameter carrying MP(IPv6 unicast), route refresh,
/// extended message, graceful restart (aware only), 4-octet AS, enhanced
/// route refresh and long-lived graceful restart (aware only).
#[rustfmt::skip]
const BIRD_OPEN: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x37, 0x01, // length 55, type OPEN
0x04, // version
0x5b, 0xa0, // AS_TRANS
0x00, 0xf0, // hold time 240
0x0a, 0xff, 0x00, 0x01, // router id 10.255.0.1
0x1a, // optional parameters length 26
0x02, 0x18, // capabilities, 24 bytes
0x01, 0x04, 0x00, 0x02, 0x00, 0x01, // MP: IPv6 unicast
0x02, 0x00, // route refresh
0x06, 0x00, // extended message
0x40, 0x02, 0x00, 0x00, // graceful restart, no address families
0x41, 0x04, 0x00, 0x03, 0x34, 0x74, // 4-octet AS 210036
0x46, 0x00, // enhanced route refresh
0x47, 0x00, // long-lived graceful restart, none
];
/// IPv6 End-of-RIB: an UPDATE whose only attribute is an empty
/// MP_UNREACH_NLRI for IPv6 unicast (RFC 4724 §2).
#[rustfmt::skip]
const IPV6_END_OF_RIB: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x1e, 0x02, // length 30, type UPDATE
0x00, 0x00, // no withdrawn routes
0x00, 0x07, // 7 bytes of path attributes
0x90, 0x0f, 0x00, 0x03, // optional + extended length, MP_UNREACH_NLRI, len 3
0x00, 0x02, 0x01, // IPv6 unicast, no prefixes
];
#[tokio::test]
async fn test_server_advertises_ipv4_and_ipv6_unicast() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer.recv_open().await?;
let caps: Vec<&Capability> = open
.options
.iter()
.flat_map(|o| match o {
OpenOption::Capabilities(c) => c.iter(),
})
.collect();
for afi in [Afi::IPV4, Afi::IPV6] {
assert!(
caps.iter().any(|c| matches!(c, Capability::MultiProtocol { afi: a, safi: Safi::UNICAST } if *a == afi)),
"missing MP capability for {afi:?}: {caps:?}"
);
}
Ok(())
}
#[tokio::test]
async fn test_bird_open_and_end_of_rib_are_accepted() -> Result<()> {
let server = TestServer::start(4200000001).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_raw(BIRD_OPEN).await?;
peer.recv_open().await?;
match peer.recv().await? {
Message::KeepAlive => {}
other => eyre::bail!("server did not accept BIRD's OPEN: {other:?}"),
}
peer.send(&Message::KeepAlive).await?;
let session = server.wait_for_session().await?;
assert_eq!(
session.peer.asn, 210036,
"4-octet ASN must come from the capability, not AS_TRANS"
);
assert!(session.four_octet_asn);
// End-of-RIB must neither break the session nor create a route.
peer.send_raw(IPV6_END_OF_RIB).await?;
peer.send(&Message::KeepAlive).await?;
tokio::time::sleep(Duration::from_millis(200)).await;
assert_eq!(
server.registry.list().len(),
1,
"session dropped after End-of-RIB"
);
assert!(server.received_routes().is_empty());
Ok(())
}
/// Over an IPv4 test transport an IPv6 route cannot be sent (no IPv6 next
/// hop), so this only checks the builder the advertiser uses: the UPDATE for
/// an IPv6 prefix carries MP_REACH_NLRI with an IPv6 next hop and no NEXT_HOP.
#[tokio::test]
async fn test_ipv6_prefix_update_shape() -> Result<()> {
let route = LocalRoute::new(
IpPrefix::try_from("2001:db8:1::/48")?,
LocalRouteAttrs::default(),
);
let nh: std::net::Ipv6Addr = "fd00:179::2".parse()?;
let update = bgp_server::advertise::build_announce(
&route,
&bgp_server::advertise::Towards {
local_asn: 4200000001,
next_hop: nh.into(),
four_octet_asn: true,
ibgp: false,
role: None,
extended_next_hop: false,
},
)
.expect("ipv6 over ipv6");
assert!(update.announced_routes.is_empty());
assert!(
!update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::NextHop(_)))
);
assert!(
update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::MpReachNlri(m)
if m.family == AfiSafi::IPV6_UNICAST && m.prefixes[0].to_string() == "2001:db8:1::/48"))
);
let _ = Ipv4Addr::LOCALHOST;
Ok(())
}
crates/integration-tests/tests/kernel.rs
+172
View File
@@ -0,0 +1,172 @@
//! The whole daemon against a real kernel: a peer announces routes over a
//! session, and `ip route` shows them.
//!
//! This changes the machine's interfaces and routes, so it is not run by
//! `cargo test`; `scripts/kernel-tests.sh` runs it in a network namespace
//! of its own.
#![cfg(target_os = "linux")]
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::process::Command;
use std::time::{Duration, Instant};
use bgp_fib::FibConfig;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::prefix;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use eyre::Result;
use serde_json::Value;
/// The kernel is given more time than the loopback tests allow.
const TIMEOUT: Duration = Duration::from_secs(10);
const TABLE: u32 = 1100;
const INTERFACE: &str = "ferrix100";
#[track_caller]
fn ip(arguments: &str) {
let output = Command::new("ip")
.args(arguments.split_whitespace())
.output()
.expect("the ip command is there");
assert!(
output.status.success(),
"ip {arguments}: {}",
String::from_utf8_lossy(&output.stderr)
);
}
fn kernel_routes() -> BTreeSet<String> {
let output = Command::new("ip")
.args(["route", "show", "table", &TABLE.to_string()])
.output()
.unwrap();
String::from_utf8_lossy(&output.stdout)
.lines()
.map(|line| line.split_whitespace().collect::<Vec<_>>().join(" "))
// The number of the nexthop object a route names is the kernel's
// to choose.
.map(|line| match line.split_once(" nhid ") {
Some((prefix, rest)) => {
let rest = rest.split_once(' ').map_or("", |(_, rest)| rest);
format!("{prefix} {rest}")
}
None => line,
})
.filter(|line| !line.is_empty())
.collect()
}
#[track_caller]
fn kernel_becomes(expected: &[&str]) {
let expected: BTreeSet<String> = expected.iter().map(|route| route.to_string()).collect();
let start = Instant::now();
while kernel_routes() != expected {
assert!(
start.elapsed() < TIMEOUT,
"the kernel has {:#?}; expected {expected:#?}",
kernel_routes()
);
std::thread::sleep(Duration::from_millis(20));
}
}
/// Removes the test's interface, and its routes with it, when dropped.
struct Interface;
impl Drop for Interface {
fn drop(&mut self) {
let _ = Command::new("ip").args(["link", "del", INTERFACE]).output();
let _ = Command::new("ip")
.args(["route", "flush", "table", &TABLE.to_string()])
.output();
}
}
async fn announce(peer: &mut TestPeer, prefixes: &[&str], next_hop: Ipv4Addr) -> Result<()> {
peer.announce(announcement(prefixes, &[65001]).with_next_hop(next_hop))
.await
}
/// A peer's routes go from the session to the kernel, and out of it again
/// when they are withdrawn and when the session ends.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn test_routes_from_a_session_reach_the_kernel() -> Result<()> {
ip(&format!("link add {INTERFACE} type dummy"));
let _interface = Interface;
ip(&format!("addr add 10.100.0.1/24 dev {INTERFACE}"));
ip(&format!("link set {INTERFACE} up"));
let server = TestServer::start_with(65000, |config| {
config.kernel = Some(FibConfig {
table: TABLE,
..FibConfig::default()
});
})
.await?;
let (base, _http) = server.serve_http().await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
// The peer speaks from the loopback and names a router on the test's
// interface as next hop, as a route server would.
let neighbor = Ipv4Addr::new(10, 100, 0, 2);
announce(&mut peer, &["203.0.113.0/24", "198.18.0.0/15"], neighbor).await?;
// One by way of a router that is not a neighbor.
announce(&mut peer, &["192.88.99.0/24"], Ipv4Addr::new(172, 31, 0, 1)).await?;
kernel_becomes(&[
"203.0.113.0/24 via 10.100.0.2 dev ferrix100 proto bgp metric 20",
"198.18.0.0/15 via 10.100.0.2 dev ferrix100 proto bgp metric 20",
]);
let kernel: Value = reqwest::get(format!("{base}/api/kernel"))
.await?
.json()
.await?;
assert_eq!(kernel["enabled"], true);
assert_eq!(kernel["table"], TABLE);
assert_eq!(kernel["installed"], 2);
// The route by way of a router that is no neighbor is not in the
// table for the kernel to be asked for: it is held, in case the
// router becomes one.
assert_eq!(kernel["unresolved"], serde_json::json!({}));
let session = server.registry.list().pop().expect("the session");
assert_eq!(session.counters.held(), 1);
assert_eq!(server.rib.len(), 2);
let refused: Value = reqwest::get(format!("{base}/api/routes/rejected"))
.await?
.json()
.await?;
assert_eq!(
refused[0]["by_reason"],
serde_json::json!({"next hop unreachable": 1})
);
let shown = kernel["interfaces"]
.as_array()
.unwrap()
.iter()
.find(|interface| interface["name"] == INTERFACE)
.expect("the test's interface is listed");
assert_eq!(shown["up"], true);
// Its own address, beside the link-local one the kernel gave it.
assert!(
shown["addresses"]
.as_array()
.unwrap()
.contains(&serde_json::json!("10.100.0.1/24")),
"{shown}"
);
// One is withdrawn.
peer.announce(withdrawal([prefix("198.18.0.0/15").into()]))
.await?;
kernel_becomes(&["203.0.113.0/24 via 10.100.0.2 dev ferrix100 proto bgp metric 20"]);
// The session ends: packets must not go on following its routes.
drop(peer);
kernel_becomes(&[]);
Ok(())
}
crates/integration-tests/tests/negotiation.rs
+279
View File
@@ -0,0 +1,279 @@
//! End-to-end tests for capability negotiation: what the two OPENs agree on
//! decides which routes are sent and accepted, and how they are encoded.
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ReceivedRoute, TestServer};
use bgp_integration_tests::{peer_config, prefix, wait_for};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, PathAttribute, UpdateMessage,
};
use bgp_packet::open::Capability;
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::{FamilySet, SUPPORTED};
use bgp_server::policy::ImportPolicy;
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
fn only(family: AfiSafi) -> FamilySet {
[family].into_iter().collect()
}
fn four_octet() -> Capability {
Capability::FourByteAsn { asn: PEER_ASN }
}
/// Waits until the server has a route, and returns what it has.
async fn some_routes(server: &TestServer) -> Result<Vec<ReceivedRoute>> {
wait_for("a route is in the table", || {
let routes = server.received_routes();
(!routes.is_empty()).then_some(routes)
})
.await
}
/// An UPDATE announcing one IPv4 route in the body and one IPv6 route in
/// MP_REACH_NLRI.
fn dual_stack_update() -> UpdateMessage {
let ipv6 = PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1)),
prefixes: vec![prefix("2001:db8:1::/48").into()],
});
announcement(&["192.0.2.0/24"], &[PEER_ASN])
.with_attributes([ipv6])
.into()
}
/// The session carries the families both OPENs offered, and the server's
/// OPEN offers every family by default.
#[tokio::test]
async fn test_families_are_the_ones_both_sides_offered() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
for family in SUPPORTED {
assert!(
open.capabilities()
.any(|c| *c == Capability::multiprotocol(family)),
"the server's OPEN offers {family}"
);
}
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
Ok(())
}
/// A route of a family the peer did not offer is not accepted from it. The
/// rest of the UPDATE is, and the session stays up.
#[tokio::test]
async fn test_routes_of_a_family_not_negotiated_are_not_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
server.wait_for_session().await?;
peer.announce(dual_stack_update()).await?;
let routes = some_routes(&server).await?;
// One UPDATE is applied as a whole, so the IPv6 route is not on its way.
assert_eq!(routes.len(), 1);
assert_eq!(routes[0].prefix, prefix("192.0.2.0/24"));
assert_eq!(server.registry.list().len(), 1, "the session survives");
Ok(())
}
/// A route of a family the peer did not offer is not sent to it.
#[tokio::test]
async fn test_routes_of_a_family_not_negotiated_are_not_sent() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.originate = vec![LocalRoute::new(
prefix("10.100.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
// This peer wants IPv6 only; the server has an IPv4 route.
let mut v6_only = TestPeer::connect(server.addr).await?;
v6_only
.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
],
)
.await?;
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV6_UNICAST));
// A second peer that takes IPv4 is sent the route. The advertiser
// handles sessions in the order they came up, so by the time this one
// has its route the first has been dealt with.
let mut dual = TestPeer::connect(server.addr).await?;
dual.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let update = dual.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
let nothing = tokio::time::timeout(Duration::from_millis(200), v6_only.recv_update()).await;
assert!(nothing.is_err(), "got {nothing:?}");
Ok(())
}
/// A configured peer is offered only the families configured for it, and a
/// peer that shares none of them is refused with the capability we needed.
#[tokio::test]
async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// Inbound connections from this address inherit the peer's settings.
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
families: only(AfiSafi::IPV6_UNICAST),
import: ImportPolicy::permissive(),
..peer_config("v6-only", at, PEER_ASN)
}];
// Keep the dialer out of the way.
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_open_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
let open = peer.recv_open().await?;
let offered: Vec<_> = open
.capabilities()
.filter(|c| matches!(c, Capability::MultiProtocol { .. }))
.collect();
assert_eq!(offered, [&Capability::multiprotocol(AfiSafi::IPV6_UNICAST)]);
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(
n.error,
BgpError::OpenMsg(OpenMsgSubcode::UnsupportedCapability)
);
// The Multiprotocol capability for IPv6 unicast.
assert_eq!(n.data, [1, 4, 0, 2, 0, 1]);
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
assert!(server.registry.list().is_empty());
Ok(())
}
/// Real routers offer families we do not support. That must not stop
/// the families we do share from working.
#[tokio::test]
async fn test_a_peer_offering_an_unsupported_family_is_still_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::new(Afi::L2VPN, Safi::EVPN)),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
assert!(
session.four_octet_asn,
"capabilities after it are still read"
);
Ok(())
}
/// A peer that offers no capabilities at all is a plain RFC 4271 speaker:
/// IPv4 unicast, two-octet AS numbers. Our four-octet AS number reaches it
/// as AS_TRANS in AS_PATH and in full in AS4_PATH (RFC 6793 §4.2.2).
#[tokio::test]
async fn test_a_peer_without_capabilities_gets_two_octet_as_paths() -> Result<()> {
const WIDE_ASN: u32 = 4_200_000_001;
let server = TestServer::start_with(WIDE_ASN, |c| {
c.originate = vec![LocalRoute::new(
prefix("10.100.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, vec![]).await?;
let session = server.wait_for_session().await?;
assert!(!session.four_octet_asn);
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
// The test peer now decodes AS_PATH two octets wide, as such a speaker
// would; an UPDATE encoded four octets wide would not parse.
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
assert!(
update
.path_attributes
.contains(&AsPathAttribute::from_asns(vec![23456])),
"{update:?}"
);
let as4_path = update.path_attributes.iter().find_map(|a| match a {
PathAttribute::As4Path(path) => Some(path.segments[0].path.clone()),
_ => None,
});
assert_eq!(as4_path, Some(vec![WIDE_ASN]));
// And its own two-octet paths are read correctly.
peer.announce(announcement(&["192.0.2.0/24"], &[PEER_ASN, 64512]))
.await?;
let routes = some_routes(&server).await?;
assert!(
routes[0]
.attributes
.contains(&AsPathAttribute::from_asns(vec![PEER_ASN, 64512]))
);
Ok(())
}
crates/integration-tests/tests/roles.rs
+338
View File
@@ -0,0 +1,338 @@
//! Roles and route leak prevention (RFC 9234), end to end: what the two
//! ends say they are to each other, and what becomes of routes that are
//! marked as only to go to customers.
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{prefix, wait_until};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::open::{BgpRole, Capability};
use bgp_primitives::constants::AfiSafi;
use bgp_rib::LocalRoute;
use bgp_server::policy::PeerRole;
use eyre::{Result, bail};
use serde_json::Value;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server with one configured peer at the address the test connects
/// from, which is `role` to it, and which originates 193.0.0.0/21.
async fn server_with(role: Option<PeerRole>, require_role: bool) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.originate = vec![LocalRoute::new(prefix("193.0.0.0/21"), Default::default())];
peer.role = role;
peer.require_role = require_role;
})
.await
}
/// The capabilities of an ordinary peer, and with `role` what it says it
/// is.
fn capabilities(role: Option<BgpRole>) -> Vec<Capability> {
let mut capabilities = vec![
Capability::FourByteAsn { asn: PEER_ASN },
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
Capability::RouteRefresh,
];
capabilities.extend(role.map(|role| Capability::BgpRole { role }));
capabilities
}
/// A session with the server, on which the peer says it is `says`.
async fn established(server: &TestServer, says: Option<BgpRole>) -> Result<TestPeer> {
server
.established_with(PEER_ASN, PEER_ID, capabilities(says))
.await
}
/// Announces `prefixes` with the AS path `path` and, with `marked_by`, an
/// Only to Customer attribute with that AS number.
async fn announce(
peer: &mut TestPeer,
prefixes: &[&str],
path: &[u32],
marked_by: Option<u32>,
) -> Result<()> {
let mut announcement = announcement(prefixes, path);
if let Some(asn) = marked_by {
announcement = announcement.with_only_to_customer(asn);
}
peer.announce(announcement).await
}
/// Waits until the table holds `count` routes, and returns the mark each
/// has, by prefix.
async fn marks(server: &TestServer, count: usize) -> Result<Vec<(String, Option<u32>)>> {
server.wait_for_routes(count).await.map_err(|_| {
eyre::eyre!(
"expected {count} routes, the table has {:?}",
server.received_routes()
)
})?;
let mut marks: Vec<(String, Option<u32>)> = server
.received_routes()
.iter()
.map(|route| {
let mark = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::OnlyToCustomer(asn) => Some(*asn),
_ => None,
});
(route.prefix.to_string(), mark)
})
.collect();
marks.sort();
Ok(marks)
}
/// The role the server's OPEN announces.
fn announced_role(open: &bgp_packet::open::OpenMessage) -> Option<BgpRole> {
open.capabilities().find_map(|capability| match capability {
Capability::BgpRole { role } => Some(*role),
_ => None,
})
}
/// Both ends say what they are, and it fits: the session comes up, and is
/// shown as one whose peer confirmed its role.
#[tokio::test]
async fn test_roles_that_fit_together_are_confirmed() -> Result<()> {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(Some(BgpRole::Customer)))
.await?;
// The peer is our customer: we are its provider.
assert_eq!(announced_role(&open), Some(BgpRole::Provider));
server.wait_for_session().await?;
let peers: Value = reqwest::get(format!("{base}/api/peers"))
.await?
.json()
.await?;
assert_eq!(peers[0]["role"], "customer");
assert_eq!(peers[0]["role_confirmed"], true);
Ok(())
}
/// The commonest way to a leak is the two ends not agreeing on what they
/// are to each other. A peer that says it is something else than it is
/// configured as is refused, with the error RFC 9234 gives for it.
#[tokio::test]
async fn test_a_peer_that_says_it_is_something_else_is_refused() -> Result<()> {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let mut peer = TestPeer::connect(server.addr).await?;
// It thinks it is our peer; we have it down as a customer.
peer.send_open_with(PEER_ASN, PEER_ID, 90, capabilities(Some(BgpRole::Peer)))
.await?;
peer.recv_open().await?;
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(n.error, BgpError::OpenMsg(OpenMsgSubcode::RoleMismatch));
// Subcode 11 on the wire.
assert_eq!(n.error.code_subcode(), (2, 11));
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
assert!(server.registry.list().is_empty());
Ok(())
}
/// A peer that says nothing does not implement RFC 9234. It is taken at
/// our word for what it is, unless the configuration requires it to say.
#[tokio::test]
async fn test_a_peer_that_does_not_say_is_refused_only_where_required() -> Result<()> {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let (base, _http) = server.serve_http().await?;
let _peer = established(&server, None).await?;
let peers: Value = reqwest::get(format!("{base}/api/peers"))
.await?
.json()
.await?;
assert_eq!(peers[0]["role"], "customer");
assert_eq!(peers[0]["role_confirmed"], false);
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), true).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_open_with(PEER_ASN, PEER_ID, 90, capabilities(None))
.await?;
peer.recv_open().await?;
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(n.error, BgpError::OpenMsg(OpenMsgSubcode::RoleMismatch));
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
Ok(())
}
/// A route marked as only to go to customers has been handed down or
/// across once. Coming up from a customer it is a leak, and is refused
/// whatever else the peer's policy takes.
#[tokio::test]
async fn test_a_marked_route_from_a_customer_is_a_leak() -> Result<()> {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let mut peer = established(&server, Some(BgpRole::Customer)).await?;
// The customer's own route, and one its other provider gave it.
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN], None).await?;
announce(
&mut peer,
&["8.8.8.0/24"],
&[PEER_ASN, 3356, 15169],
Some(3356),
)
.await?;
// A customer's routes may go anywhere: they are not marked on the way
// in.
assert_eq!(
marks(&server, 1).await?,
[("194.0.0.0/24".to_string(), None)]
);
wait_until("a route is refused", || {
!server.rejections.peers().is_empty()
})
.await?;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 1);
assert_eq!(refused.recent()[0].prefix, prefix("8.8.8.0/24"));
assert_eq!(
refused.recent()[0].reason.as_str(),
"route leak (Only to Customer)"
);
Ok(())
}
/// From a peer a marked route is in order only if the peer marked it
/// itself, on its way to us. A mark of someone else's means the peer got
/// the route from above or from another peer.
#[tokio::test]
async fn test_a_route_marked_by_someone_else_is_a_leak_from_a_peer() -> Result<()> {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Peer), false).await?;
let mut peer = established(&server, Some(BgpRole::Peer)).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN], Some(PEER_ASN)).await?;
announce(
&mut peer,
&["8.8.8.0/24"],
&[PEER_ASN, 3356, 15169],
Some(3356),
)
.await?;
// A peer that does not mark its routes has them marked for it.
announce(&mut peer, &["195.0.0.0/24"], &[PEER_ASN, 2121], None).await?;
assert_eq!(
marks(&server, 2).await?,
[
("194.0.0.0/24".to_string(), Some(PEER_ASN)),
("195.0.0.0/24".to_string(), Some(PEER_ASN)),
]
);
assert_eq!(server.rejections.peers()[0].1.total(), 1);
Ok(())
}
/// A route from a provider is marked on the way in, if it is not already:
/// from here it may only go down. That holds for both address families,
/// and whether or not the peer's attributes could be stored as they came.
#[tokio::test]
async fn test_routes_from_a_provider_are_marked_on_the_way_in() -> Result<()> {
for four_octet in [true, false] {
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Provider), false).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let mut offered = capabilities(Some(BgpRole::Provider));
if !four_octet {
offered.retain(|capability| !matches!(capability, Capability::FourByteAsn { .. }));
}
peer.handshake_with(PEER_ASN, PEER_ID, 90, offered).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121], None).await?;
// One that someone further up has marked already keeps that mark.
announce(
&mut peer,
&["8.8.8.0/24"],
&[PEER_ASN, 3356, 15169],
Some(3356),
)
.await?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN, 2121]),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0x7f8, 0, 0, 0, 0, 0, 1)),
prefixes: vec![prefix("2a00:1450::/32").into()],
}),
],
announced_routes: vec![],
}))
.await?;
assert_eq!(
marks(&server, 3).await?,
[
("194.0.0.0/24".to_string(), Some(PEER_ASN)),
("2a00:1450::/32".to_string(), Some(PEER_ASN)),
("8.8.8.0/24".to_string(), Some(3356)),
],
"four-octet AS numbers: {four_octet}"
);
assert!(server.rejections.peers().is_empty());
}
Ok(())
}
/// Our own routes are marked on their way down to a customer or across to
/// a peer, and not on their way up to a provider.
#[tokio::test]
async fn test_our_routes_are_marked_on_the_way_down_or_across() -> Result<()> {
for (peer_is, says, marked) in [
(PeerRole::Customer, BgpRole::Customer, true),
(PeerRole::Peer, BgpRole::Peer, true),
(
PeerRole::RouteServerClient,
BgpRole::RouteServerClient,
true,
),
(PeerRole::Provider, BgpRole::Provider, false),
(PeerRole::RouteServer, BgpRole::RouteServer, false),
] {
let ServerWithPeer { server, .. } = server_with(Some(peer_is), false).await?;
let mut peer = established(&server, Some(says)).await?;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, [prefix("193.0.0.0/21")]);
let mark = update
.path_attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::OnlyToCustomer(asn) => Some(*asn),
_ => None,
});
assert_eq!(mark, marked.then_some(LOCAL_ASN), "to our {peer_is:?}");
}
// Nothing is said to a peer whose role is not known.
let ServerWithPeer { server, .. } = server_with(None, false).await?;
let mut peer = established(&server, None).await?;
let update = peer.recv_update().await?;
assert!(
!update
.path_attributes
.iter()
.any(|attribute| matches!(attribute, PathAttribute::OnlyToCustomer(_)))
);
Ok(())
}
crates/integration-tests/tests/rpki.rs
+291
View File
@@ -0,0 +1,291 @@
//! Route origin validation, end to end: a server that validates against a
//! cache, a peer that announces routes, and the validation data changing
//! under both.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::test_cache::TestCache;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_packet::message::{AsPathAttribute, PathAttribute};
use bgp_rpki::{CacheState, RpkiConfig, Vrp};
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
fn vrp(p: &str, max_length: u8, asn: u32) -> Vrp {
Vrp {
prefix: prefix(p),
max_length,
asn,
}
}
/// A server that validates against `cache`, with one configured peer at
/// the address the test connects from. Returns once the server has the
/// cache's data.
async fn server_with(
cache: &TestCache,
expire: Option<Duration>,
import: ImportPolicy,
) -> Result<ServerWithPeer> {
let cache = cache.address();
let setup = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.rpki = Some(RpkiConfig {
caches: vec![cache],
// Data never expires sooner than a refresh and a retry take.
refresh: expire.map(|expire| expire / 2),
retry: Some(Duration::from_millis(100)),
expire,
});
peer.import = import;
})
.await?;
let rpki = setup.server.rpki.clone().expect("validation is configured");
wait_until("the server has the cache's data", || {
rpki.status().state == CacheState::Synchronised
})
.await?;
Ok(setup)
}
async fn established(server: &TestServer) -> Result<TestPeer> {
server.established(PEER_ASN, PEER_ID).await
}
/// Announces `prefixes` as originated by `origin`, reached through the
/// peer.
async fn announce(peer: &mut TestPeer, prefixes: &[&str], origin: u32) -> Result<()> {
let path = if origin == PEER_ASN {
vec![PEER_ASN]
} else {
vec![PEER_ASN, origin]
};
peer.announce(announcement(prefixes, &path)).await
}
/// A route whose origin is not the one the holder of the addresses
/// authorised is refused. One that is authorised is taken, and so is one
/// that nothing is known about.
#[tokio::test]
async fn test_an_invalid_route_is_refused_and_the_rest_are_taken() -> Result<()> {
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, PEER_ASN)]).await;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.0.0/21", "194.0.0.0/24"], PEER_ASN).await?;
// Someone else's AS as the origin of a covered prefix; and the right
// AS, more specific than it may announce.
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
announce(&mut peer, &["193.0.6.0/25"], PEER_ASN).await?;
server
.table_becomes(&["193.0.0.0/21", "194.0.0.0/24"], 1)
.await?;
let client = reqwest::Client::new();
let get = |path: &str| {
let request = client.get(format!("{base}{path}"));
async move { Ok::<Value, eyre::Report>(request.send().await?.error_for_status()?.json().await?) }
};
// Each route says how it stands.
let routes = get("/api/routes/received").await?["routes"].take();
let validity: Vec<(&str, &str)> = routes
.as_array()
.unwrap()
.iter()
.map(|r| (r["prefix"].as_str().unwrap(), r["rpki"].as_str().unwrap()))
.collect();
assert_eq!(
validity,
[("193.0.0.0/21", "valid"), ("194.0.0.0/24", "not found")]
);
// The refused one is on record with its reason. The too-specific one
// was refused before validation came to it.
let refused = get("/api/routes/rejected").await?;
assert_eq!(
refused[0]["by_reason"],
serde_json::json!({"RPKI invalid": 1, "prefix too specific": 1})
);
let peers = get("/api/peers").await?;
assert_eq!(peers[0]["routes"], 2);
assert_eq!(peers[0]["held_routes"], 1);
let rpki = get("/api/rpki").await?;
assert_eq!(rpki["enabled"], true);
assert_eq!(rpki["state"], "synchronised");
assert_eq!(rpki["vrps_ipv4"], 1);
assert_eq!(rpki["cache"], cache.address().to_string());
Ok(())
}
/// Validity changes with the validation data, and the routes follow: one
/// that was taken goes when it becomes invalid, and one that was refused
/// as invalid is taken when it no longer is. The peer sends nothing more
/// meanwhile; it has said all it has to say.
#[tokio::test]
async fn test_routes_follow_the_validation_data() -> Result<()> {
let ripe = vrp("193.0.0.0/21", 24, PEER_ASN);
let cache = TestCache::start(vec![ripe]).await;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let mut peer = established(&server).await?;
// Nothing is known about 8.8.8.0/24, so it is taken. 193.0.4.0/24 is
// covered, and not for this origin.
announce(&mut peer, &["8.8.8.0/24", "193.0.4.0/24"], 15169).await?;
server.table_becomes(&["8.8.8.0/24"], 1).await?;
// The holder of 8.8.8.0/24 authorises another AS: the route in the
// table is now invalid, and goes.
cache.set(vec![ripe, vrp("8.8.8.0/24", 24, 13335)]);
server.table_becomes(&[], 2).await?;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 2);
assert_eq!(refused.recent()[0].prefix, prefix("8.8.8.0/24"));
assert_eq!(&refused.recent()[0].as_path[..], [PEER_ASN, 15169]);
// Both are authorised for the origin they have: both come back, with
// the attributes they were announced with.
cache.set(vec![
ripe,
vrp("8.8.8.0/24", 24, 15169),
vrp("193.0.4.0/24", 24, 15169),
]);
server
.table_becomes(&["8.8.8.0/24", "193.0.4.0/24"], 0)
.await?;
for route in server.received_routes() {
assert!(
route
.attributes
.contains(&AsPathAttribute::from_asns(vec![PEER_ASN, 15169])),
"{route:?}"
);
}
assert_eq!(server.registry.list()[0].counters.routes(), 2);
Ok(())
}
/// A held route is the peer's to withdraw or replace like any other. One
/// it has withdrawn does not appear when it would have become valid.
#[tokio::test]
async fn test_a_held_route_the_peer_withdrew_does_not_come_back() -> Result<()> {
let ripe = vrp("193.0.0.0/21", 24, PEER_ASN);
let cache = TestCache::start(vec![ripe]).await;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24", "193.0.5.0/24"], 15169).await?;
server.table_becomes(&[], 2).await?;
// One is withdrawn, the other announced again with the right origin.
peer.announce(withdrawal([prefix("193.0.4.0/24").into()]))
.await?;
announce(&mut peer, &["193.0.5.0/24"], PEER_ASN).await?;
server.table_becomes(&["193.0.5.0/24"], 0).await?;
// Now the old origin would be valid for both. Neither old route is
// there to be brought back.
cache.set(vec![ripe, vrp("193.0.4.0/23", 24, 15169)]);
tokio::time::sleep(Duration::from_millis(500)).await;
server.table_becomes(&["193.0.5.0/24"], 0).await?;
assert!(
server.received_routes()[0]
.attributes
.contains(&AsPathAttribute::from_asns(vec![PEER_ASN])),
"the route the peer has now"
);
Ok(())
}
/// A peer can be exempt: its invalid routes are taken, and shown for what
/// they are.
#[tokio::test]
async fn test_a_peer_can_be_exempt_from_validation() -> Result<()> {
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, PEER_ASN)]).await;
let import = ImportPolicy {
accept_rpki_invalid: true,
..default_policy()
};
let ServerWithPeer { server, .. } = server_with(&cache, None, import).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
server.table_becomes(&["193.0.4.0/24"], 0).await?;
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
assert_eq!(routes["routes"][0]["rpki"], "invalid");
assert!(server.rejections.peers().is_empty());
Ok(())
}
/// Validation data that no cache has confirmed for too long is dropped,
/// and with it every verdict: routes held as invalid are taken. Refusing
/// routes on the word of data that may be long out of date would be worse.
#[tokio::test]
async fn test_held_routes_are_taken_when_the_validation_data_expires() -> Result<()> {
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, PEER_ASN)]).await;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(Duration::from_millis(1500)), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
server.table_becomes(&[], 1).await?;
drop(cache);
server.table_becomes(&["193.0.4.0/24"], 0).await?;
assert!(server.rpki.as_ref().unwrap().current().vrps.is_empty());
Ok(())
}
/// A hijack that was possible: on a session with two-octet AS numbers a
/// route has two paths, AS_PATH and AS4_PATH, and a peer can make them
/// say different things. Here AS_PATH is the peer alone, and an AS4_PATH
/// too long to be used ends at the AS that holds the addresses. The route
/// is the peer's and invalid, when it arrives and when it is looked at
/// again; nothing may take the unused path's word for where it comes from.
#[tokio::test]
async fn test_an_unused_as4_path_does_not_make_a_route_valid() -> Result<()> {
use bgp_packet::message::AsPathSegment;
const HOLDER: u32 = 15169;
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, HOLDER)]).await;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
// No capabilities: AS numbers are two octets wide on this session.
let mut peer = server.established_with(PEER_ASN, PEER_ID, vec![]).await?;
assert!(!server.registry.list()[0].four_octet_asn);
let as4_path = PathAttribute::As4Path(AsPathAttribute {
segments: vec![AsPathSegment {
ordered: true,
path: vec![3356, HOLDER],
}],
});
peer.announce(announcement(&["193.0.4.0/24"], &[PEER_ASN]).with_attributes([as4_path]))
.await?;
server.table_becomes(&[], 1).await?;
// The data changes, and every held route is looked at again.
cache.set(vec![
vrp("193.0.0.0/21", 24, HOLDER),
vrp("198.51.100.0/24", 24, 64500),
]);
let rpki = server.rpki.clone().unwrap();
wait_until("the server has the new data", || {
rpki.current().vrps.len() == 2
})
.await?;
// Time enough for the look, and then some.
tokio::time::sleep(Duration::from_millis(600)).await;
server.table_becomes(&[], 1).await?;
Ok(())
}
crates/integration-tests/tests/transit.rs
+356
View File
@@ -0,0 +1,356 @@
//! Passing routes on: what one peer announced reaches the others, as far
//! as the rules for that let it.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::message::{
CommunitiesPathAttribute, Message, NextHopPathAttribute, PathAttribute, RouteRefreshMessage,
UpdateMessage,
};
use bgp_primitives::constants::AfiSafi;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::marks::{Control, Marks};
use bgp_server::policy::ExportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 64601;
/// A server that passes on to every peer what it learned from the others.
async fn transit(customize: impl FnOnce(&mut bgp_server::config::Config)) -> Result<TestServer> {
TestServer::start_with(LOCAL_ASN, |config| {
config.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
});
customize(config);
})
.await
}
/// The peer with AS `64700 + n`, its session up.
async fn peer(server: &TestServer, n: u8) -> Result<TestPeer> {
server
.established(64700 + u32::from(n), Ipv4Addr::new(10, 0, 0, n))
.await
}
async fn announce(
peer: &mut TestPeer,
prefixes: &[&str],
path: &[u32],
communities: &[(u16, u16)],
) -> Result<()> {
peer.announce(announcement(prefixes, path).with_communities(communities))
.await
}
async fn withdraw(peer: &mut TestPeer, prefixes: &[&str]) -> Result<()> {
peer.announce(withdrawal(prefixes.iter().map(|p| prefix(p).into())))
.await
}
/// What a peer has been told so far, as the prefixes it would now have
/// and the path each came with.
#[derive(Default, Debug)]
struct Told {
routes: std::collections::BTreeMap<String, Vec<u32>>,
updates: usize,
}
impl Told {
fn take(&mut self, update: &UpdateMessage) {
self.updates += 1;
for route in &update.withdrawn_routes {
self.routes.remove(&route.prefix.to_string());
}
let path: Vec<u32> = update
.path_attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(
path.segments
.iter()
.flat_map(|segment| segment.path.clone())
.collect(),
),
_ => None,
})
.unwrap_or_default();
for route in &update.announced_routes {
self.routes.insert(route.prefix.to_string(), path.clone());
}
}
fn prefixes(&self) -> BTreeSet<&str> {
self.routes.keys().map(String::as_str).collect()
}
}
/// Reads what the peer is sent until it has exactly `expected`.
async fn told_becomes(peer: &mut TestPeer, told: &mut Told, expected: &[&str]) -> Result<()> {
let expected: BTreeSet<&str> = expected.iter().copied().collect();
let reading = async {
while told.prefixes() != expected {
let update = peer.recv_update().await?;
told.take(&update);
}
Ok::<(), eyre::Report>(())
};
match tokio::time::timeout(TIMEOUT, reading).await {
Ok(read) => read,
Err(_) => Err(eyre::eyre!(
"the peer has {:?}; expected {expected:?}",
told.prefixes()
)),
}
}
/// Nothing more is sent to the peer for a while.
async fn told_nothing(peer: &mut TestPeer) {
let more = tokio::time::timeout(Duration::from_millis(300), peer.recv_update()).await;
assert!(more.is_err(), "the peer was sent {more:?}");
}
/// What one peer announces the others are told, with our AS on the path
/// and ourselves as next hop; and what it takes back, they are told too.
#[tokio::test]
async fn test_routes_from_one_peer_reach_the_others() -> Result<()> {
let server = transit(|_| {}).await?;
let mut one = peer(&server, 1).await?;
let mut two = peer(&server, 2).await?;
announce(
&mut one,
&["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"],
&[64701, 65000],
&[],
)
.await?;
let mut told = Told::default();
told_becomes(
&mut two,
&mut told,
&["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"],
)
.await?;
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64701, 65000]);
assert_eq!(
told.updates, 1,
"routes that go out alike go in one message"
);
// Not back to where they came from.
told_nothing(&mut one).await;
// The next hop is ourselves: the address of our end of the session.
announce(&mut one, &["10.4.0.0/16"], &[64701], &[]).await?;
let update = two.recv_update().await?;
assert!(
update
.path_attributes
.contains(&PathAttribute::NextHop(NextHopPathAttribute(
Ipv4Addr::LOCALHOST
))),
"{update:?}"
);
told.take(&update);
// One is taken back; then the peer goes, and its routes with it.
withdraw(&mut one, &["10.2.0.0/16"]).await?;
told_becomes(
&mut two,
&mut told,
&["10.1.0.0/16", "10.3.0.0/16", "10.4.0.0/16"],
)
.await?;
drop(one);
told_becomes(&mut two, &mut told, &[]).await?;
Ok(())
}
/// A peer is told of the best route, and of the next best when that one
/// goes; and a peer that comes later is told what there is.
#[tokio::test]
async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
let server = transit(|_| {}).await?;
let mut near = peer(&server, 1).await?;
let mut far = peer(&server, 2).await?;
let mut watcher = peer(&server, 3).await?;
let mut told = Told::default();
announce(&mut far, &["10.1.0.0/16"], &[64702, 65000, 65001], &[]).await?;
told_becomes(&mut watcher, &mut told, &["10.1.0.0/16"]).await?;
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64702, 65000, 65001]);
// A better route for it comes: that is the one told now.
announce(&mut near, &["10.1.0.0/16"], &[64701, 65001], &[]).await?;
let update = watcher.recv_update().await?;
told.take(&update);
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64701, 65001]);
assert!(
update.withdrawn_routes.is_empty(),
"replaced, not withdrawn"
);
// And goes again: back to the other, with no gap.
withdraw(&mut near, &["10.1.0.0/16"]).await?;
let update = watcher.recv_update().await?;
told.take(&update);
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64702, 65000, 65001]);
// The other peer was told of the better route while it was the best,
// and then that it is gone: the one left is its own, which it is not
// told of.
let mut told_far = Told::default();
told_becomes(&mut far, &mut told_far, &["10.1.0.0/16"]).await?;
let gone = far.recv_update().await?;
assert_eq!(gone.withdrawn_routes, vec![prefix("10.1.0.0/16")]);
told_nothing(&mut far).await;
// A peer that comes now is told what there is; and again if it asks.
let mut late = peer(&server, 4).await?;
let mut late_told = Told::default();
told_becomes(&mut late, &mut late_told, &["10.1.0.0/16"]).await?;
late.send(&Message::RouteRefresh(RouteRefreshMessage::request(
AfiSafi::IPV4_UNICAST,
)))
.await?;
let again = late.recv_update().await?;
assert_eq!(again.announced_routes, vec![prefix("10.1.0.0/16")]);
Ok(())
}
/// A route is not sent to a peer whose AS is on its path already, nor
/// anywhere if it says NO_EXPORT; and what we originate goes to everyone
/// beside what was learned.
#[tokio::test]
async fn test_routes_stop_where_they_must() -> Result<()> {
let server = transit(|config| {
config.originate = vec![LocalRoute::new(
prefix("10.99.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
let mut one = peer(&server, 1).await?;
let mut two = peer(&server, 2).await?;
let mut told = Told::default();
told_becomes(&mut two, &mut told, &["10.99.0.0/16"]).await?;
assert_eq!(told.routes["10.99.0.0/16"], [LOCAL_ASN]);
// Through the second peer's AS already; and not for other networks.
announce(&mut one, &["10.1.0.0/16"], &[64701, 64702, 65000], &[]).await?;
announce(&mut one, &["10.2.0.0/16"], &[64701], &[(0xFFFF, 0xFF01)]).await?;
announce(&mut one, &["10.3.0.0/16"], &[64701], &[]).await?;
told_becomes(&mut two, &mut told, &["10.99.0.0/16", "10.3.0.0/16"]).await?;
told_nothing(&mut two).await;
assert_eq!(server.rib.len(), 3, "all three were taken");
Ok(())
}
/// Communities on a route say which peers it is passed on to, and are
/// gone when it is; and every route to a peer gets the marks of `export`.
#[tokio::test]
async fn test_communities_say_who_is_told() -> Result<()> {
let server = transit(|config| {
config.control = Some(Control {
local_asn: LOCAL_ASN,
});
config.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
marks: Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
prepend: 1,
});
config.originate = vec![LocalRoute::new(
prefix("10.99.0.0/16"),
LocalRouteAttrs {
// Our own route, and not for the second peer.
communities: vec![(0, 64702)],
..LocalRouteAttrs::default()
},
)];
})
.await?;
let mut one = peer(&server, 1).await?;
let mut two = peer(&server, 2).await?;
let mut three = peer(&server, 3).await?;
let (mut told_two, mut told_three) = (Told::default(), Told::default());
// To everyone but the second; to the second only; to nobody; and
// with nothing said.
announce(
&mut one,
&["10.1.0.0/16"],
&[64701],
&[(0, 64702), (3356, 3)],
)
.await?;
announce(&mut one, &["10.2.0.0/16"], &[64701], &[(64601, 64702)]).await?;
announce(&mut one, &["10.3.0.0/16"], &[64701], &[(0, 64601)]).await?;
announce(&mut one, &["10.4.0.0/16"], &[64701], &[]).await?;
told_becomes(&mut two, &mut told_two, &["10.2.0.0/16", "10.4.0.0/16"]).await?;
told_becomes(
&mut three,
&mut told_three,
&["10.99.0.0/16", "10.1.0.0/16", "10.4.0.0/16"],
)
.await?;
told_nothing(&mut two).await;
told_nothing(&mut three).await;
// Our AS once for passing it on, and once more by `export`.
assert_eq!(
told_three.routes["10.1.0.0/16"],
[LOCAL_ASN, LOCAL_ASN, 64701]
);
assert_eq!(told_three.routes["10.99.0.0/16"], [LOCAL_ASN, LOCAL_ASN]);
// What was for us to act on is gone, and the router's mark is on.
announce(
&mut one,
&["10.5.0.0/16"],
&[64701],
&[(0, 64702), (3356, 3)],
)
.await?;
let update = three.recv_update().await?;
assert!(
update
.path_attributes
.contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
(3356, 3),
(64601, 7)
]))),
"{update:?}"
);
Ok(())
}
/// Without `learned_routes` a peer is told what we originate and nothing
/// else, as before there was such a thing.
#[tokio::test]
async fn test_nothing_learned_is_passed_on_unless_asked_for() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |config| {
config.originate = vec![LocalRoute::new(
prefix("10.99.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
let mut one = peer(&server, 1).await?;
let mut two = peer(&server, 2).await?;
let mut told = Told::default();
told_becomes(&mut two, &mut told, &["10.99.0.0/16"]).await?;
announce(&mut one, &["10.1.0.0/16"], &[64701], &[]).await?;
server.wait_for_routes(1).await?;
told_nothing(&mut two).await;
Ok(())
}
crates/integration-tests/tests/update_errors.rs
+207
View File
@@ -0,0 +1,207 @@
//! End-to-end tests for errors in UPDATE messages (RFC 7606): most of them
//! cost the routes in the UPDATE, not the session.
use std::net::Ipv4Addr;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ReceivedRoute, TestServer};
use bgp_integration_tests::{prefix, wait_for};
use bgp_packet::errors::{BgpError, UpdateMsgSubcode};
use bgp_packet::message::{LocalPrefPathAttribute, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::metrics::UPDATE_ERRORS;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const WELL_KNOWN: u8 = 0x40;
const OPTIONAL_TRANSITIVE: u8 = 0xc0;
fn attr(flags: u8, type_code: u8, value: &[u8]) -> Vec<u8> {
let mut wire = vec![flags, type_code, value.len() as u8];
wire.extend_from_slice(value);
wire
}
fn origin() -> Vec<u8> {
attr(WELL_KNOWN, 1, &[0])
}
/// AS_SEQUENCE of the peer's AS, four octets wide.
fn as_path() -> Vec<u8> {
attr(WELL_KNOWN, 2, &[2, 1, 0, 0, 0xfd, 0xe9])
}
fn next_hop() -> Vec<u8> {
attr(WELL_KNOWN, 3, &[192, 0, 2, 1])
}
/// A whole UPDATE message announcing `nlri` with `attributes`, as bytes, so
/// that it can be wrong in ways the encoder would not write.
fn update(attributes: &[Vec<u8>], nlri: &[u8]) -> Vec<u8> {
let attributes = attributes.concat();
let mut body = vec![0, 0];
body.extend_from_slice(&(attributes.len() as u16).to_be_bytes());
body.extend_from_slice(&attributes);
body.extend_from_slice(nlri);
let mut wire = vec![0xff; 16];
wire.extend_from_slice(&((19 + body.len()) as u16).to_be_bytes());
wire.push(2);
wire.extend_from_slice(&body);
wire
}
/// 10.1.0.0/16 and 10.2.0.0/16.
const ROUTE_A: [u8; 3] = [16, 10, 1];
const ROUTE_B: [u8; 3] = [16, 10, 2];
async fn established(server: &TestServer, asn: u32) -> Result<TestPeer> {
server.established(asn, Ipv4Addr::new(10, 0, 0, 1)).await
}
/// Waits until the server's Adj-RIB-In holds exactly `prefixes`.
async fn wait_for_routes(server: &TestServer, prefixes: &[&str]) -> Result<Vec<ReceivedRoute>> {
let mut expected: Vec<IpPrefix> = prefixes.iter().map(|p| prefix(p)).collect();
expected.sort_by_key(|p| p.to_string());
wait_for(&format!("the table holds {prefixes:?}"), || {
let mut routes = server.received_routes();
routes.sort_by_key(|r| r.prefix.to_string());
routes
.iter()
.map(|r| &r.prefix)
.eq(expected.iter())
.then_some(routes)
})
.await
}
/// The situation RFC 7606 exists for: a route is in the table, and then an
/// UPDATE for it arrives with a broken attribute. The route goes; the
/// session, and the other routes learned over it, stay.
#[tokio::test]
async fn test_a_malformed_attribute_withdraws_the_route_and_keeps_the_session() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
let handled_before = UPDATE_ERRORS
.with_label_values(&["treat-as-withdraw"])
.get();
let good = [origin(), as_path(), next_hop()];
peer.send_raw(&update(&good, &[ROUTE_A, ROUTE_B].concat()))
.await?;
wait_for_routes(&server, &["10.1.0.0/16", "10.2.0.0/16"]).await?;
// ORIGIN with a value that is not defined.
let bad = [attr(WELL_KNOWN, 1, &[7]), as_path(), next_hop()];
peer.send_raw(&update(&bad, &ROUTE_A)).await?;
wait_for_routes(&server, &["10.2.0.0/16"]).await?;
// The session goes on: the route can be announced again.
peer.send_raw(&update(&good, &ROUTE_A)).await?;
wait_for_routes(&server, &["10.1.0.0/16", "10.2.0.0/16"]).await?;
assert_eq!(server.registry.list().len(), 1);
assert!(
UPDATE_ERRORS
.with_label_values(&["treat-as-withdraw"])
.get()
> handled_before
);
Ok(())
}
/// An announcement without a mandatory attribute is not a route we can use.
#[tokio::test]
async fn test_a_route_without_as_path_is_not_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
peer.send_raw(&update(&[origin(), next_hop()], &ROUTE_A))
.await?;
// A well-formed one after it shows that the first was dealt with.
peer.send_raw(&update(&[origin(), as_path(), next_hop()], &ROUTE_B))
.await?;
wait_for_routes(&server, &["10.2.0.0/16"]).await?;
assert_eq!(server.registry.list().len(), 1);
Ok(())
}
/// A broken attribute that plays no part in choosing a route is dropped,
/// and the route is kept.
#[tokio::test]
async fn test_a_malformed_aggregator_is_dropped_and_the_route_kept() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
let attributes = [
origin(),
as_path(),
next_hop(),
// AGGREGATOR is eight octets on this session.
attr(OPTIONAL_TRANSITIVE, 7, &[0, 0, 0xfd, 0xe9, 192]),
];
peer.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert_eq!(routes[0].attributes.len(), 3);
assert!(
!routes[0]
.attributes
.iter()
.any(|a| matches!(a, PathAttribute::Aggregator(_)))
);
Ok(())
}
/// LOCAL_PREF is a matter between routers of one AS. From an external peer
/// it is dropped; from an internal one it is kept.
#[tokio::test]
async fn test_local_pref_is_kept_only_from_an_internal_peer() -> Result<()> {
let local_pref = attr(WELL_KNOWN, 5, &[0, 0, 0, 200]);
let has_local_pref = |routes: &[ReceivedRoute]| {
routes[0]
.attributes
.contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(200)))
};
let server = TestServer::start(LOCAL_ASN).await?;
let mut external = established(&server, PEER_ASN).await?;
let attributes = [origin(), as_path(), next_hop(), local_pref.clone()];
external.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert!(!has_local_pref(&routes));
let server = TestServer::start(LOCAL_ASN).await?;
let mut internal = established(&server, LOCAL_ASN).await?;
// An internal peer has not added its AS to the path.
let attributes = [origin(), attr(WELL_KNOWN, 2, &[]), next_hop(), local_pref];
internal.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert!(has_local_pref(&routes));
Ok(())
}
/// When the prefixes themselves cannot be read there is nothing to
/// withdraw, and the session is reset as before.
#[tokio::test]
async fn test_an_unreadable_prefix_still_resets_the_session() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
// An IPv4 prefix of length 33.
peer.send_raw(&update(
&[origin(), as_path(), next_hop()],
&[33, 10, 1, 2, 3, 4],
))
.await?;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::UpdateMsg(UpdateMsgSubcode::InvalidNetworkField)
);
server.wait_for_sessions(0).await?;
Ok(())
}
crates/packet/Cargo.toml
+9 -3
View File
@@ -1,7 +1,7 @@
[package]
name = "bgp-packet"
description.workspace = true
description = "Parsing and serialisation of BGP messages"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
@@ -11,10 +11,16 @@ version.workspace = true
[dependencies]
bitfield.workspace = true
bytes.workspace = true
eyre.workspace = true
nom.workspace = true
serde.workspace = true
serde_repr.workspace = true
thiserror.workspace = true
strum.workspace = true
bgp-primitives.workspace = true
[dev-dependencies]
eyre.workspace = true
proptest.workspace = true
[lints]
workspace = true
crates/packet/README.md
+2 -3
View File
@@ -2,6 +2,5 @@
This crate implements parsers and serialisers for the Border Gateway Protocol wire format.
## ⚠️ Current status
This crate is still under development and is not yet ready for use.
It is used by the daemon in `bin/bgp`, and tested with property tests
(`tests/properties.rs`) and fuzz targets (`fuzz/`).
crates/packet/src/attributes.rs
+936
View File
@@ -0,0 +1,936 @@
//! Path attributes: what an UPDATE says about the routes it announces
//! (RFC 4271 §5 and those that added to it), each with how it is read
//! from the wire and written to it.
//!
//! Everything here is re-exported from [`crate::message`], which is where
//! the rest of the tree names it from.
use std::net::{Ipv4Addr, Ipv6Addr};
use bitfield::bitfield;
use bytes::{BufMut, BytesMut};
use nom::Parser;
use nom::number::complete::{be_u8, be_u32};
use serde::Serialize;
use serde_repr::Serialize_repr;
use bgp_primitives::constants::{AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext};
use crate::wire::{
Parsed, list, malformed, parse_afi_safi, parse_asn, put_afi_safi, put_asn, put_length, records,
whole,
};
bitfield! {
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
pub struct PathAttributeFlags(u8);
impl new;
u8;
pub optional, set_optional: 7;
pub transitive, set_transitive: 6;
pub partial, set_partial: 5;
pub extended_length, set_extended_length: 4;
}
/// Path attribute type codes.
pub mod attr_type {
pub const ORIGIN: u8 = 1;
pub const AS_PATH: u8 = 2;
pub const NEXT_HOP: u8 = 3;
pub const MULTI_EXIT_DISC: u8 = 4;
pub const LOCAL_PREF: u8 = 5;
pub const ATOMIC_AGGREGATE: u8 = 6;
pub const AGGREGATOR: u8 = 7;
pub const COMMUNITIES: u8 = 8;
pub const ORIGINATOR_ID: u8 = 9;
pub const CLUSTER_LIST: u8 = 10;
pub const MP_REACH_NLRI: u8 = 14;
pub const MP_UNREACH_NLRI: u8 = 15;
pub const EXTENDED_COMMUNITIES: u8 = 16;
pub const AS4_PATH: u8 = 17;
pub const AS4_AGGREGATOR: u8 = 18;
pub const LARGE_COMMUNITIES: u8 = 32;
pub const ONLY_TO_CUSTOMER: u8 = 35;
}
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub enum PathAttribute {
Origin(OriginPathAttribute),
AsPath(AsPathAttribute),
NextHop(NextHopPathAttribute),
MultiExitDisc(MultiExitDiscPathAttribute),
LocalPref(LocalPrefPathAttribute),
AtomicAggregate(AtomicAggregatePathAttribute),
Aggregator(AggregatorPathAttribute),
Communities(CommunitiesPathAttribute),
MpReachNlri(MpReachNlriPathAttribute),
MpUnreachNlri(MpUnreachNlriPathAttribute),
ExtendedCommunities(ExtendedCommunitiesPathAttribute),
/// The AS path with four-octet AS numbers, for a route that crosses
/// speakers that only handle two-octet ones (RFC 6793 §3).
As4Path(AsPathAttribute),
LargeCommunities(LargeCommunitiesPathAttribute),
/// Only to Customer (RFC 9234 §5): the route has been handed to a
/// customer, a peer or a route server client by the AS with this
/// number, and from there on may only go down to customers.
OnlyToCustomer(u32),
/// ORIGINATOR_ID (RFC 4456 §8): the BGP Identifier of the router in our
/// AS that learned the route from outside, put on by the first router
/// that passes the route from one router of the AS to another.
OriginatorId(Ipv4Addr),
/// CLUSTER_LIST (RFC 4456 §8): the clusters the route has been passed
/// on by inside our AS, the last first.
ClusterList(Vec<Ipv4Addr>),
/// An attribute this implementation does not interpret, kept as it
/// came so that it can be passed on.
Unknown {
flags: PathAttributeFlags,
type_code: u8,
payload: Vec<u8>,
},
}
/// The Optional and Transitive flags that the attribute with this type code
/// is defined to have, for the attributes whose definition we know. A
/// received attribute with other values is in error (RFC 7606 §3.c).
pub(crate) fn defined_flags(type_code: u8) -> Option<(bool, bool)> {
use attr_type::*;
const WELL_KNOWN: (bool, bool) = (false, true);
const OPTIONAL_TRANSITIVE: (bool, bool) = (true, true);
const OPTIONAL_NON_TRANSITIVE: (bool, bool) = (true, false);
Some(match type_code {
ORIGIN | AS_PATH | NEXT_HOP | LOCAL_PREF | ATOMIC_AGGREGATE => WELL_KNOWN,
MULTI_EXIT_DISC | ORIGINATOR_ID | CLUSTER_LIST | MP_REACH_NLRI | MP_UNREACH_NLRI => {
OPTIONAL_NON_TRANSITIVE
}
AGGREGATOR | COMMUNITIES | EXTENDED_COMMUNITIES | AS4_PATH | AS4_AGGREGATOR
| LARGE_COMMUNITIES | ONLY_TO_CUSTOMER => OPTIONAL_TRANSITIVE,
_ => return None,
})
}
/// One attribute as it lies on the wire, split out but not interpreted.
#[derive(Debug, Clone)]
pub struct RawAttribute<'a> {
pub flags: PathAttributeFlags,
pub type_code: u8,
pub value: &'a [u8],
/// The whole attribute, header and value.
pub wire: &'a [u8],
}
/// Why the bytes at the front of a buffer are not an attribute.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NotAnAttribute {
/// Too few octets for flags, type and length.
NoHeader,
/// The length says more than there is.
Truncated {
type_code: u8,
length: usize,
available: usize,
},
}
impl std::fmt::Display for NotAnAttribute {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::NoHeader => f.write_str("too few octets for an attribute"),
Self::Truncated {
type_code,
length,
available,
} => write!(
f,
"attribute {type_code} has length {length} but only {available} octets remain"
),
}
}
}
/// Splits the attribute at the front of `buf` from what follows it.
pub fn split_attribute(buf: &[u8]) -> Result<(RawAttribute<'_>, &[u8]), NotAnAttribute> {
let [flags, type_code, rest @ ..] = buf else {
return Err(NotAnAttribute::NoHeader);
};
let flags = PathAttributeFlags(*flags);
let (length, rest) = match (flags.extended_length(), rest) {
(true, [hi, lo, rest @ ..]) => (usize::from(u16::from_be_bytes([*hi, *lo])), rest),
(false, [length, rest @ ..]) => (usize::from(*length), rest),
_ => return Err(NotAnAttribute::NoHeader),
};
let Some((value, rest)) = rest.split_at_checked(length) else {
return Err(NotAnAttribute::Truncated {
type_code: *type_code,
length,
available: rest.len(),
});
};
let raw = RawAttribute {
flags,
type_code: *type_code,
value,
wire: &buf[..buf.len() - rest.len()],
};
Ok((raw, rest))
}
impl PathAttribute {
pub fn type_code(&self) -> u8 {
use attr_type::*;
match self {
Self::Origin(_) => ORIGIN,
Self::AsPath(_) => AS_PATH,
Self::NextHop(_) => NEXT_HOP,
Self::MultiExitDisc(_) => MULTI_EXIT_DISC,
Self::LocalPref(_) => LOCAL_PREF,
Self::AtomicAggregate(_) => ATOMIC_AGGREGATE,
Self::Aggregator(_) => AGGREGATOR,
Self::Communities(_) => COMMUNITIES,
Self::MpReachNlri(_) => MP_REACH_NLRI,
Self::MpUnreachNlri(_) => MP_UNREACH_NLRI,
Self::ExtendedCommunities(_) => EXTENDED_COMMUNITIES,
Self::As4Path(_) => AS4_PATH,
Self::LargeCommunities(_) => LARGE_COMMUNITIES,
Self::OnlyToCustomer(_) => ONLY_TO_CUSTOMER,
Self::OriginatorId(_) => ORIGINATOR_ID,
Self::ClusterList(_) => CLUSTER_LIST,
Self::Unknown { type_code, .. } => *type_code,
}
}
/// The flags to send the attribute with, but for Extended Length, which
/// depends on how long the value turns out to be.
fn flags(&self) -> PathAttributeFlags {
match self {
// As they came: the Partial bit has to survive (RFC 4271 §5).
Self::Unknown { flags, .. } => flags.clone(),
known => {
let (optional, transitive) = defined_flags(known.type_code())
.expect("every attribute with a variant has defined flags");
PathAttributeFlags::new(optional, transitive, false, false)
}
}
}
/// Parses one attribute, header and value, from the front of `buf`.
/// The flags are not checked against the attribute's definition;
/// [`crate::message::UpdateMessage::decode`] does that for attributes off a session.
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (raw, rest) = split_attribute(buf).map_err(|e| malformed(e.to_string()))?;
let attribute = Self::parse_value(ctx, raw.flags, raw.type_code, raw.value)?;
Ok((rest, attribute))
}
/// Makes the attribute with this type code from its value. `value` is
/// exactly the value, and all of it has to be used. A type code without
/// a variant of its own is kept as bytes.
pub(crate) fn parse_value<'a>(
ctx: &ParserContext,
flags: PathAttributeFlags,
type_code: u8,
value: &'a [u8],
) -> Result<Self, nom::Err<BgpParserError<&'a [u8]>>> {
use attr_type::*;
Ok(match type_code {
ORIGIN => Self::Origin(whole("ORIGIN", OriginPathAttribute::from_wire(value)?)?),
AS_PATH => Self::AsPath(whole("AS_PATH", AsPathAttribute::from_wire(ctx, value)?)?),
NEXT_HOP => Self::NextHop(whole("NEXT_HOP", NextHopPathAttribute::from_wire(value)?)?),
MULTI_EXIT_DISC => Self::MultiExitDisc(whole(
"MULTI_EXIT_DISC",
MultiExitDiscPathAttribute::from_wire(value)?,
)?),
LOCAL_PREF => Self::LocalPref(whole(
"LOCAL_PREF",
LocalPrefPathAttribute::from_wire(value)?,
)?),
ATOMIC_AGGREGATE => Self::AtomicAggregate(whole(
"ATOMIC_AGGREGATE",
(value, AtomicAggregatePathAttribute {}),
)?),
AGGREGATOR => Self::Aggregator(whole(
"AGGREGATOR",
AggregatorPathAttribute::from_wire(ctx, value)?,
)?),
COMMUNITIES => Self::Communities(whole(
"COMMUNITIES",
CommunitiesPathAttribute::from_wire(value)?,
)?),
MP_REACH_NLRI => Self::MpReachNlri(whole(
"MP_REACH_NLRI",
MpReachNlriPathAttribute::from_wire(ctx, value)?,
)?),
MP_UNREACH_NLRI => Self::MpUnreachNlri(whole(
"MP_UNREACH_NLRI",
MpUnreachNlriPathAttribute::from_wire(ctx, value)?,
)?),
EXTENDED_COMMUNITIES => Self::ExtendedCommunities(whole(
"EXTENDED_COMMUNITIES",
ExtendedCommunitiesPathAttribute::from_wire(value)?,
)?),
// Always four octets wide, whatever the session's AS_PATH is.
AS4_PATH => Self::As4Path(whole(
"AS4_PATH",
AsPathAttribute::from_wire_with_width(true, value)?,
)?),
LARGE_COMMUNITIES => Self::LargeCommunities(whole(
"LARGE_COMMUNITIES",
LargeCommunitiesPathAttribute::from_wire(value)?,
)?),
ONLY_TO_CUSTOMER => {
let asn = <[u8; 4]>::try_from(value).map_err(|_| {
malformed(format!(
"Only to Customer is {} octets long, not four",
value.len()
))
})?;
Self::OnlyToCustomer(u32::from_be_bytes(asn))
}
ORIGINATOR_ID => {
let id = <[u8; 4]>::try_from(value).map_err(|_| {
malformed(format!(
"ORIGINATOR_ID is {} octets long, not four",
value.len()
))
})?;
Self::OriginatorId(Ipv4Addr::from(id))
}
CLUSTER_LIST => Self::ClusterList(
records::<4>(value, "CLUSTER_LIST")?
.iter()
.map(|id| Ipv4Addr::from(*id))
.collect(),
),
_ => Self::Unknown {
flags,
type_code,
payload: value.to_vec(),
},
})
}
/// Writes the attribute, header and value, to the end of `out`.
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
// A header with a one-octet length first; whether the value needs
// the two-octet one is known once it is written.
let start = out.len();
let mut flags = self.flags();
out.put_u8(0);
out.put_u8(self.type_code());
let length_at = out.len();
out.put_u8(0);
self.value_to_wire(ctx, out)?;
let length = out.len() - length_at - 1;
if let Ok(short) = u8::try_from(length)
&& !flags.extended_length()
{
out[length_at] = short;
} else {
flags.set_extended_length(true);
// Move the value up by one octet to make room.
out.put_u8(0);
out.copy_within(length_at + 1..length_at + 1 + length, length_at + 2);
put_length(out, length_at, 2, length, u16::MAX.into(), "path attribute")?;
}
out[start] = flags.0;
Ok(())
}
fn value_to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
match self {
Self::Origin(a) => a.to_wire(out),
Self::AsPath(a) => a.to_wire(ctx, out)?,
Self::NextHop(a) => a.to_wire(out),
Self::MultiExitDisc(a) => a.to_wire(out),
Self::LocalPref(a) => a.to_wire(out),
Self::AtomicAggregate(_) => {}
Self::Aggregator(a) => a.to_wire(ctx, out),
Self::Communities(a) => a.to_wire(out),
Self::MpReachNlri(a) => a.to_wire(ctx, out),
Self::MpUnreachNlri(a) => a.to_wire(ctx, out),
Self::ExtendedCommunities(a) => a.to_wire(out),
Self::As4Path(a) => a.to_wire_with_width(true, out)?,
Self::LargeCommunities(a) => a.to_wire(out),
Self::OnlyToCustomer(asn) => out.put_u32(*asn),
Self::OriginatorId(id) => out.put_slice(&id.octets()),
Self::ClusterList(ids) => {
for id in ids {
out.put_slice(&id.octets());
}
}
Self::Unknown { payload, .. } => out.put_slice(payload),
}
Ok(())
}
}
/// An Only to Customer attribute with this AS number, as it is on the
/// wire: for adding to attributes that are kept as bytes.
pub fn only_to_customer_wire(asn: u32) -> [u8; 7] {
const OPTIONAL_TRANSITIVE: u8 = 0xc0;
let [a, b, c, d] = asn.to_be_bytes();
[
OPTIONAL_TRANSITIVE,
attr_type::ONLY_TO_CUSTOMER,
4,
a,
b,
c,
d,
]
}
/// ORIGIN: how the route came into BGP (RFC 4271 §5.1.1).
#[derive(Debug, PartialEq, Eq, Copy, Clone, Serialize_repr)]
#[repr(u8)]
pub enum OriginPathAttribute {
IGP = 0,
EGP = 1,
INCOMPLETE = 2,
}
impl OriginPathAttribute {
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, byte) = be_u8(buf)?;
let origin = match byte {
0 => Self::IGP,
1 => Self::EGP,
2 => Self::INCOMPLETE,
other => return Err(malformed(format!("unknown ORIGIN {other}"))),
};
Ok((buf, origin))
}
pub fn to_wire(&self, out: &mut BytesMut) {
out.put_u8(*self as u8);
}
}
/// AS_PATH: the ASes a route has passed through, as a list of segments
/// (RFC 4271 §4.3). Each segment is a type (1 for AS_SET, 2 for
/// AS_SEQUENCE), a count in one octet, and that many AS numbers.
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct AsPathAttribute {
pub segments: Vec<AsPathSegment>,
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct AsPathSegment {
/// True for an AS_SEQUENCE, false for an AS_SET.
pub ordered: bool,
pub path: Vec<u32>,
}
impl AsPathAttribute {
pub fn from_asns(asns: Vec<u32>) -> PathAttribute {
let segment = AsPathSegment {
ordered: true,
path: asns,
};
PathAttribute::AsPath(AsPathAttribute {
segments: vec![segment],
})
}
/// How long the path is, as RFC 4271 §9.1.2.2 counts: one for each AS
/// of a sequence, one for a whole set.
pub fn hops(&self) -> usize {
self.segments
.iter()
.map(|segment| {
if segment.ordered {
segment.path.len()
} else {
1
}
})
.sum()
}
/// The path as it really is, given what a two-octet session carried:
/// this, with AS_TRANS wherever a number did not fit, and `wide`, the
/// AS4_PATH that has the numbers themselves for the path's tail
/// (RFC 6793 §4.2.3). What `wide` does not cover is kept from the
/// front of this.
///
/// `None` if `wide` is longer than the path, which is not to be used:
/// some speaker on the way that knew nothing of AS4_PATH changed the
/// one and not the other.
pub fn with_as4_path(&self, wide: &AsPathAttribute) -> Option<AsPathAttribute> {
let mut keep = self.hops().checked_sub(wide.hops())?;
let mut segments = Vec::with_capacity(self.segments.len() + wide.segments.len());
for segment in &self.segments {
if keep == 0 {
break;
}
let (hops, taken) = if segment.ordered {
let taken = segment.path.len().min(keep);
(taken, &segment.path[..taken])
} else {
(1, &segment.path[..])
};
keep -= hops;
segments.push(AsPathSegment {
ordered: segment.ordered,
path: taken.to_vec(),
});
}
segments.extend(wide.segments.iter().cloned());
Some(AsPathAttribute { segments })
}
/// Parses an AS_PATH with AS numbers as wide as the session negotiated
/// (RFC 6793).
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
Self::from_wire_with_width(ctx.four_octet_asn, buf)
}
fn from_wire_with_width(four_octet: bool, buf: &[u8]) -> Parsed<'_, Self> {
let parse_segment = |buf| -> Parsed<'_, AsPathSegment> {
let (buf, typ) = be_u8(buf)?;
let (buf, len) = be_u8(buf)?;
// AS_SET and AS_SEQUENCE. The confederation segment types of
// RFC 5065 are an error from a peer outside a confederation,
// and this speaker is never inside one.
if typ != 1 && typ != 2 {
return Err(malformed(format!(
"unrecognised AS path segment type {typ}"
)));
}
if len == 0 {
return Err(malformed("AS path segment with no AS numbers"));
}
let count = usize::from(len);
let (buf, path) =
nom::multi::many_m_n(count, count, |buf| parse_asn(four_octet, buf)).parse(buf)?;
Ok((
buf,
AsPathSegment {
ordered: typ == 2,
path,
},
))
};
let (buf, segments) = list(buf, parse_segment)?;
Ok((buf, Self { segments }))
}
/// Writes the path with AS numbers as wide as the session negotiated.
/// On a two-octet session an AS number that does not fit is written as
/// AS_TRANS (RFC 6793 §4.2.2); [`AsPathAttribute::as4_path`] is what
/// carries the real numbers alongside.
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
self.to_wire_with_width(ctx.four_octet_asn, out)
}
fn to_wire_with_width(&self, four_octet: bool, out: &mut BytesMut) -> Result<(), EncodeError> {
for segment in &self.segments {
out.put_u8(if segment.ordered { 2 } else { 1 });
let count = u8::try_from(segment.path.len()).map_err(|_| EncodeError::TooLong {
what: "AS path segment",
len: segment.path.len(),
max: u8::MAX.into(),
})?;
out.put_u8(count);
for asn in &segment.path {
put_asn(four_octet, *asn, out);
}
}
Ok(())
}
/// The AS4_PATH to send with this path on a two-octet session, if one is
/// needed: it is when some AS number does not fit in two octets and so
/// goes out as AS_TRANS in the AS_PATH itself (RFC 6793 §4.2.2).
pub fn as4_path(&self) -> Option<PathAttribute> {
self.segments
.iter()
.flat_map(|segment| &segment.path)
.any(|asn| u16::try_from(*asn).is_err())
.then(|| PathAttribute::As4Path(self.clone()))
}
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct NextHopPathAttribute(pub Ipv4Addr);
impl NextHopPathAttribute {
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, address) = be_u32(buf)?;
Ok((buf, Self(Ipv4Addr::from(address))))
}
pub fn to_wire(&self, out: &mut BytesMut) {
out.put_u32(self.0.into());
}
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct MultiExitDiscPathAttribute(pub u32);
impl MultiExitDiscPathAttribute {
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, value) = be_u32(buf)?;
Ok((buf, Self(value)))
}
pub fn to_wire(&self, out: &mut BytesMut) {
out.put_u32(self.0);
}
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct LocalPrefPathAttribute(pub u32);
impl LocalPrefPathAttribute {
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, value) = be_u32(buf)?;
Ok((buf, Self(value)))
}
pub fn to_wire(&self, out: &mut BytesMut) {
out.put_u32(self.0);
}
}
/// ATOMIC_AGGREGATE: a marker with no value (RFC 4271 §5.1.6).
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct AtomicAggregatePathAttribute {}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct AggregatorPathAttribute {
pub asn: u32,
pub ip: Ipv4Addr,
}
impl AggregatorPathAttribute {
/// The AS number is as wide as the session's AS numbers (RFC 6793), so
/// the attribute is six or eight octets.
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, asn) = parse_asn(ctx.four_octet_asn, buf)?;
let (buf, ip) = be_u32(buf)?;
Ok((
buf,
Self {
asn,
ip: Ipv4Addr::from(ip),
},
))
}
/// On a two-octet session an AS number that does not fit is written as
/// AS_TRANS (RFC 6793 §4.2.2).
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
put_asn(ctx.four_octet_asn, self.asn, out);
out.put_u32(self.ip.into());
}
}
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct CommunitiesPathAttribute(pub Vec<(u16, u16)>);
impl CommunitiesPathAttribute {
/// Reads all of `buf` as communities, four octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let values = records::<4>(buf, "COMMUNITIES")?
.iter()
.map(|[a, b, c, d]| (u16::from_be_bytes([*a, *b]), u16::from_be_bytes([*c, *d])))
.collect();
Ok((&buf[buf.len()..], CommunitiesPathAttribute(values)))
}
pub fn to_wire(&self, out: &mut BytesMut) {
for value in &self.0 {
out.put_u16(value.0);
out.put_u16(value.1);
}
}
}
/// EXTENDED COMMUNITIES (RFC 4360).
#[derive(Debug, PartialEq, Eq, Clone, Serialize)]
pub struct ExtendedCommunitiesPathAttribute {
pub extended_communities: Vec<ExtendedCommunity>,
}
impl ExtendedCommunitiesPathAttribute {
/// Reads all of `buf` as extended communities, eight octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let communities = records::<8>(buf, "EXTENDED_COMMUNITIES")?;
Ok((
&buf[buf.len()..],
Self {
extended_communities: communities.iter().copied().map(ExtendedCommunity).collect(),
},
))
}
pub fn to_wire(&self, out: &mut BytesMut) {
for community in &self.extended_communities {
out.put_slice(&community.0);
}
}
}
/// One extended community: eight octets, of which the first is the type
/// and, for most types, the second a sub-type (RFC 4360 §2). How the rest
/// divides into an administrator and a value depends on the type, and is
/// left to whoever acts on a particular community.
#[derive(Debug, PartialEq, Eq, Clone, Copy, Serialize)]
pub struct ExtendedCommunity(pub [u8; 8]);
impl ExtendedCommunity {
pub fn typ(&self) -> u8 {
self.0[0]
}
pub fn sub_typ(&self) -> u8 {
self.0[1]
}
}
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct LargeCommunitiesPathAttribute {
pub communities: Vec<LargeCommunity>,
}
impl LargeCommunitiesPathAttribute {
/// Reads all of `buf` as large communities, twelve octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let communities = records::<12>(buf, "LARGE_COMMUNITIES")?
.iter()
.map(|community| {
let (_, community) =
LargeCommunity::from_wire(community).expect("twelve octets are one");
community
})
.collect();
Ok((&buf[buf.len()..], Self { communities }))
}
pub fn to_wire(&self, out: &mut BytesMut) {
for community in &self.communities {
community.to_wire(out);
}
}
}
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct LargeCommunity {
pub global_admin: u32,
pub data_1: u32,
pub data_2: u32,
}
impl LargeCommunity {
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, global_admin) = be_u32(buf)?;
let (buf, data_1) = be_u32(buf)?;
let (buf, data_2) = be_u32(buf)?;
Ok((
buf,
Self {
global_admin,
data_1,
data_2,
},
))
}
pub fn to_wire(&self, out: &mut BytesMut) {
out.put_u32(self.global_admin);
out.put_u32(self.data_1);
out.put_u32(self.data_2);
}
}
/// The next hop of routes in MP_REACH_NLRI.
#[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq, Hash)]
pub enum NlriNextHop {
Ipv4(Ipv4Addr),
/// A global IPv6 address.
Ipv6(Ipv6Addr),
/// A global IPv6 address and a link-local one (RFC 2545 §3).
Ipv6WithLl {
global: Ipv6Addr,
link_local: Ipv6Addr,
},
}
impl std::fmt::Display for NlriNextHop {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
NlriNextHop::Ipv4(addr) => write!(f, "{addr}"),
NlriNextHop::Ipv6(addr) => write!(f, "{addr}"),
NlriNextHop::Ipv6WithLl { global, link_local } => {
write!(f, "{global} (link-local {link_local})")
}
}
}
}
impl NlriNextHop {
/// Reads a next hop that is all of `bytes`. `None` if they are not the
/// length of one.
pub fn from_bytes(bytes: &[u8]) -> Option<Self> {
let v6 = |bytes: &[u8]| <[u8; 16]>::try_from(bytes).ok().map(Ipv6Addr::from);
match bytes.len() {
4 => <[u8; 4]>::try_from(bytes)
.ok()
.map(|a| Self::Ipv4(a.into())),
16 => v6(bytes).map(Self::Ipv6),
32 => Some(Self::Ipv6WithLl {
global: v6(&bytes[..16])?,
link_local: v6(&bytes[16..])?,
}),
_ => None,
}
}
pub fn to_wire(&self, out: &mut BytesMut) {
match self {
NlriNextHop::Ipv4(address) => out.put_slice(&address.octets()),
NlriNextHop::Ipv6(address) => out.put_slice(&address.octets()),
NlriNextHop::Ipv6WithLl { global, link_local } => {
out.put_slice(&global.octets());
out.put_slice(&link_local.octets());
}
}
}
/// How many octets the next hop is on the wire.
pub fn wire_len(&self) -> u8 {
match self {
NlriNextHop::Ipv4(_) => 4,
NlriNextHop::Ipv6(_) => 16,
NlriNextHop::Ipv6WithLl { .. } => 32,
}
}
}
/// Parses one prefix as an UPDATE names it: with a path identifier in
/// front if the session uses them for the family (RFC 7911 §3).
fn parse_nlri(version: IpVersion, with_path_id: bool, buf: &[u8]) -> Parsed<'_, Nlri> {
let (buf, path_id) = if with_path_id { be_u32(buf)? } else { (buf, 0) };
let (buf, prefix_len) = be_u8(buf)?;
let byte_len = usize::from(prefix_len.div_ceil(8));
let (buf, prefix_bytes) = nom::bytes::take(byte_len).parse(buf)?;
let prefix =
IpPrefix::new(version, prefix_bytes, prefix_len).map_err(|e| malformed(e.to_string()))?;
Ok((buf, Nlri { prefix, path_id }))
}
/// Parses a field that is nothing but prefixes, to its end.
pub(crate) fn parse_nlri_list(
version: IpVersion,
with_path_id: bool,
buf: &[u8],
) -> Parsed<'_, Vec<Nlri>> {
list(buf, |buf| parse_nlri(version, with_path_id, buf))
}
/// Writes prefixes as `parse_nlri_list` reads them: for each, its path
/// identifier if the session uses them for the family (RFC 7911 §3), its
/// length in bits, and the bytes that length covers.
pub fn put_nlri_list(nlri: &[Nlri], with_path_id: bool, out: &mut BytesMut) {
for n in nlri {
if with_path_id {
out.put_u32(n.path_id);
}
out.put_u8(n.prefix.length());
out.put(n.prefix.bytes());
}
}
/// The IP version of the prefixes that make up the NLRI of `family`.
///
/// Any family can be named in MP_REACH_NLRI and MP_UNREACH_NLRI, but only
/// unicast and multicast routes of IPv4 and IPv6 are plain prefixes, and
/// those are the only NLRI this parser reads. A peer does not send routes
/// of a family that was not negotiated, so the others do not come up.
fn ip_version_of_nlri<'a>(
family: AfiSafi,
) -> Result<IpVersion, nom::Err<BgpParserError<&'a [u8]>>> {
let unsupported = || malformed(format!("NLRI of {family} is not supported"));
if family.safi != Safi::UNICAST && family.safi != Safi::MULTICAST {
return Err(unsupported());
}
IpVersion::try_from(family.afi).map_err(|_| unsupported())
}
/// MP_REACH_NLRI: routes of a family other than the one in the message
/// body, with their next hop (RFC 4760 §3).
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct MpReachNlriPathAttribute {
pub family: AfiSafi,
pub next_hop: NlriNextHop,
pub prefixes: Vec<Nlri>,
}
impl MpReachNlriPathAttribute {
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, family) = parse_afi_safi(buf)?;
let version = ip_version_of_nlri(family)?;
let (buf, next_hop_len) = be_u8(buf)?;
let (buf, next_hop) = nom::bytes::take(usize::from(next_hop_len)).parse(buf)?;
// For IPv6 routes, a global address, optionally followed by a
// link-local one (RFC 2545). For IPv4 routes an IPv4 address, or
// the same as for IPv6: routes over a link that has only IPv6
// addresses (RFC 8950). Which of the two it is shows in its length.
let next_hop = NlriNextHop::from_bytes(next_hop)
.filter(|next_hop| {
version == IpVersion::Ipv4 || !matches!(next_hop, NlriNextHop::Ipv4(_))
})
.ok_or_else(|| {
malformed(format!(
"a next hop of {next_hop_len} octets is not valid for {family}"
))
})?;
// Reserved octet (formerly the SNPA count), for every family.
let (buf, _) = be_u8(buf)?;
let (buf, prefixes) = parse_nlri_list(version, ctx.has_add_path(family), buf)?;
Ok((
buf,
Self {
family,
next_hop,
prefixes,
},
))
}
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
put_afi_safi(self.family, out);
out.put_u8(self.next_hop.wire_len());
self.next_hop.to_wire(out);
out.put_u8(0);
put_nlri_list(&self.prefixes, ctx.has_add_path(self.family), out);
}
}
/// MP_UNREACH_NLRI: withdrawals of routes of a family other than the one
/// in the message body (RFC 4760 §4).
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct MpUnreachNlriPathAttribute {
pub family: AfiSafi,
pub prefixes: Vec<Nlri>,
}
impl MpUnreachNlriPathAttribute {
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, family) = parse_afi_safi(buf)?;
let version = ip_version_of_nlri(family)?;
let (buf, prefixes) = parse_nlri_list(version, ctx.has_add_path(family), buf)?;
Ok((buf, Self { family, prefixes }))
}
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
put_afi_safi(self.family, out);
put_nlri_list(&self.prefixes, ctx.has_add_path(self.family), out);
}
}
crates/packet/src/community.rs
+98
View File
@@ -0,0 +1,98 @@
//! Parsing and formatting of community values as text, the way they are
//! written in configuration and shown in status output.
//!
//! Standard communities are written `ASN:VALUE` (e.g. `65000:100`) or as one
//! of the well-known names from RFC 1997 / RFC 7999 / RFC 8326. Large
//! communities are written `GLOBAL:LOCAL1:LOCAL2` (RFC 8092).
use thiserror::Error;
use crate::message::LargeCommunity;
/// Text that is not a community.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum CommunityError {
#[error("community {0:?} must be ASN:VALUE with 16-bit numbers, or a well-known name")]
Standard(String),
#[error("large community {0:?} must be GLOBAL:LOCAL1:LOCAL2 with 32-bit numbers")]
Large(String),
}
/// Well-known standard communities, by name.
pub const WELL_KNOWN: &[(&str, (u16, u16))] = &[
("GRACEFUL_SHUTDOWN", (0xFFFF, 0x0000)),
("BLACKHOLE", (0xFFFF, 0x029A)),
("NO_EXPORT", (0xFFFF, 0xFF01)),
("NO_ADVERTISE", (0xFFFF, 0xFF02)),
("NO_EXPORT_SUBCONFED", (0xFFFF, 0xFF03)),
("NOPEER", (0xFFFF, 0xFF04)),
];
pub fn parse_standard(s: &str) -> Result<(u16, u16), CommunityError> {
let s = s.trim();
let invalid = || CommunityError::Standard(s.to_string());
if let Some((_, value)) = WELL_KNOWN
.iter()
.find(|(name, _)| name.eq_ignore_ascii_case(s))
{
return Ok(*value);
}
let (high, low) = s.split_once(':').ok_or_else(invalid)?;
let parse = |part: &str| part.trim().parse::<u16>().map_err(|_| invalid());
Ok((parse(high)?, parse(low)?))
}
pub fn format_standard((high, low): (u16, u16)) -> String {
format!("{high}:{low}")
}
pub fn parse_large(s: &str) -> Result<LargeCommunity, CommunityError> {
let s = s.trim();
let invalid = || CommunityError::Large(s.to_string());
let parts: Vec<&str> = s.split(':').collect();
if parts.len() != 3 {
return Err(invalid());
}
let parse = |part: &str| part.trim().parse::<u32>().map_err(|_| invalid());
Ok(LargeCommunity {
global_admin: parse(parts[0])?,
data_1: parse(parts[1])?,
data_2: parse(parts[2])?,
})
}
pub fn format_large(c: &LargeCommunity) -> String {
format!("{}:{}:{}", c.global_admin, c.data_1, c.data_2)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn standard_roundtrip_and_names() {
assert_eq!(parse_standard("65000:100").unwrap(), (65000, 100));
assert_eq!(parse_standard(" no_export ").unwrap(), (65535, 65281));
assert_eq!(parse_standard("BLACKHOLE").unwrap(), (65535, 666));
assert_eq!(format_standard((65000, 100)), "65000:100");
assert!(parse_standard("65000").is_err());
assert!(parse_standard("70000:1").is_err());
assert!(parse_standard("bogus").is_err());
}
#[test]
fn large_roundtrip() {
let c = parse_large("65000:1:2").unwrap();
assert_eq!(
c,
LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2
}
);
assert_eq!(format_large(&c), "65000:1:2");
assert!(parse_large("65000:1").is_err());
assert!(parse_large("a:b:c").is_err());
}
}
crates/packet/src/errors.rs
+74 -96
View File
@@ -1,68 +1,79 @@
use serde::{Deserialize, Serialize};
use serde::Serialize;
use strum::FromRepr;
use thiserror::Error;
#[derive(Debug, Error)]
#[error("Unknown BGP error code {code} subcode {subcode}")]
pub struct UnknownErrorCode {
pub code: u8,
pub subcode: u8,
/// A field holds a number that is not one of the values defined for it.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)]
#[error("invalid {what} value {value}")]
pub struct InvalidValue {
pub what: &'static str,
pub value: u32,
}
#[derive(Debug, Clone, PartialEq, Eq, Error, Serialize, Deserialize)]
#[repr(u8)]
/// A message that could not be decoded, together with the NOTIFICATION that
/// reports the problem to the peer (RFC 4271 §6).
#[derive(Debug, Clone, PartialEq, Eq, Error)]
#[error("{detail}")]
pub struct DecodeError {
/// The error code and subcode to send.
pub notification: BgpError,
/// The Data field of that NOTIFICATION, where the RFC defines one.
pub data: Vec<u8>,
/// What was wrong, for the log.
pub detail: String,
}
/// The error a NOTIFICATION reports: its code and subcode (RFC 4271 §4.5).
#[derive(Debug, Clone, PartialEq, Eq, Error, Serialize)]
pub enum BgpError {
#[error("Message header error")]
MsgHeader(MsgHeaderSubcode) = 1,
#[error("Message header error: {0}")]
MsgHeader(MsgHeaderSubcode),
#[error("Open message error")]
OpenMsg(OpenMsgSubcode) = 2,
#[error("Open message error: {0}")]
OpenMsg(OpenMsgSubcode),
#[error("Update message error")]
UpdateMsg(UpdateMsgSubcode) = 3,
#[error("Update message error: {0}")]
UpdateMsg(UpdateMsgSubcode),
#[error("Hold timer expired")]
HoldTimer = 4,
HoldTimer,
#[error("Finite state machine error")]
FsmError(FsmSubcode) = 5,
#[error("Finite state machine error: {0}")]
FsmError(FsmSubcode),
#[error("Cease")]
Cease(CeaseSubcode) = 6,
#[error("Cease: {0}")]
Cease(CeaseSubcode),
#[error("Route refresh message error")]
RouteRefresh(RouteRefreshSubcode) = 7,
#[error("Route refresh message error: {0}")]
RouteRefresh(RouteRefreshSubcode),
#[error("Send hold timer expired")]
SendHoldTimer = 8,
SendHoldTimer,
/// A code or subcode this implementation has no name for. A peer may
/// send any: new ones are assigned, and zero ("unspecific") is always
/// allowed. The session ends the same way, and the numbers are kept
/// for the log.
#[error("error code {code} subcode {subcode}")]
Other { code: u8, subcode: u8 },
}
impl BgpError {
pub fn from_code_subcode(code: u8, subcode: u8) -> Result<Self, UnknownErrorCode> {
let err = || UnknownErrorCode { code, subcode };
match code {
1 => Ok(Self::MsgHeader(
MsgHeaderSubcode::try_from(subcode).map_err(|_| err())?,
)),
2 => Ok(Self::OpenMsg(
OpenMsgSubcode::try_from(subcode).map_err(|_| err())?,
)),
3 => Ok(Self::UpdateMsg(
UpdateMsgSubcode::try_from(subcode).map_err(|_| err())?,
)),
4 => Ok(Self::HoldTimer),
5 => Ok(Self::FsmError(
FsmSubcode::try_from(subcode).map_err(|_| err())?,
)),
6 => Ok(Self::Cease(
CeaseSubcode::try_from(subcode).map_err(|_| err())?,
)),
7 => Ok(Self::RouteRefresh(
RouteRefreshSubcode::try_from(subcode).map_err(|_| err())?,
)),
8 => Ok(Self::SendHoldTimer),
_ => Err(err()),
}
/// The error for a code and subcode off the wire. Every pair has one,
/// so that a peer's NOTIFICATION can always be read.
pub fn from_code_subcode(code: u8, subcode: u8) -> Self {
let known = match code {
1 => MsgHeaderSubcode::from_repr(subcode).map(Self::MsgHeader),
2 => OpenMsgSubcode::from_repr(subcode).map(Self::OpenMsg),
3 => UpdateMsgSubcode::from_repr(subcode).map(Self::UpdateMsg),
4 => (subcode == 0).then_some(Self::HoldTimer),
5 => FsmSubcode::from_repr(subcode).map(Self::FsmError),
6 => CeaseSubcode::from_repr(subcode).map(Self::Cease),
7 => RouteRefreshSubcode::from_repr(subcode).map(Self::RouteRefresh),
8 => (subcode == 0).then_some(Self::SendHoldTimer),
_ => None,
};
known.unwrap_or(Self::Other { code, subcode })
}
pub fn code_subcode(&self) -> (u8, u8) {
@@ -75,11 +86,12 @@ impl BgpError {
Self::Cease(s) => (6, *s as u8),
Self::RouteRefresh(s) => (7, *s as u8),
Self::SendHoldTimer => (8, 0),
Self::Other { code, subcode } => (*code, *subcode),
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum MsgHeaderSubcode {
#[error("Connection not synchronized")]
@@ -92,16 +104,12 @@ pub enum MsgHeaderSubcode {
BadMessageType = 3,
}
impl TryFrom<u8> for MsgHeaderSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum OpenMsgSubcode {
#[error("Unspecific")]
Unspecific = 0,
#[error("Unsupported BGP version number")]
UnsupportedVersion = 1,
@@ -120,18 +128,12 @@ pub enum OpenMsgSubcode {
#[error("Unsupported capability")]
UnsupportedCapability = 7,
/// RFC 9234 §4.2.
#[error("Role mismatch")]
RoleMismatch = 8,
RoleMismatch = 11,
}
impl TryFrom<u8> for OpenMsgSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum UpdateMsgSubcode {
#[error("Malformed attribute list")]
@@ -165,16 +167,13 @@ pub enum UpdateMsgSubcode {
MalformedAsPath = 11,
}
impl TryFrom<u8> for UpdateMsgSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum FsmSubcode {
/// RFC 6608: what speakers from before it send, and FRR still does.
#[error("unspecified")]
Unspecified = 0,
#[error("Received an unexpected message in OpenSent state")]
UnexpectedOpenSent = 1,
@@ -185,14 +184,7 @@ pub enum FsmSubcode {
UnexpectedEstablished = 3,
}
impl TryFrom<u8> for FsmSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum CeaseSubcode {
#[error("Maximum number of prefixes reached")]
@@ -226,23 +218,9 @@ pub enum CeaseSubcode {
BfdDown = 10,
}
impl TryFrom<u8> for CeaseSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum RouteRefreshSubcode {
#[error("Invalid message length")]
InvalidLength = 1,
}
impl TryFrom<u8> for RouteRefreshSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
crates/packet/src/lib.rs
+4
View File
@@ -1,3 +1,7 @@
pub mod attributes;
pub mod community;
pub mod errors;
pub mod message;
pub mod open;
pub mod update_decode;
mod wire;
crates/packet/src/message.rs
+877 -1498
View File
File diff suppressed because it is too large. Load diff
crates/packet/src/open.rs
+894 -606
View File
File diff suppressed because it is too large. Load diff
crates/packet/src/update_decode.rs
+1219
View File
File diff suppressed because it is too large. Load diff
crates/packet/src/wire.rs
+198
View File
@@ -0,0 +1,198 @@
//! What the readers and writers of the wire format share: the shape of a
//! parser's result and its errors, the fields that several messages and
//! attributes have in common, and the lengths that are written before what
//! they count.
use bytes::{BufMut, BytesMut};
use nom::Err::Failure;
use nom::IResult;
use nom::number::complete::{be_u8, be_u16, be_u32};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext};
use crate::message::AS_TRANS;
/// A parser's result: what is left of the input, and what was read.
pub(crate) type Parsed<'a, T> = IResult<&'a [u8], T, BgpParserError<&'a [u8]>>;
/// The error for input that does not follow the format; `detail` says how.
pub(crate) fn malformed<'a>(detail: impl Into<String>) -> nom::Err<BgpParserError<&'a [u8]>> {
Failure(BgpParserError::Malformed(detail.into()))
}
/// Parses `buf` to its end as a list of `item`s. Unlike nom's `many0`,
/// which stops quietly at an item it cannot read, an error in an item is an
/// error of the list.
pub(crate) fn list<'a, T>(
mut buf: &'a [u8],
item: impl Fn(&'a [u8]) -> Parsed<'a, T>,
) -> Parsed<'a, Vec<T>> {
let mut items = Vec::new();
while !buf.is_empty() {
let (rest, parsed) = item(buf)?;
items.push(parsed);
buf = rest;
}
Ok((buf, items))
}
/// A parsed value, provided the parser used all of the input. `what` names
/// the input in the error.
pub(crate) fn whole<T>(
what: impl std::fmt::Display,
(rest, parsed): (&[u8], T),
) -> Result<T, nom::Err<BgpParserError<&[u8]>>> {
if rest.is_empty() {
Ok(parsed)
} else {
Err(malformed(format!(
"{} octets left over in {what}",
rest.len()
)))
}
}
/// `value` as a list of one or more records of `N` octets each: how the
/// community attributes and CLUSTER_LIST are laid out. An empty list is
/// malformed (RFC 7606 §7.8 for COMMUNITIES, and the others alike). `what`
/// names the attribute in the error.
pub(crate) fn records<'a, const N: usize>(
value: &'a [u8],
what: &str,
) -> Result<&'a [[u8; N]], nom::Err<BgpParserError<&'a [u8]>>> {
let (records, rest) = value.as_chunks::<N>();
if records.is_empty() || !rest.is_empty() {
return Err(malformed(format!(
"{what} is {} octets long, not a non-empty multiple of {N}",
value.len()
)));
}
Ok(records)
}
/// Parses an AS number as wide as the session's: four octets or two
/// (RFC 6793).
pub(crate) fn parse_asn(four_octet: bool, buf: &[u8]) -> Parsed<'_, u32> {
if four_octet {
be_u32(buf)
} else {
be_u16(buf).map(|(buf, asn)| (buf, u32::from(asn)))
}
}
/// Writes an AS number as wide as the session's. On a two-octet session
/// one that does not fit is written as AS_TRANS (RFC 6793 §4.2.2).
pub(crate) fn put_asn(four_octet: bool, asn: u32, out: &mut BytesMut) {
if four_octet {
out.put_u32(asn);
} else {
out.put_u16(u16::try_from(asn).unwrap_or(AS_TRANS));
}
}
/// Whether the prefixes in the body of an UPDATE carry path identifiers
/// on a session with `ctx`. The body holds IPv4 unicast routes and nothing
/// else (RFC 4271 §4.3), so that is the family whose ADD-PATH agreement
/// decides it; every other family's prefixes are in MP_REACH_NLRI and
/// MP_UNREACH_NLRI, which name their family.
pub(crate) fn body_has_path_id(ctx: &ParserContext) -> bool {
ctx.has_add_path(AfiSafi::IPV4_UNICAST)
}
/// Parses an address family as the multiprotocol attributes and most
/// capabilities name it: a two-octet AFI and a one-octet SAFI.
pub(crate) fn parse_afi_safi(buf: &[u8]) -> Parsed<'_, AfiSafi> {
let (buf, afi) = be_u16(buf)?;
let (buf, safi) = be_u8(buf)?;
Ok((buf, AfiSafi::new(Afi(afi), Safi(safi))))
}
/// Writes an address family as [`parse_afi_safi`] reads it.
pub(crate) fn put_afi_safi(family: AfiSafi, out: &mut BytesMut) {
out.put_u16(family.afi.0);
out.put_u8(family.safi.0);
}
/// Writes `length` into the field of `width` octets reserved at `at`, if
/// it is no more than `max`. `what` names the field in the error.
pub(crate) fn put_length(
out: &mut BytesMut,
at: usize,
width: usize,
length: usize,
max: usize,
what: &'static str,
) -> Result<(), EncodeError> {
if length > max {
return Err(EncodeError::TooLong {
what,
len: length,
max,
});
}
let bytes = length.to_be_bytes();
out[at..at + width].copy_from_slice(&bytes[bytes.len() - width..]);
Ok(())
}
/// Fills in the field of `width` octets reserved at `at` with how much
/// has been written after it, if that is no more than `max`: for a length
/// that is known only once what it counts has been written. `what` names
/// the field in the error.
pub(crate) fn fill_length(
out: &mut BytesMut,
at: usize,
width: usize,
max: usize,
what: &'static str,
) -> Result<(), EncodeError> {
let length = out.len() - at - width;
put_length(out, at, width, length, max, what)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_length_is_filled_in_after_what_it_counts() {
let mut out = BytesMut::new();
out.put_u8(0xaa);
let at = out.len();
out.put_u16(0);
out.put_slice(&[1, 2, 3]);
fill_length(&mut out, at, 2, u16::MAX.into(), "field").unwrap();
assert_eq!(&out[..], [0xaa, 0, 3, 1, 2, 3]);
let mut out = BytesMut::new();
out.put_u8(0);
out.put_slice(&[0; 300]);
assert_eq!(
fill_length(&mut out, 0, 1, u8::MAX.into(), "field"),
Err(EncodeError::TooLong {
what: "field",
len: 300,
max: 255,
})
);
}
#[test]
fn records_are_a_whole_number_of_them_and_at_least_one() {
assert_eq!(
records::<4>(&[1, 2, 3, 4, 5, 6, 7, 8], "X").unwrap().len(),
2
);
for value in [&[][..], &[1, 2, 3][..], &[1, 2, 3, 4, 5][..]] {
let error = records::<4>(value, "X").unwrap_err();
assert_eq!(
crate::message::describe(&error),
format!(
"X is {} octets long, not a non-empty multiple of 4",
value.len()
)
);
}
}
}
crates/packet/tests/properties.rs
+763
View File
@@ -0,0 +1,763 @@
//! Property tests for the wire codec.
//!
//! Two things are checked for every kind of message:
//!
//! - **Round trip.** A message that can be written can be read back, comes
//! back equal.
//! - **Hostile input.** Whatever bytes arrive, decoding returns a message or
//! an error: it never panics. What it does accept can be shown, written
//! again, and read back as the same message.
//!
//! The second is what a peer can do to us, so it is run on three kinds of
//! input: plain noise, noise behind a valid header (so it reaches the body
//! parsers), and valid messages with a few bytes damaged (so it reaches deep
//! into them).
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::errors::BgpError;
use bgp_packet::message::{
AggregatorPathAttribute, AsPathAttribute, AsPathSegment, AtomicAggregatePathAttribute,
BGP_HEADER_LEN, BGP_MAX_MSG_LEN, CommunitiesPathAttribute, ExtendedCommunitiesPathAttribute,
ExtendedCommunity, LargeCommunitiesPathAttribute, LargeCommunity, LocalPrefPathAttribute,
Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute, MultiExitDiscPathAttribute,
NextHopPathAttribute, NlriNextHop, NotificationMessage, OriginPathAttribute, PathAttribute,
PathAttributeFlags, RouteRefreshMessage, UpdateMessage,
};
use bgp_packet::open::{
AddPathEntry, BgpRole, BgpSecHeader, Capability, ExtendedNextHopEntry, GracefulRestartEntry,
LongLivedGracefulRestartEntry, MultiLabelEntry, OpenMessage, OpenOption, OrfFamily, OrfType,
OutboundRouteFilterEntry, SendReceive,
};
use bgp_primitives::constants::{Afi, AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::{EncodeError, ParserContext};
use bytes::BytesMut;
use proptest::collection::vec;
use proptest::prelude::*;
/// What a session settled that the codec depends on.
#[derive(Debug, Clone, Copy)]
struct Session {
four_octet_asn: bool,
ibgp: bool,
/// Whether prefixes carry path identifiers (RFC 7911), for IPv4 and
/// for IPv6 unicast.
add_path: [bool; 2],
}
impl Session {
fn ctx(self) -> ParserContext {
ParserContext {
four_octet_asn: self.four_octet_asn,
ibgp: self.ibgp,
add_path: [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST]
.into_iter()
.zip(self.add_path)
.filter_map(|(family, in_use)| in_use.then_some(family))
.collect(),
}
}
fn has_add_path(self, version: IpVersion, safi: Safi) -> bool {
safi == Safi::UNICAST
&& match version {
IpVersion::Ipv4 => self.add_path[0],
IpVersion::Ipv6 => self.add_path[1],
}
}
}
fn session() -> impl Strategy<Value = Session> {
any::<(bool, bool, [bool; 2])>().prop_map(|(four_octet_asn, ibgp, add_path)| Session {
four_octet_asn,
ibgp,
add_path,
})
}
// ---------------------------------------------------------------------------
// Generators
// ---------------------------------------------------------------------------
fn prefix(version: IpVersion) -> impl Strategy<Value = IpPrefix> {
let max_len = match version {
IpVersion::Ipv4 => 32u8,
IpVersion::Ipv6 => 128,
};
(0..=max_len, any::<[u8; 16]>()).prop_map(move |(length, bytes)| {
IpPrefix::new(version, bytes, length).expect("16 bytes are enough for any prefix")
})
}
/// Prefixes as an UPDATE names them. Without ADD-PATH for the family there
/// is no identifier on the wire, and it reads back as zero.
fn nlri(session: Session, version: IpVersion, safi: Safi) -> impl Strategy<Value = Vec<Nlri>> {
let with_path_id = session.has_add_path(version, safi);
vec(
(prefix(version), any::<u32>()).prop_map(move |(prefix, path_id)| Nlri {
prefix,
path_id: if with_path_id { path_id } else { 0 },
}),
0..5,
)
}
fn afi() -> impl Strategy<Value = Afi> {
any::<u16>().prop_map(Afi)
}
fn safi() -> impl Strategy<Value = Safi> {
any::<u8>().prop_map(Safi)
}
/// AS numbers that fit the session: any on a four-octet one, two-octet ones
/// otherwise (wider ones are replaced by AS_TRANS on the wire, by design).
fn asn(four_octet_asn: bool) -> BoxedStrategy<u32> {
if four_octet_asn {
any::<u32>().boxed()
} else {
any::<u16>().prop_map(u32::from).boxed()
}
}
fn as_path(asn: BoxedStrategy<u32>) -> impl Strategy<Value = AsPathAttribute> {
// A segment has at least one AS number.
vec((any::<bool>(), vec(asn, 1..6)), 0..4).prop_map(|segments| AsPathAttribute {
segments: segments
.into_iter()
.map(|(ordered, path)| AsPathSegment { ordered, path })
.collect(),
})
}
fn extended_community() -> impl Strategy<Value = ExtendedCommunity> {
any::<[u8; 8]>().prop_map(ExtendedCommunity)
}
fn mp_reach(session: Session) -> impl Strategy<Value = MpReachNlriPathAttribute> {
let ip_safi = || prop_oneof![Just(Safi::UNICAST), Just(Safi::MULTICAST)];
let v6_next_hop = || {
prop_oneof![
any::<[u8; 16]>().prop_map(|a| NlriNextHop::Ipv6(Ipv6Addr::from(a))),
(any::<[u8; 16]>(), any::<[u8; 16]>()).prop_map(|(global, link_local)| {
NlriNextHop::Ipv6WithLl {
global: Ipv6Addr::from(global),
link_local: Ipv6Addr::from(link_local),
}
}),
]
};
// IPv4 routes have an IPv4 next hop, or an IPv6 one over a link that
// has only IPv6 addresses (RFC 8950).
let v4_next_hop = move || {
prop_oneof![
3 => any::<[u8; 4]>().prop_map(|a| NlriNextHop::Ipv4(Ipv4Addr::from(a))),
1 => v6_next_hop(),
]
};
let v4 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
v4_next_hop(),
nlri(session, IpVersion::Ipv4, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV4, safi),
next_hop,
prefixes,
});
let v6 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
v6_next_hop(),
nlri(session, IpVersion::Ipv6, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV6, safi),
next_hop,
prefixes,
});
prop_oneof![v4, v6]
}
fn mp_unreach(session: Session) -> impl Strategy<Value = MpUnreachNlriPathAttribute> {
let of = move |version: IpVersion| {
nlri(session, version, Safi::UNICAST).prop_map(move |prefixes| MpUnreachNlriPathAttribute {
family: AfiSafi::new(version.into(), Safi::UNICAST),
prefixes,
})
};
prop_oneof![of(IpVersion::Ipv4), of(IpVersion::Ipv6)]
}
/// Type codes whose definition the decoder knows; an unknown attribute has
/// none of these.
const DEFINED_ATTRIBUTES: [u8; 17] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 32, 35];
/// Unknown attributes, each with a type code of its own: a repeated
/// attribute is dropped by the decoder.
fn unknown_attributes() -> impl Strategy<Value = Vec<PathAttribute>> {
proptest::collection::btree_map(
any::<u8>().prop_filter("a type code without a definition", |code| {
!DEFINED_ATTRIBUTES.contains(code)
}),
(any::<u8>(), vec(any::<u8>(), 0..300)),
0..3,
)
.prop_map(|attributes| {
attributes
.into_iter()
.map(|(type_code, (flags, payload))| {
// Optional (bit 7): an unknown well-known attribute is an
// error. Bit 4 selects the two-octet length, which a payload
// this long needs.
let mut flags = flags | 0x80;
if payload.len() > 255 {
flags |= 0x10;
}
PathAttribute::Unknown {
flags: PathAttributeFlags(flags),
type_code,
payload,
}
})
.collect()
})
}
/// The attributes of a well-formed UPDATE on `session`: the mandatory ones,
/// and any of the others at most once.
fn path_attributes(session: Session) -> impl Strategy<Value = Vec<PathAttribute>> {
let mandatory = (
prop_oneof![
Just(OriginPathAttribute::IGP),
Just(OriginPathAttribute::EGP),
Just(OriginPathAttribute::INCOMPLETE)
],
as_path(asn(session.four_octet_asn)),
any::<[u8; 4]>(),
any::<u32>(),
)
.prop_map(move |(origin, as_path, next_hop, local_pref)| {
let mut attributes = vec![
PathAttribute::Origin(origin),
PathAttribute::AsPath(as_path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::from(next_hop))),
];
// Mandatory between internal peers, dropped from external ones.
if session.ibgp {
attributes.push(PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)));
}
attributes
});
let optional = (
proptest::option::of(
any::<u32>().prop_map(|v| PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(v))),
),
proptest::option::of(Just(PathAttribute::AtomicAggregate(
AtomicAggregatePathAttribute {},
))),
proptest::option::of((asn(session.four_octet_asn), any::<[u8; 4]>()).prop_map(
|(asn, ip)| {
PathAttribute::Aggregator(AggregatorPathAttribute {
asn,
ip: Ipv4Addr::from(ip),
})
},
)),
proptest::option::of(
vec(any::<(u16, u16)>(), 1..8)
.prop_map(|c| PathAttribute::Communities(CommunitiesPathAttribute(c))),
),
proptest::option::of(mp_reach(session).prop_map(PathAttribute::MpReachNlri)),
proptest::option::of(mp_unreach(session).prop_map(PathAttribute::MpUnreachNlri)),
proptest::option::of(
vec(extended_community(), 1..6).prop_map(|extended_communities| {
PathAttribute::ExtendedCommunities(ExtendedCommunitiesPathAttribute {
extended_communities,
})
}),
),
// AS4_PATH is four octets wide, and only a two-octet speaker sends
// it.
proptest::option::of(as_path(any::<u32>().boxed()).prop_map(PathAttribute::As4Path))
.prop_map(move |as4_path| as4_path.filter(|_| !session.four_octet_asn)),
proptest::option::of(vec(any::<(u32, u32, u32)>(), 1..6).prop_map(|communities| {
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: communities
.into_iter()
.map(|(global_admin, data_1, data_2)| LargeCommunity {
global_admin,
data_1,
data_2,
})
.collect(),
})
})),
proptest::option::of(any::<u32>().prop_map(PathAttribute::OnlyToCustomer)),
// Route reflection's, which only an internal peer sends.
proptest::option::of((any::<[u8; 4]>(), vec(any::<[u8; 4]>(), 1..4))).prop_map(
move |reflected| {
reflected
.filter(|_| session.ibgp)
.map(|(originator, clusters)| {
vec![
PathAttribute::OriginatorId(Ipv4Addr::from(originator)),
PathAttribute::ClusterList(
clusters.into_iter().map(Ipv4Addr::from).collect(),
),
]
})
.unwrap_or_default()
},
),
unknown_attributes(),
);
(mandatory, optional).prop_map(|(mut attributes, optional)| {
let (
med,
atomic,
aggregator,
communities,
reach,
unreach,
extended,
as4,
large,
otc,
reflected,
unknown,
) = optional;
attributes.extend(
[
med,
atomic,
aggregator,
communities,
reach,
unreach,
extended,
as4,
large,
otc,
]
.into_iter()
.flatten(),
);
attributes.extend(reflected);
attributes.extend(unknown);
attributes
})
}
fn update(session: Session) -> impl Strategy<Value = UpdateMessage> {
(
nlri(session, IpVersion::Ipv4, Safi::UNICAST),
path_attributes(session),
nlri(session, IpVersion::Ipv4, Safi::UNICAST),
)
.prop_map(
|(withdrawn_routes, path_attributes, announced_routes)| UpdateMessage {
withdrawn_routes,
path_attributes,
announced_routes,
},
)
}
/// Capability codes that have their own variant.
const KNOWN_CAPABILITIES: [u8; 13] = [1, 2, 3, 5, 6, 7, 8, 9, 64, 65, 69, 70, 71];
fn capability() -> impl Strategy<Value = Capability> {
let orf = (
any::<u8>().prop_map(OrfType),
prop_oneof![
Just(SendReceive::Receive),
Just(SendReceive::Send),
Just(SendReceive::Both)
],
)
.prop_map(|(r#type, send_recv)| OutboundRouteFilterEntry { r#type, send_recv });
let add_path = prop_oneof![
Just(SendReceive::Receive),
Just(SendReceive::Send),
Just(SendReceive::Both)
];
prop_oneof![
(afi(), safi()).prop_map(|(afi, safi)| Capability::MultiProtocol { afi, safi }),
Just(Capability::RouteRefresh),
vec((afi(), safi(), vec(orf, 0..4)), 0..3).prop_map(|families| {
Capability::OutboundRouteFilter(
families
.into_iter()
.map(|(afi, safi, orfs)| OrfFamily { afi, safi, orfs })
.collect(),
)
}),
vec((afi(), safi(), afi()), 1..4).prop_map(|entries| Capability::ExtendedNextHop {
entries: entries
.into_iter()
.map(|(afi, safi, nh_afi)| ExtendedNextHopEntry { afi, safi, nh_afi })
.collect(),
}),
Just(Capability::ExtendedMessage),
(any::<u8>(), afi()).prop_map(|(header, afi)| Capability::BgpSec {
header: BgpSecHeader(header),
afi,
}),
vec((afi(), safi(), any::<u8>()), 1..4).prop_map(|entries| {
Capability::MultiLabel {
entries: entries
.into_iter()
.map(|(afi, safi, count)| MultiLabelEntry { afi, safi, count })
.collect(),
}
}),
any::<u8>().prop_map(|role| Capability::BgpRole {
role: BgpRole::from(role),
}),
(
any::<bool>(),
any::<bool>(),
0u16..4096,
vec((afi(), safi(), any::<bool>()), 0..4)
)
.prop_map(|(restart_state, notification, restart_time, entries)| {
Capability::GracefulRestart {
restart_state,
notification,
restart_time,
entries: entries
.into_iter()
.map(|(afi, safi, preserve_forwarding)| GracefulRestartEntry {
afi,
safi,
preserve_forwarding,
})
.collect(),
}
}),
any::<u32>().prop_map(|asn| Capability::FourByteAsn { asn }),
vec((afi(), safi(), add_path), 1..4).prop_map(|entries| Capability::AddPath {
entries: entries
.into_iter()
.map(|(afi, safi, send_recv)| AddPathEntry {
afi,
safi,
send_recv,
})
.collect(),
}),
Just(Capability::EnhancedRouteRefresh),
vec((afi(), safi(), any::<u8>(), 0u32..(1 << 24)), 0..4).prop_map(|entries| {
Capability::LongLivedGracefulRestart(
entries
.into_iter()
.map(
|(afi, safi, flags, stale_time)| LongLivedGracefulRestartEntry {
afi,
safi,
flags,
stale_time,
},
)
.collect(),
)
}),
(
any::<u8>().prop_filter("a code without a variant", |code| {
!KNOWN_CAPABILITIES.contains(code)
}),
vec(any::<u8>(), 0..20)
)
.prop_map(|(type_code, value)| Capability::Unknown { type_code, value }),
]
}
fn open() -> impl Strategy<Value = OpenMessage> {
(
any::<u16>(),
any::<u16>(),
any::<[u8; 4]>(),
// Enough to go past 255 octets of options now and then, which
// switches to the extended length of RFC 9072.
vec(vec(capability(), 0..10), 0..8),
)
.prop_map(|(asn, hold_time, identifier, options)| OpenMessage {
version: 4,
asn,
hold_time,
identifier: Ipv4Addr::from(identifier),
options: options.into_iter().map(OpenOption::Capabilities).collect(),
})
}
fn notification() -> impl Strategy<Value = NotificationMessage> {
// Any code and subcode: a peer may send ones we have no name for.
(any::<u8>(), any::<u8>(), vec(any::<u8>(), 0..40)).prop_map(|(code, subcode, data)| {
NotificationMessage {
error: BgpError::from_code_subcode(code, subcode),
data,
}
})
}
fn message(session: Session) -> impl Strategy<Value = Message> {
prop_oneof![
open().prop_map(Message::Open),
update(session).prop_map(Message::Update),
notification().prop_map(Message::Notification),
Just(Message::KeepAlive),
(afi(), any::<u8>(), safi()).prop_map(|(afi, subtype, safi)| {
Message::RouteRefresh(RouteRefreshMessage { afi, subtype, safi })
}),
]
}
/// A session, and a well-formed message that fits it.
fn session_and_message() -> impl Strategy<Value = (Session, Message)> {
session().prop_flat_map(|session| (Just(session), message(session)))
}
// ---------------------------------------------------------------------------
// Properties
// ---------------------------------------------------------------------------
/// Decodes `wire` and checks what must hold for anything the decoder
/// accepts: it can be shown and written, and unless the decoder had to
/// repair it, it reads back as itself.
fn check_decoding(ctx: &ParserContext, wire: &[u8]) -> Result<(), TestCaseError> {
let Ok(decoded) = Message::decode_with_report(ctx, wire) else {
return Ok(());
};
let (msg, repairs) = (decoded.message, decoded.update_errors);
// Display of what a peer sent must not be able to panic: it is logged.
let _ = format!("{msg:?} {repairs:?}");
if let Message::Update(update) = &msg {
for prefix in update
.withdrawn_routes
.iter()
.chain(&update.announced_routes)
{
let _ = prefix.to_string();
}
for attribute in &update.path_attributes {
match attribute {
PathAttribute::MpReachNlri(mp) => {
mp.prefixes.iter().for_each(|p| drop(p.to_string()))
}
PathAttribute::MpUnreachNlri(mp) => {
mp.prefixes.iter().for_each(|p| drop(p.to_string()))
}
_ => {}
}
}
}
let mut rewritten = BytesMut::new();
match msg.to_wire(ctx, &mut rewritten) {
Ok(()) => {}
// Withdrawing what a full-size UPDATE announced can need more room
// than one message has. The result is acted on, never sent.
Err(EncodeError::TooLong { .. }) if !repairs.is_empty() => return Ok(()),
Err(e) => {
return Err(TestCaseError::fail(format!(
"accepted but cannot be written: {e}"
)));
}
}
if repairs.is_empty() {
let again = Message::decode_with_report(ctx, &rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e}")))?;
prop_assert!(again.update_errors.is_empty(), "{:?}", again.update_errors);
prop_assert_eq!(again.message, msg);
}
Ok(())
}
/// Writes a generated message. `None` when it is too big for one BGP
/// message, or some part of it for its length field, which the generators
/// can produce and the encoder must refuse.
fn written(ctx: &ParserContext, msg: &Message) -> Option<BytesMut> {
let mut wire = BytesMut::new();
match msg.to_wire(ctx, &mut wire) {
Ok(()) => Some(wire),
Err(EncodeError::TooLong { .. }) => None,
}
}
fn framed(msg_type: u8, body: &[u8]) -> Vec<u8> {
let mut wire = vec![0xff; 16];
let length = u16::try_from(usize::from(BGP_HEADER_LEN) + body.len()).unwrap();
wire.extend_from_slice(&length.to_be_bytes());
wire.push(msg_type);
wire.extend_from_slice(body);
wire
}
/// Ways to damage a message.
#[derive(Debug, Clone)]
enum Damage {
Set { at: prop::sample::Index, to: u8 },
Remove { at: prop::sample::Index },
Insert { at: prop::sample::Index, byte: u8 },
Truncate { to: prop::sample::Index },
}
fn damage() -> impl Strategy<Value = Damage> {
prop_oneof![
(any::<prop::sample::Index>(), any::<u8>()).prop_map(|(at, to)| Damage::Set { at, to }),
any::<prop::sample::Index>().prop_map(|at| Damage::Remove { at }),
(any::<prop::sample::Index>(), any::<u8>())
.prop_map(|(at, byte)| Damage::Insert { at, byte }),
any::<prop::sample::Index>().prop_map(|to| Damage::Truncate { to }),
]
}
/// Applies `damage` to the body of `wire` and makes the header's length
/// right again, so that the damage is the body parsers' problem.
fn damaged(wire: &[u8], damage: &[Damage]) -> Vec<u8> {
let header = usize::from(BGP_HEADER_LEN);
let msg_type = wire[header - 1];
let mut body = wire[header..].to_vec();
for d in damage {
match d {
Damage::Set { at, to } if !body.is_empty() => {
let i = at.index(body.len());
body[i] = *to;
}
Damage::Remove { at } if !body.is_empty() => {
body.remove(at.index(body.len()));
}
Damage::Insert { at, byte } => body.insert(at.index(body.len() + 1), *byte),
Damage::Truncate { to } if !body.is_empty() => body.truncate(to.index(body.len())),
_ => {}
}
}
body.truncate(usize::from(BGP_MAX_MSG_LEN - BGP_HEADER_LEN));
framed(msg_type, &body)
}
/// 2000 cases a property, which takes about a second. For a longer hunt
/// set `PROPTEST_CASES`, e.g. `PROPTEST_CASES=1000000 cargo test --release
/// -p bgp-packet --test properties`.
fn config() -> ProptestConfig {
let cases = std::env::var("PROPTEST_CASES")
.ok()
.and_then(|cases| cases.parse().ok())
.unwrap_or(2000);
ProptestConfig {
cases,
// Failures are not saved to a file: one that is found becomes a
// named test next to the code it broke.
failure_persistence: None,
..ProptestConfig::default()
}
}
proptest! {
#![proptest_config(config())]
#[test]
fn a_message_that_can_be_written_reads_back_equal((session, msg) in session_and_message()) {
let ctx = session.ctx();
let Some(wire) = written(&ctx, &msg) else {
return Ok(());
};
let decoded = Message::decode_with_report(&ctx, &wire)
.map_err(|e| TestCaseError::fail(format!("{e}")))?;
// What we write is well formed: nothing for the decoder to repair.
prop_assert!(decoded.update_errors.is_empty(), "{:?}", decoded.update_errors);
prop_assert_eq!(&decoded.message, &msg);
}
#[test]
fn noise_is_never_more_than_an_error(session in session(), bytes in vec(any::<u8>(), 0..200)) {
check_decoding(&session.ctx(), &bytes)?;
}
#[test]
fn noise_behind_a_valid_header_is_never_more_than_an_error(
session in session(),
msg_type in 0u8..7,
body in vec(any::<u8>(), 0..300),
) {
check_decoding(&session.ctx(), &framed(msg_type, &body))?;
}
#[test]
fn a_damaged_message_is_never_more_than_an_error(
(session, msg) in session_and_message(),
damage in vec(damage(), 1..4),
) {
let ctx = session.ctx();
let Some(wire) = written(&ctx, &msg) else {
return Ok(());
};
check_decoding(&ctx, &damaged(&wire, &damage))?;
}
/// The parsers below the message level are public too. What they
/// accept must write back as it was read, where the format leaves
/// nothing for the reader to normalise.
#[test]
fn noise_is_never_more_than_an_error_for_the_parts(
session in session(),
bytes in vec(any::<u8>(), 0..120),
) {
let ctx = session.ctx();
let _ = OpenMessage::from_wire(&bytes);
let _ = UpdateMessage::decode(&ctx, &bytes);
if let Ok((rest, capability)) = Capability::from_wire(&bytes) {
let consumed = &bytes[..bytes.len() - rest.len()];
let mut rewritten = BytesMut::new();
capability.to_wire(&mut rewritten)
.map_err(|e| TestCaseError::fail(format!("accepted but cannot be written: {e}")))?;
// ADD-PATH families with a Send/Receive value nobody defined
// are left out (RFC 7911 §4), and the reserved octet of
// Multiprotocol and ORF and the reserved bits of Graceful
// Restart are read as zero whatever they were. Everything else
// is written as it was read.
let normalised = matches!(
capability,
Capability::AddPath { .. }
| Capability::MultiProtocol { .. }
| Capability::OutboundRouteFilter(_)
| Capability::GracefulRestart { .. }
);
if !normalised {
prop_assert_eq!(&rewritten[..], consumed, "{:?}", capability);
}
let again = Capability::from_wire(&rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e:?}")))?;
prop_assert_eq!(again, (&[][..], capability));
}
if let Ok((rest, attribute)) = PathAttribute::from_wire(&ctx, &bytes) {
let consumed = &bytes[..bytes.len() - rest.len()];
let mut rewritten = BytesMut::new();
attribute.to_wire(&ctx, &mut rewritten)
.map_err(|e| TestCaseError::fail(format!("accepted but cannot be written: {e}")))?;
// The flags are not checked against the attribute's definition
// here, and are written as defined; and the bits of a prefix
// past its length are cleared. With the flags as defined and
// no prefixes, the attribute is written as it was read.
let names_prefixes = matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
);
if rewritten[0] == consumed[0] && !names_prefixes {
prop_assert_eq!(&rewritten[..], consumed, "{:?}", attribute);
}
let again = PathAttribute::from_wire(&ctx, &rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e:?}")))?;
prop_assert_eq!(again, (&[][..], attribute));
}
}
}
crates/primitives/Cargo.toml
+7 -4
View File
@@ -1,7 +1,7 @@
[package]
name = "bgp-primitives"
description = "Primitive types for working with routing information"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
@@ -9,11 +9,14 @@ repository.workspace = true
version.workspace = true
[dependencies]
bitfield.workspace = true
bytes.workspace = true
eyre.workspace = true
nom.workspace = true
serde.workspace = true
serde_repr.workspace = true
thiserror.workspace = true
strum.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/primitives/src/constants.rs
+150 -56
View File
@@ -1,80 +1,174 @@
use std::fmt::Display;
use eyre::bail;
use serde_repr::{Deserialize_repr, Serialize_repr};
use serde::Serialize;
use serde_repr::Serialize_repr;
use thiserror::Error;
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Serialize_repr, Deserialize_repr)]
/// An address family number this implementation has no name for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)]
#[error("unknown address family identifier {0}")]
pub struct UnknownAfi(pub u16);
/// The IP version of a prefix. For the address family field of a capability
/// or an MP_REACH_NLRI, which can name families that are not IP, see [`Afi`].
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize_repr)]
#[repr(u16)]
pub enum AddressFamilyId {
pub enum IpVersion {
Ipv4 = 1,
Ipv6 = 2,
}
impl TryFrom<u16> for AddressFamilyId {
type Error = eyre::Error;
fn try_from(value: u16) -> Result<Self, Self::Error> {
Ok(match value {
x if x == Self::Ipv4 as u16 => Self::Ipv4,
x if x == Self::Ipv6 as u16 => Self::Ipv6,
other => bail!("Unknown AddressFamily: {}", other),
})
}
}
impl Display for AddressFamilyId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
impl IpVersion {
/// How many bits an address of this version has.
pub fn bits(self) -> u8 {
match self {
AddressFamilyId::Ipv4 => write!(f, "Ipv4"),
AddressFamilyId::Ipv6 => write!(f, "Ipv6"),
Self::Ipv4 => 32,
Self::Ipv6 => 128,
}
}
}
/// Represents a Subsequent Address Family Identifier.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Serialize_repr, Deserialize_repr)]
#[repr(u8)]
pub enum SubsequentAfi {
Unicast = 1,
Multicast = 2,
NlriWithMpls = 4,
MplsLabelledVpn = 128,
MulticastMplsVpn = 129,
impl TryFrom<u16> for IpVersion {
type Error = UnknownAfi;
fn try_from(value: u16) -> Result<Self, Self::Error> {
match value {
x if x == Self::Ipv4 as u16 => Ok(Self::Ipv4),
x if x == Self::Ipv6 as u16 => Ok(Self::Ipv6),
other => Err(UnknownAfi(other)),
}
}
}
impl TryFrom<u8> for SubsequentAfi {
type Error = eyre::Error;
impl Display for IpVersion {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
IpVersion::Ipv4 => write!(f, "Ipv4"),
IpVersion::Ipv6 => write!(f, "Ipv6"),
}
}
}
fn try_from(value: u8) -> Result<Self, Self::Error> {
Ok(match value {
x if x == Self::Unicast as u8 => Self::Unicast,
x if x == Self::Multicast as u8 => Self::Multicast,
x if x == Self::NlriWithMpls as u8 => Self::NlriWithMpls,
x if x == Self::MplsLabelledVpn as u8 => Self::MplsLabelledVpn,
x if x == Self::MulticastMplsVpn as u8 => Self::MulticastMplsVpn,
other => bail!("Unknown SubsequentAfi: {}", other),
/// An Address Family Identifier as it appears on the wire (RFC 4760).
///
/// Any number is a valid value: a peer may name a family this implementation
/// has never heard of, and that must be readable so it can be ignored. Which
/// families are *supported* is a separate question, answered where routes
/// are stored.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
#[serde(transparent)]
pub struct Afi(pub u16);
impl Afi {
pub const IPV4: Self = Self(1);
pub const IPV6: Self = Self(2);
pub const L2VPN: Self = Self(25);
pub const BGP_LS: Self = Self(16388);
}
impl From<IpVersion> for Afi {
fn from(version: IpVersion) -> Self {
Self(version as u16)
}
}
impl TryFrom<Afi> for IpVersion {
type Error = UnknownAfi;
fn try_from(afi: Afi) -> Result<Self, Self::Error> {
Self::try_from(afi.0)
}
}
/// A Subsequent Address Family Identifier as it appears on the wire. Like
/// [`Afi`], any number is a valid value.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
#[serde(transparent)]
pub struct Safi(pub u8);
impl Safi {
pub const UNICAST: Self = Self(1);
pub const MULTICAST: Self = Self(2);
pub const MPLS_LABEL: Self = Self(4);
pub const EVPN: Self = Self(70);
pub const BGP_LS: Self = Self(71);
pub const MPLS_VPN: Self = Self(128);
pub const MULTICAST_MPLS_VPN: Self = Self(129);
pub const FLOWSPEC: Self = Self(133);
}
/// An address family: the pair that says what kind of routes a capability
/// or an MP_REACH_NLRI is about.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub struct AfiSafi {
pub afi: Afi,
pub safi: Safi,
}
impl AfiSafi {
pub const IPV4_UNICAST: Self = Self::new(Afi::IPV4, Safi::UNICAST);
pub const IPV6_UNICAST: Self = Self::new(Afi::IPV6, Safi::UNICAST);
pub const fn new(afi: Afi, safi: Safi) -> Self {
Self { afi, safi }
}
/// The unicast family of an IP version.
pub fn unicast(version: IpVersion) -> Self {
Self::new(version.into(), Safi::UNICAST)
}
/// The usual name of the family, if it has one here.
fn name(self) -> Option<&'static str> {
Some(match (self.afi, self.safi) {
(Afi::IPV4, Safi::UNICAST) => "IPv4 unicast",
(Afi::IPV6, Safi::UNICAST) => "IPv6 unicast",
(Afi::IPV4, Safi::MULTICAST) => "IPv4 multicast",
(Afi::IPV6, Safi::MULTICAST) => "IPv6 multicast",
(Afi::IPV4, Safi::MPLS_LABEL) => "IPv4 labeled unicast",
(Afi::IPV6, Safi::MPLS_LABEL) => "IPv6 labeled unicast",
(Afi::IPV4, Safi::MPLS_VPN) => "VPNv4 unicast",
(Afi::IPV6, Safi::MPLS_VPN) => "VPNv6 unicast",
(Afi::IPV4, Safi::FLOWSPEC) => "IPv4 flowspec",
(Afi::IPV6, Safi::FLOWSPEC) => "IPv6 flowspec",
(Afi::L2VPN, Safi::EVPN) => "L2VPN EVPN",
(Afi::BGP_LS, Safi::BGP_LS) => "BGP-LS",
_ => return None,
})
}
}
/// The implementation try_from for u16 just reuses the u8 implementation since
/// we don't have any cases where the SAFI is longer than u8::MAX yet.
impl TryFrom<u16> for SubsequentAfi {
type Error = eyre::Error;
fn try_from(value: u16) -> Result<Self, Self::Error> {
TryFrom::<u8>::try_from(value as u8)
}
}
impl Display for SubsequentAfi {
impl Display for AfiSafi {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
SubsequentAfi::Unicast => write!(f, "Unicast"),
SubsequentAfi::Multicast => write!(f, "Multicast"),
SubsequentAfi::NlriWithMpls => write!(f, "NlriWithMpls"),
SubsequentAfi::MplsLabelledVpn => write!(f, "MplsLabelledVpn"),
SubsequentAfi::MulticastMplsVpn => write!(f, "MulticastMplsVpn"),
match self.name() {
Some(name) => f.write_str(name),
None => write!(f, "AFI {} SAFI {}", self.afi.0, self.safi.0),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn any_family_is_a_value_and_the_common_ones_have_names() {
assert_eq!(AfiSafi::IPV6_UNICAST.to_string(), "IPv6 unicast");
assert_eq!(
AfiSafi::new(Afi::L2VPN, Safi::EVPN).to_string(),
"L2VPN EVPN"
);
assert_eq!(
AfiSafi::new(Afi(999), Safi(7)).to_string(),
"AFI 999 SAFI 7"
);
assert_eq!(AfiSafi::unicast(IpVersion::Ipv4), AfiSafi::IPV4_UNICAST);
}
#[test]
fn only_ipv4_and_ipv6_are_ip_versions() {
assert_eq!(IpVersion::try_from(Afi::IPV6), Ok(IpVersion::Ipv6));
assert_eq!(IpVersion::try_from(Afi::L2VPN), Err(UnknownAfi(25)));
assert_eq!(IpVersion::try_from(3u16), Err(UnknownAfi(3)));
}
}
crates/primitives/src/ip_prefix.rs
+310 -111
View File
@@ -2,130 +2,208 @@ use std::fmt::Display;
use std::net::{Ipv4Addr, Ipv6Addr};
use std::str::FromStr;
use bytes::{Buf, BufMut, BytesMut};
use eyre::{Result, bail};
use serde::{Deserialize, Serialize};
use serde::Serialize;
use thiserror::Error;
use crate::constants::AddressFamilyId;
use crate::parser::{ParserContext, ToWireError};
use crate::constants::IpVersion;
/// IpPrefix represents some IP address prefix, for a specific AddressFamilyId.
#[derive(Debug, PartialEq, Eq, Clone, Hash, Serialize, Deserialize)]
/// Why something is not a valid prefix.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum PrefixError {
#[error("prefix {0:?} is not in address/length form")]
Format(String),
#[error("prefix {0:?} has an address that is neither IPv4 nor IPv6")]
Address(String),
#[error("prefix length {length} is longer than the {max} bits of an address")]
LengthTooLong { length: u8, max: u8 },
#[error("prefix length {length} needs {needed} address bytes, got {got}")]
TooFewBytes {
length: u8,
needed: usize,
got: usize,
},
}
/// An IPv4 or IPv6 prefix.
///
/// It is a small value with no allocation behind it, cheap to copy, compare
/// and hash: a routing table holds millions of them and looks them up by
/// value. A prefix has one representation: the bits past its length are
/// always zero, so two prefixes are equal exactly when they are the same
/// prefix.
#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Hash, Serialize)]
pub struct IpPrefix {
pub address_family: AddressFamilyId,
pub prefix: Vec<u8>,
pub length: u8,
address_family: IpVersion,
length: u8,
/// The address, left-aligned, zero past `length` bits.
address: [u8; 16],
}
impl IpPrefix {
pub fn new(address_family: AddressFamilyId, prefix: Vec<u8>, length: u8) -> Result<Self> {
// Ensure that the prefix we are given contains the right number of bytes corresponding to the prefix length.
if prefix.len() < ((length + 7) / 8).into() {
bail!(
"Mismatched prefix {:?} for given prefix length: {}",
prefix,
length
);
/// Makes a prefix from the leading bytes of its address. `prefix` must
/// hold at least the bytes that `length` bits cover; anything past
/// those bits is ignored (RFC 4271 §4.3: such bits are irrelevant).
pub fn new(
address_family: IpVersion,
prefix: impl AsRef<[u8]>,
length: u8,
) -> Result<Self, PrefixError> {
let prefix = prefix.as_ref();
let max = address_family.bits();
if length > max {
return Err(PrefixError::LengthTooLong { length, max });
}
let needed = usize::from(length.div_ceil(8));
if prefix.len() < needed {
return Err(PrefixError::TooFewBytes {
length,
needed,
got: prefix.len(),
});
}
let mut address = [0u8; 16];
address[..needed].copy_from_slice(&prefix[..needed]);
if let (Some(last), rem @ 1..) = (address[..needed].last_mut(), length % 8) {
*last &= high_bits_mask(rem);
}
Ok(Self {
address_family,
prefix,
length,
address,
})
}
pub fn to_wire(&self, _: &ParserContext, out: &mut BytesMut) -> Result<(), ToWireError> {
// Verify that there is enough space to write the IpPrefix.
if out.remaining() < (self.prefix.len() + 1) {
Err(ToWireError::OutBufferOverflow)?;
pub fn address_family(&self) -> IpVersion {
self.address_family
}
/// The prefix length in bits.
pub fn length(&self) -> u8 {
self.length
}
/// The bytes the prefix length covers, as they go on the wire: none for
/// a default route, three for a /24.
pub fn bytes(&self) -> &[u8] {
&self.address[..usize::from(self.length.div_ceil(8))]
}
/// The prefix of `length` bits that covers this one: itself, if it is
/// no longer than that.
pub fn shortened(&self, length: u8) -> Self {
if length >= self.length {
return *self;
}
let mut address = [0u8; 16];
let whole = usize::from(length / 8);
address[..whole].copy_from_slice(&self.address[..whole]);
if !length.is_multiple_of(8) {
address[whole] = self.address[whole] & high_bits_mask(length % 8);
}
Self {
address_family: self.address_family,
length,
address,
}
}
// Write length and prefix.
out.put_u8(self.length);
out.put(self.prefix.as_slice());
/// Returns true if `other` is equal to or more specific than `self`, i.e.
/// `self` covers `other`. Prefixes of different address families never
/// contain each other.
pub fn contains(&self, other: &IpPrefix) -> bool {
if self.address_family != other.address_family || other.length < self.length {
return false;
}
let full_bytes = usize::from(self.length / 8);
if self.address[..full_bytes] != other.address[..full_bytes] {
return false;
}
let rem_bits = self.length % 8;
if rem_bits == 0 {
return true;
}
let mask = high_bits_mask(rem_bits);
(self.address[full_bytes] & mask) == (other.address[full_bytes] & mask)
}
}
Ok(())
/// The mask that keeps the first `bits` bits of an octet and clears the
/// rest, for `bits` from 1 to 7: what the last octet of a prefix whose
/// length is not a whole number of octets is masked with.
fn high_bits_mask(bits: u8) -> u8 {
u8::MAX << (8 - bits)
}
/// A prefix as an UPDATE names it: the prefix and, on a session where
/// ADD-PATH is in use for the address family (RFC 7911), the identifier
/// that tells apart the several routes a peer may announce for it.
///
/// Whether the identifier is on the wire is a matter of what the session
/// negotiated, not of its value. Where it is not, it is zero here.
#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash, Serialize)]
pub struct Nlri {
pub prefix: IpPrefix,
pub path_id: u32,
}
impl Nlri {
/// `prefixes` as they are named on a session without ADD-PATH.
pub fn plain(prefixes: impl IntoIterator<Item = IpPrefix>) -> Vec<Self> {
prefixes.into_iter().map(Self::from).collect()
}
}
impl From<IpPrefix> for Nlri {
fn from(prefix: IpPrefix) -> Self {
Self { prefix, path_id: 0 }
}
}
/// So that a list of these compares with a list of plain prefixes.
impl PartialEq<IpPrefix> for Nlri {
fn eq(&self, prefix: &IpPrefix) -> bool {
self.path_id == 0 && self.prefix == *prefix
}
}
impl Display for Nlri {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self.path_id {
0 => write!(f, "{}", self.prefix),
id => write!(f, "{} (path {id})", self.prefix),
}
}
}
impl TryFrom<&str> for IpPrefix {
type Error = eyre::Error;
type Error = PrefixError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
let parts: Vec<&str> = value.split("/").collect();
let format = || PrefixError::Format(value.to_string());
let address = || PrefixError::Address(value.to_string());
let (addr, length) = value.split_once('/').ok_or_else(format)?;
let length: u8 = length.parse().map_err(|_| format())?;
if parts.len() != 2 {
bail!(
"Expected IpPrefix in format prefix/length but got: {}",
value
);
}
let length: u8 = u8::from_str_radix(parts[1], 10).map_err(eyre::Error::from)?;
let mut octets;
let afi: AddressFamilyId;
if parts[0].contains(":") {
afi = AddressFamilyId::Ipv6;
let addr: Ipv6Addr = Ipv6Addr::from_str(parts[0]).map_err(eyre::Error::from)?;
octets = addr.octets().to_vec();
} else if parts[0].contains(".") {
afi = AddressFamilyId::Ipv4;
let addr: Ipv4Addr = Ipv4Addr::from_str(parts[0]).map_err(eyre::Error::from)?;
octets = addr.octets().to_vec();
let (afi, octets) = if addr.contains(':') {
let addr = Ipv6Addr::from_str(addr).map_err(|_| address())?;
(IpVersion::Ipv6, addr.octets().to_vec())
} else {
bail!("Could not figure out address type")
}
// Truncate the octets we have to the right number of bytes to match the prefix length..
if length % 8 == 0 {
// We can cleanly truncate the number of bytes since we are at a byte boundary.
octets.truncate((length / 8).into());
} else {
// We need to keep length % 8 bits of the last byte.
let num_bytes = (length / 8) + 1;
let mask = u8::MAX << (8 - (length % 8));
octets.truncate(num_bytes.into());
// Fix up the last byte.
let last_pos = octets.len() - 1;
octets[last_pos] &= mask;
}
Ok(IpPrefix {
address_family: afi,
prefix: octets,
length,
})
let addr = Ipv4Addr::from_str(addr).map_err(|_| address())?;
(IpVersion::Ipv4, addr.octets().to_vec())
};
IpPrefix::new(afi, octets, length)
}
}
impl Display for IpPrefix {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self.address_family {
AddressFamilyId::Ipv4 => {
// Pad the prefix with 0 bytes if it's less than 4 bytes long.
let bytes = &mut self.prefix.clone();
if bytes.len() < 4 {
bytes.extend(std::iter::repeat(0).take(4 - bytes.len()));
}
let four_bytes: [u8; 4] = bytes
.as_slice()
.try_into()
.map_err(|_| std::fmt::Error {})?;
let ipv4_addr = Ipv4Addr::from(four_bytes);
write!(f, "{}/{}", ipv4_addr, self.length)
IpVersion::Ipv4 => {
let [a, b, c, d, ..] = self.address;
write!(f, "{}/{}", Ipv4Addr::new(a, b, c, d), self.length)
}
AddressFamilyId::Ipv6 => {
let bytes = &mut self.prefix.clone();
if bytes.len() < 16 {
bytes.extend(std::iter::repeat(0).take(16 - bytes.len()));
}
let sixteen_bytes: [u8; 16] = bytes
.as_slice()
.try_into()
.map_err(|_| std::fmt::Error {})?;
let ipv6_addr = Ipv6Addr::from(sixteen_bytes);
write!(f, "{}/{}", ipv6_addr, self.length)
IpVersion::Ipv6 => {
write!(f, "{}/{}", Ipv6Addr::from(self.address), self.length)
}
}
}
@@ -133,9 +211,11 @@ impl Display for IpPrefix {
#[cfg(test)]
mod tests {
use crate::constants::AddressFamilyId;
use crate::constants::IpVersion;
use super::IpPrefix;
use proptest::prelude::*;
use super::{IpPrefix, PrefixError};
macro_rules! verify_roundtrip {
($name:ident, $prefix_str:expr, $afi:expr, $bytes:expr, $length:expr) => {
@@ -143,9 +223,9 @@ mod tests {
fn $name() {
let ip_prefix = IpPrefix::try_from($prefix_str).unwrap();
assert_eq!(ip_prefix.address_family, $afi);
assert_eq!(ip_prefix.prefix, $bytes);
assert_eq!(ip_prefix.length, $length);
assert_eq!(ip_prefix.address_family(), $afi);
assert_eq!(ip_prefix.bytes(), $bytes);
assert_eq!(ip_prefix.length(), $length);
let to_str: &str = &ip_prefix.to_string();
assert_eq!(IpPrefix::try_from(to_str).unwrap(), ip_prefix);
@@ -156,7 +236,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_24,
"10.1.2.0/24",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![10, 1, 2],
24
);
@@ -165,7 +245,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_19,
"10.245.123.0/19",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![10, 245, 96],
19
);
@@ -174,7 +254,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_3,
"192.168.1.0/3",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![192],
3
);
@@ -183,7 +263,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_0,
"0.0.0.0/0",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![],
0
);
@@ -191,7 +271,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv6_48,
"2001:db8:cafe::/48",
AddressFamilyId::Ipv6,
IpVersion::Ipv6,
vec![32, 1, 13, 184, 202, 254],
48
);
@@ -200,16 +280,135 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv6_32,
"2001:db8:cafe::/32",
AddressFamilyId::Ipv6,
IpVersion::Ipv6,
vec![32, 1, 13, 184],
32
);
verify_roundtrip!(
verify_roundtrip_ipv6_0,
"::/0",
AddressFamilyId::Ipv6,
vec![],
0
);
#[test]
fn text_that_is_not_a_prefix_is_rejected_with_the_reason() {
for (text, expected) in [
("10.0.0.0", PrefixError::Format("10.0.0.0".into())),
("10.0.0.0/x", PrefixError::Format("10.0.0.0/x".into())),
("10.0.0/8", PrefixError::Address("10.0.0/8".into())),
("nonsense/8", PrefixError::Address("nonsense/8".into())),
(
"10.0.0.0/33",
PrefixError::LengthTooLong {
length: 33,
max: 32,
},
),
(
"2001:db8::/129",
PrefixError::LengthTooLong {
length: 129,
max: 128,
},
),
] {
assert_eq!(IpPrefix::try_from(text), Err(expected), "{text}");
}
assert!(IpPrefix::try_from("0.0.0.0/0").is_ok());
assert!(IpPrefix::try_from("2001:db8::1/128").is_ok());
assert_eq!(
IpPrefix::new(IpVersion::Ipv4, vec![10], 16),
Err(PrefixError::TooFewBytes {
length: 16,
needed: 2,
got: 1
})
);
}
#[test]
fn a_prefix_has_one_representation() {
// Extra bytes and stray bits past the length are dropped.
let noisy = IpPrefix::new(IpVersion::Ipv4, vec![10, 0xff, 3, 4], 12).unwrap();
assert_eq!(noisy, IpPrefix::try_from("10.240.0.0/12").unwrap());
assert_eq!(noisy.bytes(), [10, 0xf0]);
assert_eq!(
IpPrefix::new(IpVersion::Ipv4, vec![0xff; 13], 99),
Err(PrefixError::LengthTooLong {
length: 99,
max: 32
})
);
}
#[test]
fn a_prefix_is_a_small_value() {
assert!(size_of::<IpPrefix>() <= 20);
let prefix = IpPrefix::try_from("10.1.2.0/24").unwrap();
let copy = prefix;
assert_eq!(copy, prefix);
assert_eq!(prefix.bytes(), [10, 1, 2]);
assert_eq!(IpPrefix::try_from("0.0.0.0/0").unwrap().bytes(), [0u8; 0]);
}
proptest! {
#[test]
fn text_round_trips(version: bool, bytes: [u8; 16], length in 0u8..=128) {
let version = if version { IpVersion::Ipv6 } else { IpVersion::Ipv4 };
let length = length.min(version.bits());
let prefix = IpPrefix::new(version, bytes, length).unwrap();
let text = prefix.to_string();
prop_assert_eq!(IpPrefix::try_from(text.as_str()), Ok(prefix));
prop_assert!(prefix.contains(&prefix));
}
#[test]
fn text_that_is_noise_is_an_error_or_a_prefix_that_round_trips(text in "\\PC{0,40}") {
if let Ok(prefix) = IpPrefix::try_from(text.as_str()) {
let shown = prefix.to_string();
prop_assert_eq!(IpPrefix::try_from(shown.as_str()), Ok(prefix));
}
}
/// A shortened prefix is the same as one made from the same
/// address with the shorter length, and covers the original.
#[test]
fn shortening_keeps_the_leading_bits(
version: bool, bytes: [u8; 16], length in 0u8..=128, shorter in 0u8..=128,
) {
let version = if version { IpVersion::Ipv6 } else { IpVersion::Ipv4 };
let length = length.min(version.bits());
let prefix = IpPrefix::new(version, bytes, length).unwrap();
let shortened = prefix.shortened(shorter);
let expected = IpPrefix::new(version, bytes, shorter.min(length)).unwrap();
prop_assert_eq!(shortened, expected);
prop_assert!(shortened.contains(&prefix));
}
/// `contains` agrees with comparing the leading bits by hand.
#[test]
fn contains_is_a_comparison_of_leading_bits(
a: [u8; 4], a_len in 0u8..=32, b: [u8; 4], b_len in 0u8..=32,
) {
let outer = IpPrefix::new(IpVersion::Ipv4, a, a_len).unwrap();
let inner = IpPrefix::new(IpVersion::Ipv4, b, b_len).unwrap();
let mask = if a_len == 0 { 0 } else { u32::MAX << (32 - a_len) };
let expected = b_len >= a_len
&& (u32::from_be_bytes(a) & mask) == (u32::from_be_bytes(b) & mask);
prop_assert_eq!(outer.contains(&inner), expected);
}
}
#[test]
fn contains_covers_more_specifics() {
let p = |s: &str| IpPrefix::try_from(s).unwrap();
assert!(p("10.100.0.0/16").contains(&p("10.100.7.0/24")));
assert!(p("10.100.0.0/16").contains(&p("10.100.0.0/16")));
assert!(!p("10.100.0.0/16").contains(&p("10.101.0.0/24")));
assert!(!p("10.100.7.0/24").contains(&p("10.100.0.0/16")));
assert!(p("0.0.0.0/0").contains(&p("192.0.2.0/24")));
// Non-byte-aligned boundary.
assert!(p("10.100.0.0/19").contains(&p("10.100.31.0/24")));
assert!(!p("10.100.0.0/19").contains(&p("10.100.32.0/24")));
// Cross-family never matches.
assert!(!p("::/0").contains(&p("10.0.0.0/8")));
assert!(p("2001:db8::/32").contains(&p("2001:db8:cafe::/48")));
}
verify_roundtrip!(verify_roundtrip_ipv6_0, "::/0", IpVersion::Ipv6, vec![], 0);
}
crates/primitives/src/parser.rs
+33 -25
View File
@@ -1,35 +1,43 @@
use std::fmt::Display;
use nom::error::{ErrorKind, ParseError};
use crate::constants::AddressFamilyId;
use crate::constants::AfiSafi;
use thiserror::Error;
#[derive(Debug, Default)]
/// What reading or writing a message has to know about the session it
/// belongs to. The default is a session before its OPENs are exchanged;
/// only an UPDATE depends on anything here, and there are none until then.
#[derive(Debug, Clone, Default)]
pub struct ParserContext {
/// Whether thi parser is being run with a peer that is RFC6793 compliant.
pub four_octet_asn: Option<bool>,
/// Which address family should be parsed by default with this parser.
pub address_family: Option<AddressFamilyId>,
/// Whether AS numbers are four octets wide on this session (RFC 6793).
pub four_octet_asn: bool,
/// Whether the peer is in our own AS. Some attributes mean something
/// only between internal peers and are dropped when an external one
/// sends them (RFC 7606 §7).
pub ibgp: bool,
/// The address families whose prefixes carry a path identifier in the
/// direction this context is for (RFC 7911). What a session sends and
/// what it receives are negotiated separately, so a session has one
/// context for reading and one for writing.
pub add_path: Vec<AfiSafi>,
}
#[derive(Debug)]
pub enum ToWireError {
/// There was not enough space in the output buffer to serialize the data into.
OutBufferOverflow,
/// Another error.
Other(eyre::Error),
}
impl Display for ToWireError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ToWireError::OutBufferOverflow => write!(f, "OutBufferOverflow"),
ToWireError::Other(report) => report.fmt(f),
}
impl ParserContext {
pub fn has_add_path(&self, family: AfiSafi) -> bool {
self.add_path.contains(&family)
}
}
impl std::error::Error for ToWireError {}
/// Why a value could not be written to the wire.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum EncodeError {
/// A length does not fit the field that has to carry it.
#[error("{what} is too long to encode: {len} exceeds {max}")]
TooLong {
what: &'static str,
len: usize,
max: usize,
},
}
// Custom error type for the parser.
#[derive(Debug)]
@@ -37,8 +45,8 @@ pub enum BgpParserError<I> {
/// ProtocolError encapsulates protocol level semantics problems that require
/// action from the caller of the parser.
ProtocolError(ProtocolErrorValues),
CustomText(&'static str),
Eyre(eyre::ErrReport),
/// The input does not follow the format; the text says how.
Malformed(String),
Nom(I, ErrorKind),
}
crates/primitives/src/poptrie/mod.rs
+14 -5
View File
@@ -1,3 +1,10 @@
//! A Poptrie (Asai and Ohara, SIGCOMM 2015): a compressed multiway trie for
//! longest-prefix matching.
//!
//! Work in progress. Nothing in the daemon uses it yet, and the construction
//! side has no public entry point, hence the lint expectation below.
#![expect(dead_code, reason = "not yet wired into a RIB")]
use std::collections::BTreeMap;
#[derive(Debug)]
@@ -75,7 +82,8 @@ impl<T: Clone> Poptrie<T> {
result
}
fn new(&mut self, rt: BTreeMap<(u32, u8), T>) {
/// Fills the trie from a routing table keyed by (address, prefix length).
fn build(&mut self, rt: BTreeMap<(u32, u8), T>) {
const DEPTH: u8 = 6;
let mut forest = vec![(0, rt)];
@@ -168,9 +176,10 @@ mod tests {
leaves: vec![],
};
poptrie.new(map);
println!("Success");
dbg!(poptrie);
poptrie.build(map);
assert!(
!poptrie.inners.is_empty(),
"building from a non-empty table yields nodes"
);
}
}
crates/rib/Cargo.toml
+16 -1
View File
@@ -1,9 +1,24 @@
[package]
name = "bgp-rib"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
bgp-packet.workspace = true
bgp-primitives.workspace = true
bytes.workspace = true
foldhash.workspace = true
hashbrown.workspace = true
serde.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["full"] }
tracing.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/rib/proptest-regressions/table/journal_tests.txt
+7
View File
@@ -0,0 +1,7 @@
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 7107cf5580fe7bb95e138f1fead386d89273532f83aa1d11808aa9586aa9da63 # shrinks to ops = [Withdraw { peer: 0, prefix: 1 }, Withdraw { peer: 0, prefix: 1 }, Poll { reader: 2, budget: 2 }, Withdraw { peer: 0, prefix: 10 }, Poll { reader: 2, budget: 4 }, Announce { peer: 2, prefix: 7, route: Route { local_pref: None, path: [1, 2], origin: 0, med: Some(0) } }, Announce { peer: 3, prefix: 6, route: Route { local_pref: None, path: [1], origin: 0, med: None } }, Announce { peer: 3, prefix: 6, route: Route { local_pref: None, path: [1, 1], origin: 0, med: None } }, Withdraw { peer: 1, prefix: 5 }, Withdraw { peer: 0, prefix: 11 }, Withdraw { peer: 3, prefix: 6 }, Announce { peer: 0, prefix: 3, route: Route { local_pref: None, path: [1, 2], origin: 1, med: None } }, Poll { reader: 2, budget: 3 }, Withdraw { peer: 0, prefix: 8 }, Flap { peer: 3 }, Withdraw { peer: 1, prefix: 1 }, Announce { peer: 2, prefix: 5, route: Route { local_pref: None, path: [1], origin: 1, med: Some(0) } }, Announce { peer: 0, prefix: 1, route: Route { local_pref: None, path: [1, 1], origin: 1, med: None } }, Withdraw { peer: 0, prefix: 1 }, Withdraw { peer: 2, prefix: 8 }, Poll { reader: 1, budget: 2 }, Flap { peer: 1 }, Poll { reader: 2, budget: 2 }, Announce { peer: 0, prefix: 9, route: Route { local_pref: None, path: [1], origin: 0, med: Some(1) } }, Announce { peer: 2, prefix: 2, route: Route { local_pref: Some(100), path: [2], origin: 0, med: Some(1) } }, Poll { reader: 1, budget: 3 }, Withdraw { peer: 3, prefix: 1 }, Announce { peer: 1, prefix: 4, route: Route { local_pref: None, path: [2], origin: 1, med: None } }, Announce { peer: 3, prefix: 7, route: Route { local_pref: None, path: [2], origin: 0, med: None } }, Announce { peer: 1, prefix: 11, route: Route { local_pref: None, path: [1], origin: 1, med: Some(1) } }, Withdraw { peer: 0, prefix: 6 }, Withdraw { peer: 0, prefix: 4 }, Announce { peer: 3, prefix: 10, route: Route { local_pref: Some(100), path: [2, 2], origin: 1, med: None } }, Announce { peer: 1, prefix: 2, route: Route { local_pref: None, path: [2], origin: 0, med: Some(2) } }, Announce { peer: 1, prefix: 3, route: Route { local_pref: None, path: [1], origin: 1, med: None } }, Withdraw { peer: 0, prefix: 2 }, Announce { peer: 1, prefix: 4, route: Route { local_pref: None, path: [1, 1], origin: 0, med: None } }, Poll { reader: 1, budget: 2 }, Poll { reader: 1, budget: 4 }, Poll { reader: 2, budget: 1 }, Poll { reader: 1, budget: 3 }, Poll { reader: 1, budget: 5 }, Poll { reader: 1, budget: 1 }, Withdraw { peer: 3, prefix: 5 }, Withdraw { peer: 3, prefix: 9 }, Withdraw { peer: 2, prefix: 10 }], shards = 3
crates/rib/proptest-regressions/table/tests.txt
+7
View File
@@ -0,0 +1,7 @@
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 6bcfc317c976928c2fbb29e943f385127b91770a76f18e7061b80b31f4276955 # shrinks to ops = [Announce { peer: 1, prefix: 1, route: Route { local_pref: None, path: [2], origin: 0, med: None } }, Announce { peer: 2, prefix: 1, route: Route { local_pref: None, path: [1], origin: 0, med: Some(2) } }, Announce { peer: 3, prefix: 1, route: Route { local_pref: None, path: [1], origin: 0, med: None } }, Announce { peer: 3, prefix: 1, route: Route { local_pref: None, path: [1], origin: 1, med: None } }]
crates/rib/src/adj_rib_in.rs
-96
View File
@@ -1,96 +0,0 @@
use std::collections::HashMap;
use bgp_primitives::ip_prefix::IpPrefix;
use tokio::sync::{broadcast, mpsc, oneshot};
use tracing::warn;
use crate::entry::{PeerId, RibEntry, SnapshotEntry};
use crate::event::RibEvent;
const EVENT_CHANNEL_CAPACITY: usize = 1024;
const CMD_CHANNEL_CAPACITY: usize = 256;
enum RibCommand {
Announce { peer: PeerId, prefix: IpPrefix, entry: RibEntry },
Withdraw { peer: PeerId, prefix: IpPrefix },
Snapshot { reply: oneshot::Sender<Vec<SnapshotEntry>> },
}
/// Handle to the Adj-RIB-In actor. Cheap to clone — all clones share the same
/// backing task. The actor is stopped when all handles are dropped.
#[derive(Clone)]
pub struct AdjRibIn {
tx: mpsc::Sender<RibCommand>,
events: broadcast::Sender<RibEvent>,
}
impl AdjRibIn {
/// Spawns the backing actor and returns a handle.
pub fn spawn() -> Self {
let (tx, rx) = mpsc::channel(CMD_CHANNEL_CAPACITY);
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let events_clone = events.clone();
tokio::spawn(run_actor(rx, events_clone));
Self { tx, events }
}
pub async fn announce(&self, peer: PeerId, prefix: IpPrefix, entry: RibEntry) {
let _ = self.tx.send(RibCommand::Announce { peer, prefix, entry }).await;
}
pub async fn withdraw(&self, peer: PeerId, prefix: IpPrefix) {
let _ = self.tx.send(RibCommand::Withdraw { peer, prefix }).await;
}
/// Returns a point-in-time snapshot of all routes in the RIB.
pub async fn snapshot(&self) -> Vec<SnapshotEntry> {
let (reply_tx, reply_rx) = oneshot::channel();
let _ = self.tx.send(RibCommand::Snapshot { reply: reply_tx }).await;
reply_rx.await.unwrap_or_default()
}
/// Subscribe to a stream of RIB mutation events.
///
/// The receiver will see every [`RibEvent`] from the point of subscription.
/// If the consumer falls behind by more than [`EVENT_CHANNEL_CAPACITY`]
/// events it will receive a [`broadcast::error::RecvError::Lagged`] error.
pub fn subscribe(&self) -> broadcast::Receiver<RibEvent> {
self.events.subscribe()
}
}
async fn run_actor(mut rx: mpsc::Receiver<RibCommand>, events: broadcast::Sender<RibEvent>) {
let mut table: HashMap<(PeerId, IpPrefix), RibEntry> = HashMap::new();
while let Some(cmd) = rx.recv().await {
match cmd {
RibCommand::Announce { peer, prefix, entry } => {
table.insert((peer, prefix.clone()), entry.clone());
if let Err(e) = events.send(RibEvent::Announced { peer, prefix, entry }) {
// No active subscribers — not an error.
let _ = e;
}
}
RibCommand::Withdraw { peer, prefix } => {
if table.remove(&(peer, prefix.clone())).is_some() {
if let Err(e) = events.send(RibEvent::Withdrawn { peer, prefix }) {
let _ = e;
}
}
}
RibCommand::Snapshot { reply } => {
let snap = table
.iter()
.map(|((peer, prefix), entry)| SnapshotEntry {
peer: *peer,
prefix: prefix.clone(),
entry: entry.clone(),
})
.collect();
if reply.send(snap).is_err() {
warn!("Snapshot reply channel closed before send");
}
}
}
}
}
crates/rib/src/entry.rs
-39
View File
@@ -1,39 +0,0 @@
use std::net::SocketAddr;
use std::time::SystemTime;
use bgp_packet::message::PathAttribute;
use bgp_primitives::ip_prefix::IpPrefix;
/// Identifies a BGP peer by its TCP address and negotiated ASN.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PeerId {
pub addr: SocketAddr,
pub asn: u32,
}
impl std::fmt::Display for PeerId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}(AS{})", self.addr, self.asn)
}
}
/// A single route stored in the Adj-RIB-In.
#[derive(Debug, Clone)]
pub struct RibEntry {
pub path_attributes: Vec<PathAttribute>,
pub received_at: SystemTime,
}
impl RibEntry {
pub fn new(path_attributes: Vec<PathAttribute>) -> Self {
Self { path_attributes, received_at: SystemTime::now() }
}
}
/// A flattened row returned by [`crate::AdjRibIn::snapshot`].
#[derive(Debug, Clone)]
pub struct SnapshotEntry {
pub peer: PeerId,
pub prefix: IpPrefix,
pub entry: RibEntry,
}
crates/rib/src/error.rs
+14
View File
@@ -0,0 +1,14 @@
/// A RIB's actor is no longer running, so the table can be neither read nor
/// changed.
///
/// The actor runs for as long as a handle to it exists; it stops early only
/// by panicking. That is a bug, and one the server cannot work around: it
/// could no longer say what it originates. Callers pass this up
/// until the process exits and is restarted, rather than carry on with a
/// table that looks empty.
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("the {rib} has stopped")]
pub struct RibStopped {
/// Which table.
pub rib: &'static str,
}
crates/rib/src/event.rs
-9
View File
@@ -1,9 +0,0 @@
use bgp_primitives::ip_prefix::IpPrefix;
use crate::entry::{PeerId, RibEntry};
#[derive(Debug, Clone)]
pub enum RibEvent {
Announced { peer: PeerId, prefix: IpPrefix, entry: RibEntry },
Withdrawn { peer: PeerId, prefix: IpPrefix },
}
crates/rib/src/lib.rs
+8 -6
View File
@@ -1,7 +1,9 @@
mod adj_rib_in;
mod entry;
mod event;
//! The router's tables: the routes it has learned and chosen
//! ([`table::RibTable`]) and the routes it originates ([`LocalRib`]).
pub use adj_rib_in::AdjRibIn;
pub use entry::{PeerId, RibEntry, SnapshotEntry};
pub use event::RibEvent;
mod error;
mod local_rib;
pub mod table;
pub use error::RibStopped;
pub use local_rib::{LocalRib, LocalRibEvent, LocalRoute, LocalRouteAttrs};
crates/rib/src/local_rib.rs
+262
View File
@@ -0,0 +1,262 @@
//! Locally originated routes (the "Loc-RIB" of routes this router announces
//! on its own behalf, as opposed to routes learned from peers).
//!
//! The actor stores one route per prefix together with its attributes. It
//! does not know about peers or sessions: the server subscribes to
//! [`LocalRibEvent`]s and turns them into UPDATEs.
use std::collections::HashMap;
use std::time::SystemTime;
use bgp_packet::message::{LargeCommunity, OriginPathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use serde::Serialize;
use tokio::sync::{broadcast, mpsc, oneshot};
use tokio::task::AbortHandle;
use crate::error::RibStopped;
const EVENT_CHANNEL_CAPACITY: usize = 1024;
const CMD_CHANNEL_CAPACITY: usize = 256;
/// Attributes of a locally originated route.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct LocalRouteAttrs {
pub origin: OriginPathAttribute,
/// Number of *additional* copies of the local ASN to prepend to the
/// AS_PATH. `0` yields the plain `[local_asn]` path.
pub prepend: u8,
pub med: Option<u32>,
/// RFC 1997 standard communities as `(high, low)` 16-bit halves.
pub communities: Vec<(u16, u16)>,
/// RFC 8092 large communities.
pub large_communities: Vec<LargeCommunity>,
}
impl Default for LocalRouteAttrs {
fn default() -> Self {
Self {
origin: OriginPathAttribute::IGP,
prepend: 0,
med: None,
communities: vec![],
large_communities: vec![],
}
}
}
/// A route originated by this router.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct LocalRoute {
pub prefix: IpPrefix,
pub attrs: LocalRouteAttrs,
pub announced_at: SystemTime,
}
impl LocalRoute {
pub fn new(prefix: IpPrefix, attrs: LocalRouteAttrs) -> Self {
Self {
prefix,
attrs,
announced_at: SystemTime::now(),
}
}
}
const STOPPED: RibStopped = RibStopped { rib: "local RIB" };
#[derive(Debug, Clone)]
pub enum LocalRibEvent {
/// A route was added or its attributes replaced.
Announced(LocalRoute),
Withdrawn {
prefix: IpPrefix,
},
}
enum Command {
Announce {
route: LocalRoute,
reply: oneshot::Sender<()>,
},
Withdraw {
prefix: IpPrefix,
reply: oneshot::Sender<Option<LocalRoute>>,
},
Snapshot {
reply: oneshot::Sender<Vec<LocalRoute>>,
},
Len {
reply: oneshot::Sender<usize>,
},
}
/// Handle to the local RIB actor. Cheap to clone; all clones share one table.
///
/// Every operation fails with [`RibStopped`] if the actor is gone; see
/// there for what that means.
#[derive(Clone)]
pub struct LocalRib {
tx: mpsc::Sender<Command>,
events: broadcast::Sender<LocalRibEvent>,
actor: AbortHandle,
}
impl LocalRib {
pub fn spawn() -> Self {
let (tx, rx) = mpsc::channel(CMD_CHANNEL_CAPACITY);
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let actor = tokio::spawn(run_actor(rx, events.clone())).abort_handle();
Self { tx, events, actor }
}
/// Adds a route, or replaces the existing route for the same prefix.
pub async fn announce(&self, route: LocalRoute) -> Result<(), RibStopped> {
self.ask(|reply| Command::Announce { route, reply }).await
}
/// Removes a route. Returns the removed route, or `None` if the prefix
/// was not present.
pub async fn withdraw(&self, prefix: IpPrefix) -> Result<Option<LocalRoute>, RibStopped> {
self.ask(|reply| Command::Withdraw { prefix, reply }).await
}
/// Point-in-time copy of every route, in no particular order.
pub async fn snapshot(&self) -> Result<Vec<LocalRoute>, RibStopped> {
self.ask(|reply| Command::Snapshot { reply }).await
}
/// How many routes the RIB holds, without copying them.
pub async fn len(&self) -> Result<usize, RibStopped> {
self.ask(|reply| Command::Len { reply }).await
}
/// Whether the RIB holds no routes.
pub async fn is_empty(&self) -> Result<bool, RibStopped> {
Ok(self.len().await? == 0)
}
/// Sends a command and waits for its answer.
async fn ask<T>(
&self,
command: impl FnOnce(oneshot::Sender<T>) -> Command,
) -> Result<T, RibStopped> {
let (reply, answer) = oneshot::channel();
self.tx.send(command(reply)).await.map_err(|_| STOPPED)?;
answer.await.map_err(|_| STOPPED)
}
/// Resolves when the actor has stopped. It never does in a healthy
/// process; whoever owns the RIB waits on this to notice a failure even
/// when nothing happens to be using the table.
pub async fn stopped(&self) {
self.tx.closed().await
}
/// Stops the actor, as a panic in it would. For tests of how the rest
/// of the server copes.
#[doc(hidden)]
pub fn stop_for_testing(&self) {
self.actor.abort();
}
/// Subscribe to mutations. A subscriber that falls too far behind is
/// told so with [`broadcast::error::RecvError::Lagged`].
pub fn subscribe(&self) -> broadcast::Receiver<LocalRibEvent> {
self.events.subscribe()
}
}
async fn run_actor(mut rx: mpsc::Receiver<Command>, events: broadcast::Sender<LocalRibEvent>) {
let mut table: HashMap<IpPrefix, LocalRoute> = HashMap::new();
while let Some(cmd) = rx.recv().await {
match cmd {
Command::Announce { route, reply } => {
table.insert(route.prefix, route.clone());
let _ = events.send(LocalRibEvent::Announced(route));
let _ = reply.send(());
}
Command::Withdraw { prefix, reply } => {
let removed = table.remove(&prefix);
if removed.is_some() {
let _ = events.send(LocalRibEvent::Withdrawn { prefix });
}
let _ = reply.send(removed);
}
Command::Snapshot { reply } => {
// A failed reply means the asker stopped waiting.
let _ = reply.send(table.values().cloned().collect());
}
Command::Len { reply } => {
let _ = reply.send(table.len());
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(p: &str) -> LocalRoute {
LocalRoute::new(prefix(p), LocalRouteAttrs::default())
}
#[tokio::test]
async fn announce_and_withdraw_emit_events() {
let rib = LocalRib::spawn();
let mut events = rib.subscribe();
rib.announce(route("10.0.0.0/24")).await.unwrap();
assert!(
matches!(events.recv().await.unwrap(), LocalRibEvent::Announced(r) if r.prefix == prefix("10.0.0.0/24"))
);
assert_eq!(rib.len().await, Ok(1));
let removed = rib.withdraw(prefix("10.0.0.0/24")).await.unwrap();
assert_eq!(removed.unwrap().prefix, prefix("10.0.0.0/24"));
assert!(
matches!(events.recv().await.unwrap(), LocalRibEvent::Withdrawn { prefix: p } if p == prefix("10.0.0.0/24"))
);
assert_eq!(rib.is_empty().await, Ok(true));
// Withdrawing an absent prefix is a no-op, not an error.
assert_eq!(rib.withdraw(prefix("10.0.0.0/24")).await.unwrap(), None);
}
#[tokio::test]
async fn announcing_a_prefix_again_replaces_its_attributes() {
let rib = LocalRib::spawn();
rib.announce(route("10.0.0.0/24")).await.unwrap();
let attrs = LocalRouteAttrs {
med: Some(50),
..Default::default()
};
rib.announce(LocalRoute::new(prefix("10.0.0.0/24"), attrs.clone()))
.await
.unwrap();
let table = rib.snapshot().await.unwrap();
assert_eq!(table.len(), 1);
assert_eq!(table[0].attrs, attrs);
}
#[tokio::test]
async fn a_stopped_actor_is_an_error_and_not_an_empty_table() {
let rib = LocalRib::spawn();
rib.announce(route("10.0.0.0/24")).await.unwrap();
rib.stop_for_testing();
rib.stopped().await;
assert_eq!(rib.announce(route("10.0.1.0/24")).await, Err(STOPPED));
assert_eq!(rib.withdraw(prefix("10.0.0.0/24")).await, Err(STOPPED));
assert_eq!(rib.snapshot().await, Err(STOPPED));
assert_eq!(rib.len().await, Err(STOPPED));
}
}
crates/rib/src/table/attrs.rs
+780
View File
@@ -0,0 +1,780 @@
//! Attribute sets, stored once.
//!
//! A full table has millions of routes and far fewer distinct lists of path
//! attributes: a peer announces prefixes that share their attributes
//! together, about six to an UPDATE on the Internet today. A route in the
//! table is therefore a pointer to an [`AttrSet`], and each distinct set
//! exists once, in the [`AttrStore`].
//!
//! A set is kept as its bytes on the wire, which is a fifth the size of the
//! parsed form. What choosing between routes needs from it is worked out
//! once, when the set is stored, and kept beside the bytes.
use std::hash::BuildHasher;
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use bgp_packet::message::{NlriNextHop, PathAttribute, RawAttribute, attr_type, split_attribute};
use bgp_primitives::parser::{EncodeError, ParserContext};
use bytes::{BufMut, BytesMut};
use foldhash::fast::RandomState;
use hashbrown::HashTable;
const OPTIONAL: u8 = 0x80;
/// The type of an AS_PATH segment whose AS numbers are in order; the other
/// type, AS_SET, is an unordered handful (RFC 4271 §4.3).
const AS_SEQUENCE: u8 = 2;
/// What route selection reads from a set of attributes (RFC 4271 §9.1.2.2).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SelectionKey {
/// LOCAL_PREF, if the set has one.
pub local_pref: Option<u32>,
/// MULTI_EXIT_DISC, if the set has one.
pub med: Option<u32>,
/// The AS that the route was learned from: the first AS of the path, or
/// zero if the path is empty or begins with a set. MEDs are comparable
/// only between routes with the same one.
pub neighbor_as: u32,
/// The length of the AS path as selection counts it: an AS_SET is one
/// hop however many AS numbers it holds.
pub as_path_len: u16,
/// ORIGIN: 0 for IGP, 1 for EGP, 2 for INCOMPLETE.
pub origin: u8,
/// The AS the route originates from: the last AS of the path. Zero,
/// which is no AS's number, if the path is empty or ends in a set and
/// so has no one origin. Not part of selection; kept here because it
/// is what a route's validity is asked about (RFC 6811), for every
/// route, when the validation data changes.
pub origin_as: u32,
}
/// Reads the selection key straight off attributes in the form an
/// [`AttrSet`] keeps, without building anything. `None` if the bytes are
/// not a well-formed list of attributes.
///
/// A list with the same attribute twice is not well formed (RFC 4271
/// §5). The decoder keeps only the first of such, so none comes from a
/// session; were one to get in some other way, the readers of a set, of
/// which this is one, could each take a different one of the two for the
/// route's path or next hop.
fn scan(mut wire: &[u8]) -> Option<SelectionKey> {
// One bit for each type of attribute there can be.
let mut seen = [0u64; 4];
let mut key = SelectionKey {
local_pref: None,
med: None,
neighbor_as: 0,
as_path_len: 0,
// A route without ORIGIN does not get this far; if one did, it
// would be the least preferred.
origin: 2,
origin_as: 0,
};
let number = |value: &[u8]| Some(u32::from_be_bytes(<[u8; 4]>::try_from(value).ok()?));
while !wire.is_empty() {
let (attribute, rest) = split_attribute(wire).ok()?;
wire = rest;
let value = attribute.value;
let (word, bit) = (
usize::from(attribute.type_code / 64),
1u64 << (attribute.type_code % 64),
);
if seen[word] & bit != 0 {
return None;
}
seen[word] |= bit;
match attribute.type_code {
attr_type::ORIGIN => key.origin = *value.first()?,
attr_type::AS_PATH => {
let path = scan_as_path(value)?;
key.as_path_len = path.hops;
key.neighbor_as = path.first;
key.origin_as = path.last;
}
attr_type::MULTI_EXIT_DISC => key.med = Some(number(value)?),
attr_type::LOCAL_PREF => key.local_pref = Some(number(value)?),
_ => {}
}
}
Some(key)
}
/// What the selection key takes from an AS path.
struct PathScan {
/// The length as selection counts it.
hops: u16,
/// The first AS, if the path begins with a sequence; else zero.
first: u32,
/// The last AS, if the path ends in a sequence; else zero.
last: u32,
}
impl SelectionKey {
/// The AS the route originates from: the last of its path. `None` if
/// the path is empty or ends in a set, which has no one last AS.
pub fn origin(&self) -> Option<u32> {
Some(self.origin_as).filter(|origin| *origin != 0)
}
}
/// Reads an AS_PATH value with four-octet AS numbers.
fn scan_as_path(mut segments: &[u8]) -> Option<PathScan> {
let mut hops = 0usize;
let (mut first, mut last) = (0, 0);
let mut at_start = true;
while let [kind, count, rest @ ..] = segments {
let (asns, rest) = rest.split_at_checked(usize::from(*count) * 4)?;
let ordered = *kind == AS_SEQUENCE;
hops += if ordered { usize::from(*count) } else { 1 };
let asn = |bytes: Option<&[u8; 4]>| bytes.map_or(0, |asn| u32::from_be_bytes(*asn));
if at_start && ordered {
first = asn(asns.first_chunk::<4>());
}
last = if ordered {
asn(asns.last_chunk::<4>())
} else {
0
};
at_start = false;
segments = rest;
}
if !segments.is_empty() {
return None;
}
Some(PathScan {
hops: u16::try_from(hops).unwrap_or(u16::MAX),
first,
last,
})
}
/// Appends the AS numbers of an AS_PATH or AS4_PATH value with four-octet
/// AS numbers to `out`, and returns whether any segment is not a sequence.
/// Stops at what is not a segment.
fn read_as_path(mut segments: &[u8], out: &mut Vec<u32>) -> bool {
let mut has_set = false;
while let [kind, count, rest @ ..] = segments {
let Some((asns, rest)) = rest.split_at_checked(usize::from(*count) * 4) else {
break;
};
has_set |= *kind != AS_SEQUENCE;
let (asns, _) = asns.as_chunks::<4>();
out.extend(asns.iter().map(|asn| u32::from_be_bytes(*asn)));
segments = rest;
}
has_set
}
/// One distinct list of path attributes, with the next hop that goes with
/// it. Made by an [`AttrStore`]; two routes have the same attributes
/// exactly when they point to the same `AttrSet`.
#[derive(Debug)]
pub struct AttrSet {
hash: u64,
key: SelectionKey,
/// The attributes as on the wire, with four-octet AS numbers. A next
/// hop that arrived in MP_REACH_NLRI is kept as an MP_REACH_NLRI that
/// holds nothing else, the form RFC 6396 §4.3.4 uses in table dumps.
wire: Box<[u8]>,
}
impl AttrSet {
pub fn key(&self) -> &SelectionKey {
&self.key
}
/// The attributes as they are on the wire.
pub fn wire(&self) -> &[u8] {
&self.wire
}
/// The attributes one by one, as they lie in the stored bytes. The
/// bytes were checked when the set was made, so this is all of them;
/// it stops at the first that does not split as an attribute all the
/// same, rather than go on from the middle of one.
fn raw_attributes(&self) -> impl Iterator<Item = RawAttribute<'_>> {
let mut wire = &*self.wire;
std::iter::from_fn(move || {
let (attribute, rest) = split_attribute(wire).ok()?;
wire = rest;
Some(attribute)
})
}
/// The value of the attribute of type `type_code`, if the set has one.
fn attribute(&self, type_code: u8) -> Option<&[u8]> {
self.raw_attributes()
.find(|attribute| attribute.type_code == type_code)
.map(|attribute| attribute.value)
}
/// The route's next hop: the one that came in MP_REACH_NLRI, or else
/// the NEXT_HOP attribute. Read off the stored bytes without building
/// anything else, for those who ask it of every route.
pub fn next_hop(&self) -> Option<NlriNextHop> {
let mut plain = None;
for attribute in self.raw_attributes() {
match attribute.type_code {
attr_type::MP_REACH_NLRI => return decode_next_hop(attribute.value),
attr_type::NEXT_HOP => plain = NlriNextHop::from_bytes(attribute.value),
_ => {}
}
}
plain
}
/// Whether the route carries the OTC attribute (RFC 9234): it came
/// down from a provider or across from a peer, and is to go on down
/// only. Read off the stored bytes without building anything else,
/// for those who ask it of every route: a session to a peer or a
/// provider asks it of most of the table, to pass over them.
pub fn is_only_to_customer(&self) -> bool {
self.attribute(attr_type::ONLY_TO_CUSTOMER).is_some()
}
/// Whether the route carries the BLACKHOLE community (RFC 7999), which
/// asks for the packets to its prefix to be dropped. Read off the
/// stored bytes without building anything else, for those who ask it
/// of every route.
pub fn is_blackhole(&self) -> bool {
const BLACKHOLE: [u8; 4] = [0xFF, 0xFF, 0x02, 0x9A];
self.attribute(attr_type::COMMUNITIES)
.is_some_and(|communities| communities.as_chunks::<4>().0.contains(&BLACKHOLE))
}
/// Puts the AS numbers of the route's path in `out`, in place of what
/// was there: in the order of the path, the neighbor's first and the
/// origin's last, sets and all. Returns whether the path has a set in
/// it.
///
/// This is the whole path: what a session with two-octet AS numbers
/// brought in AS4_PATH was put in its place when the UPDATE was read
/// (RFC 6793 §4.2.3). It reads the stored bytes and builds nothing
/// else, for those who ask it of every route.
pub fn as_path(&self, out: &mut Vec<u32>) -> bool {
out.clear();
match self.attribute(attr_type::AS_PATH) {
Some(segments) => read_as_path(segments, out),
None => false,
}
}
/// The attributes, parsed, and the next hop if it came in
/// MP_REACH_NLRI.
///
/// `None` if they do not parse. Sets made from parsed attributes or
/// from an UPDATE the decoder accepted always do; one made from bytes
/// of another source ([`AttrStore::intern_wire`]) was checked only as
/// far as selection reads it.
pub fn attributes(&self) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
parse(&self.wire)
}
}
/// How the attributes of a set are written: with four-octet AS numbers.
fn four_octet() -> ParserContext {
ParserContext {
four_octet_asn: true,
ibgp: true,
add_path: Vec::new(),
}
}
/// Writes `attributes` and `next_hop` in the form an [`AttrSet`] keeps.
fn encode(
attributes: &[PathAttribute],
next_hop: Option<&NlriNextHop>,
out: &mut BytesMut,
) -> Result<(), EncodeError> {
let ctx = four_octet();
for attribute in attributes {
// These name prefixes, which are not a property of the route.
if matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
) {
continue;
}
attribute.to_wire(&ctx, out)?;
}
if let Some(next_hop) = next_hop {
encode_next_hop(next_hop, out);
}
Ok(())
}
/// Writes a next hop as an MP_REACH_NLRI that holds nothing else: the
/// next hop's length, and the next hop.
fn encode_next_hop(next_hop: &NlriNextHop, out: &mut BytesMut) {
out.put_u8(OPTIONAL);
out.put_u8(attr_type::MP_REACH_NLRI);
out.put_u8(1 + next_hop.wire_len());
out.put_u8(next_hop.wire_len());
next_hop.to_wire(out);
}
/// Reads the next hop out of the value of an MP_REACH_NLRI as
/// [`encode_next_hop`] writes it. `None` if the value is not one.
fn decode_next_hop(value: &[u8]) -> Option<NlriNextHop> {
let (length, address) = value.split_first()?;
NlriNextHop::from_bytes(address).filter(|_| usize::from(*length) == address.len())
}
/// Parses path attributes in the form [`AttrSet`] keeps and table dumps
/// use (RFC 6396 §4.3.4): four-octet AS numbers, and an MP_REACH_NLRI that
/// holds only a next hop. `None` if they are malformed.
fn parse(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
let ctx = four_octet();
let mut attributes = Vec::new();
let mut next_hop = None;
while !wire.is_empty() {
let (raw, rest) = split_attribute(wire).ok()?;
if raw.type_code == attr_type::MP_REACH_NLRI {
next_hop = Some(decode_next_hop(raw.value)?);
} else {
let (_, attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()?;
attributes.push(attribute);
}
wire = rest;
}
Some((attributes, next_hop))
}
/// How many locks the store is split over. Sessions intern concurrently,
/// once per UPDATE; this many locks keep them out of each other's way.
const STRIPES: usize = 64;
/// The one place attribute sets are kept. Safe to share between threads.
pub struct AttrStore {
stripes: Box<[Mutex<HashTable<Arc<AttrSet>>>]>,
hasher: RandomState,
}
impl Default for AttrStore {
fn default() -> Self {
Self::new()
}
}
impl AttrStore {
pub fn new() -> Self {
Self {
stripes: (0..STRIPES).map(|_| Mutex::default()).collect(),
hasher: RandomState::default(),
}
}
/// The stored set equal to `attributes` and `next_hop`, added if it was
/// not there. `next_hop` is the one from MP_REACH_NLRI; routes whose
/// next hop is the NEXT_HOP attribute pass `None`.
///
/// Fails only if the attributes cannot be encoded, which attributes that
/// were decoded from a message always can.
pub fn intern(
&self,
attributes: &[PathAttribute],
next_hop: Option<&NlriNextHop>,
) -> Result<Arc<AttrSet>, EncodeError> {
let mut wire = BytesMut::with_capacity(256);
encode(attributes, next_hop, &mut wire)?;
Ok(self
.intern_wire(&wire)
.expect("attributes written here are well formed"))
}
/// As [`AttrStore::intern`], for attributes that are already in the
/// form a set keeps: on the wire, with four-octet AS numbers, no
/// MP_UNREACH_NLRI, and an MP_REACH_NLRI (if any) that holds only the
/// next hop. That is how a table dump has them (RFC 6396 §4.3.4).
///
/// Nothing is parsed: finding a set that is already stored costs a hash
/// of the bytes and a comparison. `None` if the bytes are not a
/// well-formed list of attributes.
pub fn intern_wire(&self, wire: &[u8]) -> Option<Arc<AttrSet>> {
let hash = self.hasher.hash_one(wire);
let mut stripe = self.stripe(hash);
if let Some(existing) = stripe.find(hash, |set| *set.wire == *wire) {
return Some(Arc::clone(existing));
}
let set = Arc::new(AttrSet {
hash,
key: scan(wire)?,
wire: wire.into(),
});
stripe.insert_unique(hash, Arc::clone(&set), |set| set.hash);
Some(set)
}
/// The set for attributes as a session received them: `wire` is what
/// the decoder kept of the UPDATE's attributes
/// (`DecodedUpdate::attributes_wire`), and `next_hop` the next hop of
/// the routes if they came in MP_REACH_NLRI.
pub fn intern_received(
&self,
wire: &[u8],
next_hop: Option<&NlriNextHop>,
) -> Option<Arc<AttrSet>> {
self.intern_received_with(wire, &[], next_hop)
}
/// As [`AttrStore::intern_received`], with attributes of our own after
/// the peer's: `added` is one or more whole attributes in the same
/// form, of types that `wire` does not have.
pub fn intern_received_with(
&self,
wire: &[u8],
added: &[u8],
next_hop: Option<&NlriNextHop>,
) -> Option<Arc<AttrSet>> {
if added.is_empty() && next_hop.is_none() {
return self.intern_wire(wire);
}
let mut whole = BytesMut::with_capacity(wire.len() + added.len() + 36);
whole.put_slice(wire);
whole.put_slice(added);
if let Some(next_hop) = next_hop {
encode_next_hop(next_hop, &mut whole);
}
self.intern_wire(&whole)
}
/// How many distinct sets are stored.
pub fn len(&self) -> usize {
(0..STRIPES).map(|i| self.stripe_at(i).len()).sum()
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// Drops the sets that no route refers to any more, and returns how
/// many that was. Sets are not dropped as their last route goes, since
/// that would mean a lock here for every route removed; this is called
/// now and then instead.
pub fn sweep(&self) -> usize {
let mut dropped = 0;
for i in 0..STRIPES {
let mut stripe = self.stripe_at(i);
let before = stripe.len();
// The store's own reference is the only one left. Nobody can
// be taking another: that is done under this lock.
stripe.retain(|set| Arc::strong_count(set) > 1);
dropped += before - stripe.len();
}
dropped
}
fn stripe(&self, hash: u64) -> MutexGuard<'_, HashTable<Arc<AttrSet>>> {
self.stripe_at(super::spread(hash, STRIPES))
}
fn stripe_at(&self, index: usize) -> MutexGuard<'_, HashTable<Arc<AttrSet>>> {
// A panic while holding the lock cannot leave the table half
// changed in a way that matters: see `IgnorePoison` in the server.
self.stripes[index]
.lock()
.unwrap_or_else(PoisonError::into_inner)
}
}
#[cfg(test)]
mod tests {
use std::net::Ipv4Addr;
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LocalPrefPathAttribute,
MultiExitDiscPathAttribute, NextHopPathAttribute, OriginPathAttribute,
};
use super::*;
fn attributes(path: Vec<u32>) -> Vec<PathAttribute> {
vec![
PathAttribute::Origin(OriginPathAttribute::EGP),
AsPathAttribute::from_asns(path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65000, 1)])),
]
}
#[test]
fn equal_attributes_are_one_set() {
let store = AttrStore::new();
let a = store.intern(&attributes(vec![65001, 65002]), None).unwrap();
let b = store.intern(&attributes(vec![65001, 65002]), None).unwrap();
let c = store.intern(&attributes(vec![65001, 65003]), None).unwrap();
assert!(Arc::ptr_eq(&a, &b));
assert!(!Arc::ptr_eq(&a, &c));
assert_eq!(store.len(), 2);
// The next hop is part of what a set is.
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let d = store
.intern(&attributes(vec![65001, 65002]), Some(&next_hop))
.unwrap();
assert!(!Arc::ptr_eq(&a, &d));
assert_eq!(store.len(), 3);
}
#[test]
fn a_set_gives_back_what_went_in() {
let store = AttrStore::new();
let next_hop = NlriNextHop::Ipv6WithLl {
global: "2001:db8::1".parse().unwrap(),
link_local: "fe80::1".parse().unwrap(),
};
let original = attributes(vec![65001, 4_200_000_001]);
let set = store.intern(&original, Some(&next_hop)).unwrap();
assert_eq!(set.attributes(), Some((original, Some(next_hop))));
}
#[test]
fn the_selection_key_is_read_off_the_attributes() {
let store = AttrStore::new();
let mut attrs = vec![
PathAttribute::Origin(OriginPathAttribute::INCOMPLETE),
PathAttribute::AsPath(AsPathAttribute {
segments: vec![
AsPathSegment {
ordered: true,
path: vec![65001, 65002],
},
// A set counts as one hop.
AsPathSegment {
ordered: false,
path: vec![65003, 65004, 65005],
},
],
}),
PathAttribute::LocalPref(LocalPrefPathAttribute(200)),
];
let key = *store.intern(&attrs, None).unwrap().key();
assert_eq!(
key,
SelectionKey {
local_pref: Some(200),
med: None,
neighbor_as: 65001,
as_path_len: 3,
origin: 2,
// The path ends in a set: it has no one origin.
origin_as: 0,
}
);
attrs.push(PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(7)));
assert_eq!(store.intern(&attrs, None).unwrap().key().med, Some(7));
}
#[test]
fn the_next_hop_is_read_off_a_set() {
let store = AttrStore::new();
let plain = store.intern(&attributes(vec![3333]), None).unwrap();
assert_eq!(
plain.next_hop(),
Some(NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 1)))
);
// The one that came with the routes counts, where there is one.
let with_link_local = NlriNextHop::Ipv6WithLl {
global: "2001:db8::1".parse().unwrap(),
link_local: "fe80::1".parse().unwrap(),
};
let v6 = store
.intern(&attributes(vec![3333]), Some(&with_link_local))
.unwrap();
assert_eq!(v6.next_hop(), Some(with_link_local));
// A route of our own making may have none.
let none = store
.intern(&[PathAttribute::Origin(OriginPathAttribute::IGP)], None)
.unwrap();
assert_eq!(none.next_hop(), None);
}
#[test]
fn the_path_is_read_off_a_set_as_it_really_is() {
use bgp_packet::message::{AsPathAttribute, AsPathSegment};
let store = AttrStore::new();
let mut path = Vec::new();
let set = store
.intern(&attributes(vec![3333, 3333, 1299]), None)
.unwrap();
assert!(!set.as_path(&mut path));
assert_eq!(path, [3333, 3333, 1299]);
// A set is said to be there.
let sequence = |path: Vec<u32>| AsPathSegment {
ordered: true,
path,
};
let mut with_set = attributes(vec![]);
with_set[1] = PathAttribute::AsPath(AsPathAttribute {
segments: vec![
sequence(vec![3333]),
AsPathSegment {
ordered: false,
path: vec![64500, 64501],
},
],
});
let set = store.intern(&with_set, None).unwrap();
assert!(set.as_path(&mut path));
assert_eq!(path, [3333, 64500, 64501]);
// No path at all.
let set = store.intern(&attributes(vec![]), None).unwrap();
assert!(!set.as_path(&mut path));
assert!(path.is_empty());
}
/// The BLACKHOLE community is read off the stored bytes.
#[test]
fn a_blackhole_request_is_told_from_the_bytes() {
let store = AttrStore::new();
let with = |communities: Vec<(u16, u16)>| {
let mut attributes = attributes(vec![3333]);
attributes.retain(|attribute| !matches!(attribute, PathAttribute::Communities(_)));
attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
communities,
)));
store.intern(&attributes, None).unwrap()
};
assert!(with(vec![(3333, 100), (0xFFFF, 0x029A)]).is_blackhole());
assert!(!with(vec![(3333, 666), (0xFFFF, 0xFF01)]).is_blackhole());
assert!(
!store
.intern(&attributes(vec![3333]), None)
.unwrap()
.is_blackhole()
);
}
/// The OTC attribute is read off the stored bytes, wherever in them it
/// lies.
#[test]
fn otc_is_told_from_the_bytes() {
let store = AttrStore::new();
let without = store.intern(&attributes(vec![3333]), None).unwrap();
assert!(!without.is_only_to_customer());
let mut with = attributes(vec![3333]);
with.insert(1, PathAttribute::OnlyToCustomer(3333));
assert!(store.intern(&with, None).unwrap().is_only_to_customer());
// After COMMUNITIES, which is the last of the usual attributes.
let mut after = attributes(vec![3333]);
after.push(PathAttribute::OnlyToCustomer(3333));
assert!(store.intern(&after, None).unwrap().is_only_to_customer());
}
/// What a session received, with attributes of our own added after it,
/// is one set with the parsed form of the whole; the bytes are not
/// parsed to find that out, so the bytes must say the same thing.
#[test]
fn received_attributes_with_ours_added_are_the_same_set_as_the_parsed_form() {
let store = AttrStore::new();
let ctx = four_octet();
let received = attributes(vec![65001, 65002]);
let mut wire = BytesMut::new();
encode(&received, None, &mut wire).unwrap();
let ours = [
PathAttribute::OnlyToCustomer(65001),
PathAttribute::LocalPref(LocalPrefPathAttribute(150)),
];
let mut added = BytesMut::new();
for attribute in &ours {
attribute.to_wire(&ctx, &mut added).unwrap();
}
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let mut whole = received.clone();
whole.extend(ours.iter().cloned());
let parsed = store.intern(&whole, Some(&next_hop)).unwrap();
let from_bytes = store
.intern_received_with(&wire, &added, Some(&next_hop))
.unwrap();
assert!(Arc::ptr_eq(&parsed, &from_bytes));
assert!(from_bytes.is_only_to_customer());
assert_eq!(from_bytes.next_hop(), Some(next_hop));
// Without a next hop too.
let parsed = store.intern(&whole, None).unwrap();
let from_bytes = store.intern_received_with(&wire, &added, None).unwrap();
assert!(Arc::ptr_eq(&parsed, &from_bytes));
// An attribute of ours of a type the peer sent is not a set.
let mut repeated = BytesMut::new();
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(1))
.to_wire(&ctx, &mut repeated)
.unwrap();
assert!(store.intern_received_with(&wire, &repeated, None).is_none());
}
/// Two of the same attribute are not a set: its readers could each
/// take a different one.
#[test]
fn the_same_attribute_twice_is_not_a_set() {
let store = AttrStore::new();
let origin = [0x40, 1, 1, 0];
let path = |asn: u8| [0x40, 2, 6, 2, 1, 0, 0, 0, asn];
let once = [&origin[..], &path(1)].concat();
assert!(store.intern_wire(&once).is_some());
let twice = [&origin[..], &path(1), &path(2)].concat();
assert!(store.intern_wire(&twice).is_none());
// Whatever the type, known here or not.
let unknown = [0xc0, 200, 1, 7];
assert!(store.intern_wire(&[&once[..], &unknown].concat()).is_some());
assert!(
store
.intern_wire(&[&once[..], &unknown, &unknown].concat())
.is_none()
);
}
#[test]
fn attributes_on_the_wire_are_interned_without_parsing_them() {
let store = AttrStore::new();
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let mut attrs = attributes(vec![65001, 65002, 65003]);
attrs.push(PathAttribute::LocalPref(LocalPrefPathAttribute(300)));
let parsed = store.intern(&attrs, Some(&next_hop)).unwrap();
// The same set by its bytes is the same set, key and all.
let by_wire = store.intern_wire(parsed.wire()).unwrap();
assert!(Arc::ptr_eq(&parsed, &by_wire));
// And one that arrives as bytes first gets the same key as it would
// have had from the parsed form.
let other = AttrStore::new();
let fresh = other.intern_wire(parsed.wire()).unwrap();
assert_eq!(fresh.key(), parsed.key());
assert_eq!(fresh.key().as_path_len, 3);
assert_eq!(fresh.key().neighbor_as, 65001);
assert_eq!(fresh.key().origin_as, 65003);
assert_eq!(fresh.attributes(), Some((attrs, Some(next_hop))));
// Bytes that are not attributes are refused.
assert!(other.intern_wire(&[0x40, 0x01, 0x05, 0x00]).is_none());
assert!(
other
.intern_wire(&[0x40, 0x02, 0x03, 0x02, 0x01, 0x00])
.is_none()
);
}
#[test]
fn sweeping_drops_only_sets_nobody_holds() {
let store = AttrStore::new();
let kept = store.intern(&attributes(vec![65001]), None).unwrap();
drop(store.intern(&attributes(vec![65002]), None).unwrap());
assert_eq!(store.len(), 2);
assert_eq!(store.sweep(), 1);
assert_eq!(store.len(), 1);
// The one that is left is still the one that is found.
let again = store.intern(&attributes(vec![65001]), None).unwrap();
assert!(Arc::ptr_eq(&kept, &again));
}
}
crates/rib/src/table/journal.rs
+950
View File
@@ -0,0 +1,950 @@
//! The journal: how the rest of the router follows the best routes.
//!
//! Whatever acts on the table's choices (the peers that are sent the best
//! routes, the forwarding table, the status page) has to learn when a best
//! route changes, at its own pace, without the table waiting for it. Each
//! shard keeps a journal for that, and each follower reads it through a
//! [`Reader`].
//!
//! # What the journal holds
//!
//! It records *that* a prefix's best route changed, not what it changed to.
//! A reader that comes to an entry looks the prefix up and is given the best
//! route as it is at that moment. A follower does not want the history; it
//! wants to end up with what the table has now.
//!
//! This is what keeps the journal small. If a prefix changes again before a
//! reader has got to its entry, the older entry no longer says anything: the
//! newer one will deliver the current route. Each row remembers the
//! sequence number of its latest change, an entry with any other number is
//! dead, and dead entries are thrown out when the journal is tidied. So a
//! prefix that flaps a thousand times costs a reader one event, and the
//! journal never has more live entries than the table has prefixes.
//!
//! # Readers that fall behind
//!
//! Entries are dropped once every reader has passed them. A reader that
//! stops reading would hold them all back, so there is a limit: when the
//! journal is too long even after tidying, its oldest half is dropped
//! regardless. A reader that was behind that point is told so the next time
//! it reads ([`Event::Reset`]) and is given the whole table again. Reading
//! too slowly costs the slow reader a resynchronisation; it costs the
//! table, and the other readers, nothing.
//!
//! A new reader starts the same way: `Reset`, every best route, `Synced`,
//! and changes from then on.
//!
//! # All routes, or only the best
//!
//! There are two journals in each shard. One records changes of best route,
//! for followers that want one route for each prefix: the forwarding table,
//! and peers that are sent the best route. The other records every change
//! to any route, for followers that want them all: peers that are sent
//! several routes for a prefix (ADD-PATH, RFC 7911), or a choice made for
//! them alone. A [`Reader`] follows one or the other. The second journal
//! costs nothing while nobody follows it.
//!
//! # Order
//!
//! For any one prefix, a reader is given its states in the order they came
//! about, and always, in the end, the latest. Between prefixes there is no
//! order worth relying on.
use std::collections::VecDeque;
use std::hash::BuildHasher;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use bgp_primitives::ip_prefix::IpPrefix;
use foldhash::fast::RandomState;
use hashbrown::{HashMap, HashTable};
use super::{AttrSet, PeerIndex, RibTable, RouteRef, Row, Shard, read, write};
/// A cursor that holds nothing back: a reader slot that is free, or a
/// reader that has not read this shard yet.
const NO_CURSOR: u64 = u64::MAX;
/// When a journal is tidied, and when it is too long.
#[derive(Debug, Clone, Copy)]
pub(super) struct Limits {
/// Below this many entries a journal is not worth tidying.
pub(super) tidy_at: usize,
/// Room allowed beyond the number of prefixes before the journal is
/// thought too long.
pub(super) slack: usize,
}
impl Default for Limits {
fn default() -> Self {
Self {
tidy_at: 1024,
slack: 64,
}
}
}
struct Entry {
seq: u64,
prefix: IpPrefix,
/// When, by the table's clock: the same for every change one call
/// made, in whatever shard. A reader gives out all the changes up to
/// a time or none of them, so that what was changed together is read
/// together.
stamp: u64,
}
/// One journal of one shard.
pub(super) struct Journal {
kind: Kind,
/// The sequence number of the latest change. Counts every change this
/// journal is for, whether or not anyone was reading.
seq: u64,
/// Entries up to and including this number were dropped without regard
/// to the readers. A reader whose cursor is below it missed something.
floor: u64,
/// In order of sequence number. Not every number is here: dead entries
/// are removed, and nothing is recorded while there are no readers.
entries: VecDeque<Entry>,
/// Prefixes that have no route any more and whose disappearance is
/// still to be read, with the sequence number of the entry that says
/// so. A row keeps this number itself; a prefix without a row cannot.
gone: HashMap<IpPrefix, u64, RandomState>,
/// Where each reader has read to, by reader slot. Written by the
/// readers themselves while they hold the shard for reading.
cursors: Vec<AtomicU64>,
/// How many reader slots are in use.
readers: usize,
limits: Limits,
/// The time the entries written now are given; see [`Entry::stamp`].
stamp: u64,
}
impl Journal {
fn new(kind: Kind, limits: Limits) -> Self {
Self {
kind,
limits,
seq: 0,
floor: 0,
entries: VecDeque::new(),
gone: HashMap::default(),
cursors: Vec::new(),
readers: 0,
stamp: 0,
}
}
/// Notes a change to `prefix`, which has a row. Returns the sequence
/// number for the row to remember.
fn changed(&mut self, prefix: IpPrefix) -> u64 {
self.seq += 1;
if self.readers > 0 {
self.entries.push_back(Entry {
seq: self.seq,
prefix,
stamp: self.stamp,
});
}
self.seq
}
/// As [`Journal::changed`], for a prefix whose row was just made:
/// if it had gone and is back, the note that it had gone is void.
fn appeared(&mut self, prefix: IpPrefix) -> u64 {
if !self.gone.is_empty() {
self.gone.remove(&prefix);
}
self.changed(prefix)
}
/// Notes that `prefix` lost its last route and has no row any more.
fn disappeared(&mut self, prefix: IpPrefix) {
self.seq += 1;
if self.readers > 0 {
self.entries.push_back(Entry {
seq: self.seq,
prefix,
stamp: self.stamp,
});
self.gone.insert(prefix, self.seq);
}
}
/// Voids the note that `entry`'s prefix had gone, if `entry` is what
/// said so: an entry that is dropped must not leave the note behind,
/// or the prefix would be given out as gone on the strength of a later
/// entry that says something else.
fn void_gone_note(gone: &mut HashMap<IpPrefix, u64, RandomState>, entry: &Entry) {
if gone.get(&entry.prefix) == Some(&entry.seq) {
gone.remove(&entry.prefix);
}
}
/// What `entry` says: something only if it is the latest change of its
/// prefix.
fn says<'a>(&self, entry: &Entry, rows: &'a HashTable<Row>, hasher: &RandomState) -> Says<'a> {
let hash = hasher.hash_one(entry.prefix);
match rows.find(hash, |row| row.prefix == entry.prefix) {
Some(row) if *row.seqs.of(self.kind) == entry.seq => Says::Row(row),
None if self.gone.get(&entry.prefix) == Some(&entry.seq) => Says::Gone,
_ => Says::Nothing,
}
}
/// Where a reader whose cursor is `cursor` is to read from.
fn place_of(&self, cursor: u64) -> Place {
if cursor < self.floor {
Place::Lagged
} else if cursor == self.seq {
Place::Nothing
} else {
Place::From(self.entries.partition_point(|entry| entry.seq <= cursor))
}
}
/// Keeps the journal from growing. Called after every change to the
/// shard; does nothing until there is something worth doing.
fn tidy(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
if self.entries.len() <= self.limits.tidy_at {
return;
}
self.drop_passed();
// Entries for prefixes that have a row are at most as many as the
// rows once the dead ones are out. Entries for prefixes that have
// gone are not counted in: they are what a reader that has stopped
// reading makes pile up. Twice that, so that the pass over every
// entry that finding the dead ones takes is made only once the
// journal has doubled, and the cost per change stays small.
let limit = 2 * (rows.len() + self.limits.slack);
if self.entries.len() <= limit {
return;
}
self.drop_dead(rows, hasher);
if self.entries.len() <= limit {
return;
}
self.drop_oldest_half();
}
/// Drops what every reader has passed.
fn drop_passed(&mut self) {
let passed = self
.cursors
.iter()
.map(|cursor| cursor.load(Ordering::Acquire))
.min()
.unwrap_or(NO_CURSOR);
while let Some(front) = self.entries.front() {
if front.seq > passed {
break;
}
Self::void_gone_note(&mut self.gone, front);
self.entries.pop_front();
}
}
/// Drops the dead entries: those whose prefix has changed again since,
/// so that a later entry says what they said. This looks every entry
/// up.
fn drop_dead(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
let mut entries = std::mem::take(&mut self.entries);
entries.retain(|entry| !matches!(self.says(entry, rows, hasher), Says::Nothing));
self.entries = entries;
}
/// Drops the older half of the entries without regard to the readers,
/// for a journal that is too long even without its dead entries: a
/// reader is not reading while prefixes come and go, or more prefixes
/// went at once than are left. A reader that had not read what goes
/// loses its place. Half, so that readers that are only a little
/// behind keep theirs, and so that getting a long journal down to size
/// takes few passes.
fn drop_oldest_half(&mut self) {
for entry in self.entries.drain(..self.entries.len() / 2) {
Self::void_gone_note(&mut self.gone, &entry);
self.floor = entry.seq;
}
}
}
/// What an entry says; see [`Journal::says`].
enum Says<'a> {
/// The prefix's row, whose latest change the entry is.
Row(&'a Row),
/// The prefix has no route any more, and the entry is what says so.
Gone,
/// Nothing: the prefix has changed again since, and a later entry has
/// it.
Nothing,
}
/// Where a reader is to read a journal from; see [`Journal::place_of`].
enum Place {
/// Entries the reader had not read were dropped; it has lost its
/// place.
Lagged,
/// The reader has read everything up to the latest change.
Nothing,
/// The index of the first entry the reader has not read.
From(usize),
}
/// Which of a shard's two journals.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum Kind {
/// Changes of best route.
Best,
/// Changes of any route.
All,
}
/// One of something for each [`Kind`] of journal.
#[derive(Debug, Clone, Copy, Default)]
pub(super) struct ByKind<T> {
best: T,
all: T,
}
impl<T> ByKind<T> {
pub(super) fn of(&self, kind: Kind) -> &T {
match kind {
Kind::Best => &self.best,
Kind::All => &self.all,
}
}
pub(super) fn of_mut(&mut self, kind: Kind) -> &mut T {
match kind {
Kind::Best => &mut self.best,
Kind::All => &mut self.all,
}
}
}
/// The sequence numbers a row remembers, one for each journal: of the
/// latest change of its best route, and of any of its routes.
pub(super) type Seqs = ByKind<u64>;
/// The two journals of a shard, written together: every change to a row is
/// a change of one of its routes, and some are a change of its best route.
pub(super) type Journals = ByKind<Journal>;
impl Journals {
pub(super) fn new(limits: Limits) -> Self {
Self {
best: Journal::new(Kind::Best, limits),
all: Journal::new(Kind::All, limits),
}
}
/// Copies the journals' sequence numbers to `latest`, which the table
/// keeps outside the shard's lock for readers to look at
/// ([`RibTable::latest_seq`]). Called with the lock held, after every
/// change to the shard.
pub(super) fn publish(&self, latest: &ByKind<AtomicU64>) {
latest.best.store(self.best.seq, Ordering::Release);
latest.all.store(self.all.seq, Ordering::Release);
}
/// The time by the table's clock that the changes made from now on
/// are given; see [`Entry::stamp`].
pub(super) fn set_stamp(&mut self, stamp: u64) {
self.best.stamp = stamp;
self.all.stamp = stamp;
}
/// Notes that `prefix` has its first route, and so a row. Returns the
/// sequence numbers for the new row.
pub(super) fn appeared(&mut self, prefix: IpPrefix) -> Seqs {
Seqs {
best: self.best.appeared(prefix),
all: self.all.appeared(prefix),
}
}
/// Notes that a route of `row` changed, and whether its best route did.
pub(super) fn changed(&mut self, row: &mut Row, best_changed: bool) {
if best_changed {
row.seqs.best = self.best.changed(row.prefix);
}
row.seqs.all = self.all.changed(row.prefix);
}
/// Notes that `prefix` lost its last route and has no row any more.
pub(super) fn disappeared(&mut self, prefix: IpPrefix) {
self.best.disappeared(prefix);
self.all.disappeared(prefix);
}
/// Keeps the journals from growing; see [`Journal::tidy`].
pub(super) fn tidy(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
self.best.tidy(rows, hasher);
self.all.tidy(rows, hasher);
}
/// Where the journals are now, to ask [`Journals::has_news_since`]
/// about later.
pub(super) fn mark(&self) -> Mark {
Mark(self.best.seq, self.all.seq)
}
/// Whether anything was written since `mark` that some reader is there
/// to read.
pub(super) fn has_news_since(&self, mark: Mark) -> bool {
(self.best.readers > 0 && self.best.seq != mark.0)
|| (self.all.readers > 0 && self.all.seq != mark.1)
}
fn len(&self) -> usize {
self.best.entries.len() + self.all.entries.len()
}
}
/// A point in a shard's journals; see [`Journals::mark`].
#[derive(Clone, Copy)]
pub(super) struct Mark(u64, u64);
/// The routes of one prefix, as a follower of all routes is given them.
#[derive(Clone, Copy)]
pub struct Routes<'a>(&'a Row);
impl<'a> Routes<'a> {
/// The routes: at least one, since a prefix with none has no row, and
/// is given out as [`Event::Withdrawn`] instead.
pub fn iter(&self) -> impl Iterator<Item = RouteRef<'a>> + use<'a> {
self.0.routes()
}
}
impl std::fmt::Debug for Routes<'_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_list().entries(self.iter()).finish()
}
}
/// What a [`Reader`] is given.
#[derive(Debug, Clone, Copy)]
pub enum Event<'a> {
/// Forget what you know. The whole table follows, as [`Event::Best`]
/// or [`Event::Routes`] for every prefix, and then [`Event::Synced`].
/// The first thing a new reader is given, and what a reader that fell
/// too far behind is given in place of what it missed.
Reset,
/// This is now the best route for `prefix`. Given to a follower of the
/// best routes.
Best {
prefix: IpPrefix,
peer: PeerIndex,
/// The peer's identifier for the route: zero unless the peer
/// sends several routes for a prefix (ADD-PATH, RFC 7911).
path_id: u32,
attrs: &'a Arc<AttrSet>,
},
/// These are now all the routes for `prefix`: whatever was known of its
/// routes before is replaced. Given to a follower of all routes.
Routes {
prefix: IpPrefix,
routes: Routes<'a>,
},
/// `prefix` has no route any more.
Withdrawn { prefix: IpPrefix },
/// The table that began with [`Event::Reset`] is complete: a prefix
/// not heard of since the reset has no route.
Synced,
}
/// What a call to [`Reader::poll`] got done.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Progress {
/// How many events were given out.
pub events: usize,
/// Whether the reader has now been given everything there was when the
/// call began. If not, call again.
pub caught_up: bool,
}
enum Phase {
/// Giving out the whole table; `next` is the shard to do next. `upto`
/// is the time on the table's clock the table is given out as of, and
/// is taken when [`Event::Reset`] is given, so `None` until then: the
/// shards are gone through one at a time while the table changes, and
/// the place taken in each shard's journal is as of that one time, so
/// that what follows is consistent from shard to shard. A change after
/// it that a later shard's rows already show is read again from the
/// journal; that is harmless.
Resync { next: usize, upto: Option<u64> },
/// Following the journals; `next` is the shard to read next, and
/// `upto` the time by the table's clock up to which this pass reads
/// (see [`Entry::stamp`]).
Follow { next: usize, upto: u64 },
}
impl Phase {
/// The beginning of a resynchronisation.
const RESYNC: Self = Self::Resync {
next: 0,
upto: None,
};
/// The beginning of a pass over the journals.
const FOLLOW: Self = Self::Follow { next: 0, upto: 0 };
}
/// Follows the routes of a table: the best of each prefix, or all of them.
/// Made by [`RibTable::reader`] and its siblings.
///
/// A reader holds its place in every shard's journal. Dropping it gives
/// the place up, so that the journals need not keep entries for it.
pub struct Reader {
table: Arc<RibTable>,
/// Which journal it follows.
kind: Kind,
slot: usize,
/// Where this reader has read to in each shard; the copy in the shard
/// is for the shard's benefit.
cursors: Vec<u64>,
phase: Phase,
/// The table's version as it was when the current pass over the
/// journals began. [`Reader::changed`] waits while the version is still
/// this one: whatever was written since the pass began has moved it
/// on, and is there to be read.
seen: u64,
caught_up: bool,
}
impl RibTable {
/// A new follower of this table's best routes. It is given the whole
/// table first and every change after.
pub fn reader(self: &Arc<Self>) -> Reader {
self.register(Kind::Best, Phase::RESYNC)
}
/// A new follower that is given only the changes from now on, for
/// those who want the news and not the table. If it falls too far
/// behind it is reset like any other, and is then given the table.
pub fn reader_from_now(self: &Arc<Self>) -> Reader {
let mut reader = self.register(Kind::Best, Phase::FOLLOW);
for (index, shard) in self.shards.iter().enumerate() {
let shard = read(shard);
reader.set_cursor(index, &shard, shard.journals.best.seq);
}
reader
}
/// A new follower of every route in the table, not only the best: it
/// is told whenever any route of a prefix changes, with all the
/// prefix's routes ([`Event::Routes`]). The whole table first, like
/// [`RibTable::reader`].
pub fn reader_of_all_routes(self: &Arc<Self>) -> Reader {
self.register(Kind::All, Phase::RESYNC)
}
fn register(self: &Arc<Self>, kind: Kind, phase: Phase) -> Reader {
let mut slots = self
.reader_slots
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let slot = match slots.iter().position(|used| !used) {
Some(free) => free,
None => {
slots.push(false);
slots.len() - 1
}
};
slots[slot] = true;
for shard in &self.shards {
let mut shard = write(shard);
let journal = shard.journals.of_mut(kind);
while journal.cursors.len() <= slot {
journal.cursors.push(AtomicU64::new(NO_CURSOR));
}
journal.cursors[slot].store(NO_CURSOR, Ordering::Release);
journal.readers += 1;
}
self.readers.fetch_add(1, Ordering::AcqRel);
Reader {
table: Arc::clone(self),
kind,
slot,
cursors: vec![NO_CURSOR; self.shards.len()],
phase,
seen: 0,
caught_up: false,
}
}
/// How many entries the journals hold, over all shards. For tests and
/// for watching: it stays bounded whatever the readers do.
pub fn journal_len(&self) -> usize {
self.shards
.iter()
.map(|shard| read(shard).journals.len())
.sum()
}
/// Tells readers that are waiting that there is something to read.
pub(super) fn wake_readers(&self) {
if self.readers.load(Ordering::Acquire) > 0 {
self.version.fetch_add(1, Ordering::AcqRel);
self.wake.notify_waiters();
}
}
}
impl Drop for Reader {
fn drop(&mut self) {
for shard in &self.table.shards {
let mut shard = write(shard);
let journal = shard.journals.of_mut(self.kind);
journal.cursors[self.slot].store(NO_CURSOR, Ordering::Release);
journal.readers -= 1;
if journal.readers == 0 {
// Nobody left to read it.
journal.entries.clear();
journal.gone.clear();
}
}
self.table.readers.fetch_sub(1, Ordering::AcqRel);
let mut slots = self
.table
.reader_slots
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
slots[self.slot] = false;
}
}
enum Drained {
/// Entries were dropped that this reader had not read.
Lagged,
/// `events` were given out; `consumed` if the reader's place in the
/// journal moved at all; `more` if entries were left past the time
/// read up to.
Read {
events: usize,
consumed: bool,
more: bool,
},
}
impl Reader {
/// Gives `visit` what has changed since the last call, in up to about
/// `budget` events (a shard's part of a resynchronisation is not cut
/// short, so it can be more).
///
/// `visit` is called with a shard of the table held for reading. It
/// should be quick, and it must not call into the table, even to read:
/// taking the shard's lock again can wait for ever on a writer that is
/// waiting for this hold to end.
pub fn poll(&mut self, budget: usize, mut visit: impl FnMut(Event<'_>)) -> Progress {
// A call that may give nothing out would never get anywhere.
let budget = budget.max(1);
// Held once for the call: a count taken on the table for every
// shard of every pass, by every reader, was the most of what a
// router with many peers did.
let table = Arc::clone(&self.table);
let mut events = 0;
loop {
let done = match self.phase {
Phase::Resync { next, upto } => {
self.resync_step(&table, next, upto, budget, &mut events, &mut visit)
}
Phase::Follow { next, upto } => {
self.follow_pass(&table, next, upto, budget, &mut events, &mut visit)
}
};
if let Some(progress) = done {
return progress;
}
}
}
/// One step of a resynchronisation: the shard `next`, or the end of it
/// once every shard has been given out. Returns what to tell the
/// caller if the budget is spent, and `None` to go on.
fn resync_step(
&mut self,
table: &RibTable,
next: usize,
upto: Option<u64>,
budget: usize,
events: &mut usize,
visit: &mut impl FnMut(Event<'_>),
) -> Option<Progress> {
let upto = upto.unwrap_or_else(|| {
self.forget_place();
visit(Event::Reset);
*events += 1;
table.settled()
});
if next == table.shards.len() {
visit(Event::Synced);
*events += 1;
self.phase = Phase::FOLLOW;
return None;
}
*events += self.scan(table, next, upto, visit);
self.phase = Phase::Resync {
next: next + 1,
upto: Some(upto),
};
(*events >= budget).then(|| self.progress(*events, false))
}
/// A pass over the journals from shard `next` on, or the rest of one
/// that an earlier call cut short. Returns what to tell the caller if
/// the pass is to be the call's last, and `None` to go round again.
fn follow_pass(
&mut self,
table: &RibTable,
next: usize,
upto: u64,
budget: usize,
events: &mut usize,
visit: &mut impl FnMut(Event<'_>),
) -> Option<Progress> {
// Read before the pass: a change made during it shows as a newer
// version, and `changed` then returns at once. And the pass reads
// up to one time in every shard alike, chosen so that about the
// budget is read.
let (version, upto) = if next == 0 {
(
table.version.load(Ordering::Acquire),
self.time_for(budget - *events),
)
} else {
(self.seen, upto)
};
self.seen = version;
let mut consumed_any = false;
let mut more_somewhere = false;
for shard in next..table.shards.len() {
if *events >= budget {
self.phase = Phase::Follow { next: shard, upto };
return Some(self.progress(*events, false));
}
// Nothing new in this shard, by the number that is read
// without its lock: it is left alone, and so are the sessions
// writing to it.
if self.cursors[shard] == table.latest_seq(shard, self.kind) {
continue;
}
match self.drain(table, shard, upto, visit) {
Drained::Lagged => {
self.phase = Phase::RESYNC;
return None;
}
Drained::Read {
events: read,
consumed,
more,
} => {
*events += read;
consumed_any |= consumed;
more_somewhere |= more;
}
}
}
self.phase = Phase::FOLLOW;
if next == 0 && !consumed_any && more_somewhere {
// A whole pass got nowhere, though there is more: it is all
// beyond the earliest change still under way, which no pass
// can read past until it ends. Asking again at once would spin
// until then. The change wakes the readers when it ends.
return Some(self.progress(*events, true));
}
if next != 0 || more_somewhere {
// This pass took up where an earlier call stopped, so the
// shards before that point have not been looked at in this
// call; or some shard had more than its share, or changes
// later than this pass's time. Go round again from the first.
return (*events >= budget).then(|| self.progress(*events, false));
}
Some(self.progress(*events, true))
}
fn progress(&mut self, events: usize, caught_up: bool) -> Progress {
self.caught_up = caught_up;
Progress { events, caught_up }
}
/// Whether the last [`Reader::poll`] read everything there was.
pub fn caught_up(&self) -> bool {
self.caught_up
}
/// Waits until there may be something to read. Returns at once if the
/// last [`Reader::poll`] did not catch up.
pub async fn changed(&mut self) {
loop {
// Registered before the check, so that a change between the
// check and the wait is not missed.
let woken = self.table.wake.notified();
tokio::pin!(woken);
woken.as_mut().enable();
if !self.caught_up || self.table.version.load(Ordering::Acquire) != self.seen {
return;
}
woken.await;
}
}
/// What this reader is told about a row.
fn event_for<'a>(&self, row: &'a Row) -> Event<'a> {
match self.kind {
Kind::Best => {
let best = row.route(row.best());
Event::Best {
prefix: row.prefix,
peer: best.peer,
path_id: best.path_id,
attrs: best.attrs,
}
}
Kind::All => Event::Routes {
prefix: row.prefix,
routes: Routes(row),
},
}
}
/// Gives up this reader's place in every journal, before a
/// resynchronisation takes a new one.
fn forget_place(&mut self) {
for (index, shard) in self.table.shards.iter().enumerate() {
read(shard).journals.of(self.kind).cursors[self.slot]
.store(NO_CURSOR, Ordering::Release);
self.cursors[index] = NO_CURSOR;
}
}
fn set_cursor(&mut self, index: usize, shard: &Shard, cursor: u64) {
self.cursors[index] = cursor;
shard.journals.of(self.kind).cursors[self.slot].store(cursor, Ordering::Release);
}
/// Gives out every prefix in a shard and takes a place in its journal
/// as of `upto` on the table's clock, in one hold of the shard: what
/// changed after that time is in the journal after that place, whether
/// or not the rows given out show it already.
fn scan(
&mut self,
table: &RibTable,
index: usize,
upto: u64,
visit: &mut impl FnMut(Event<'_>),
) -> usize {
let shard = read(&table.shards[index]);
for row in &shard.rows {
visit(self.event_for(row));
}
let journal = shard.journals.of(self.kind);
let beyond = journal.entries.partition_point(|entry| entry.stamp <= upto);
let cursor = match (beyond.checked_sub(1), journal.entries.get(beyond)) {
// Nothing in the journal is later than `upto`.
(_, None) => journal.seq,
// The last entry that is not later; or, if every entry is
// later, just before the first.
(Some(last), Some(_)) => journal.entries[last].seq,
(None, Some(first)) => first.seq - 1,
};
self.set_cursor(index, &shard, cursor);
shard.rows.len()
}
/// The time on the table's clock up to which a pass over the shards
/// is to read, for about `budget` events in all: changes made together
/// are read together, in whatever shards they are (see
/// [`Entry::stamp`]). Taken shard by shard to the budget, the first
/// shards would be read far ahead of the last, and the routes of one
/// UPDATE, spread over the shards by their prefixes, given out far
/// apart.
fn time_for(&self, budget: usize) -> u64 {
let shards = self.table.shards.len();
let share = (budget / shards).max(1);
// No further than the changes that are in every shard already.
let settled = self.table.settled();
let mut upto = settled;
// The earliest change not yet read: at least that one is, however
// many changes were made with it.
let mut earliest = None;
for (index, shard) in self.table.shards.iter().enumerate() {
let cursor = self.cursors[index];
if cursor == self.table.latest_seq(index, self.kind) {
continue;
}
let shard = read(shard);
let journal = shard.journals.of(self.kind);
let Place::From(start) = journal.place_of(cursor) else {
continue;
};
if let Some(next) = journal.entries.get(start) {
earliest = Some(earliest.map_or(next.stamp, |e: u64| e.min(next.stamp)));
}
// Everything before the change that would go over the share.
if let Some(beyond) = journal.entries.get(start + share) {
upto = upto.min(beyond.stamp - 1);
}
}
upto.max(earliest.unwrap_or(upto)).min(settled)
}
/// Reads a shard's journal from this reader's place, no further than
/// the changes made by `upto` on the table's clock.
fn drain(
&mut self,
table: &RibTable,
index: usize,
upto: u64,
visit: &mut impl FnMut(Event<'_>),
) -> Drained {
let shard = read(&table.shards[index]);
let journal = shard.journals.of(self.kind);
let mut cursor = self.cursors[index];
let start = match journal.place_of(cursor) {
Place::Lagged => return Drained::Lagged,
Place::Nothing => {
return Drained::Read {
events: 0,
consumed: false,
more: false,
};
}
Place::From(start) => start,
};
let mut events = 0;
let mut more = false;
for entry in journal.entries.range(start..) {
if entry.stamp > upto {
more = true;
break;
}
cursor = entry.seq;
let event = match journal.says(entry, &shard.rows, &shard.hasher) {
Says::Row(row) => self.event_for(row),
Says::Gone => Event::Withdrawn {
prefix: entry.prefix,
},
Says::Nothing => continue,
};
visit(event);
events += 1;
}
if !more {
// Past the last entry there is nothing up to the latest change.
cursor = journal.seq;
}
let consumed = cursor != self.cursors[index];
self.set_cursor(index, &shard, cursor);
Drained::Read {
events,
consumed,
more,
}
}
}
#[cfg(test)]
#[path = "journal_tests.rs"]
mod tests;
crates/rib/src/table/journal_tests.rs
+906
View File
@@ -0,0 +1,906 @@
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use proptest::prelude::*;
use super::super::tests::{Fixture, Route, external, internal, prefix, route};
use super::super::{AttrSet, PeerIndex, PeerInfo, RibTable};
use super::{Event, Kind, Limits, Reader, read};
/// What a follower would keep: its own copy of the best routes, brought
/// up to date from the events it is given.
#[derive(Default)]
struct Mirror {
best: HashMap<IpPrefix, (PeerIndex, Arc<AttrSet>)>,
/// What was held before a reset and has not been heard of since.
stale: HashMap<IpPrefix, (PeerIndex, Arc<AttrSet>)>,
/// For a follower of all routes: every route of every prefix, as
/// (peer, path identifier, attributes, whether best).
routes: HashMap<IpPrefix, Vec<Held>>,
resets: usize,
synced: usize,
best_events: usize,
routes_events: usize,
withdrawn_events: usize,
}
/// A route as a mirror holds it. The attributes are held, so that the
/// address used to compare them stays theirs.
type Held = (PeerIndex, u32, Arc<AttrSet>, bool);
fn same_routes(a: &[Held], b: &[Held]) -> bool {
a.len() == b.len()
&& a.iter()
.zip(b)
.all(|(a, b)| (a.0, a.1, a.3) == (b.0, b.1, b.3) && Arc::ptr_eq(&a.2, &b.2))
}
fn sorted(mut routes: Vec<Held>) -> Vec<Held> {
routes.sort_by_key(|(peer, path_id, _, _)| (*peer, *path_id));
routes
}
impl Mirror {
fn apply(&mut self, event: Event<'_>) {
match event {
Event::Reset => {
self.resets += 1;
self.stale = std::mem::take(&mut self.best);
self.routes.clear();
}
Event::Routes { prefix, routes } => {
self.routes_events += 1;
assert!(
routes.iter().next().is_some(),
"a prefix with no routes is withdrawn"
);
self.routes.insert(
prefix,
sorted(
routes
.iter()
.map(|r| (r.peer, r.path_id, Arc::clone(r.attrs), r.best))
.collect(),
),
);
}
Event::Best {
prefix,
peer,
attrs,
..
} => {
self.best_events += 1;
self.stale.remove(&prefix);
self.best.insert(prefix, (peer, Arc::clone(attrs)));
}
Event::Withdrawn { prefix } => {
self.withdrawn_events += 1;
self.best.remove(&prefix);
self.routes.remove(&prefix);
}
Event::Synced => {
self.synced += 1;
self.stale.clear();
}
}
}
/// Reads until there is nothing more, in steps of `budget`.
fn catch_up(&mut self, reader: &mut Reader, budget: usize) {
while !reader.poll(budget, |event| self.apply(event)).caught_up {}
}
#[track_caller]
fn assert_matches(&self, table: &RibTable) {
let mut expected = HashMap::new();
table.for_each_best(|prefix, peer, attrs| {
expected.insert(*prefix, (peer, Arc::clone(attrs)));
});
assert_eq!(
self.best.len(),
expected.len(),
"the mirror has {} prefixes, the table {}",
self.best.len(),
expected.len()
);
for (prefix, (peer, attrs)) in &expected {
let Some((mirrored_peer, mirrored_attrs)) = self.best.get(prefix) else {
panic!("{prefix} is in the table and not in the mirror");
};
assert_eq!(mirrored_peer, peer, "{prefix}: from the wrong peer");
assert!(
Arc::ptr_eq(mirrored_attrs, attrs),
"{prefix}: wrong attributes"
);
}
}
/// For the mirror of a follower of all routes: it holds every route in
/// the table, and knows which is best.
#[track_caller]
fn assert_has_all_routes_of(&self, table: &RibTable) {
let mut expected: HashMap<IpPrefix, Vec<Held>> = HashMap::new();
table.for_each_route(|r| {
expected.entry(r.prefix).or_default().push((
r.peer,
r.path_id,
Arc::clone(r.attrs),
r.best,
));
});
assert_eq!(self.routes.len(), expected.len(), "prefixes");
for (prefix, routes) in expected {
let mirrored = self
.routes
.get(&prefix)
.unwrap_or_else(|| panic!("{prefix} is in the table and not in the mirror"));
assert!(
same_routes(mirrored, &sorted(routes)),
"{prefix}: the mirror's routes are not the table's"
);
}
}
}
fn nth_prefix(n: u32) -> IpPrefix {
IpPrefix::new(IpVersion::Ipv4, (n << 8).to_be_bytes(), 24).unwrap()
}
#[test]
fn a_new_reader_is_given_the_table_and_then_the_changes() {
let fixture = Fixture::new(&[external(1), external(2)]);
let (a, b) = (prefix("10.0.0.0/8"), prefix("2001:db8::/32"));
fixture.announce(0, &a, &Route::via(&[1, 2]));
fixture.announce(0, &b, &Route::via(&[1, 2]));
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
let mut events = Vec::new();
let progress = reader.poll(100, |event| {
events.push(
format!("{event:?}")
.split([' ', '{'])
.next()
.unwrap()
.to_string(),
);
mirror.apply(event);
});
assert!(progress.caught_up);
assert_eq!(events, ["Reset", "Best", "Best", "Synced"]);
mirror.assert_matches(&fixture.table);
// Nothing has happened: nothing to read.
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
// A change that does not change the best route is not an event.
fixture.announce(1, &a, &Route::via(&[3, 4, 5]));
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(1, &a, &Route::via(&[3]));
fixture.withdraw(0, &b);
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (3, 1));
assert_eq!((mirror.resets, mirror.synced), (1, 1));
mirror.assert_matches(&fixture.table);
}
#[test]
fn a_reader_from_now_is_given_the_changes_and_not_the_table() {
let fixture = Fixture::new(&[external(1)]);
let (old, new) = (prefix("10.0.0.0/8"), prefix("10.1.0.0/16"));
fixture.announce(0, &old, &Route::via(&[1]));
let mut reader = fixture.table.reader_from_now();
assert_eq!(
reader
.poll(100, |event| panic!("unexpected {event:?}"))
.events,
0
);
fixture.announce(0, &new, &Route::via(&[1]));
fixture.withdraw(0, &old);
let (mut best, mut withdrawn) = (Vec::new(), Vec::new());
reader.poll(100, |event| match event {
Event::Best { prefix, .. } => best.push(prefix),
Event::Withdrawn { prefix } => withdrawn.push(prefix),
other => panic!("unexpected {other:?}"),
});
assert_eq!((best, withdrawn), (vec![new], vec![old]));
}
/// The number a reader checks without a shard's lock must be the journal's
/// own after every kind of change, whether or not anyone is reading: a
/// reader skips a shard on the strength of it, and a shard with news that
/// said it had none would never be read until the next change.
#[test]
fn the_lock_free_number_is_the_journals_after_every_kind_of_change() {
let mut fixture = Fixture::new(&[external(1), external(2)]);
let prefixes: Vec<IpPrefix> = (0..16).map(nth_prefix).collect();
let agree = |table: &RibTable| {
for (index, shard) in table.shards.iter().enumerate() {
let shard = read(shard);
for kind in [Kind::Best, Kind::All] {
assert_eq!(
table.latest_seq(index, kind),
shard.journals.of(kind).seq,
"shard {index}, {kind:?}"
);
}
}
};
// Without a reader, with one of each kind, and with both.
for readers in 0..4 {
let _best = (readers & 1 != 0).then(|| fixture.table.reader());
let _all = (readers & 2 != 0).then(|| fixture.table.reader_of_all_routes());
agree(&fixture.table);
for p in &prefixes {
fixture.announce(0, p, &Route::via(&[1]));
}
agree(&fixture.table);
// The same prefixes again, with other attributes.
for p in &prefixes {
fixture.announce(0, p, &Route::via(&[1, 2]));
}
agree(&fixture.table);
// A second route for each, then the last route of one taken away.
for p in &prefixes {
fixture.announce(1, p, &Route::via(&[2, 3]));
}
agree(&fixture.table);
fixture.withdraw(0, &prefixes[0]);
agree(&fixture.table);
fixture.withdraw(1, &prefixes[0]);
agree(&fixture.table);
fixture.table.remove_peer(fixture.peers[1]);
agree(&fixture.table);
fixture.peers[1] = fixture.table.add_peer(external(2)).unwrap();
fixture
.table
.mark_stale(fixture.peers[0], &[IpVersion::Ipv4]);
agree(&fixture.table);
fixture
.table
.sweep_stale(fixture.peers[0], &[IpVersion::Ipv4]);
agree(&fixture.table);
assert!(
fixture.table.is_empty(),
"round {readers} left routes behind"
);
}
}
/// A shard that had nothing new for a reader is passed over without its
/// lock. The next change to it must bring the reader back.
#[test]
fn a_shard_passed_over_is_read_again_once_it_has_news() {
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(0, &prefix("10.0.0.0/8"), &Route::via(&[1]));
let mut events = Vec::new();
let progress = reader.poll(100, |event| events.push(format!("{event:?}")));
assert!(progress.caught_up);
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].starts_with("Best"), "{events:?}");
}
/// As [`a_shard_passed_over_is_read_again_once_it_has_news`], for a reader
/// that begins where the journals are rather than with the table.
#[test]
fn a_shard_passed_over_by_a_reader_from_now_is_read_again_once_it_has_news() {
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
fixture.announce(0, &prefix("192.0.2.0/24"), &Route::via(&[1]));
let mut reader = fixture.table.reader_from_now();
assert!(reader.poll(100, |_| panic!("no event expected")).caught_up);
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(0, &prefix("10.0.0.0/8"), &Route::via(&[1]));
let mut events = Vec::new();
let progress = reader.poll(100, |event| events.push(format!("{event:?}")));
assert!(progress.caught_up);
assert_eq!(events.len(), 1, "{events:?}");
assert!(events[0].starts_with("Best"), "{events:?}");
}
/// A change to a route that is not the best is nothing to a follower of
/// the best routes, and is what a follower of all routes is there for.
#[test]
fn a_follower_of_all_routes_is_told_of_every_change() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
fixture.announce(0, &p, &Route::via(&[1]));
let mut best = fixture.table.reader();
let mut all = fixture.table.reader_of_all_routes();
let (mut best_mirror, mut all_mirror) = (Mirror::default(), Mirror::default());
best_mirror.catch_up(&mut best, 100);
all_mirror.catch_up(&mut all, 100);
assert_eq!((best_mirror.best_events, all_mirror.routes_events), (1, 1));
// A worse route from another peer, then two more from that peer under
// other path identifiers, one of them changed, one withdrawn.
fixture.announce(1, &p, &Route::via(&[2, 3, 4]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 3, 5]));
fixture.announce_path(1, &p, 8, &Route::via(&[2, 3, 6]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 9, 5]));
fixture.withdraw_path(1, &p, 8);
assert_eq!(
best.poll(100, |event| panic!("unexpected {event:?}"))
.events,
0
);
all_mirror.catch_up(&mut all, 100);
// All of it between two reads is one event, with the routes as they
// are now.
assert_eq!(all_mirror.routes_events, 2);
assert_eq!(all_mirror.routes[&p].len(), 3);
all_mirror.assert_has_all_routes_of(&fixture.table);
// The last route going is a withdrawal for both.
fixture.table.remove_peer(fixture.peers[1]);
fixture.withdraw(0, &p);
best_mirror.catch_up(&mut best, 100);
all_mirror.catch_up(&mut all, 100);
assert_eq!(
(best_mirror.withdrawn_events, all_mirror.withdrawn_events),
(1, 1)
);
all_mirror.assert_has_all_routes_of(&fixture.table);
// Nobody following all routes: that journal keeps nothing. A change
// that is not of the best route is then recorded nowhere.
drop(all);
fixture.announce(0, &p, &Route::via(&[1]));
fixture.announce(1, &p, &Route::via(&[2, 3]));
let before = fixture.table.journal_len();
fixture.announce(1, &p, &Route::via(&[2, 4]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 5]));
assert_eq!(fixture.table.journal_len(), before);
}
/// A follower wants what the table has now, not how it got there.
#[test]
fn many_changes_to_a_prefix_between_reads_are_one_event() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
for round in 0..50 {
fixture.announce(0, &p, &Route::via(&[1, 2, 3]));
fixture.announce(1, &p, &Route::via(&[4, 5]));
fixture.withdraw(1, &p);
if round % 2 == 0 {
fixture.withdraw(0, &p);
}
}
// It ended with peer 0's route in place.
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (1, 0));
mirror.assert_matches(&fixture.table);
// And a prefix that came and went again is, at most, a withdrawal.
let q = prefix("10.1.0.0/16");
for _ in 0..50 {
fixture.announce(0, &q, &Route::via(&[1]));
fixture.withdraw(0, &q);
}
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (1, 1));
mirror.assert_matches(&fixture.table);
}
#[test]
fn reading_in_small_steps_gives_the_same_as_reading_at_once() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..500 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1, n % 7 + 1]));
}
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 7);
mirror.assert_matches(&fixture.table);
for n in 0..500 {
if n % 3 == 0 {
fixture.withdraw(0, &nth_prefix(n));
} else {
fixture.announce(0, &nth_prefix(n), &Route::via(&[2, n % 5 + 1]));
}
}
let mut steps = 0;
while !reader.poll(7, |event| mirror.apply(event)).caught_up {
steps += 1;
}
assert!(steps > 50, "the budget was kept to: {steps} steps");
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.resets, 1);
}
#[test]
fn with_no_reader_nothing_is_kept() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..5000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
fixture.withdraw(0, &nth_prefix(n));
}
assert_eq!(fixture.table.journal_len(), 0);
// And what a reader was holding back goes when the reader does.
let reader = fixture.table.reader();
for n in 0..100 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
assert_eq!(fixture.table.journal_len(), 100);
drop(reader);
assert_eq!(fixture.table.journal_len(), 0);
}
/// A reader that stops reading must not make the table keep a growing
/// journal for it. It is the one that pays: it is given the table again.
#[test]
fn a_reader_that_stops_reading_is_reset_and_holds_nothing_up() {
// One shard, so that the journal's length is that shard's.
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
fixture.announce(0, &prefix("192.0.2.0/24"), &Route::via(&[1]));
let mut stuck = fixture.table.reader();
let mut stuck_mirror = Mirror::default();
stuck_mirror.catch_up(&mut stuck, 100);
let mut diligent = fixture.table.reader();
let mut diligent_mirror = Mirror::default();
// Prefixes come and go, each a different one, so no entry is made
// dead by a later one for the same prefix.
let limits = Limits::default();
let mut longest = 0;
for n in 0..20_000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
fixture.withdraw(0, &nth_prefix(n));
longest = longest.max(fixture.table.journal_len());
if n % 100 == 0 {
diligent_mirror.catch_up(&mut diligent, 1000);
}
}
assert!(
longest <= limits.tidy_at.max(2 * (1 + limits.slack)) + 2,
"the journal reached {longest} entries"
);
// The one that kept up never noticed.
diligent_mirror.catch_up(&mut diligent, 1000);
assert_eq!(
diligent_mirror.resets, 1,
"only the reset every reader starts with"
);
diligent_mirror.assert_matches(&fixture.table);
// The one that did not is told, and ends up right all the same.
stuck_mirror.catch_up(&mut stuck, 1000);
assert_eq!(stuck_mirror.resets, 2);
assert_eq!(stuck_mirror.synced, 2);
stuck_mirror.assert_matches(&fixture.table);
}
/// When a peer that was the only source of most prefixes goes, more
/// prefixes disappear at once than are left. The journal still comes down
/// to its bound, and a reader ends up with what the table has.
#[test]
fn many_prefixes_going_at_once_leaves_the_journal_bounded() {
let fixture = Fixture::on(RibTable::new(1), &[external(1), external(2)]);
for n in 0..5000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
fixture.announce(1, &nth_prefix(0), &Route::via(&[2, 3]));
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 1000);
fixture.table.remove_peer(fixture.peers[0]);
// Some later change, as there always is, lets the journal be tidied.
for _ in 0..8 {
fixture.announce(1, &nth_prefix(1), &Route::via(&[2]));
fixture.withdraw(1, &nth_prefix(1));
}
let limits = Limits::default();
assert!(
fixture.table.journal_len() <= limits.tidy_at.max(2 * (1 + limits.slack)) + 2,
"the journal holds {} entries",
fixture.table.journal_len()
);
mirror.catch_up(&mut reader, 1000);
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.best.len(), 1);
}
/// Readers are woken to read. An announcement that changes nothing, or
/// nothing a reader follows, writes nothing for them and must not wake
/// them: each would look through every shard to find that out.
#[test]
fn readers_are_not_woken_when_nothing_was_written_for_them() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let version = || fixture.table.version.load(Ordering::Acquire);
// Nobody reading: nobody to wake.
fixture.announce(0, &p, &Route::via(&[1]));
assert_eq!(version(), 0);
let _best = fixture.table.reader();
let before = version();
// The same route again.
fixture.announce(0, &p, &Route::via(&[1]));
assert_eq!(version(), before);
// A worse route from another peer: the best route stands.
fixture.announce(1, &p, &Route::via(&[2, 3, 4]));
assert_eq!(version(), before);
// A better one.
fixture.announce(1, &p, &Route::via(&[]));
assert_eq!(version(), before + 1);
// A follower of all routes is told of any change to a route.
let _all = fixture.table.reader_of_all_routes();
let before = version();
fixture.announce(0, &p, &Route::via(&[1, 1, 1]));
assert_eq!(version(), before + 1);
fixture.announce(0, &p, &Route::via(&[1, 1, 1]));
assert_eq!(version(), before + 1);
}
/// A budget of nothing would let a reader ask for ever without getting
/// anywhere; it is taken as the smallest that does.
#[test]
fn a_budget_of_nothing_still_makes_progress() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..20 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 0);
for n in 0..20 {
fixture.withdraw(0, &nth_prefix(n));
}
mirror.catch_up(&mut reader, 0);
mirror.assert_matches(&fixture.table);
assert!(mirror.best.is_empty());
}
#[tokio::test]
async fn a_waiting_reader_is_woken_by_a_change_of_best_route() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
// Not caught up yet: no waiting.
reader.changed().await;
mirror.catch_up(&mut reader, 100);
let waiting = tokio::time::timeout(std::time::Duration::from_millis(50), reader.changed());
assert!(waiting.await.is_err(), "nothing changed, so it waits");
let table = Arc::clone(&fixture.table);
let attrs = table
.attrs()
.intern(&Route::via(&[1]).attributes(), None)
.unwrap();
let peer = fixture.peers[0];
let writer = tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
table.announce(peer, &attrs, &[p]);
});
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("woken by the announcement");
writer.await.unwrap();
mirror.catch_up(&mut reader, 100);
mirror.assert_matches(&fixture.table);
// A change made before the wait began is not missed either.
fixture.withdraw(0, &p);
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("the change was already there");
}
/// A reader does not read past a change that is still under way, which
/// a change to every shard (a peer's whole table going) is for a while.
/// Changes made by others meanwhile are beyond it. A reader that finds
/// only those must say it is caught up and wait, not ask again at once
/// for as long as the change takes; and when the change ends, it is
/// woken, though the change itself wrote nothing for it.
#[tokio::test]
async fn a_change_under_way_does_not_keep_a_reader_spinning() {
let fixture = Fixture::new(&[external(1), external(2)]);
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
// A long change begins, and another writer announces meanwhile.
let change = fixture.table.begin_change();
fixture.announce(1, &prefix("10.0.0.0/8"), &Route::via(&[2]));
let progress = reader.poll(100, |_| {});
assert_eq!(progress.events, 0, "nothing can be read past the change");
assert!(
progress.caught_up,
"and the reader is to wait, not ask again"
);
let waiting = tokio::time::timeout(std::time::Duration::from_millis(50), reader.changed());
assert!(
waiting.await.is_err(),
"nothing to be had until the change ends"
);
// The change ends without writing anything itself.
drop(change);
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("woken when the change ended");
mirror.catch_up(&mut reader, 100);
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.best.len(), 1);
}
/// Writers on several threads, a reader on another, reading while they
/// write. When the writers are done and the reader has caught up, it has
/// what the table has.
#[test]
fn a_reader_keeps_up_with_writers_on_other_threads() {
const WRITERS: usize = 4;
const PREFIXES: u32 = 300;
let infos: Vec<PeerInfo> = (0..WRITERS).map(|i| external(i as u8 + 1)).collect();
// Small journals, so that tidying and falling behind happen too.
let fixture = Fixture::on(
RibTable::with_journal_limits(
8,
Limits {
tidy_at: 16,
slack: 4,
},
),
&infos,
);
let table = &fixture.table;
let done = AtomicBool::new(false);
let mut mirror = Mirror::default();
std::thread::scope(|scope| {
let writers: Vec<_> = (0..WRITERS)
.map(|writer| {
let peer = fixture.peers[writer];
scope.spawn(move || {
// A simple generator; each writer its own sequence.
let mut state = 0x9e37_79b9_7f4a_7c15u64.wrapping_mul(writer as u64 + 1);
let mut next = || {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
state
};
for _ in 0..30_000 {
let p = nth_prefix((next() % u64::from(PREFIXES)) as u32);
if next() % 3 == 0 {
table.withdraw(peer, &[p]);
} else {
let route = Route {
med: Some((next() % 3) as u32),
..Route::via(
&[(next() % 3) as u32 + 1, 9][..(next() % 2 + 1) as usize],
)
};
let attrs = table.attrs().intern(&route.attributes(), None).unwrap();
table.announce(peer, &attrs, &[p]);
}
}
})
})
.collect();
let mirror = &mut mirror;
let done = &done;
let reading = scope.spawn(move || {
let mut reader = table.reader();
while !done.load(Ordering::Acquire) {
reader.poll(50, |event| mirror.apply(event));
}
mirror.catch_up(&mut reader, 50);
});
for writer in writers {
writer.join().unwrap();
}
done.store(true, Ordering::Release);
reading.join().unwrap();
});
mirror.assert_matches(table);
assert!(mirror.best_events > 0);
}
// ---------------------------------------------------------------------------
// Readers against the table, under any interleaving
// ---------------------------------------------------------------------------
#[derive(Debug, Clone)]
enum Op {
Announce {
peer: usize,
prefix: usize,
path_id: u32,
route: Route,
},
Withdraw {
peer: usize,
prefix: usize,
path_id: u32,
},
Flap {
peer: usize,
},
Poll {
reader: usize,
budget: usize,
},
/// The reader is dropped and a new one takes its place.
Replace {
reader: usize,
},
}
const PEERS: usize = 4;
const PREFIXES: usize = 12;
// The first three follow the best routes, the others all routes.
const READERS: usize = 5;
const OF_BEST: usize = 3;
fn op() -> impl Strategy<Value = Op> {
prop_oneof![
8 => (0..PEERS, 0..PREFIXES, 0u32..3, route()).prop_map(
|(peer, prefix, path_id, route)| Op::Announce { peer, prefix, path_id, route }
),
5 => (0..PEERS, 0..PREFIXES, 0u32..3)
.prop_map(|(peer, prefix, path_id)| Op::Withdraw { peer, prefix, path_id }),
1 => (0..PEERS).prop_map(|peer| Op::Flap { peer }),
// Reader 0 is polled rarely: it is the one that falls behind.
4 => (1..READERS, 1usize..6).prop_map(|(reader, budget)| Op::Poll { reader, budget }),
1 => (0..READERS, 1usize..6).prop_map(|(reader, budget)| Op::Poll { reader, budget }),
1 => (1..READERS).prop_map(|reader| Op::Replace { reader }),
]
}
proptest! {
#![proptest_config(ProptestConfig::with_cases(1500))]
/// However changes and reads are interleaved, and however small the
/// journals: a reader that has caught up holds exactly the table's best
/// routes, and every reader gets there in the end.
#[test]
fn a_reader_that_has_caught_up_has_what_the_table_has(
ops in proptest::collection::vec(op(), 1..150),
shards in 1usize..4,
) {
let infos: Vec<PeerInfo> = (0..PEERS)
.map(|i| if i == 0 { internal(9) } else { external(i as u8) })
.collect();
// Journals that are tidied, and found too long, almost at once.
let limits = Limits { tidy_at: 3, slack: 1 };
let mut fixture = Fixture::on(RibTable::with_journal_limits(shards, limits), &infos);
let prefixes: Vec<IpPrefix> = (0..PREFIXES as u32).map(nth_prefix).collect();
let new_reader = |table: &Arc<RibTable>, index: usize| {
if index < OF_BEST { table.reader() } else { table.reader_of_all_routes() }
};
let check = |index: usize, mirror: &Mirror, table: &RibTable| {
if index < OF_BEST {
mirror.assert_matches(table);
} else {
mirror.assert_has_all_routes_of(table);
}
};
let mut readers: Vec<(Reader, Mirror)> = (0..READERS)
.map(|index| (new_reader(&fixture.table, index), Mirror::default()))
.collect();
for op in ops {
match op {
Op::Announce { peer, prefix, path_id, route } => {
fixture.announce_path(peer, &prefixes[prefix], path_id, &route);
}
Op::Withdraw { peer, prefix, path_id } => {
fixture.withdraw_path(peer, &prefixes[prefix], path_id);
}
Op::Flap { peer } => {
fixture.table.remove_peer(fixture.peers[peer]);
fixture.peers[peer] = fixture.table.add_peer(infos[peer]).unwrap();
}
Op::Poll { reader: index, budget } => {
let (reader, mirror) = &mut readers[index];
let progress = reader.poll(budget, |event| mirror.apply(event));
if progress.caught_up {
check(index, mirror, &fixture.table);
}
}
Op::Replace { reader } => {
readers[reader] = (new_reader(&fixture.table, reader), Mirror::default());
}
}
// Never much longer than the table, whatever the readers do.
// Two journals to a shard.
let bound = 2 * shards * (2 * (PREFIXES + limits.slack) + limits.tidy_at + PREFIXES);
prop_assert!(fixture.table.journal_len() <= bound);
}
for (index, (reader, mirror)) in readers.iter_mut().enumerate() {
mirror.catch_up(reader, 5);
check(index, mirror, &fixture.table);
prop_assert_eq!(mirror.resets, mirror.synced);
}
}
}
/// Routes that were announced together, with one set of attributes, are
/// spread over the shards by their prefixes. A reader that takes its
/// budget from the shards evenly is given them together all the same,
/// which is what lets whoever passes them on send them in one message.
#[test]
fn changes_made_together_are_read_together() {
let fixture = Fixture::on(RibTable::new(128), &[external(1)]);
let mut reader = fixture.table.reader();
assert!(reader.poll(10, |_| {}).caught_up);
// 6000 UPDATEs of six prefixes each, every one with attributes of its
// own.
let groups = 6000u32;
for group in 0..groups {
let attrs = fixture
.table
.attrs()
.intern(&Route::via(&[1, 100_000 + group]).attributes(), None)
.unwrap();
let prefixes: Vec<IpPrefix> = (0..6).map(|n| nth_prefix(group * 6 + n)).collect();
fixture.table.announce(fixture.peers[0], &attrs, &prefixes);
}
// Read in batches a tenth of that size: each should hold whole groups,
// but for the ones cut by its edges.
let mut batches = 0;
let mut split_groups = 0;
let mut seen_total = 0;
while seen_total < groups * 6 {
let mut groups_in_batch: HashMap<usize, usize> = HashMap::new();
let progress = reader.poll(3600, |event| {
if let Event::Best { attrs, .. } = event {
*groups_in_batch
.entry(Arc::as_ptr(attrs) as usize)
.or_default() += 1;
}
});
seen_total += groups_in_batch.values().sum::<usize>() as u32;
split_groups += groups_in_batch.values().filter(|n| **n < 6).count();
batches += 1;
if progress.events == 0 {
break;
}
}
assert_eq!(seen_total, groups * 6);
// A few are, where a batch's time falls inside one: the reader cannot
// know what was written together until it is all there.
assert!(
split_groups < 100,
"{split_groups} of {groups} groups were split over {batches} batches"
);
}
crates/rib/src/table/mod.rs
+1053
View File
File diff suppressed because it is too large. Load diff
crates/rib/src/table/select.rs
+281
View File
@@ -0,0 +1,281 @@
//! Choosing the best route for a prefix (RFC 4271 §9.1.2.2).
//!
//! The steps, in order. A route wins at the first step where it differs:
//!
//! 1. highest LOCAL_PREF;
//! 2. shortest AS path;
//! 3. lowest ORIGIN;
//! 4. lowest MULTI_EXIT_DISC, among routes from the same neighbouring AS;
//! 5. a route from an external peer over one from an internal peer;
//! 6. the peer with the lowest BGP Identifier;
//! 7. the peer with the lowest address.
//!
//! (The RFC's step between 5 and 6, the interior cost to the next hop, has
//! nothing to compare until there is an IGP to ask.)
//!
//! Steps 1 to 3 are put into one number per route when it is stored,
//! [`preference`]. Most of the time that number alone settles it: a new
//! route with a larger one than the best is the new best, with a smaller
//! one changes nothing, and neither needs a look at any other route. Only a
//! tie goes on to [`choose`], which looks at the whole row, and that begins
//! with a pass over the numbers to find the routes still in the running.
//!
//! [`after_add`], [`after_replace`] and [`after_remove`] are what the table
//! calls when a row has changed. Each says whether the best route is now
//! another, and takes the short way where there is one. They rely on, and
//! keep, one fact about a row: its best route has the highest preference
//! of all its routes.
use super::attrs::SelectionKey;
use super::{DEFAULT_LOCAL_PREF, Path, PeerInfo, Row};
/// Steps 1 to 3 as one number of 48 bits: the larger, the better.
///
/// LOCAL_PREF fills the high 32 bits. Below it come the AS path length in
/// 12 bits and the ORIGIN in 2, each turned upside down so that shorter and
/// lower compare as larger. Twelve bits are enough for any path: a BGP
/// message of 4096 octets has room for about a thousand AS numbers.
pub(super) fn preference(key: &SelectionKey) -> u64 {
// LOCAL_PREF is a matter between routers of one AS. What an external
// peer sent as one never gets as far as the table (the decoder drops
// it, RFC 7606 §7.5); a route from such a peer that has one was given
// it by our own import policy, which is who is to say.
let local_pref = key.local_pref.unwrap_or(DEFAULT_LOCAL_PREF);
const LONGEST: u16 = (1 << 12) - 1;
(u64::from(local_pref) << 16)
| (u64::from(LONGEST - key.as_path_len.min(LONGEST)) << 4)
| (u64::from(3 - key.origin.min(3)) << 2)
}
/// Buffers that [`choose`] reuses.
#[derive(Default)]
pub(super) struct Scratch {
/// Routes still in the running: (neighbouring AS, MED, index in row).
candidates: Vec<(u32, u32, u32)>,
}
/// The best route changed after a route was added at the end of `row`?
pub(super) fn after_add(row: &mut Row, peers: &[Option<PeerInfo>], scratch: &mut Scratch) -> bool {
let added = row.paths.len() - 1;
let preference = row.paths[added].preference();
let best = row.paths[row.best()].preference();
// Better than the best in the first steps of selection is better than
// everything; worse than it changes nothing. A tie has to be settled by
// the later steps.
if preference > best {
row.set_best(added);
true
} else if preference < best {
false
} else if let Some(wins) = settles_tie(row, added, peers) {
if wins {
row.set_best(added);
}
wins
} else {
choose(row, peers, scratch)
}
}
/// The best route changed after the route at `at` was replaced by another
/// from the same peer? `before` is the preference of the route it replaced.
/// The two are known to differ.
pub(super) fn after_replace(
row: &mut Row,
at: usize,
before: u64,
peers: &[Option<PeerInfo>],
scratch: &mut Scratch,
) -> bool {
let preference = row.paths[at].preference();
if at == row.best() {
// The best route itself changed. If it got better in the first
// steps it is still the best, but it is a different route now.
if preference <= before {
choose(row, peers, scratch);
}
return true;
}
let best = row.paths[row.best()].preference();
if preference > best {
row.set_best(at);
true
} else if preference < best && before < best {
// It was not in the running and still is not.
false
} else {
choose(row, peers, scratch)
}
}
/// Takes the route at `at` out of `row`, which has others, and returns
/// whether the best route changed.
pub(super) fn after_remove(
row: &mut Row,
at: usize,
peers: &[Option<PeerInfo>],
scratch: &mut Scratch,
) -> bool {
let best = row.best();
let best_preference = row.paths[best].preference();
let last = row.paths.len() - 1;
let removed = row.swap_remove(at);
if at == best {
choose(row, peers, scratch);
return true;
}
// The last route moved into the gap.
if best == last {
row.set_best(at);
}
// A route that tied with the best in the first steps may have been
// what kept another from being best, by its MED.
removed.preference() == best_preference && choose(row, peers, scratch)
}
/// Chooses the best route of `row` from all its routes, records it in the
/// row, and returns whether it is another route than before.
///
/// The row must have at least one route.
fn choose(row: &mut Row, peers: &[Option<PeerInfo>], scratch: &mut Scratch) -> bool {
let before = row.best;
let top = row
.paths
.iter()
.map(|path| path.preference())
.max()
.unwrap_or(0);
let candidates = &mut scratch.candidates;
candidates.clear();
for (index, path) in row.paths.iter().enumerate() {
if path.preference() == top {
let key = path.attrs.key();
// A route without a MED is taken to have the lowest (RFC 4271
// §9.1.2.2 c).
candidates.push((key.neighbor_as, key.med.unwrap_or(0), index as u32));
}
}
// Step 4. MEDs are compared only between routes from the same
// neighbouring AS, so "lower MED" is not an order over all the routes,
// and comparing them pair by pair can go round in circles. Instead:
// within each neighbouring AS only the routes with its lowest MED stay
// in, and the steps after are an order over what is left.
candidates.sort_unstable();
let mut best: Option<(TieBreak, u32)> = None;
let mut group = None;
for &(neighbor_as, med, index) in candidates.iter() {
// Sorted by AS and then MED: the first of each AS has its lowest.
let lowest = match group {
Some((group_as, lowest)) if group_as == neighbor_as => lowest,
_ => {
group = Some((neighbor_as, med));
med
}
};
if med != lowest {
continue;
}
let tie_break = TieBreak::of(row, index as usize, peers);
if best.as_ref().is_none_or(|(least, _)| tie_break < *least) {
best = Some((tie_break, index));
}
}
row.best = best.map_or(0, |(_, index)| index);
row.best != before
}
/// The routes of `row` from the best down, as indices into its routes:
/// after the best, the one that would be chosen were the best gone, and
/// so on. What an operator asks when looking at a prefix: which route
/// would take over.
pub(super) fn ranked(row: &Row, peers: &[Option<PeerInfo>]) -> Vec<usize> {
// A copy to take the winners out of one by one, and where each of its
// routes is in the row itself.
let mut rest = Row {
prefix: row.prefix,
best: 0,
seqs: row.seqs,
paths: row
.paths
.iter()
.map(|path| Path {
word: path.word,
attrs: std::sync::Arc::clone(&path.attrs),
})
.collect(),
path_ids: row.path_ids.clone(),
};
let mut places: Vec<usize> = (0..row.paths.len()).collect();
let mut scratch = Scratch::default();
let mut order = Vec::with_capacity(row.paths.len());
while !rest.paths.is_empty() {
choose(&mut rest, peers, &mut scratch);
let at = rest.best();
order.push(places[at]);
rest.swap_remove(at);
places.swap_remove(at);
}
order
}
/// For a route just added at `added` that ties with the best in steps 1 to
/// 3: whether it is the new best, if that can be said from the two routes
/// alone. `None` if the whole row has to be looked at.
///
/// It can be said when MED cannot knock either of them out. The new route
/// has the lowest MED there is, so nothing knocks it out. It knocks out
/// only routes of its own neighbouring AS with a higher MED, and if the
/// best is not one of those, the best is still the best of all the others.
/// So the better of the two is the best of all. This is the usual case:
/// most routes have no MED.
fn settles_tie(row: &Row, added: usize, peers: &[Option<PeerInfo>]) -> Option<bool> {
let new = row.paths[added].attrs.key();
let best = row.paths[row.best()].attrs.key();
if new.med.unwrap_or(0) != 0 {
return None;
}
if best.neighbor_as == new.neighbor_as && best.med.unwrap_or(0) != 0 {
return None;
}
Some(TieBreak::of(row, added, peers) < TieBreak::of(row, row.best(), peers))
}
/// Steps 5 to 7, then the peer's number, and then the path identifier for
/// a peer that sent several routes: so that the choice never depends on the
/// order routes arrived in. The least wins.
#[derive(PartialEq, Eq, PartialOrd, Ord)]
struct TieBreak {
internal: bool,
router_id: u32,
address: std::net::IpAddr,
peer: u16,
path_id: u32,
}
impl TieBreak {
fn of(row: &Row, index: usize, peers: &[Option<PeerInfo>]) -> Self {
let peer = row.paths[index].peer();
let path_id = row.path_id(index);
match peers.get(peer.as_usize()).copied().flatten() {
Some(info) => Self {
internal: info.ibgp,
router_id: info.router_id.into(),
address: info.address,
peer: peer.0,
path_id,
},
// Not expected: a peer is forgotten only once its routes are
// gone. If it happened, anything known would be preferred.
None => Self {
internal: true,
router_id: u32::MAX,
address: std::net::Ipv6Addr::from(u128::MAX).into(),
peer: peer.0,
path_id,
},
}
}
}
crates/rib/src/table/tests.rs
+1032
View File
File diff suppressed because it is too large. Load diff
crates/rpki/Cargo.toml
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "bgp-rpki"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
bgp-primitives.workspace = true
bytes.workspace = true
foldhash.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["full"] }
tracing.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/rpki/src/aspa.rs
+426
View File
@@ -0,0 +1,426 @@
//! Autonomous System Provider Authorisations, and what they say about a
//! route's AS path (draft-ietf-sidrops-aspa-verification).
//!
//! An ASPA is a signed statement by an AS of who its providers are: the
//! networks it buys transit from. With those statements a router can tell
//! a path that goes the way traffic is paid to go (up from the origin
//! through providers, across at most one peering, and down through
//! customers) from one that does not. A path that climbs again after it
//! has come down, or crosses a second peering, is a route leak or a forged
//! path, and this is the check that finds them.
use std::collections::HashMap;
use std::sync::Arc;
/// What the ASPAs say about a path.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum PathValidity {
/// Some AS on the path has no ASPA, and nothing on the path is known
/// to be wrong. Most paths are in this state while few networks have
/// an ASPA, and are used.
Unknown,
/// Every hop that has to be from a customer to its provider is
/// attested to be one.
Valid,
/// Some AS on the path has said who its providers are, and the path
/// went from it to an AS that is not one of them where only a provider
/// could come next.
Invalid,
}
impl PathValidity {
pub fn as_str(self) -> &'static str {
match self {
Self::Unknown => "unknown",
Self::Valid => "valid",
Self::Invalid => "invalid",
}
}
}
/// Which of the two procedures verifies a path. That depends on what the
/// neighbor it came from is to us.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Algorithm {
/// For a route from a customer, a peer, or across a route server: all
/// of its path must lead upwards, from each AS to a provider of it
/// (§6.1 of the draft).
Upstream,
/// For a route from a provider: its path may lead up from the origin
/// and then down to us, and nothing else (§6.2).
Downstream,
}
/// Whether one AS is a provider of another, as far as the ASPAs say.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Hop {
/// The customer has no ASPA.
NoAttestation,
/// The customer's ASPA lists the other AS.
Provider,
/// The customer has an ASPA, and the other AS is not on it.
NotProvider,
}
/// The ASPAs there are: for each customer AS that has one, its providers.
///
/// Cloning copies the list of customers and shares the lists of providers.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct AspaTable {
providers: HashMap<u32, Arc<[u32]>, foldhash::fast::FixedState>,
}
impl AspaTable {
/// How many customer ASes have an ASPA.
pub fn len(&self) -> usize {
self.providers.len()
}
pub fn is_empty(&self) -> bool {
self.providers.is_empty()
}
/// Records that `customer`'s providers are `providers`, in place of
/// what was recorded for it before, which is returned.
pub fn set(
&mut self,
customer: u32,
providers: impl IntoIterator<Item = u32>,
) -> Option<Arc<[u32]>> {
let mut providers: Vec<u32> = providers.into_iter().collect();
providers.sort_unstable();
providers.dedup();
self.providers.insert(customer, providers.into())
}
/// Forgets `customer`'s ASPA, and returns the providers it listed.
pub fn remove(&mut self, customer: u32) -> Option<Arc<[u32]>> {
self.providers.remove(&customer)
}
/// Puts back what [`AspaTable::set`] or [`AspaTable::remove`] returned.
pub(crate) fn restore(&mut self, customer: u32, providers: Option<Arc<[u32]>>) {
match providers {
Some(providers) => self.providers.insert(customer, providers),
None => self.providers.remove(&customer),
};
}
/// The providers `customer` lists, in increasing order, if it has an
/// ASPA.
pub fn providers(&self, customer: u32) -> Option<&[u32]> {
self.providers.get(&customer).map(|providers| &**providers)
}
/// The customers and their providers, in no particular order.
pub fn iter(&self) -> impl Iterator<Item = (u32, &[u32])> {
self.providers
.iter()
.map(|(customer, providers)| (*customer, &**providers))
}
fn hop(&self, customer: u32, provider: u32) -> Hop {
match self.providers.get(&customer) {
None => Hop::NoAttestation,
Some(providers) if providers.binary_search(&provider).is_ok() => Hop::Provider,
Some(_) => Hop::NotProvider,
}
}
/// Verifies `path`: the AS numbers of an AS_PATH in the order they are
/// in it, the neighbor's first and the origin's last, with whatever
/// repeats it has.
///
/// This is the part of the draft's procedures that the ASPAs decide.
/// What comes before it there is the caller's to check: that the path
/// has no AS_SET and begins with the neighbor's AS. An empty path is
/// invalid.
pub fn verify(&self, path: &[u32], algorithm: Algorithm) -> PathValidity {
// The draft numbers the path's distinct ASes from the origin,
// A(1), to the neighbor, A(N). Here they are `hops[0]` to
// `hops[n - 1]`.
let mut hops: Vec<u32> = Vec::with_capacity(path.len());
for asn in path.iter().rev() {
if hops.last() != Some(asn) {
hops.push(*asn);
}
}
let n = hops.len();
if n == 0 {
return PathValidity::Invalid;
}
// How far up from the origin the path can be a chain of customers
// and their providers at most (as far as nothing says otherwise),
// and at least (as far as it is attested).
let (mut max_up, mut min_up) = (n, n);
for (at, pair) in hops.windows(2).enumerate() {
match self.hop(pair[0], pair[1]) {
Hop::Provider => continue,
Hop::NoAttestation => min_up = min_up.min(at + 1),
Hop::NotProvider => {
min_up = min_up.min(at + 1);
max_up = at + 1;
break;
}
}
}
if algorithm == Algorithm::Upstream {
return match (max_up < n, min_up < n) {
(true, _) => PathValidity::Invalid,
(false, true) => PathValidity::Unknown,
(false, false) => PathValidity::Valid,
};
}
// The same from the neighbor's end, looking down: each AS is to be
// the customer of the one after it.
let (mut max_down, mut min_down) = (n, n);
for (at, pair) in hops.windows(2).rev().enumerate() {
match self.hop(pair[1], pair[0]) {
Hop::Provider => continue,
Hop::NoAttestation => min_down = min_down.min(at + 1),
Hop::NotProvider => {
min_down = min_down.min(at + 1);
max_down = at + 1;
break;
}
}
}
// The way up and the way down have to meet, or overlap.
if max_up + max_down < n {
PathValidity::Invalid
} else if min_up + min_down < n {
PathValidity::Unknown
} else {
PathValidity::Valid
}
}
}
impl FromIterator<(u32, Vec<u32>)> for AspaTable {
fn from_iter<T: IntoIterator<Item = (u32, Vec<u32>)>>(aspas: T) -> Self {
let mut table = Self::default();
for (customer, providers) in aspas {
table.set(customer, providers);
}
table
}
}
#[cfg(test)]
mod tests {
use proptest::prelude::*;
use super::Algorithm::{Downstream, Upstream};
use super::PathValidity::{Invalid, Unknown, Valid};
use super::*;
fn table(aspas: &[(u32, &[u32])]) -> AspaTable {
aspas
.iter()
.map(|(customer, providers)| (*customer, providers.to_vec()))
.collect()
}
/// A small Internet. 10 and 20 are the large networks, which peer and
/// have no providers (an ASPA that lists AS 0). 100 buys from 10, 200
/// from 20, 300 from both. 1000 buys from 100; 2000 from 200 and has
/// no ASPA.
fn internet() -> AspaTable {
table(&[
(10, &[0]),
(20, &[0]),
(100, &[10]),
(200, &[20]),
(300, &[10, 20]),
(1000, &[100]),
])
}
#[test]
fn a_path_that_only_climbs_is_valid_from_a_customer_or_peer() {
let aspas = internet();
// 1000 -> 100 -> 10, as 10 hears it from its customer 100.
assert_eq!(aspas.verify(&[100, 1000], Upstream), Valid);
// As 20 hears it from its peer 10.
assert_eq!(aspas.verify(&[10, 100, 1000], Upstream), Valid);
// A route the neighbor originates is as far up as it has got.
assert_eq!(aspas.verify(&[1000], Upstream), Valid);
// Prepending changes nothing.
assert_eq!(
aspas.verify(&[10, 10, 100, 100, 100, 1000, 1000], Upstream),
Valid
);
}
#[test]
fn a_leak_is_invalid_from_a_customer_or_peer() {
let aspas = internet();
// 300 buys from 10 and 20, and passes 10's routes on to 20: the
// classic leak. 20 hears the path 300 10 100 1000 from its
// customer; 10 says it has no providers, so 300 is not one.
assert_eq!(aspas.verify(&[300, 10, 100, 1000], Upstream), Invalid);
// 20 passes a route of its peer 10's on to another peer: 20 is not
// a provider of 10.
assert_eq!(aspas.verify(&[20, 10, 100, 1000], Upstream), Invalid);
// An origin that is not where the path says: 1000 lists only 100.
assert_eq!(aspas.verify(&[200, 1000], Upstream), Invalid);
}
#[test]
fn a_hop_nobody_has_attested_leaves_the_path_unknown() {
let aspas = internet();
// 2000 has no ASPA.
assert_eq!(aspas.verify(&[200, 2000], Upstream), Unknown);
assert_eq!(aspas.verify(&[20, 200, 2000], Upstream), Unknown);
// But what is known to be wrong is invalid whatever else is not
// known.
assert_eq!(aspas.verify(&[300, 20, 200, 2000], Upstream), Invalid);
// With no ASPAs at all, nothing is known.
assert_eq!(AspaTable::default().verify(&[1, 2, 3], Upstream), Unknown);
assert_eq!(AspaTable::default().verify(&[1, 2, 3], Downstream), Unknown);
// Except of the path that is no path.
assert_eq!(AspaTable::default().verify(&[], Upstream), Invalid);
assert_eq!(AspaTable::default().verify(&[], Downstream), Invalid);
}
#[test]
fn from_a_provider_a_path_may_climb_and_then_descend() {
let aspas = internet();
// 1000's route, as 200 hears it from its provider 20: up through
// 100 to 10, across to 20.
assert_eq!(aspas.verify(&[20, 10, 100, 1000], Downstream), Valid);
// As 300 hears it from 10: only up.
assert_eq!(aspas.verify(&[10, 100, 1000], Downstream), Valid);
// As 1000 would hear 300's route from 100: up to 10, down to 100.
assert_eq!(aspas.verify(&[100, 10, 300], Downstream), Valid);
// Short paths have no room to be wrong in.
assert_eq!(aspas.verify(&[20, 1000], Downstream), Valid);
assert_eq!(aspas.verify(&[20], Downstream), Valid);
}
#[test]
fn from_a_provider_a_path_that_descends_and_climbs_again_is_invalid() {
let aspas = internet();
// 300 leaks 10's routes to 20, and 20 passes them down to 200,
// which hears the path 20 300 10 100 1000. The way up from 1000
// ends at 10, three ASes in: 300 is not a provider of 10. The way
// down to 200 begins no earlier than 20, one AS: 20 has no
// providers, so it did not get the route from one. Three and one
// do not make five.
assert_eq!(aspas.verify(&[20, 300, 10, 100, 1000], Downstream), Invalid);
// Two peerings in a row: 30 is another large network, which hands
// its customer a route it got from its peer 20, which got it from
// its peer 10.
let mut aspas = internet();
aspas.set(30, [0]);
assert_eq!(aspas.verify(&[30, 20, 10, 100, 1000], Downstream), Invalid);
}
#[test]
fn from_a_provider_what_is_not_attested_leaves_the_path_unknown() {
let aspas = internet();
// 2000 (no ASPA) up to 200, to 20, across to 10, down to 100: the
// first hop is not attested either way.
assert_eq!(aspas.verify(&[10, 20, 200, 2000], Downstream), Unknown);
// With 2000's ASPA it is valid.
let mut with = aspas.clone();
with.set(2000, [200]);
assert_eq!(with.verify(&[10, 20, 200, 2000], Downstream), Valid);
}
#[test]
fn an_aspa_is_replaced_and_removed_whole() {
let mut aspas = internet();
assert_eq!(aspas.providers(300), Some(&[10, 20][..]));
assert_eq!(
aspas.set(300, [20, 10, 20, 5]).as_deref(),
Some(&[10, 20][..])
);
assert_eq!(aspas.providers(300), Some(&[5, 10, 20][..]));
assert_eq!(aspas.remove(300).as_deref(), Some(&[5, 10, 20][..]));
assert_eq!(aspas.providers(300), None);
assert_eq!(aspas.remove(300), None);
assert_eq!(aspas.len(), 5);
}
/// The procedures as the draft words them, with its numbering: A(1) is
/// the origin, A(N) the neighbor.
fn by_the_draft(aspas: &AspaTable, path: &[u32], algorithm: Algorithm) -> PathValidity {
let mut a: Vec<u32> = path.iter().rev().copied().collect();
a.dedup();
let n = a.len();
if n == 0 {
return Invalid;
}
// One-based, as written.
let hop = |i: usize, j: usize| aspas.hop(a[i - 1], a[j - 1]);
let first_up = |wanted: &[Hop]| {
(1..n)
.find(|i| wanted.contains(&hop(*i, *i + 1)))
.unwrap_or(n)
};
let last_down = |wanted: &[Hop]| {
(2..=n)
.rev()
.find(|j| wanted.contains(&hop(*j, *j - 1)))
.map_or(n, |j| n - j + 1)
};
let not = [Hop::NotProvider];
let not_attested = [Hop::NoAttestation, Hop::NotProvider];
let (max_up, min_up) = (first_up(&not), first_up(&not_attested));
match algorithm {
Upstream if max_up < n => Invalid,
Upstream if min_up < n => Unknown,
Upstream => Valid,
Downstream => {
let (max_down, min_down) = (last_down(&not), last_down(&not_attested));
if max_up + max_down < n {
Invalid
} else if min_up + min_down < n {
Unknown
} else {
Valid
}
}
}
}
proptest! {
/// Over a handful of AS numbers, so that paths and ASPAs meet.
#[test]
fn verification_agrees_with_the_draft_as_written(
aspas in proptest::collection::vec(
(0u32..8, proptest::collection::vec(0u32..8, 1..4)), 0..8),
path in proptest::collection::vec(0u32..8, 0..10),
downstream: bool,
) {
let aspas: AspaTable = aspas.into_iter().collect();
let algorithm = if downstream { Downstream } else { Upstream };
prop_assert_eq!(
aspas.verify(&path, algorithm),
by_the_draft(&aspas, &path, algorithm)
);
}
/// What is invalid from a provider is invalid from anyone, and
/// what is valid from a customer is valid from anyone: the
/// procedure for providers allows more.
#[test]
fn the_procedure_for_providers_is_the_more_lenient(
aspas in proptest::collection::vec(
(0u32..8, proptest::collection::vec(0u32..8, 1..4)), 0..8),
path in proptest::collection::vec(0u32..8, 1..10),
) {
let aspas: AspaTable = aspas.into_iter().collect();
let (up, down) = (aspas.verify(&path, Upstream), aspas.verify(&path, Downstream));
if down == Invalid {
prop_assert_eq!(up, Invalid);
}
if up == Valid {
prop_assert_eq!(down, Valid);
}
}
}
}
crates/rpki/src/client.rs
+953
View File
@@ -0,0 +1,953 @@
//! Keeps the VRPs and ASPAs in step with a validator, over the
//! RPKI-to-Router protocol (RFC 8210 and draft-ietf-sidrops-8210bis).
//!
//! The client connects to a cache (the validator's RTR server), asks for
//! everything once, and from then on for what has changed: when the cache
//! says there is something new, and otherwise every so often. Each time the
//! data changes an [`Update`] is published for whoever validates routes.
//!
//! The newest version of the protocol is asked for first, and an older one
//! if that is what the cache speaks. Only version 2 carries ASPAs: with an
//! older cache there are none, and paths are not verified.
//!
//! A cache that cannot be reached is tried again, and the other caches in
//! turn. Meanwhile the data last received is used, but not for ever: a list
//! that has not been confirmed for the expire interval is dropped, and
//! routes are then neither valid nor invalid. Using stale data would be
//! worse: it would go on refusing routes whose holders have since
//! authorised them.
use std::io;
use std::net::SocketAddr;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::{Duration, SystemTime};
use bgp_primitives::constants::IpVersion;
use bytes::BytesMut;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio::sync::watch;
use tokio::task::JoinHandle;
use tokio::time::{Instant, sleep_until, timeout};
use tracing::{info, warn};
use crate::aspa::AspaTable;
use crate::rtr::{Pdu, RtrError, Timers, VERSION_0, VERSION_2, error_code};
use crate::vrp::{Vrp, VrpTable};
// Hashed by the hundred thousand with every answer, and told apart from
// what there was with every reset: a quick hash matters here.
type HashSet<K> = std::collections::HashSet<K, foldhash::fast::RandomState>;
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How long a cache may take to send the next part of an answer.
const RESPONSE_TIMEOUT: Duration = Duration::from_secs(60);
/// How long a whole answer may take. Everything a validator has takes it
/// seconds to send.
const ANSWER_TIMEOUT: Duration = Duration::from_secs(600);
/// The most records taken in one answer: several times what the whole
/// RPKI holds. A cache that sends more is not sending the RPKI, and
/// would otherwise be listened to until there is no memory left.
const MAX_RECORDS: usize = 8_000_000;
/// How long to wait before trying the next cache after one has failed.
const NEXT_CACHE_DELAY: Duration = Duration::from_secs(1);
/// How long to wait before asking a cache again that said it has no data
/// yet, rather than the retry interval: it is up and validating, which
/// takes minutes after it starts, and the retry interval is ten unless it
/// has said otherwise, which it has not had the chance to.
const NO_DATA_RETRY: Duration = Duration::from_secs(30);
/// How long a cache that is being left is given to take the reason.
const ERROR_REPORT_TIMEOUT: Duration = Duration::from_secs(1);
/// Where the VRPs come from, and how patiently.
#[derive(Debug, Clone, Default)]
pub struct RpkiConfig {
/// The caches, in order of preference. One is used at a time; the next
/// is tried when it fails.
pub caches: Vec<SocketAddr>,
/// If set, used in place of what the cache says (or of the defaults of
/// RFC 8210 §6, with a cache that says nothing).
pub refresh: Option<Duration>,
pub retry: Option<Duration>,
pub expire: Option<Duration>,
}
/// The VRPs and ASPAs as they are after a change.
#[derive(Debug, Clone)]
pub struct Update {
/// Goes up by one with every update.
pub version: u64,
pub vrps: Arc<VrpTable>,
pub aspas: Arc<AspaTable>,
/// What was added or removed to get here from the version before.
/// `None` if any route's validity may have changed.
pub changed: Option<Changed>,
}
/// What differs between one [`Update`] and the next.
#[derive(Debug, Clone)]
pub struct Changed {
/// The VRPs that came or went: only routes that these cover can have
/// another origin validity than before.
pub vrps: Arc<VrpTable>,
/// The customer ASes whose ASPA came, went or lists other providers:
/// only paths with one of these on them can verify otherwise than
/// before.
pub aspa_customers: Arc<HashSet<u32>>,
}
/// How things stand with the caches.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum CacheState {
/// No cache has been reached yet, or the one in use was lost.
Connecting,
/// Connected and waiting for the data.
Synchronising,
/// The data is complete as of the cache's serial.
Synchronised,
}
/// What there is to say about the client, for showing.
#[derive(Debug, Clone)]
pub struct Status {
/// The cache in use, or last tried.
pub cache: Option<SocketAddr>,
pub state: CacheState,
/// The cache's session and the serial the data is at.
pub session_id: Option<u16>,
pub serial: Option<u32>,
pub protocol_version: u8,
pub vrps_ipv4: usize,
pub vrps_ipv6: usize,
/// How many ASes have an ASPA. None do if the cache speaks a version
/// of the protocol before 2.
pub aspas: usize,
/// When the data was last confirmed by a cache.
pub updated_at: Option<SystemTime>,
/// When it will be dropped unless confirmed again.
pub expires_at: Option<SystemTime>,
/// Why the last attempt failed, if it did.
pub last_error: Option<String>,
}
struct Shared {
updates: watch::Sender<Update>,
/// The version of the latest update, for a look that costs nothing.
version: AtomicU64,
status: Mutex<Status>,
}
/// Handle to the VRPs and to the client that keeps them. Cheap to clone.
#[derive(Clone)]
pub struct Rpki {
shared: Arc<Shared>,
}
impl Rpki {
/// Starts a client for `config` and returns the handle to its data,
/// with the task that runs it. Until the first cache has answered
/// there are no VRPs and no ASPAs: every route's origin is
/// [`crate::Validity::NotFound`] and every path
/// [`crate::PathValidity::Unknown`].
pub fn spawn(config: RpkiConfig) -> (Self, JoinHandle<()>) {
let (updates, _) = watch::channel(Update {
version: 0,
vrps: Arc::default(),
aspas: Arc::default(),
changed: None,
});
let rpki = Self {
shared: Arc::new(Shared {
updates,
version: AtomicU64::new(0),
status: Mutex::new(Status {
cache: None,
state: CacheState::Connecting,
session_id: None,
serial: None,
protocol_version: VERSION_2,
vrps_ipv4: 0,
vrps_ipv6: 0,
aspas: 0,
updated_at: None,
expires_at: None,
last_error: None,
}),
}),
};
let client = Client {
shared: Arc::clone(&rpki.shared),
config,
data: Data::default(),
place: None,
protocol_version: VERSION_2,
timers: Timers::default(),
expires: None,
};
(rpki, tokio::spawn(client.run()))
}
/// The VRPs and ASPAs as they are now.
pub fn current(&self) -> Update {
self.shared.updates.borrow().clone()
}
/// The version of the latest [`Update`]. Costs a load of one number,
/// for those who ask before every use whether theirs is still current.
pub fn version(&self) -> u64 {
self.shared.version.load(Ordering::Acquire)
}
/// Follows the updates. A follower that is busy sees the latest only;
/// [`Update::version`] tells it whether it skipped any.
pub fn subscribe(&self) -> watch::Receiver<Update> {
self.shared.updates.subscribe()
}
pub fn status(&self) -> Status {
self.shared
.status
.lock()
.unwrap_or_else(PoisonError::into_inner)
.clone()
}
}
/// Why a session with a cache ended.
#[derive(Debug, thiserror::Error)]
enum SessionError {
#[error("{0}")]
Io(#[from] io::Error),
#[error("the cache sent something that is not RTR: {0}")]
Malformed(#[from] RtrError),
#[error("the cache closed the connection")]
Closed,
#[error("the cache did not accept the connection within {0:?}")]
ConnectTimeout(Duration),
#[error("the cache sent nothing for {0:?} in the middle of an answer")]
ResponseTimeout(Duration),
#[error("the cache did not finish its answer within {0:?}")]
AnswerTimeout(Duration),
#[error("the cache confirmed nothing for so long that the data expired")]
Expired,
#[error("the cache reported an error: {text} (code {code})")]
Reported { code: u16, text: String },
#[error("the cache sent {0} where it has no place")]
Unexpected(&'static str),
#[error("the cache {0}")]
Inconsistent(Inconsistency),
/// Not a failure of the cache: it speaks an older version, and is to
/// be asked again in that.
#[error("the cache speaks protocol version {0}")]
Downgrade(u8),
}
impl SessionError {
/// The error code that tells the cache why it is being left, if that
/// is something it did and can be told (RFC 8210 §10). An Error Report
/// is never answered with another.
fn report_code(&self) -> Option<u16> {
match self {
Self::Malformed(e) => Some(e.code()),
Self::Unexpected(_) => Some(error_code::CORRUPT_DATA),
Self::Inconsistent(problem) => Some(problem.code),
Self::Io(_)
| Self::Closed
| Self::ConnectTimeout(_)
| Self::ResponseTimeout(_)
| Self::AnswerTimeout(_)
| Self::Expired
| Self::Reported { .. }
| Self::Downgrade(_) => None,
}
}
}
/// Something a cache said that does not fit with what it said before.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct Inconsistency {
what: &'static str,
/// The error code it is reported to the cache with.
code: u16,
}
impl std::fmt::Display for Inconsistency {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str(self.what)
}
}
const fn inconsistent(what: &'static str, code: u16) -> Inconsistency {
Inconsistency { what, code }
}
const DUPLICATE_VRP: Inconsistency =
inconsistent("announced a VRP twice", error_code::DUPLICATE_ANNOUNCEMENT);
const UNKNOWN_VRP: Inconsistency = inconsistent(
"withdrew a VRP it had not announced",
error_code::WITHDRAWAL_OF_UNKNOWN_RECORD,
);
const UNKNOWN_ASPA: Inconsistency = inconsistent(
"withdrew an ASPA it had not announced",
error_code::WITHDRAWAL_OF_UNKNOWN_RECORD,
);
const OTHER_SESSION: Inconsistency = inconsistent(
"ended an answer in another session than it began it",
error_code::CORRUPT_DATA,
);
const OTHER_VERSION: Inconsistency = inconsistent(
"changed protocol version in the middle of a session",
error_code::UNEXPECTED_VERSION,
);
const INTERNAL: Inconsistency = inconsistent(
"could not be followed (a bug here)",
error_code::INTERNAL_ERROR,
);
/// Where the data is at: which of the cache's sessions, and which serial.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct Place {
cache: SocketAddr,
/// The version of the protocol the data came in. What another version
/// carries is not the same data.
protocol_version: u8,
session_id: u16,
serial: u32,
}
/// What a cache has given: the data validation rests on.
#[derive(Debug, Clone, Default)]
struct Data {
vrps: HashSet<Vrp>,
aspas: AspaTable,
}
/// One thing an answer says: a VRP or an ASPA, come or gone.
#[derive(Debug, Clone)]
enum Record {
Vrp {
announce: bool,
vrp: Vrp,
},
/// With providers, the ASPA that is now `customer`'s; without, that
/// it has none any more.
Aspa {
customer: u32,
providers: Option<Vec<u32>>,
},
}
struct Client {
shared: Arc<Shared>,
config: RpkiConfig,
/// The data as last received.
data: Data,
/// Where `data` is at, if it came from a cache that can be asked for
/// what has changed since.
place: Option<Place>,
protocol_version: u8,
/// As the cache last gave them, or the defaults.
timers: Timers,
/// When `data` is to be dropped unless a cache confirms it first.
expires: Option<Instant>,
}
/// A connection to a cache, read a PDU at a time.
struct Connection {
stream: TcpStream,
buf: BytesMut,
}
impl Connection {
async fn send(&mut self, version: u8, pdu: &Pdu) -> io::Result<()> {
let mut out = BytesMut::new();
pdu.encode(version, &mut out);
self.stream.write_all(&out).await
}
/// The next PDU, and the protocol version it was sent with.
/// Cancel-safe: what has arrived of a PDU stays in the buffer.
async fn next(&mut self) -> Result<(u8, Pdu), SessionError> {
loop {
if let Some((version, pdu, used)) = Pdu::decode(&self.buf)? {
let _ = self.buf.split_to(used);
return Ok((version, pdu));
}
if self.stream.read_buf(&mut self.buf).await? == 0 {
return Err(SessionError::Closed);
}
}
}
}
impl Client {
fn status(&self, change: impl FnOnce(&mut Status)) {
change(
&mut self
.shared
.status
.lock()
.unwrap_or_else(PoisonError::into_inner),
);
}
fn refresh(&self) -> Duration {
self.config.refresh.unwrap_or(Duration::from_secs(
self.timers.refresh.clamp(1, 86400).into(),
))
}
fn retry(&self) -> Duration {
self.config
.retry
.unwrap_or(Duration::from_secs(self.timers.retry.clamp(1, 7200).into()))
}
/// Never sooner than a refresh that failed and was tried once more
/// would take, whatever the cache or the configuration says: data that
/// ran out between two refreshes would leave nothing validated for a
/// part of every round (RFC 8210 §6 wants it longer than either).
fn expire(&self) -> Duration {
self.config
.expire
.unwrap_or(Duration::from_secs(
self.timers.expire.clamp(600, 172_800).into(),
))
.max(self.refresh() + self.retry())
}
async fn run(mut self) {
if self.config.caches.is_empty() {
return std::future::pending().await;
}
let mut attempt = 0usize;
loop {
let cache = self.config.caches[attempt % self.config.caches.len()];
self.status(|status| {
status.cache = Some(cache);
status.state = CacheState::Connecting;
});
let ended = self.session(cache).await;
if let SessionError::Downgrade(version) = ended {
info!(%cache, version, "RPKI cache speaks an older RTR version; asking again in that");
self.protocol_version = version;
continue;
}
let no_data = matches!(
ended,
SessionError::Reported {
code: error_code::NO_DATA_AVAILABLE,
..
}
);
if no_data {
info!(%cache, "RPKI cache has no data yet; asking again soon");
} else {
warn!(%cache, err = %ended, "RPKI cache session ended");
}
self.status(|status| {
status.state = CacheState::Connecting;
status.last_error = Some(ended.to_string());
});
attempt += 1;
// The next cache, or this one when it is back, may speak a
// newer version.
self.protocol_version = VERSION_2;
let all_tried = attempt.is_multiple_of(self.config.caches.len());
self.wait(rest(all_tried, no_data, self.retry())).await;
}
}
/// Waits for `duration`, dropping the data on the way if its time runs
/// out.
async fn wait(&mut self, duration: Duration) {
let until = Instant::now() + duration;
if let Some(expires) = self.expires
&& expires <= until
{
sleep_until(expires).await;
self.expired();
}
sleep_until(until).await;
}
/// The data has gone unconfirmed for too long to be used.
fn expired(&mut self) {
warn!(
vrps = self.data.vrps.len(),
aspas = self.data.aspas.len(),
"RPKI data has expired without a cache to confirm it; dropping it"
);
self.data = Data::default();
self.place = None;
self.expires = None;
self.publish(Arc::default(), None);
self.status(|status| {
status.session_id = None;
status.serial = None;
status.expires_at = None;
});
}
/// Runs a session with `cache` until it ends, which it does only by
/// failing.
async fn session(&mut self, cache: SocketAddr) -> SessionError {
let stream = match timeout(CONNECT_TIMEOUT, TcpStream::connect(cache)).await {
Ok(Ok(stream)) => stream,
Ok(Err(e)) => return e.into(),
Err(_) => return SessionError::ConnectTimeout(CONNECT_TIMEOUT),
};
let mut connection = Connection {
stream,
buf: BytesMut::with_capacity(64 * 1024),
};
info!(%cache, "Connected to RPKI cache");
self.status(|status| status.state = CacheState::Synchronising);
loop {
if let Err(e) = self.synchronise(cache, &mut connection).await {
return self.leave(&mut connection, e).await;
}
// Until the cache says there is something new, or it is time
// to ask anyway. If the data runs out first, it is the cache
// that has stopped confirming it.
let ask = Instant::now() + self.refresh();
let expires = self.expires.unwrap_or(ask);
tokio::select! {
pdu = connection.next() => {
let problem = match pdu {
Ok((_, Pdu::ErrorReport { code, text })) => {
SessionError::Reported { code, text }
}
Ok((version, _)) if version != self.protocol_version => {
SessionError::Inconsistent(OTHER_VERSION)
}
Ok((_, Pdu::SerialNotify { .. })) => continue,
Ok(_) => SessionError::Unexpected("an answer nobody asked for"),
Err(e) => e,
};
return self.leave(&mut connection, problem).await;
}
() = sleep_until(ask) => {}
() = sleep_until(expires), if expires < ask => {
self.expired();
return SessionError::Expired;
}
}
}
}
/// Tells the cache why the session is ending, if it is for something
/// the cache did, and gives the reason back.
async fn leave(&self, connection: &mut Connection, why: SessionError) -> SessionError {
if let Some(code) = why.report_code() {
let report = Pdu::ErrorReport {
code,
text: why.to_string(),
};
// If it cannot be told, it cannot.
let _ = timeout(
ERROR_REPORT_TIMEOUT,
connection.send(self.protocol_version, &report),
)
.await;
}
why
}
/// Asks the cache for what has changed, or for everything, and takes
/// the answer.
async fn synchronise(
&mut self,
cache: SocketAddr,
connection: &mut Connection,
) -> Result<(), SessionError> {
// Changes can be asked for only of the cache the data came from,
// in the version of the protocol it came in.
let mut since = self.place.filter(|place| {
place.cache == cache && place.protocol_version == self.protocol_version
});
loop {
let query = match since {
Some(place) => Pdu::SerialQuery {
session_id: place.session_id,
serial: place.serial,
},
None => Pdu::ResetQuery,
};
connection.send(self.protocol_version, &query).await?;
match self.answer(cache, connection, since).await? {
Answer::Complete => return Ok(()),
// The cache cannot say what changed: ask for everything.
Answer::Reset => since = None,
}
}
}
/// Reads the answer to a query. `since` is the place a Serial Query
/// asked about; `None` after a Reset Query.
async fn answer(
&mut self,
cache: SocketAddr,
connection: &mut Connection,
since: Option<Place>,
) -> Result<Answer, SessionError> {
// Each PDU in good time, and all of them too: a cache that sends
// one a minute would keep the old data in use for ever.
let deadline = Instant::now() + ANSWER_TIMEOUT;
let mut next = async || {
let left = deadline.saturating_duration_since(Instant::now());
match timeout(RESPONSE_TIMEOUT.min(left), connection.next()).await {
Ok(pdu) => pdu,
Err(_) if left <= RESPONSE_TIMEOUT => {
Err(SessionError::AnswerTimeout(ANSWER_TIMEOUT))
}
Err(_) => Err(SessionError::ResponseTimeout(RESPONSE_TIMEOUT)),
}
};
// The answer begins with a Cache Response, or is a Cache Reset.
let session_id = loop {
let (version, pdu) = next().await?;
if version < self.protocol_version {
// An older cache answers in its own version, with an error
// or as if it had been asked in that; the query has to be
// put again in that version (RFC 8210 §7).
return Err(SessionError::Downgrade(version));
}
if version > self.protocol_version {
return Err(SessionError::Inconsistent(OTHER_VERSION));
}
match pdu {
Pdu::CacheResponse { session_id } => break session_id,
Pdu::CacheReset if since.is_some() => return Ok(Answer::Reset),
// News of what is being asked for anyway.
Pdu::SerialNotify { .. } => {}
// A cache that refuses the version in the version it
// refuses: the next one down, then.
Pdu::ErrorReport { code, .. }
if code == error_code::UNSUPPORTED_VERSION
&& self.protocol_version != VERSION_0 =>
{
return Err(SessionError::Downgrade(self.protocol_version - 1));
}
Pdu::ErrorReport { code, text } => {
return Err(SessionError::Reported { code, text });
}
_ => return Err(SessionError::Unexpected("data before a Cache Response")),
}
};
// The cache has answered in the version asked for: that is the
// version of the session.
let protocol_version = self.protocol_version;
self.status(|status| status.protocol_version = protocol_version);
if since.is_some_and(|place| place.session_id != session_id) {
// The cache has started over; its serials mean something else
// now.
return Ok(Answer::Reset);
}
// Then the VRPs and ASPAs, and End of Data. Nothing takes effect
// until the answer is complete (RFC 8210 §8.1), so what it says is
// kept aside in the order it came.
let mut records: Vec<Record> = Vec::new();
let (serial, timers) = loop {
let (version, pdu) = next().await?;
if version != self.protocol_version && !matches!(pdu, Pdu::ErrorReport { .. }) {
return Err(SessionError::Inconsistent(OTHER_VERSION));
}
if records.len() >= MAX_RECORDS {
return Err(SessionError::Unexpected("more data than any cache has"));
}
match pdu {
Pdu::Prefix { announce, vrp } => records.push(Record::Vrp { announce, vrp }),
Pdu::Aspa {
announce,
customer,
providers,
} => records.push(Record::Aspa {
customer,
providers: announce.then_some(providers),
}),
Pdu::RouterKey => {}
Pdu::EndOfData {
session_id: ended,
serial,
timers,
} => {
if ended != session_id {
return Err(SessionError::Inconsistent(OTHER_SESSION));
}
break (serial, timers);
}
Pdu::ErrorReport { code, text } => {
return Err(SessionError::Reported { code, text });
}
_ => {
return Err(SessionError::Unexpected(
"a PDU that is not part of an answer",
));
}
}
};
// The first data there is can change any route's validity; after
// that, only where something came or went.
let first = self.place.is_none() && self.expires.is_none();
let (before, replaced) = match since {
// Changes, to what there is.
Some(_) => (std::mem::take(&mut self.data), None),
// Everything anew.
None => (Data::default(), Some(std::mem::take(&mut self.data))),
};
let vrps = Arc::clone(&self.shared.updates.borrow().vrps);
// Building the tables is work for a thread that may take its time.
let built = tokio::task::spawn_blocking(move || apply(before, replaced, &records, vrps))
.await
.map_err(|_| SessionError::Inconsistent(INTERNAL))?;
let built = match built {
Ok(built) => built,
Err((data, problem)) => {
// What there was stays as it was.
self.data = data;
return Err(SessionError::Inconsistent(problem));
}
};
self.data = built.data;
self.place = Some(Place {
cache,
protocol_version: self.protocol_version,
session_id,
serial,
});
self.timers = timers.unwrap_or_default();
let expires_in = self.expire();
self.expires = Some(Instant::now() + expires_in);
let changes = built.changed.vrps.len() + built.changed.aspa_customers.len();
if first || changes > 0 {
self.publish(built.vrps, (!first).then_some(built.changed));
}
info!(
%cache,
serial,
vrps = self.data.vrps.len(),
aspas = self.data.aspas.len(),
changes,
"RPKI data is up to date"
);
let updated_at = SystemTime::now();
self.status(|status| {
status.state = CacheState::Synchronised;
status.session_id = Some(session_id);
status.serial = Some(serial);
status.updated_at = Some(updated_at);
status.expires_at = Some(updated_at + expires_in);
status.last_error = None;
});
Ok(Answer::Complete)
}
/// Publishes the data as it now is. `vrps` holds its VRPs.
fn publish(&self, vrps: Arc<VrpTable>, changed: Option<Changed>) {
let count = |version| {
self.data
.vrps
.iter()
.filter(|vrp| vrp.prefix.address_family() == version)
.count()
};
let (ipv4, ipv6) = (count(IpVersion::Ipv4), count(IpVersion::Ipv6));
self.status(|status| {
status.vrps_ipv4 = ipv4;
status.vrps_ipv6 = ipv6;
status.aspas = self.data.aspas.len();
});
let aspas = Arc::new(self.data.aspas.clone());
self.shared.updates.send_modify(|update| {
update.version += 1;
update.vrps = vrps;
update.aspas = aspas;
update.changed = changed;
self.shared.version.store(update.version, Ordering::Release);
});
}
}
/// The data after an answer, ready to be used.
struct Built {
data: Data,
/// The VRPs of `data`, arranged for validating.
vrps: Arc<VrpTable>,
changed: Changed,
}
/// Applies the `records` of a complete answer, in order, to `data`.
/// `replaced` is the data that an answer to a Reset Query takes the place
/// of, and `vrps` the table of the VRPs as they were before the answer,
/// which is kept if the answer changes none.
///
/// If the answer contradicts itself or what there was, what there was is
/// given back with the reason.
fn apply(
mut data: Data,
replaced: Option<Data>,
records: &[Record],
vrps: Arc<VrpTable>,
) -> Result<Built, (Data, Inconsistency)> {
// What each record displaced, to put back if a later one cannot be
// applied.
let mut displaced: Vec<Option<Arc<[u32]>>> = Vec::new();
let mut problem = None;
for record in records {
let ok = match record {
Record::Vrp {
announce: true,
vrp,
} => data.vrps.insert(*vrp).then_some(None).ok_or(DUPLICATE_VRP),
Record::Vrp {
announce: false,
vrp,
} => data.vrps.remove(vrp).then_some(None).ok_or(UNKNOWN_VRP),
// An ASPA that is announced takes the place of the one there
// was (draft-ietf-sidrops-8210bis §5.12).
Record::Aspa {
customer,
providers: Some(providers),
} => Ok(data.aspas.set(*customer, providers.iter().copied())),
Record::Aspa {
customer,
providers: None,
} => data.aspas.remove(*customer).map(Some).ok_or(UNKNOWN_ASPA),
};
match ok {
Ok(was) => displaced.push(was),
Err(e) => {
problem = Some(e);
break;
}
}
}
if let Some(problem) = problem {
return Err(match replaced {
Some(replaced) => (replaced, problem),
None => {
// Undo, last first.
for (record, was) in records.iter().zip(displaced).rev() {
match record {
Record::Vrp {
announce: true,
vrp,
} => {
data.vrps.remove(vrp);
}
Record::Vrp {
announce: false,
vrp,
} => {
data.vrps.insert(*vrp);
}
Record::Aspa { customer, .. } => data.aspas.restore(*customer, was),
}
}
(data, problem)
}
});
}
// What came or went: against the data replaced, or else whatever the
// answer touched.
let (changed_vrps, aspa_customers): (Vec<Vrp>, HashSet<u32>) = match &replaced {
Some(replaced) => (
data.vrps
.symmetric_difference(&replaced.vrps)
.copied()
.collect(),
differing_customers(&data.aspas, &replaced.aspas),
),
None => (
records
.iter()
.filter_map(|record| match record {
Record::Vrp { vrp, .. } => Some(*vrp),
Record::Aspa { .. } => None,
})
.collect(),
records
.iter()
.filter_map(|record| match record {
Record::Aspa { customer, .. } => Some(*customer),
Record::Vrp { .. } => None,
})
.collect(),
),
};
Ok(Built {
vrps: match changed_vrps.is_empty() {
true => vrps,
false => Arc::new(VrpTable::new(data.vrps.iter().copied())),
},
changed: Changed {
vrps: Arc::new(VrpTable::new(changed_vrps)),
aspa_customers: Arc::new(aspa_customers),
},
data,
})
}
/// The customers whose ASPA is not the same in `a` as in `b`.
fn differing_customers(a: &AspaTable, b: &AspaTable) -> HashSet<u32> {
let differs = |(customer, providers): (u32, &[u32]), other: &AspaTable| {
(other.providers(customer) != Some(providers)).then_some(customer)
};
a.iter()
.filter_map(|aspa| differs(aspa, b))
.chain(b.iter().filter_map(|aspa| differs(aspa, a)))
.collect()
}
/// How a query was answered.
enum Answer {
/// With data, which is now in use.
Complete,
/// With a request to ask for everything instead.
Reset,
}
/// How long to rest after a session ended: the next cache at once; when
/// all have been tried, the retry interval, or less if the last one is
/// only still validating.
fn rest(all_tried: bool, no_data: bool, retry: Duration) -> Duration {
match (all_tried, no_data) {
(false, _) => NEXT_CACHE_DELAY,
(true, true) => NO_DATA_RETRY.min(retry),
(true, false) => retry,
}
}
#[cfg(test)]
mod tests {
use super::*;
/// A cache that is still validating after it started is asked again
/// in half a minute, not the ten the retry interval is by default.
#[test]
fn a_cache_without_data_yet_is_asked_again_soon() {
let ten_minutes = Duration::from_secs(600);
assert_eq!(rest(true, true, ten_minutes), NO_DATA_RETRY);
assert_eq!(rest(true, false, ten_minutes), ten_minutes);
// Never later than the retry interval says.
let short = Duration::from_secs(5);
assert_eq!(rest(true, true, short), short);
// Another cache first, whatever this one said.
assert_eq!(rest(false, true, ten_minutes), NEXT_CACHE_DELAY);
}
}
crates/rpki/src/lib.rs
+29
View File
@@ -0,0 +1,29 @@
//! Route origin validation (RFC 6811) and AS path verification
//! (draft-ietf-sidrops-aspa-verification) with the RPKI.
//!
//! The holder of an address block can sign a statement of which AS may
//! originate routes for it. Validators collect and check those statements
//! and hand routers the result, a list of validated ROA payloads (VRPs),
//! over the RPKI-to-Router protocol (RFC 8210). With that list a router can
//! tell a route whose origin is authorised from one whose origin is not:
//! the commonest kind of hijack and of mistake.
//!
//! The same way, a network can sign a statement of who its providers are
//! (an ASPA). With those a router can tell a path that could have come
//! about by networks passing routes on as they are paid to from one that
//! could not: a route leak, or a path made up to get past origin
//! validation.
//!
//! - [`vrp`] holds the VRPs and answers whether a route's origin is valid.
//! - [`aspa`] holds the ASPAs and answers whether a path is.
//! - [`rtr`] is the protocol's messages.
//! - [`client`] keeps both in step with a validator.
pub mod aspa;
pub mod client;
pub mod rtr;
pub mod vrp;
pub use aspa::{Algorithm, AspaTable, PathValidity};
pub use client::{CacheState, Changed, Rpki, RpkiConfig, Status, Update};
pub use vrp::{Validity, Vrp, VrpTable};
Loaded 100 of 268 files, more files were not shown because too many files have changed in this diff. Show more