A review of the wire codec and the routing table, with the fixes.
Bugs in the codec:
- A NOTIFICATION with a code or subcode we have no name for (subcode 0,
"unspecific", is common) failed to decode and was answered as a bad
message. Every pair now reads, as BgpError::Other.
- Optional parameters in the extended form of RFC 9072 were read and
written with one-octet parameter lengths; they are two.
- An ORF capability listing more than one address family made the whole
OPEN malformed.
- Bytes after an OPEN's optional parameters were accepted.
- Extended communities were split into fields by rules that were wrong
for the common types. They are kept as their eight octets.
- Several length fields were computed in u16 and could wrap on encode.
Simplifications in the codec:
- Lengths are filled in after the value is written, for messages,
attributes, capabilities and the UPDATE's two fields. The parallel
wire_len functions, which had to be kept in step by hand, are gone.
Message::to_wire no longer encodes into a second buffer.
- Type codes for messages, attributes and capabilities are each defined
once. The strum discriminant enums and their hand-written conversions
are removed.
- One function splits an attribute off a buffer; the UPDATE decoder, the
plain attribute parser and the table's key reader all use it.
- The UPDATE decoder's 185-line loop is an AttributeReader with one
method that decides what becomes of an attribute. Duplicate detection
is a bitset, not a HashSet allocated per UPDATE.
- ParserContext::four_octet_asn is a bool. The "context not set" error
path, which treated a programming error as a malformed route, is gone.
- Codec functions take a context only if they read it, and return a
Result only if they can fail.
- MP_REACH_NLRI and MP_UNREACH_NLRI hold an AfiSafi.
- AddressFamilyId is IpVersion, which is what it was.
- Deserialize is no longer derived on codec types: nothing used it, and
it bypassed IpPrefix's invariants. IpPrefix::to_wire, which was
broken and unused, is removed.
Bugs in the table:
- The index of a row's best route was sixteen bits. A peer using
ADD-PATH could send more routes for a prefix than that counts, and
the index wrapped. It is 32 bits, at no cost in row size.
- Shards and attribute-store stripes were picked from the top bits of
the hash, which are the bits hashbrown keeps as the tag it compares
first. With 64 shards each shard's table had 2 of 128 tag values.
They are now picked from bits the table does not use.
- remove_peer held the peer list's lock for the whole scan of the
table, so no session could establish while a peer was removed.
- When more prefixes went at once than were left, the forced trim
dropped a fixed few entries, reset every reader, and re-ran the tidy
on every later write. It drops the older half.
- Readers were woken by every change, including a repeated announcement
and changes their journal does not record.
- Reader::poll with a budget of zero never made progress.
Simplifications in the table:
- The two journals of a shard are one Journals value written through
three calls; the five places that kept them in step by hand are gone.
Whether an entry is still live is decided in one place.
- The shortcuts of route selection moved to select.rs, next to the
invariant they depend on, which is now written down.
- announce and withdraw take routes with or without path identifiers;
the _paths variants are removed.
- The selection key is read off the wire form only; the second
implementation over parsed attributes is removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer can now send several routes for one prefix, told apart by a path
identifier, and each is a route of its own in the table.
Wire:
- The ADD-PATH capability lists several address families. It was read as
exactly one, so the OPEN of a router offering it for IPv4 and IPv6 was
malformed and the session never came up.
- Prefixes in UPDATEs are `Nlri`: a prefix and a path identifier. The
identifier is on the wire, in the body and in MP_REACH/MP_UNREACH, for
the families the session uses ADD-PATH for in that direction. Sending
and receiving are settled separately, so a session has one codec context
for each.
Negotiation:
- `add_path` on a peer in the configuration: off (the default), receive,
send or both, offered for every family offered to the peer.
- In use for a family and a direction when one side offered to send and
the other to receive. Shown on /statusz and in /api/peers.
Table:
- A route is known by its peer and path identifier together. Identifiers
are stored only in rows that have a non-zero one, so routes of peers
that send one route each cost nothing more.
- Equal routes from one peer are chosen between by the lower identifier.
- /api/routes/received gives each route's path_id.
Journal:
- A second journal in each shard records every change to any route, for
followers that want all the routes of a prefix and not only the best:
RibTable::reader_of_all_routes and Event::Routes. This is what sending
several routes to a peer will read. It costs nothing while nobody
follows it.
Sending: prefixes go out with a path identifier where that was agreed, but
there is still one route for each prefix to send: only originated routes
are announced.
Tested with the table and journal model tests extended to path identifiers
and to followers of all routes, the codec property tests and fuzz targets
extended to ADD-PATH sessions, and end to end in both directions and for
the case where only one side offers it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The server now keeps what it learns in table::RibTable. The Adj-RIB-In
actor is gone, and with it the queue every UPDATE from every session went
through.
- A session takes a number in the table when it is established and writes
its UPDATEs into the table itself. Its routes are removed when it ends,
on a blocking thread since that looks at every prefix; a session torn
down without cleaning up still has them removed.
- For every prefix the best route is chosen (RFC 4271 §9.1.2.2) as routes
come and go. Nothing acts on the choice yet beyond showing it.
- The UPDATE decoder hands over the bytes of the attributes it kept, so the
table finds the attribute set from them without writing the attributes
out again. On a session with two-octet AS numbers the bytes are not in
the table's form, and the parsed attributes are used.
- /api/routes/received marks the best route of each prefix and returns at
most `limit` routes (default 1000): the table can hold hundreds of
millions. /api/status has the totals, and the status page says how many
it is showing. The time a route was received is no longer kept or shown.
- The event stream reports changes of best route (best_changed,
best_withdrawn) in place of every route received. Each client follows
the table's journal with a reader of its own, from the moment it
connects: table::RibTable::reader_from_now.
- Metrics: bgp_rib_routes, bgp_rib_prefixes, bgp_rib_attribute_sets and
bgp_rib_journal_entries replace bgp_adj_rib_in_routes.
- A line is no longer logged at INFO for every UPDATE sent or received; a
full table is a million of them. They are at DEBUG.
- A table with no room for another peer ends the session with Cease / Out
of Resources.
Four peers sending a synthetic full table each over TCP, 4.9 million
routes: 0.80 s, against 4.76 s before. One peer: 0.45 s against 0.69 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bgp_rib::table::RibTable holds every route from every peer and the best
one for each prefix. It is not used by the server yet; this adds it with
its tests and benchmarks.
- Organised by prefix. A row holds the prefix's routes from all peers in
one array; choosing again after a change reads that row and nothing else.
- A route is sixteen bytes: a pointer to its attribute set and one word
holding the peer and the first three steps of selection (LOCAL_PREF, AS
path length, ORIGIN) as a single number. That number alone settles most
changes; a tie goes on to MED, compared only within a neighbouring AS
and in a way that does not depend on arrival order, and the tie-breaks.
- Attribute sets are stored once, in wire form, with what selection needs
from them worked out when they are stored.
- Sharded by hash of the prefix, a lock per shard. A session applies its
own UPDATEs; there is no queue or owning thread in front of the table.
- Readers take one shard at a time for reading and look at rows in place.
Route selection is checked against a plain implementation with no
shortcuts by a property test over random announcements, withdrawals and
session flaps, for both the chosen route and the count of changes.
On a RIPE RIS dump from AMS-IX (148 peers, 39.3 million routes), against
the Adj-RIB-In actor, which does no selection:
actor table
memory 11.0 GiB 1.8 GiB (279 -> 48 bytes a route)
load, 1 thread 10.2 s 12.3 s
load, 12 threads (no gain) 2.1 s (19 million routes a second)
read every route 1.6 s, copying 8 GiB, table stopped
0.2 s, in place, table running
The benchmark crate counts memory only with --features count-memory: the
counters are shared between threads and distort the timings.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A prefix was a Vec of its bytes: a heap allocation for every prefix parsed,
stored or copied, and a variable-length key to hash. It is now twenty
bytes, Copy, with private fields and one representation; `bytes()` gives
the wire form.
On the full-table benchmark this alone takes decoding from 9.3 to 10.7
million routes a second and inserting into the Adj-RIB-In from 2.9 to 3.6.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
crates/bench makes a synthetic full table (a million IPv4 and 230,000 IPv6
prefixes, announced by several peers in UPDATEs shaped like a router's)
and times the steps of taking it in: decoding, the Adj-RIB-In, snapshot,
peer removal, and the whole path over TCP. It also counts the memory the
table holds, and fills a plain packed hash map with the same routes as a
point of comparison.
Run with `cargo bench -p bgp-bench`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
