A review of the wire codec and the routing table, with the fixes.
Bugs in the codec:
- A NOTIFICATION with a code or subcode we have no name for (subcode 0,
"unspecific", is common) failed to decode and was answered as a bad
message. Every pair now reads, as BgpError::Other.
- Optional parameters in the extended form of RFC 9072 were read and
written with one-octet parameter lengths; they are two.
- An ORF capability listing more than one address family made the whole
OPEN malformed.
- Bytes after an OPEN's optional parameters were accepted.
- Extended communities were split into fields by rules that were wrong
for the common types. They are kept as their eight octets.
- Several length fields were computed in u16 and could wrap on encode.
Simplifications in the codec:
- Lengths are filled in after the value is written, for messages,
attributes, capabilities and the UPDATE's two fields. The parallel
wire_len functions, which had to be kept in step by hand, are gone.
Message::to_wire no longer encodes into a second buffer.
- Type codes for messages, attributes and capabilities are each defined
once. The strum discriminant enums and their hand-written conversions
are removed.
- One function splits an attribute off a buffer; the UPDATE decoder, the
plain attribute parser and the table's key reader all use it.
- The UPDATE decoder's 185-line loop is an AttributeReader with one
method that decides what becomes of an attribute. Duplicate detection
is a bitset, not a HashSet allocated per UPDATE.
- ParserContext::four_octet_asn is a bool. The "context not set" error
path, which treated a programming error as a malformed route, is gone.
- Codec functions take a context only if they read it, and return a
Result only if they can fail.
- MP_REACH_NLRI and MP_UNREACH_NLRI hold an AfiSafi.
- AddressFamilyId is IpVersion, which is what it was.
- Deserialize is no longer derived on codec types: nothing used it, and
it bypassed IpPrefix's invariants. IpPrefix::to_wire, which was
broken and unused, is removed.
Bugs in the table:
- The index of a row's best route was sixteen bits. A peer using
ADD-PATH could send more routes for a prefix than that counts, and
the index wrapped. It is 32 bits, at no cost in row size.
- Shards and attribute-store stripes were picked from the top bits of
the hash, which are the bits hashbrown keeps as the tag it compares
first. With 64 shards each shard's table had 2 of 128 tag values.
They are now picked from bits the table does not use.
- remove_peer held the peer list's lock for the whole scan of the
table, so no session could establish while a peer was removed.
- When more prefixes went at once than were left, the forced trim
dropped a fixed few entries, reset every reader, and re-ran the tidy
on every later write. It drops the older half.
- Readers were woken by every change, including a repeated announcement
and changes their journal does not record.
- Reader::poll with a budget of zero never made progress.
Simplifications in the table:
- The two journals of a shard are one Journals value written through
three calls; the five places that kept them in step by hand are gone.
Whether an entry is still live is decided in one place.
- The shortcuts of route selection moved to select.rs, next to the
invariant they depend on, which is now written down.
- announce and withdraw take routes with or without path identifiers;
the _paths variants are removed.
- The selection key is read off the wire form only; the second
implementation over parsed attributes is removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer can now send several routes for one prefix, told apart by a path
identifier, and each is a route of its own in the table.
Wire:
- The ADD-PATH capability lists several address families. It was read as
exactly one, so the OPEN of a router offering it for IPv4 and IPv6 was
malformed and the session never came up.
- Prefixes in UPDATEs are `Nlri`: a prefix and a path identifier. The
identifier is on the wire, in the body and in MP_REACH/MP_UNREACH, for
the families the session uses ADD-PATH for in that direction. Sending
and receiving are settled separately, so a session has one codec context
for each.
Negotiation:
- `add_path` on a peer in the configuration: off (the default), receive,
send or both, offered for every family offered to the peer.
- In use for a family and a direction when one side offered to send and
the other to receive. Shown on /statusz and in /api/peers.
Table:
- A route is known by its peer and path identifier together. Identifiers
are stored only in rows that have a non-zero one, so routes of peers
that send one route each cost nothing more.
- Equal routes from one peer are chosen between by the lower identifier.
- /api/routes/received gives each route's path_id.
Journal:
- A second journal in each shard records every change to any route, for
followers that want all the routes of a prefix and not only the best:
RibTable::reader_of_all_routes and Event::Routes. This is what sending
several routes to a peer will read. It costs nothing while nobody
follows it.
Sending: prefixes go out with a path identifier where that was agreed, but
there is still one route for each prefix to send: only originated routes
are announced.
Tested with the table and journal model tests extended to path identifiers
and to followers of all routes, the codec property tests and fuzz targets
extended to ADD-PATH sessions, and end to end in both directions and for
the case where only one side offers it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The server now keeps what it learns in table::RibTable. The Adj-RIB-In
actor is gone, and with it the queue every UPDATE from every session went
through.
- A session takes a number in the table when it is established and writes
its UPDATEs into the table itself. Its routes are removed when it ends,
on a blocking thread since that looks at every prefix; a session torn
down without cleaning up still has them removed.
- For every prefix the best route is chosen (RFC 4271 §9.1.2.2) as routes
come and go. Nothing acts on the choice yet beyond showing it.
- The UPDATE decoder hands over the bytes of the attributes it kept, so the
table finds the attribute set from them without writing the attributes
out again. On a session with two-octet AS numbers the bytes are not in
the table's form, and the parsed attributes are used.
- /api/routes/received marks the best route of each prefix and returns at
most `limit` routes (default 1000): the table can hold hundreds of
millions. /api/status has the totals, and the status page says how many
it is showing. The time a route was received is no longer kept or shown.
- The event stream reports changes of best route (best_changed,
best_withdrawn) in place of every route received. Each client follows
the table's journal with a reader of its own, from the moment it
connects: table::RibTable::reader_from_now.
- Metrics: bgp_rib_routes, bgp_rib_prefixes, bgp_rib_attribute_sets and
bgp_rib_journal_entries replace bgp_adj_rib_in_routes.
- A line is no longer logged at INFO for every UPDATE sent or received; a
full table is a million of them. They are at DEBUG.
- A table with no room for another peer ends the session with Cease / Out
of Resources.
Four peers sending a synthetic full table each over TCP, 4.9 million
routes: 0.80 s, against 4.76 s before. One peer: 0.45 s against 0.69 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A prefix was a Vec of its bytes: a heap allocation for every prefix parsed,
stored or copied, and a variable-length key to hash. It is now twenty
bytes, Copy, with private fields and one representation; `bytes()` gives
the wire form.
On the full-table benchmark this alone takes decoding from 9.3 to 10.7
million routes a second and inserting into the Adj-RIB-In from 2.9 to 3.6.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The RIB handles swallowed every failure of their actor: an UPDATE applied
to a stopped Adj-RIB-In vanished, and a snapshot of it came back empty.
Sessions stayed up, peers' routes went nowhere, and the status page showed
a healthy router with no routes. An actor stops only by panicking, so this
is the state the server would be in after a bug.
- Every operation on AdjRibIn and LocalRib returns Result<_, RibStopped>.
LocalRibError is replaced by it.
- The server waits on both actors, the advertiser and the dialers. If one
stops it sends every peer Cease / Administrative Shutdown and returns an
error, so the process exits and can be restarted. Before, the tasks'
handles were held and never looked at.
- A session that cannot store an UPDATE ends with the same Cease.
- The HTTP API answers 503 with the reason for anything that needs a table
it cannot read, including /metricz: a scrape that fails is noticed, last
known sizes are not.
Also:
- The routes of a session are now removed even if its task is torn down
without running its cleanup (a panic), instead of staying in the table
as routes of a peer that is gone.
- Table sizes for the status page and metrics are asked for as a count
instead of copying the whole table to take its length.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Any malformed UPDATE used to reset the session, withdrawing every route
learned over it. One bad attribute, typically originated ASes away and
passed on by a peer that did not look at it, could take down a session
carrying a full table.
UpdateMessage::decode now applies RFC 7606 and returns the UPDATE as it is
to be acted on, plus a report of what was wrong:
- Treat-as-withdraw: a malformed or misflagged attribute, a missing
mandatory one, or an attribute that overruns the attribute list. The
routes the UPDATE announced (in the body and in MP_REACH_NLRI) are
withdrawn.
- Attribute discard: malformed ATOMIC_AGGREGATE, AGGREGATOR, AS4_PATH and
AS4_AGGREGATOR; an attribute sent twice (the first is kept); LOCAL_PREF,
ORIGINATOR_ID and CLUSTER_LIST from an external peer; AS4_PATH between
four-octet speakers.
- Session reset remains where the prefixes are in doubt: the lengths that
frame the message, an unreadable prefix, a malformed or repeated
MP_REACH_NLRI or MP_UNREACH_NLRI, an unrecognised well-known attribute.
Each now sends its own subcode instead of always Malformed Attribute
List.
The session logs each handled error and counts it in
bgp_update_errors_total{handling}. The parser context says whether the
peer is internal (replacing a field nothing read).
Stricter than before, because the checks did not exist:
- An announcement must carry ORIGIN and AS_PATH, NEXT_HOP for routes in the
body, and LOCAL_PREF from an internal peer.
- An AS_PATH segment must be an AS_SET or AS_SEQUENCE and not empty.
- An attribute value must be used up entirely by its parser; a NEXT_HOP of
six octets, say, was accepted before.
Bugs found on the way: we sent ATOMIC_AGGREGATE flagged optional and
AGGREGATOR flagged non-transitive. The flags now come from one table used
for both writing and checking.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AFI and SAFI were closed enums, so a family without a variant could not be
represented at all, and the set of negotiated families was a struct with
one bool per family. Supporting another family meant touching both.
- `Afi`, `Safi` and `AfiSafi` in bgp-primitives are newtypes over the wire
numbers, with constants and names for the common families. Capabilities
and MP_REACH/MP_UNREACH use them, so any family a peer names is read as
what it is. The special case that turned a capability for an unlisted
family into an unknown capability is gone.
- `FamilySet` is a set of `AfiSafi`. What we can exchange routes for is the
explicit `family::SUPPORTED` list; only those are offered in our OPEN.
- `AddressFamilyId` stays as the IP version of a prefix.
- NLRI is still a list of IP prefixes. MP_REACH/MP_UNREACH for a family
whose NLRI is something else (labelled, VPN, EVPN, ...) is refused
rather than misread as prefixes, which is what happened before for the
SAFIs the old enum listed.
Bug found on the way: the reserved octet in MP_REACH_NLRI was skipped only
for IPv6, so an IPv4 MP_REACH_NLRI was parsed with an extra 0.0.0.0/0
route that the peer never sent.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A session was one function that walked OpenSent, OpenConfirm and
Established with the socket, the timers, collision detection, the registry
and the RIB all in the same loop. There was no state or event type, and no
way to test a transition without a TCP connection.
It is now two pieces:
- fsm.rs is the state machine of RFC 4271 section 8 for a connection. It
takes an Event and returns the Actions to carry out. It owns no socket,
timer or task. What the OPENs negotiated lives in the states that have
it, so there is nothing to unwrap. Thirteen tests cover the handshake,
hold-time negotiation, refused OPENs, messages out of turn, timer expiry,
collisions and shutdown by feeding it events.
- connection.rs is the driver: it turns the socket, the timers and the
outbound queue into events and carries the actions out.
Behaviour changes:
- A message that arrived in pieces was lost if a timer fired between the
pieces: read_exact was used inside select!, which discards what it has
read when another branch wins. Messages are now framed from a buffer
that survives cancellation.
- A NOTIFICATION from the peer during the handshake is no longer answered
with an FSM error; RFC 4271 says to close without replying.
- A session the peer ended is reported as 'peer sent NOTIFICATION: ...'
with its error, where the dialer used to record 'session closed by peer'.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The library crates returned eyre reports, so a failure was a string and the
session code could not tell a truncated UPDATE from a bad header. Whatever
went wrong, the peer got an FSM error or nothing.
Each crate now has its own error types:
- bgp-primitives: PrefixError, UnknownAfi and UnknownSafi, and EncodeError
for every to_wire.
- bgp-packet: Message::decode returns a DecodeError naming the NOTIFICATION
code, subcode and data for the failure (header errors per section 6.1,
OPEN per 6.2, UPDATE per 6.3). InvalidValue and CommunityError replace
the remaining strings. from_wire is now a thin wrapper around decode.
- bgp-server: SessionError, ServerError and ConfigError. Every abnormal end
of a session goes through one function that sends the NOTIFICATION the
error maps to. The message length is checked against the 4096-octet
maximum before the body is read.
- bgp-web: serve returns io::Error.
eyre remains in the binary and in tests.
Bugs found and fixed on the way:
- An OPEN with no optional parameters, which is valid, failed to parse.
- An unsupported BGP version was reported as an unknown optional parameter.
It now gets Unsupported Version Number with the version we speak as data.
- A SAFI above 255 was truncated to its low octet and could match a known
one.
- Prefix text with a length beyond the address size (10.0.0.0/99) was
accepted.
- Message::to_wire truncated the body length to 16 bits before checking it.
- A peer hold time above 65535 seconds in the configuration wrapped.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Default settings. No change in behaviour; the tree was last formatted
before the lecture work and had drifted to a mix of line widths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Implement passive-side BGP FSM (OpenSent → OpenConfirm → Established)
with hold timer, keepalive timer, and NOTIFICATION on expiry
- Add bgp-server crate: config validation, TCP listener accept loop
(BgpServerBuilder → BgpServer), connection dispatch
- Add bgp-rib crate: Adj-RIB-In actor owning route state with event
broadcast for streaming consumers and snapshot for introspection
- Wire Adj-RIB-In into FSM: announce/withdraw on UPDATE for both legacy
IPv4 NLRI fields and MP_REACH/MP_UNREACH path attributes
- Add bgp-integration-tests crate with 15 end-to-end tests covering
session establishment, timers, UPDATE handling, and RIB introspection
- Fix require_empty! macro bug in open.rs (checked wrong buffer)
- Fix FourByteAsn capability parser returning consumed buffer as remainder
- Fix tracing subscriber not being installed in TracingArgs::install
- Expand docs/RFCs.md with full implementation status across 15 RFCs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
