A review of the session driver, the server loop, the dialer, the
advertiser and the HTTP API, with the fixes.
Bugs:
- A peer that stopped reading held its session, and our routes, for as
long as TCP kept retrying: a blocked write was awaited with no timer
running. Writes now give up after the Send Hold Time of RFC 9687.
- Routes we originate were sent to a peer in our own AS with our AS
number on the path and no LOCAL_PREF, which such a peer refuses. They
now go with an empty path and a LOCAL_PREF.
- Collision detection knew a peer by its BGP Identifier alone. A
neighbour with one session over IPv4 and one over IPv6 had the second
closed as a collision, and any host that could connect could displace
a peer's connection by claiming its identifier. A peer is now its
address and identifier together.
- The dialer reset its backoff whenever TCP connected, so a peer that
accepts the connection and refuses the session was dialled every few
seconds for ever. It resets only after a session was established.
- A failure on our side (routes that cannot be stored) dropped the
connection without a NOTIFICATION. It goes through the state machine
and the peer is sent Cease. A queued message that cannot be encoded
is logged and skipped instead of ending the session.
- The accept loop retried at once after an error, spinning when the
process was out of file descriptors.
- Attribute sets no route used any more were dropped only when a peer
went away. A task now does it every minute.
- The advertiser subscribed to session events inside its task, after the
dialers were started, and could miss the first session.
- Shutdown waited without limit on a session whose queue was full.
- /api/routes/received took any limit and built the reply on a runtime
thread. The limit is capped and the work is done off those threads.
- The default log level was TRACE, which logs every UPDATE.
- IPv6 listeners are IPv6-only: listening on 0.0.0.0 and [::] fails on
systems where the second would take IPv4 too, and IPv4 peers arriving
under mapped addresses matched no configured peer.
- A connection closed after a NOTIFICATION is kept for a moment so that
the peer can read it.
- The configuration accepted two peers with one name or one address,
and a router id of 0.0.0.0.
- An HTTP address that cannot be bound now stops the daemon at startup.
Simplifications:
- SessionConfig mixed what a server's sessions share with what is
configured for one peer, and was cloned and patched field by field in
two places. It is ServerContext, shared, and PeerSettings, made from a
PeerConfig in one place.
- The attributes of an UPDATE as they were on the wire travelled from
the reader to the table through a field of the session, around the
state machine. They go through it, with the message.
- run_session returns how the session ended and whether it had been
established; Ok no longer reads as "closed by peer".
- Session is not generic over its reader.
- BgpServerBuilder had nothing to build: BgpServer::bind takes the
configuration.
- One shutdown path, close_sessions, for the signal handler and for a
failed server.
- The advertiser's three copies of "send every route" are one.
- PeerId moved from the rib crate, which did not use it, to the server.
- Unused API removed or made private to the crate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A review of the wire codec and the routing table, with the fixes.
Bugs in the codec:
- A NOTIFICATION with a code or subcode we have no name for (subcode 0,
"unspecific", is common) failed to decode and was answered as a bad
message. Every pair now reads, as BgpError::Other.
- Optional parameters in the extended form of RFC 9072 were read and
written with one-octet parameter lengths; they are two.
- An ORF capability listing more than one address family made the whole
OPEN malformed.
- Bytes after an OPEN's optional parameters were accepted.
- Extended communities were split into fields by rules that were wrong
for the common types. They are kept as their eight octets.
- Several length fields were computed in u16 and could wrap on encode.
Simplifications in the codec:
- Lengths are filled in after the value is written, for messages,
attributes, capabilities and the UPDATE's two fields. The parallel
wire_len functions, which had to be kept in step by hand, are gone.
Message::to_wire no longer encodes into a second buffer.
- Type codes for messages, attributes and capabilities are each defined
once. The strum discriminant enums and their hand-written conversions
are removed.
- One function splits an attribute off a buffer; the UPDATE decoder, the
plain attribute parser and the table's key reader all use it.
- The UPDATE decoder's 185-line loop is an AttributeReader with one
method that decides what becomes of an attribute. Duplicate detection
is a bitset, not a HashSet allocated per UPDATE.
- ParserContext::four_octet_asn is a bool. The "context not set" error
path, which treated a programming error as a malformed route, is gone.
- Codec functions take a context only if they read it, and return a
Result only if they can fail.
- MP_REACH_NLRI and MP_UNREACH_NLRI hold an AfiSafi.
- AddressFamilyId is IpVersion, which is what it was.
- Deserialize is no longer derived on codec types: nothing used it, and
it bypassed IpPrefix's invariants. IpPrefix::to_wire, which was
broken and unused, is removed.
Bugs in the table:
- The index of a row's best route was sixteen bits. A peer using
ADD-PATH could send more routes for a prefix than that counts, and
the index wrapped. It is 32 bits, at no cost in row size.
- Shards and attribute-store stripes were picked from the top bits of
the hash, which are the bits hashbrown keeps as the tag it compares
first. With 64 shards each shard's table had 2 of 128 tag values.
They are now picked from bits the table does not use.
- remove_peer held the peer list's lock for the whole scan of the
table, so no session could establish while a peer was removed.
- When more prefixes went at once than were left, the forced trim
dropped a fixed few entries, reset every reader, and re-ran the tidy
on every later write. It drops the older half.
- Readers were woken by every change, including a repeated announcement
and changes their journal does not record.
- Reader::poll with a budget of zero never made progress.
Simplifications in the table:
- The two journals of a shard are one Journals value written through
three calls; the five places that kept them in step by hand are gone.
Whether an entry is still live is decided in one place.
- The shortcuts of route selection moved to select.rs, next to the
invariant they depend on, which is now written down.
- announce and withdraw take routes with or without path identifiers;
the _paths variants are removed.
- The selection key is read off the wire form only; the second
implementation over parsed attributes is removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The server now keeps what it learns in table::RibTable. The Adj-RIB-In
actor is gone, and with it the queue every UPDATE from every session went
through.
- A session takes a number in the table when it is established and writes
its UPDATEs into the table itself. Its routes are removed when it ends,
on a blocking thread since that looks at every prefix; a session torn
down without cleaning up still has them removed.
- For every prefix the best route is chosen (RFC 4271 §9.1.2.2) as routes
come and go. Nothing acts on the choice yet beyond showing it.
- The UPDATE decoder hands over the bytes of the attributes it kept, so the
table finds the attribute set from them without writing the attributes
out again. On a session with two-octet AS numbers the bytes are not in
the table's form, and the parsed attributes are used.
- /api/routes/received marks the best route of each prefix and returns at
most `limit` routes (default 1000): the table can hold hundreds of
millions. /api/status has the totals, and the status page says how many
it is showing. The time a route was received is no longer kept or shown.
- The event stream reports changes of best route (best_changed,
best_withdrawn) in place of every route received. Each client follows
the table's journal with a reader of its own, from the moment it
connects: table::RibTable::reader_from_now.
- Metrics: bgp_rib_routes, bgp_rib_prefixes, bgp_rib_attribute_sets and
bgp_rib_journal_entries replace bgp_adj_rib_in_routes.
- A line is no longer logged at INFO for every UPDATE sent or received; a
full table is a million of them. They are at DEBUG.
- A table with no room for another peer ends the session with Cease / Out
of Resources.
Four peers sending a synthetic full table each over TCP, 4.9 million
routes: 0.80 s, against 4.76 s before. One peer: 0.45 s against 0.69 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The RIB handles swallowed every failure of their actor: an UPDATE applied
to a stopped Adj-RIB-In vanished, and a snapshot of it came back empty.
Sessions stayed up, peers' routes went nowhere, and the status page showed
a healthy router with no routes. An actor stops only by panicking, so this
is the state the server would be in after a bug.
- Every operation on AdjRibIn and LocalRib returns Result<_, RibStopped>.
LocalRibError is replaced by it.
- The server waits on both actors, the advertiser and the dialers. If one
stops it sends every peer Cease / Administrative Shutdown and returns an
error, so the process exits and can be restarted. Before, the tasks'
handles were held and never looked at.
- A session that cannot store an UPDATE ends with the same Cease.
- The HTTP API answers 503 with the reason for anything that needs a table
it cannot read, including /metricz: a scrape that fails is noticed, last
known sizes are not.
Also:
- The routes of a session are now removed even if its task is torn down
without running its cleanup (a panic), instead of staying in the table
as routes of a peer that is gone.
- Table sizes for the status page and metrics are asked for as a count
instead of copying the whole table to take its length.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AFI and SAFI were closed enums, so a family without a variant could not be
represented at all, and the set of negotiated families was a struct with
one bool per family. Supporting another family meant touching both.
- `Afi`, `Safi` and `AfiSafi` in bgp-primitives are newtypes over the wire
numbers, with constants and names for the common families. Capabilities
and MP_REACH/MP_UNREACH use them, so any family a peer names is read as
what it is. The special case that turned a capability for an unlisted
family into an unknown capability is gone.
- `FamilySet` is a set of `AfiSafi`. What we can exchange routes for is the
explicit `family::SUPPORTED` list; only those are offered in our OPEN.
- `AddressFamilyId` stays as the IP version of a prefix.
- NLRI is still a list of IP prefixes. MP_REACH/MP_UNREACH for a family
whose NLRI is something else (labelled, VPN, EVPN, ...) is refused
rather than misread as prefixes, which is what happened before for the
SAFIs the old enum listed.
Bug found on the way: the reserved octet in MP_REACH_NLRI was skipped only
for IPv6, so an IPv4 MP_REACH_NLRI was parsed with an extra 0.0.0.0/0
route that the peer never sent.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The OPEN used to offer IPv4 and IPv6 unicast unconditionally, and what the
peer offered was ignored apart from its four-octet AS number.
Address families (RFC 4760):
- A peer's `address_families` in the configuration says which to offer it;
all of them by default.
- A session carries the families both OPENs offered. A peer that sends no
Multiprotocol capability is a plain RFC 4271 speaker: IPv4 unicast.
- Routes of a family that was not negotiated are not sent, and are ignored
when received; the rest of the UPDATE is applied and the session stays up.
- A session with no family in common is refused with Unsupported
Capability, naming the capabilities we needed (RFC 5492).
- The negotiated families are shown on /statusz and in /api/peers.
Bugs found on the way:
- A capability naming an address family we have no constant for (EVPN,
VPNv4, ...) failed the whole OPEN, so such a peer could never establish.
It is now kept as an unknown capability and ignored.
- An UPDATE could not be encoded for a peer without four-octet AS support,
which ended the session on the first route. AS_PATH is now written two
octets wide with AS_TRANS where needed, and AS4_PATH carries the real
numbers (RFC 6793).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Default settings. No change in behaviour; the tree was last formatted
before the lecture work and had drifted to a mix of line widths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Routes were announced and withdrawn through an HTTP form, each tagged with a
free-text owner and checked against an allow-list and a per-owner limit.
That was the lab's interface, not a router's.
Routes now come from 'originate' entries in the configuration file and are
in the local RIB before any session starts. The local RIB holds one route
per prefix with no notion of owners or policy. local_route_policy is gone
from the configuration.
The HTTP API keeps its GET endpoints (status, config, peers, originated and
received routes, events, metrics). The announce, withdraw, wire-bytes and
well-known-community endpoints are removed, and the status page loses the
announce form, the community presets and the byte decoder. Community text
parsing moves from the web crate to bgp-packet so the configuration can use
it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Preparing to peer with a production BIRD router turned up two problems.
- packet: most Capability::from_wire arms returned the per-capability
buffer (always empty by then) instead of the bytes following the
capability. The list parser then saw an empty remainder and silently
dropped every later capability. Any OPEN listing extended message, role,
graceful restart, add-path, enhanced route refresh, LLGR, extended next
hop or an unknown capability before the 4-octet ASN capability lost the
ASN, so the peer appeared as AS_TRANS (23456) and a configured
remote_asn was refused with Bad Peer AS. BIRD and FRR both do this.
Single-capability round-trip tests could not catch it; new tests parse
every capability with another following it, and a full BIRD capability
list.
- server: advertise MultiProtocol IPv6 unicast alongside IPv4. Without it
a peer will not accept the IPv6 routes we originate via MP_REACH_NLRI.
- Interop test: byte-for-byte BIRD 2 OPEN is accepted with the right ASN,
and an IPv6 End-of-RIB neither drops the session nor creates a route.
- docs/lab/prod: BIRD config for an announce-only ferrix neighbour
(strict /48 import filter, community scrubbing, unreachable next hop,
import limit, export none, passive) and the matching ferrix config.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
