A peer can now send several routes for one prefix, told apart by a path
identifier, and each is a route of its own in the table.
Wire:
- The ADD-PATH capability lists several address families. It was read as
exactly one, so the OPEN of a router offering it for IPv4 and IPv6 was
malformed and the session never came up.
- Prefixes in UPDATEs are `Nlri`: a prefix and a path identifier. The
identifier is on the wire, in the body and in MP_REACH/MP_UNREACH, for
the families the session uses ADD-PATH for in that direction. Sending
and receiving are settled separately, so a session has one codec context
for each.
Negotiation:
- `add_path` on a peer in the configuration: off (the default), receive,
send or both, offered for every family offered to the peer.
- In use for a family and a direction when one side offered to send and
the other to receive. Shown on /statusz and in /api/peers.
Table:
- A route is known by its peer and path identifier together. Identifiers
are stored only in rows that have a non-zero one, so routes of peers
that send one route each cost nothing more.
- Equal routes from one peer are chosen between by the lower identifier.
- /api/routes/received gives each route's path_id.
Journal:
- A second journal in each shard records every change to any route, for
followers that want all the routes of a prefix and not only the best:
RibTable::reader_of_all_routes and Event::Routes. This is what sending
several routes to a peer will read. It costs nothing while nobody
follows it.
Sending: prefixes go out with a path identifier where that was agreed, but
there is still one route for each prefix to send: only originated routes
are announced.
Tested with the table and journal model tests extended to path identifiers
and to followers of all routes, the codec property tests and fuzz targets
extended to ADD-PATH sessions, and end to end in both directions and for
the case where only one side offers it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The server now keeps what it learns in table::RibTable. The Adj-RIB-In
actor is gone, and with it the queue every UPDATE from every session went
through.
- A session takes a number in the table when it is established and writes
its UPDATEs into the table itself. Its routes are removed when it ends,
on a blocking thread since that looks at every prefix; a session torn
down without cleaning up still has them removed.
- For every prefix the best route is chosen (RFC 4271 §9.1.2.2) as routes
come and go. Nothing acts on the choice yet beyond showing it.
- The UPDATE decoder hands over the bytes of the attributes it kept, so the
table finds the attribute set from them without writing the attributes
out again. On a session with two-octet AS numbers the bytes are not in
the table's form, and the parsed attributes are used.
- /api/routes/received marks the best route of each prefix and returns at
most `limit` routes (default 1000): the table can hold hundreds of
millions. /api/status has the totals, and the status page says how many
it is showing. The time a route was received is no longer kept or shown.
- The event stream reports changes of best route (best_changed,
best_withdrawn) in place of every route received. Each client follows
the table's journal with a reader of its own, from the moment it
connects: table::RibTable::reader_from_now.
- Metrics: bgp_rib_routes, bgp_rib_prefixes, bgp_rib_attribute_sets and
bgp_rib_journal_entries replace bgp_adj_rib_in_routes.
- A line is no longer logged at INFO for every UPDATE sent or received; a
full table is a million of them. They are at DEBUG.
- A table with no room for another peer ends the session with Cease / Out
of Resources.
Four peers sending a synthetic full table each over TCP, 4.9 million
routes: 0.80 s, against 4.76 s before. One peer: 0.45 s against 0.69 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Whatever acts on the best routes (export to peers, the forwarding table,
the status page) reads their changes from a per-shard journal through a
table::Reader. Nothing uses it yet.
- The journal records that a prefix's best route changed, not what to. A
reader that comes to an entry is given the route as it is then. A prefix
that changes a thousand times between reads is one event, and the
journal never holds more live entries than the table has prefixes.
- Each row keeps the sequence number of its latest change; an entry with
another number is dead and is dropped when the journal is tidied.
- Entries go once every reader has passed them. A reader that does not
read cannot hold them back for ever: past a limit the oldest half is
dropped, and that reader is given the whole table again (Reset, every
best route, Synced). A new reader starts the same way.
- With no reader nothing is recorded.
- Reader::changed waits for there to be something to read.
Tested against a mirror kept from the events: after any interleaving of
announcements, withdrawals, session flaps, reads with small budgets and
readers coming and going, with journals small enough to be tidied and
overrun constantly, a reader that has caught up holds exactly the table's
best routes. Also with writers on four threads and a reader on a fifth.
The property test found one fault (a pass that resumed part-way claimed
to have caught up) and catches three more put in on purpose.
On the AMS-IX dump, 12 threads: loading takes 2.29 s with a follower
reading throughout against 2.17 s with none, the journal peaking at
170,000 entries for 1.39 million prefixes. With a follower that does not
read, the journal ends at 1.74 million entries, and the follower then
catches up in 0.08 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bgp_rib::table::RibTable holds every route from every peer and the best
one for each prefix. It is not used by the server yet; this adds it with
its tests and benchmarks.
- Organised by prefix. A row holds the prefix's routes from all peers in
one array; choosing again after a change reads that row and nothing else.
- A route is sixteen bytes: a pointer to its attribute set and one word
holding the peer and the first three steps of selection (LOCAL_PREF, AS
path length, ORIGIN) as a single number. That number alone settles most
changes; a tie goes on to MED, compared only within a neighbouring AS
and in a way that does not depend on arrival order, and the tie-breaks.
- Attribute sets are stored once, in wire form, with what selection needs
from them worked out when they are stored.
- Sharded by hash of the prefix, a lock per shard. A session applies its
own UPDATEs; there is no queue or owning thread in front of the table.
- Readers take one shard at a time for reading and look at rows in place.
Route selection is checked against a plain implementation with no
shortcuts by a property test over random announcements, withdrawals and
session flaps, for both the chosen route and the count of changes.
On a RIPE RIS dump from AMS-IX (148 peers, 39.3 million routes), against
the Adj-RIB-In actor, which does no selection:
actor table
memory 11.0 GiB 1.8 GiB (279 -> 48 bytes a route)
load, 1 thread 10.2 s 12.3 s
load, 12 threads (no gain) 2.1 s (19 million routes a second)
read every route 1.6 s, copying 8 GiB, table stopped
0.2 s, in place, table running
The benchmark crate counts memory only with --features count-memory: the
counters are shared between threads and distort the timings.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
crates/bench reads TABLE_DUMP_V2 dumps (RFC 6396) as published by RIPE RIS
and RouteViews, and `--bench real_table` runs the Adj-RIB-In on one. The
dumps are not checked in; crates/bench/data/README.md says how to fetch
one. Without a dump the benchmark says so and does nothing.
It also prints what in the table decides the cost of a RIB: routes per
prefix, distinct attribute sets, how many routes share a set, and how much
the sets are shared between peers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
