A table given out anew to a session, at its start or when the journal
had let changes go, was gathered as a verdict for each prefix, each
with the route dressed for the session, and sent once the table was
all there. Dressing a route parses its attributes and encodes them for
the session, so a table of five million routes took seconds before
the first UPDATE went, on one worker thread without a yield. And the
routes that share attributes went out in separate UPDATEs after all:
they were grouped by the dressed form, of which only so many are kept,
and the routes of a set are spread over the whole table, so by the
time the next was reached the form had been let go of and made anew.
The mesh benchmark's new peer got four million routes in 3.8 million
UPDATEs; on a RIPE RIS table, grouped whole, a full table is 160,000
UPDATEs and 21 MB, and grouped in slices of 4,096 routes it is 845,000
UPDATEs and 72 MB, so slices are not the answer either.
The table is now gathered by what dresses its routes: the attribute
set as the table has it, who the route is from, and its IP version.
Nothing is dressed while gathering, which is a map insert a route.
Once the table is all there, each set is dressed once, with the thread
lent to the other tasks every few thousand, and then the sets go out
one after the other, each as few UPDATEs as its routes fit in, with
the peer reading the first while the rest are sent. A prefix heard of
again while the table is read supersedes what was heard before: each
word on a prefix is numbered, and only the latest goes. Path
identifiers are given at gathering, in the order the table has a
prefix's routes, so that they are as before. What was announced before
and not heard of again is withdrawn after the table, as before. The
exporter never blocks the thread on a table: the steady state's batch
is small, and the gathered table is gone through with yields.
Testing: a new test gives a session a table of 300 routes sharing
three sets of attributes, spread over the shards, and checks they go
in three UPDATEs. The mesh benchmark has a new phase in which a peer
takes the first router's table once the mesh is complete, timing the
first UPDATE and the whole; its feeders now negotiate no hold timer,
since they say nothing after their tables, and `BENCH_LOG` prints the
routers' logs. A new example, packing, counts the UPDATEs and octets
a RIS table takes grouped whole and in slices. The server tests, the
integration tests, the kernel tests and every lab phase pass. In the
Linux VM, three rounds alternating with the previous commit, the new
peer's first UPDATE comes 2.9 s after its session is up instead of
5.3 s, and it has all four million routes in 4.0 s instead of 8.0 s,
in 667,000 UPDATEs instead of 3.76 million; the whole run takes
22.4 s of CPU time instead of 26.2 s. The routers' exchange of their
tables, which does not go through this path once their sessions are
up, is unchanged within its noise.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An exporter following the table took up to 65,536 changes at a time,
dressed them all, and only then sent them; and before each batch it
slept the 50 ms that lets a run of changes be gathered, whether or not
the table was already far ahead of it. The session's queue holds about
a thousand UPDATEs, so once the batch went out the exporter waited on
the peer reading it, and while the next batch was dressed the peer had
nothing to read. A profile of two routers exchanging their tables,
sampled in 25 ms windows, showed the two stages taking turns: the
exporters at two cores while the sessions were at none, then the other
way round, with a gap of 50 to 75 ms at the start of each cycle for the
sleep. Each task ran at about half a core.
A batch is now 4,096 changes, so that what one batch sends is being
read by the peer while the next is being dressed, and the gathering
sleep is skipped while the reader is behind the table: a run is in hand
already. A reader says whether its last poll caught up.
Testing: the server tests, the integration tests, the kernel tests and
every lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the larger table
(4.9 million routes) is across in 5.6 s instead of 6.7 s and the
smaller (2.5 million) in 4.4 s instead of 6.4 s; the whole exchange
takes 7.3 s of wall-clock time instead of 8.6 s, and 18.9 s of CPU time
instead of 20.5 s.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The exporter built each UPDATE as a message: the attributes of the
route's set, parsed from the table, were cloned, had the next hop
added, were sorted, and were handed to the session, which encoded them
to the wire with the rest of the message. For a table whose attribute
sets each cover a few routes that is an encoding of the same octets for
nearly every route, and a copy of the message's bytes on the way to the
writer besides. A profile of a production router taking the other
router's table with ADD-PATH had the exporters at 55% of the CPU time,
their dressing of routes at 23%, and memcpy at 12%.
A route's attributes are now encoded once for a session, when it is
first dressed for it, and kept with where in the octets a NEXT_HOP or
an MP_REACH_NLRI belongs so that the attributes stay in order of type
code. An UPDATE is then its header, the encoded octets with the next
hop or the MP_REACH_NLRI spliced in, and the prefixes: written straight
into a buffer that goes to the session as it is, and from the session
to the writer without a copy. The session's queue takes either a
message or an encoded UPDATE; an encoded one for a session that is no
longer established is dropped, as the exporter made it for the session
as it was. What is kept for a route, now a few hundred octets instead
of a kilobyte, lets twice as many be kept. The exporter's maps hash
with foldhash instead of SipHash, and the parser context for an UPDATE
is made once a flush instead of once an UPDATE.
Testing: a new test encodes a thousand routes with each of the forms
an UPDATE takes (IPv4 with the next hop in its own attribute, IPv6 in
MP_REACH_NLRI, with and without path identifiers, to two-octet and
four-octet AS speakers, over several messages) both ways and checks the
octets are the same byte for byte; the message-building form is kept
for the test. The exporter tests decode what the session is handed.
The server tests, the integration tests, the kernel tests and every
lab phase pass. On the mesh benchmark in the Linux VM, three
rounds each alternating with the previous commit, the two routers'
exchange of 7.4 million routes takes 20.6 s of CPU time instead of
25.4 s (-19%) over 9.0 s of wall-clock time instead of 11.1 s, and the
larger of the two tables is across in 7.1 s instead of 9.3 s. The
smaller table's time varies too much from run to run (5-8 s) for a
difference to be read from it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A prefix's row of paths is given room for as many paths as there are
peers, and from there grows by a quarter at a time, one path at a time
while it is short. A peer sending every path it has (ADD-PATH) gives a
prefix several paths from one peer, so a row goes past the peers' count
from the first few routes and was then reallocated for each path.
A row shorter than the first allocation's room doubles instead, and
grows by a quarter only from there. Measured on the mesh benchmark, the
reallocations in the table's insert path fall by half; the memory held
is as before, as the rows that reach the quarter-growth are the ones
that were already allocated for it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The full-table benchmark measures one router taking in tables from its
peers. What two routers of one AS do with each other, each sending the
other every path it has (ADD-PATH both ways with all_paths, as a mesh
of routers that each hold the network's whole view is configured), was
measured nowhere: the exporter following the table's journal while the
sessions fill the table, at the size of the Internet. In production
that exchange took 70 seconds for a table that arrives from the
Internet in 20.
A new benchmark, mesh, starts two servers in one AS with distinct
router IDs, peers them over IPv6 loopback with ADD-PATH both ways and
all_paths, and feeds the first from four synthetic peers and the second
from two over IPv4 loopback (BENCH_PEERS, BENCH_PEERS_B,
BENCH_SCALE_DOWN). It prints how long each router takes to hold its
own peers' routes and then the other's, with the rate of the latter.
BENCH_DEADLINE prints the sessions' state if a router is not there in
time.
The benchmarks now allocate with jemalloc, as the daemon does, except
when memory is being counted: the C library's allocator put its own
locks in the profiles. Two routers on one host have each other's
addresses as next hops, loopback addresses, which the import check
refuses as it refuses a next hop that is this machine; the permissive
import policy, which only tests use and no configuration can ask for,
now takes them.
Testing: the next-hop test covers both policies; the benchmark runs on
macOS and in the Linux VM under perf.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A session read from and wrote to its socket in one loop: while it was
writing, it was not reading. Writing waits when the peer's receive
window is closed, and the peer's window closes when the peer is not
reading, which it is not while it is writing to us and waiting on our
window in turn. Two routers sending each other every path at once, as
they do with ADD-PATH both ways after a restart, each stopped reading
within seconds and waited on the other until the send hold timer ended
both sessions eight minutes later. The status page's TCP row showed it:
both windows zero, the send queue full, the receive queue unread. The
batching of writes in the previous change made a session spend far
longer writing and so made this certain; it was possible before.
Now each session has a writer task. The session encodes its messages
into batches and hands them to the task through a bounded queue without
ever waiting: a batch the queue has no room for is kept, and while any
is kept the session takes no more messages from the rest of the server
(which then waits, as before, in its own task) but goes on reading the
peer. The writer task's failure, or a write that makes no progress for
the send hold time, comes back to the session as a local failure and
ends it. When the session ends, what is still to go out, a NOTIFICATION
among it, is handed over and the writer shuts our side once it has
written it.
Testing: a new integration test has one peer's table in the server and
a second peer send a table of its own while reading nothing of what the
server sends it, and checks that the server takes the second table all
the same; against the previous commit the test hangs until its timeout.
The send hold timer test now reads the writer's failure. The session
tests, the integration tests, the kernel tests and every lab phase
pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The last change built jemalloc with its background thread on, so that
freed pages go back to the kernel on time. jemalloc has that thread on
Linux only: on macOS every start of the daemon, and every test that
starts one, printed "option background_thread currently supports
pthread only".
The feature now comes from a Linux-only dependency entry of the daemon
crate, so the macOS build asks for nothing jemalloc cannot do. The
Linux build is as before: measured in the VM with four peers and a
collector, resident memory falls after the load without any setting
in the environment.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A session wrote every message to its socket as it came: one write and,
with Nagle's algorithm, mostly one segment for each UPDATE, a hundred or
two octets. On the IPv6 cross-connect between the two routers, whose
round trip is 50 microseconds, a million routes took 42 seconds to go
out: the status page's new TCP row showed the connection busy for that
long with neither window nor buffer limiting it, a congestion window
stuck at ten segments, and a smoothed round-trip time of 18 ms. That is
the receiver's delayed acknowledgements, 40 ms at a time, on a stream
of small segments, holding the sender to a few segments a round trip.
Now a session gathers what is waiting for it, up to the outbound queue's
length, in one wake, encodes the messages into one buffer, and writes
the buffer when it reaches 64 KiB or when the actions of the wake are
done; the socket has TCP_NODELAY, so a batch's last, partial segment is
not held back for the acknowledgement of the one before. The
NOTIFICATION that ends a session, including one sent after a panic, is
flushed on its own.
Measured in the Linux VM with four peers' tables going to a collector
over a veth pair, 151 MiB: 121,850 segments instead of 313,773, the
connection busy 0.8 s instead of 2.0 s, and the congestion window grown
to 47 segments instead of 13. Over loopback the transit benchmark is
unchanged at 2.5 s for the second neighbor to have the table: there the
receiver's parsing sets the pace, not the segments.
Testing: the session tests, including the one that sends a NOTIFICATION
after a panic; the integration tests; the kernel tests and every lab
phase.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A router feeding a collector every path (all_paths) was 4.8 GB resident
with a table that should take a third of that. A heap profile of the
daemon in the Linux VM, with four full tables and such a collector,
showed where it went: the exporter kept, for every route it had
advertised, the route's outgoing form, the parsed attribute list it was
dressed with, through an `Arc<Dressed>` in the per-route record. One
such form lived for every distinct attribute set advertised, about a
kilobyte each, for the life of the session, on top of the exporter's
cache of 262,144 of them. The record needs only to tell whether a route
changed, and the table's own attribute set, which is in memory anyway,
tells it that: the outgoing form is a function of the set, the peer the
route came from and the policy.
This change:
* `Sent` holds an `Arc<AttrSet>` instead of an `Arc<Dressed>`, and a
route is sent again when its set differs. `Verdict::All` carries both
while a flush is on.
* `DRESSED_KEPT` goes from 262,144 to 65,536: a kilobyte a set, so 64 MB
rather than 256 MB a session at most; what a flush is sending holds
its own.
* The status page's header shows the daemon's memory as jemalloc
accounts for it (allocated and resident), `/api/status` has the six
figures (`memory`: allocated, active, metadata, resident, mapped,
retained), and `bgp_memory_bytes{kind}` exports them. jemalloc is built
with statistics for this, and `bgp-web` reads them through
`tikv-jemalloc-ctl`.
* `scripts/profile/full-table.sh` gains `COLLECTOR=1`, a collector
session over IPv6 that takes both families with ADD-PATH, and `HEAP=1`,
which builds the daemon with jemalloc's heap profiler (the new
`heap-profiling` feature of the daemon crate, not for production) and
turns the profile into a flame graph with jeprof; and it reports
memory with the table in and again 30 s later. The feeder
(`crates/bench/examples/feed.rs`) plays the collector with
`FEED_COLLECTOR=1`.
Measured in the VM, four peers' full tables and a collector sent every
path (4.9 million routes advertised): 1,723 MiB resident before, 1,286
MiB after, with jemalloc's allocated figure from 1,500 MiB to 946 MiB.
Without the collector the same tables take 630 MiB, so the collector's
bookkeeping is now about 50 bytes a route advertised. Of the resident
figure, some 400 MiB was pages jemalloc had freed and not yet returned:
it returns them on allocator activity only, and a router that has its
tables in allocates little. jemalloc is now built with its background
thread on (the `background_threads` feature), which returns them on
time: 1,020 MiB resident 30 s after the load, against 1,286 MiB without
it.
Documentation: `memory` in the HTTP API reference, `bgp_memory_bytes`
in the metrics reference and the monitoring guide, the script's new
settings in the testing guide.
Testing: the ADD-PATH integration tests pass, including the collector
ones; the heap profile and the figures above are from
`PEERS=4 COLLECTOR=1 HEAP=1 scripts/profile/full-table.sh` in the VM.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kernel's view of a session's TCP connection, which the status page
shows on request since the last change, could not be looked at after
the fact: a feed that crawled overnight left no record of whether the
peer's window or our own side held it up.
This change samples it at every scrape of `/metricz`: for each
established session, one `NETLINK_SOCK_DIAG` round trip, and twenty
gauges labelled by peer, `bgp_peer_tcp_*`: smoothed, least and variance
of the round-trip time; the congestion window and segment size; the
peer's receive window and ours; the send queue, the part of it not yet
sent, the receive queue and the segments in flight; retransmitted and
lost segments; the busy time and the time held up by the peer's window
and by our send buffer, cumulative, so that their rates give the share
of time a session is held up; octets sent and received; and the
delivery and pacing rates. A session the kernel has no connection for
loses its series, as does one that ends.
`SessionHandle::metric_label` is the one place a peer's metric label is
made (its name, or its address), used by the advertiser's gauge and here.
Documentation: the metrics reference lists them; the monitoring guide
gives the query for the share of time a session is held up by the
neighbor's receive window, which is the question the overnight feed
raised.
Testing: a daemon in the Linux VM with one session shows the series and
drops them when the session ends; the lab's basics phase checks the
scrape for the FRR session's round-trip time; the workspace tests pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When a session advertised a full table to a route collector at two
thousand routes a second, nothing on the status page said whether the
daemon or the peer set that pace. The answer is in the kernel: the
peer's receive window, the queues, and how long the connection was held
up by the window or by the send buffer, which `ss -ti` shows.
This change asks the kernel the same way `ss` does, over the socket
diagnostics netlink family (`NETLINK_SOCK_DIAG`, `SOCK_DIAG_BY_FAMILY`
with the connection's addresses and ports), and reads `tcp_info` and
the congestion control name from the answer. No external program is
run and no unsafe code is needed: the request and the answer are byte
layouts of the kernel's ABI, read field by field, with fields an older
kernel does not send left at zero. A socket on a link-local address is
found by its interface, which the request names.
* `bgp_fib::sockdiag`: the request, the parser, `TcpConnection`, and
`tcp_connection(local, peer)`; `Ok(None)` where there is no such
connection or the system is not Linux.
* `GET /api/tcp?peer=ADDRESS` in the HTTP API: the connection of the
session with that peer, with times in milliseconds, the two windows
named by what they allow, and the held-up times also as shares of the
busy time.
* The status page: a "tcp" link on each established session, which
expands a row below it with the figures, re-asked on every refresh
while open. The group rows' span is corrected for the "sent" column.
Also on the status page: the attributes of a received route that have
no column of their own were badges with their value in a tooltip, which
is hard to read. Clicking the attrs cell now expands a row under the
route with every attribute by its name (ORIGIN, AS_PATH, NEXT_HOP, MED,
LOCAL_PREF, COMMUNITIES, LARGE_COMMUNITIES, AGGREGATOR and the rest),
the path identifier, the selection rank and the RPKI and ASPA states;
the badges stay. Both expansions use the terms of `ss` and the RFCs
rather than sentences.
Documentation: the endpoint in the HTTP API reference, and the link and
the expansion in the monitoring guide.
Testing: unit tests of the request and of the parser on a reply built
field by field, and of a "no such socket" and a refused answer; a
Linux-only test that asks the kernel about a loopback connection of the
test's own and about one that does not exist; the lab's basics phase
checks the endpoint for the FRR session; the page was checked by hand
against the daemon in the Linux VM.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A collector at an exchange is fed over one IPv6 session that carries
both families, with ADD-PATH for both and IPv4 routes with IPv6 next
hops (RFC 8950). The integration test for that has the collector on
IPv6 loopback with those capabilities and two feeders sending 20,000
IPv4 and 1,000 IPv6 prefixes each, and checks that every route of both
families arrives with its identifier. The test helper now reads the
prefixes that come and go in the multiprotocol attributes too.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The number of routes advertised to a peer was written once a flush was
over, after every UPDATE of it had been handed to the session. The first
flush to a route collector is the whole table, two million routes or
more to a peer that reads at the pace of the Internet, so the status
page showed the peer holding one route for as long as that took, which
read as nothing going out.
Now the count is written after each batch of a flush, so that the page
and `bgp_peer_advertised_routes` show the table going out.
Also, an exporter that had the whole table and is given it again,
because the table changed faster than the exporter read it and the
journal let the oldest changes go, now logs that it is going through the
table again, at info. Until now that was invisible, and a collector
session during a full-table load from several peers is where it can
happen: the exporter is held up sending while the shards' journals,
cut back to twice their rows, run past its place.
Testing: a new integration test has a collector connected before two
feeders send 40,000 prefixes each, and checks it is told all 80,000
routes with their identifiers, which takes the exporter's initial walk
of the table through many batches; the advertiser and ADD-PATH tests
pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Before this change, a session on which ADD-PATH (RFC 7911) was
negotiated in the sending direction was sent path identifiers, but only
the best route for each prefix. A route collector such as bgp.tools
wants every route a router has accepted, from every neighbor, and from
one session: without this the collector saw one path per prefix, nearly
always the same neighbor's.
This change adds `all_paths` to the export policy. On a session with it,
and ADD-PATH send negotiated for a family, every accepted route for a
prefix of that family is advertised, each with a path identifier of its
own; for a family without ADD-PATH send the best route goes as before.
`all_paths` needs `learned_routes` (a configuration error otherwise) and
the peer's `add_path` to offer sending. The export policy, the role
rules, the well-known communities and the export limits apply to every
route alike, and the export limits count every route.
Implementation, in the advertiser:
* An exporter with `all_paths` follows the table's journal of all routes
(`RibTable::reader_of_all_routes`, until now unused) instead of the
best-route journal, and is given all the routes of a prefix whenever
one of them changes (`News::Routes`).
* What the peer holds is `Announced`: the set of prefixes for a
best-route session, as before; or for an all-paths session, for each
prefix the routes announced, keyed by the peer they came from and that
peer's own identifier, with the identifier they went out with and the
form they went out in. A route keeps its identifier for as long as it
is announced, goes out again only if it changed, and is withdrawn by
its identifier; a new route gets the lowest identifier not in use for
the prefix, from 1.
* Verdicts carry either kind (`Verdict::Best`, `Verdict::All`), and a
reset of the table puts what was announced in doubt until heard of
again, as before, with the same identifiers kept across the reset.
* The record for an all-paths session is about 50 bytes a route, some
tens of megabytes for a full table from several neighbors; the
best-route bookkeeping is unchanged in size.
Also, so that a collector session can be watched: the number of routes
advertised to each peer and not withdrawn since, every route counted
where several are sent for a prefix. The exporter keeps it in the
session's counters after every change; it is `advertised_routes` in
`/api/peers`, the "sent" column of the status page's neighbor table,
and the gauge `bgp_peer_advertised_routes{peer}`.
Documentation: a "Feeding a Route Collector" section in the guide to
passing routes on, with the configuration for a collector; `all_paths`
in the configuration reference; the ADD-PATH sections of the neighbor
guide and the standards reference; the limitation that was removed;
the new field, column and metric in the HTTP API, monitoring and
metrics references.
Testing: an integration test with two feeders and a collector that
negotiated ADD-PATH receive, which checks the two routes with their
identifiers, the withdrawal of one by its identifier, a changed route
keeping its identifier and a new route taking the free one; a
configuration test for `all_paths` without `learned_routes`; a new
`collector` phase of the interoperability lab in which FRR, as a route
collector, is told both of BIRD's routes for a prefix from BIRD's two
sessions and holds them apart, and loses one when its session is
disabled, with the advertised count checked at each step. The workspace tests, the kernel tests and every lab phase pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A row's routes are in a vector that grew by one route at a time, so with
four peers sending full tables every row's vector was reallocated three
times as their routes arrived, and the reallocations were 7% of the
daemon's CPU time in a profile of a four-peer load.
This change has the first growth of a row take it to as many routes as
the table has peers, at most 8 (`FIRST_ROOM`), since a prefix that one
full table has, the others nearly always have too; after that it grows
by a quarter as before. A table with one peer never grows a row and is
unchanged. A route server's clients do not all announce every prefix,
hence the cap: a row of a prefix that fewer peers announce has at most
seven spare slots of 16 bytes.
Measured in the Linux VM with four peers sending the synthetic table of
1.23 million prefixes: the kernel is in step 1.9 s after the first route
instead of 2.2 s, the daemon's CPU time over the load is 5% lower, and
it is 604 MiB resident instead of about 630, since there are no
reallocation leftovers. With one peer nothing changes. Memory per route
in the benchmarks, allocations counted: 77 bytes before and after with
the synthetic table from four peers; 64 before and 65 after with the
RIPE RIS dump's eight largest peers, whose load went from 2.8 s to
1.8 s on one thread, the allocation counting making every reallocation
dearer than it is in the daemon.
Also tried and not adopted, with the new `JEMALLOC_CONF` setting of the
profiling script: transparent huge pages for the allocator
(`thp:always,metadata_thp:always`). Over three pairs of runs with one
peer the table arrived in 0.66, 1.17 and 0.70 s with them against 0.87,
0.75 and 0.87 s without, the CPU time was within the noise, and the
daemon was 60 to 100 MiB larger. An arena for the routes would not help
either: jemalloc's size classes already give rows exact 16, 32, 48 and
64 byte allocations, and the cost was the number of reallocations, not
the allocator.
Testing: the table tests, including the one that grows a row past
65,535 routes, pass; the kernel tests and the interoperability lab pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The routing table's shards started empty and grew as routes came in,
each growth moving every row of the shard into a table twice the size.
In a profile of the daemon taking in a full table, the growth showed as
up to a seventh of the session's time, and the daemon's resident memory
had the old and the new table of each shard side by side for a moment.
This change adds `RibTable::expect_prefixes`, which reserves room for
that many prefixes spread over the shards, and has the server call it at
startup with the largest `expected_routes` among the configured peers:
the prefixes of several full tables are mostly the same ones. Without
`expected_routes` nothing changes. The `full_table` benchmark does the
same unless `BENCH_PRESIZE=0`.
Measured, the saving in time is within the run-to-run noise: in the
Linux VM one peer's table is in the routing table in 0.81 s instead of
0.89 s, and the kernel is in step at about the same time as before;
three pairs of runs of the benchmark on a Mac average 0.33 s against
0.35 s. The profile's attribution to the growth was mostly sampling skid.
The memory saving is real: 442 MiB resident with one peer's table in
instead of 483 MiB, and 626 MiB instead of 632 MiB with four.
Also: the profiling script printed nothing for the kernel's change
counts because it asked `/metrics` for them; the endpoint is `/metricz`.
With that fixed, a four-peer load shows 1.25 million installs for 1.23
million prefixes: the journal already gives the kernel task only the
latest change of a prefix when it lags, so no further coalescing of
installs is called for.
Documentation: `expected_routes` in the configuration reference, the
neighbor guide and the proto comment say that the table is sized by it;
the testing guide's VM results are brought up to date.
Testing: a table test checks that the room is there and is kept; the
workspace tests pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The kernel task read changes of best route from the routing table 1024
at a time. A poll of the table does two binary searches and takes a read
lock in every shard, however few events the shard has; with the 128
shards of an 8-core machine, that overhead was paid for every 8 events a
shard, and the profile of a full-table load showed 8% to 9% of the
daemon's CPU time in the reader's searches and locks.
This change raises the batch to 16384. In the Linux VM, with four peers
sending the synthetic table of 1.23 million prefixes, the kernel is in
step 2.2 s after the first route instead of 2.9 s, and the daemon's CPU
time over the load falls 13%; the reader's share drops from 9% to under
1%. With one peer the kernel-side share falls the same way and the CPU
time 3%. A batch holds up to 16384 small records between polls, which is
about a megabyte.
Testing: the fib unit tests pass; measured with
scripts/profile/full-table.sh with one and four peers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The full-table profiling script took its peers as unconfigured ones under
the permissive policy, which is not how a router is set up. Now it writes
a peer block for each of them, with `expected_routes` set to the table's
size, so that the default import policy and anything that sizes itself by
the expected routes are in the measurement. The few bogon prefixes in the
synthetic table are refused by that policy, as the output says.
After the table is in, the script also prints
`bgp_kernel_route_changes_total`, so that a run shows how many installs
and removals the kernel was asked for and not only how many routes it
ended up with.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.
This change:
* Keeps counts of blackhole routes and of routes kept unconfirmed from
an earlier run in `Plan`, in step with the map through `Plan::put`
and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
cost nothing. `kept()` had the same scan; it ran only while routes of
an earlier run were being kept, which is exactly during a full-table
load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
new `profiling` Cargo profile (release with line tables) and frame
pointers, runs it in a network namespace joined by a veth pair to a
second namespace, feeds it the synthetic table from there as one or
more neighbors, samples it with `perf`, and writes a flame graph
(inferno), flat profiles by self and inclusive time, the time until
the routes are in the table and in the kernel, and the daemon's
resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
and `OUT` control it. From macOS it runs in the Linux VM through
scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
from the peers' addresses, completes the handshakes, writes each
neighbor's table, drains what the daemon sends back and holds the
sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
"Profiling" section, and the daemon's numbers in the VM.
Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.
Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Before this change, a route carrying the BLACKHOLE community (65535:666,
RFC 7999) from an EBGP neighbor was always rejected with the reason
"blackhole request", and nothing discarded traffic. A customer under a
denial-of-service attack had no way to ask this router to drop traffic
to an address.
This change adds `accept_blackholes` to the import policy. When it is
set, a route with the BLACKHOLE community is accepted if:
* its prefix is a single address, a /32 or a /128 (otherwise it is
rejected with the new reason "blackhole for more than one address");
the `max_ipv4_prefix_length` and `max_ipv6_prefix_length` limits do
not apply, as RFC 7999 section 3.2 recommends;
* the address is inside a prefix the neighbor may announce, if
`prefixes` is configured, regardless of the entry's `max_length`
(new `PrefixSet::covers`); without a prefix list, an address inside
the local AS's own prefixes is rejected as before ("prefix is our
own");
* the AS path, origin and RPKI checks that apply to every route pass.
The next hop of an accepted blackhole route is not checked for
third-party addresses and is not resolved, because traffic is never
forwarded to it. The route takes part in best path selection as usual;
a host route is normally the active route for its address.
The kernel task installs the active route as a kernel blackhole route
(`RTN_BLACKHOLE`, shown by `ip route` as `blackhole PREFIX proto bgp`).
Blackhole routes left by an earlier run are adopted at startup like
other routes of ours. The route is advertised to other neighbors as the
export policy and export limits permit, with the community intact, so a
provider that honors BLACKHOLE can discard the traffic upstream.
Implementation:
* `plan::Target` becomes an enum: `Via { gateway, interface }` or
`Discard`. `plan::resolve` takes a `blackhole` flag and returns
`Discard` for such routes without looking at the next hop (the
own-network check still applies). No nexthop object is made for a
discard target.
* `AttrSet::is_blackhole` reads the community off the stored attribute
bytes, like `next_hop` and `as_path`, so the kernel task, the
revalidator and the RPKI revalidation can ask it of every route.
* `ImportPolicy::is_blackhole_request` and `check_blackhole_prefix`
carry the import side; the import flow (`routes.rs`) computes once per
UPDATE whether its routes are an accepted blackhole request and skips
the next hop checks for them.
* `Status::blackholes` and the `blackholes` field of `/api/kernel` count
the installed blackhole routes; the status page shows the count next
to the installed routes.
Not covered: a prefix this router originates with the BLACKHOLE
community is advertised but not installed as a blackhole route. The
documentation says to add a kernel blackhole route by hand for that
case.
Documentation: the "Blackhole Routes" section of the import policy
guide replaces "Blackhole Requests", the kernel guide gains a "Blackhole
Routes" section, the configuration reference describes
`accept_blackholes`, the HTTP API reference describes `blackholes`, and
the RFC 7999 rows of the standards reference are updated.
Testing: unit tests for the attribute check, the prefix checks and the
plan; integration tests that a blackhole request is refused by default
and, with `accept_blackholes`, taken for one address inside the peer's
space and refused otherwise, with the recorded reasons; a kernel test
that a blackhole request becomes a kernel blackhole route whatever its
next hop, is removed on withdrawal and is adopted after a restart. The
existing kernel tests, the interoperability lab and the workspace tests
pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.
Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:
* The route is looked up in one kernel table, the "resolution table"
(`kernel { resolution_table: N }`, default 254 = main). The longest
prefix that contains the next hop wins; of several routes for that
prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
route is installed with the gateway and interface of the kernel route
that covers the next hop. A kernel route without a gateway (a route
onto a point-to-point or tunnel interface) makes the next hop itself
the gateway on that interface. A kernel route that names a nexthop
object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
own or another daemon's. A next hop that is reachable only through a
BGP route depends on another BGP next hop, and such chains are not
predictable.
* The default route does not resolve a next hop unless
`kernel { resolve_via_default_route: true }` is set. On a router the
default route points at a transit provider, and a next hop that is
reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
unresolvable, as does a route whose interface is down.
The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.
Implementation:
* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
and the kernel's nexthop objects into one value. The kernel task
publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
kernels get a dump of every table, filtered in user space). It asks
again, debounced, whenever a dedicated netlink socket that is a member
of the route multicast groups receives anything. That socket is given a
64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
installs on it too, up to a million at a time, and an ENOBUFS overflow
is treated as the same hint as a message. Its contents are never
parsed. An earlier attempt to receive route notifications on the
socket that writes routes failed, because the kernel only excludes the
requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
`Discarded`, which appear as the reasons in `/api/kernel` and on the
status page.
Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.
Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Of <stem>-v6 and <stem>-v4, the IPv6 session is listed first and carries
the name in full; IPv4 is under it. The same order in the lookup's list of
peers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The attention strip's names lead to the peer's row, which lights up for a
moment. The IPv4 and IPv6 sessions of one neighbour (NAME-v4, NAME-v6) are
listed together, joined down the side, the name said once in full. A route
count that moved by more than 2% between refreshes carries ▲ or ▼ for two
minutes; a row whose state or count changed while the tab was in the
background carries ● until it is clicked. A compact mode drops the next
hop and session columns, and is remembered by the browser. Colours are
variables, with a second set for prefers-color-scheme: dark.
A new per-peer `expected_routes` says about how many routes a peer sends
when all is well. The daemon only passes it to the API; the page lists an
established peer with under half of it as needing attention, a minute
after the session came up, with the expected count beside the actual one.
An established session with next to no routes looked, until now, exactly
like one that was fine.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The documentation sources are Zola Markdown: front matter between +++
lines, shortcodes that insert the example configurations, and @/ internal
links. A code forge shows all three unrendered, which the README used to
deny. A docs/README.md now explains each, and the README points to it.
"Configuring BGP Neighbors" becomes "BGP Neighbors", as the other feature
pages are titled; "When a Neighbor Restarts" becomes "Neighbor Restart", a
noun phrase as the style guide asks.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What Ferrix is and does, that it routes AS210036, how to build it and try
it without a router, the security points an operator must know, and a note
that the code is written with an AI coding agent under the maintainer's
review.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Received routes are listed from the best down: the RIB gains ranked_routes,
which runs selection again without the winners, so the second route is the
one that would be installed were the best withdrawn. The API gives each
route its rank, and the page shows it as a numbered badge after the star.
The lookup box moves to the received-routes table, which is what it filters,
and takes an AS number: a new `origin` query parameter finds every route
whose path ends in that AS, with an empty path counting as the local AS.
Long cells are clamped to a few lines with a control to open them, and an
AS repeated along a path (prepending) is shown once with a ×N badge. Peer
names are bold and dark, not underlined links, so a row can be found by
its name among the addresses and numbers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The page opened with every session in bold green, the normal state, and
the brightest numbers on it were the rejection counts, which RPKI doing
its job produces by the thousand. The two sessions that were actually
down were in the same red, thirty rows down. Now:
- A status line at the top says what needs attention, by name and with
the reason: sessions down, peers restarting, the kernel behind or
refusing routes, RPKI data missing or about to expire. When nothing
does, it says so. A second line gives the normal state in grey.
- Colour means one thing each. Red: failed, needs a hand. Amber:
degraded, worth watching. Green, set small: a system doing its job.
Grey: inactive and expected. Normal is black; "Established" is no
longer green, and a rejection count no longer red.
- Peers that need attention come first, then the others grouped by what
they are to us, with a "problems only" switch. Numbers are right
aligned with thousands separators; the established time and uptime
are one cell; the session's properties are glyphs (v4, v6, GR, ENH,
AP, 2B) with the meaning as a tooltip, as are the roles.
- Received routes: the best route leads, marked ★; attributes without a
column are glyphs (OTC, AA, AGG, AS4, OID, CL); validity is ✓ ✗ –.
- Rejected routes: those alike (same prefix, reason and path) are one
row with a count and the last time; each peer's reasons are a bar.
- Communities with a configured name (community_names, new) are badges
with that name, the number as the tooltip: "comrack", "SwissIX",
"interconnection 2011" rather than 210036:100:2 and so on. A name for
a pattern shows the open part after it. The daemon does nothing with
the names but pass them to the page.
- One lookup box at the top, which takes a peer's name too; a peer's
name anywhere on the page leads to its routes; table headers and the
status line stay in view while scrolling.
- The host name comes from gethostname(2), not from running `hostname`,
which a static binary on a bare machine could not find: comrack said
"unknown host".
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API, the metrics, the status page and the documentation say a route
is "rejected" (/api/routes/rejected, rejected_routes,
bgp_routes_rejected_total, "Rejected routes"), and the reason type is
Rejection. The code that keeps the record said "refused": RefusalLog,
PeerRefusals, RefusedRoute, RefusalsView, RefusedRouteView, the refusals
module. Those are RejectionLog, PeerRejections, RejectedRoute,
RejectionsView, RejectedRouteView and rejections now, and the log line
says "Rejected a route". Nothing the API or the metrics expose changes.
Refusing stays what it was: a connection turned away at the door
(server::Refusal), a kernel that would not take a route, an OPEN we
would not have.
The two aliases of SendReceive, AddPathSendRecv and OrfSendRecv, are
gone; it is one type.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The test files had grown their own copies of the same helpers: prefix()
twelve times, established() seven, an UPDATE builder six, a seventeen
field PeerConfig literal in fifteen places, a timeout constant in twenty
one. Each is in crates/integration-tests/src once now: peer_config() and
TestServer::with_peer for a configured peer, Announcement for building
UPDATEs, established(), the waits (wait_until, wait_for, wait_for_routes,
wait_for_sessions), get_json, and TIMEOUT. A new field on PeerConfig is
one line to add, not fifteen.
TestPeer::recv has a deadline, so a test that gets nothing fails instead
of hanging; is_closed, which read one byte and took a KEEPALIVE for an
open connection, is closed_within and closed_unanswered.
Net 788 lines fewer; every test checks what it did.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- PathAttribute::UnknownPathAttribute and Capability::UnknownCapability
are PathAttribute::Unknown and Capability::Unknown: the enum already
says what kind of thing is unknown.
- The capability entry types are all named XxxEntry now
(ExtendedNextHopEntry, LongLivedGracefulRestartEntry), rather than two
of them sharing a name with the variant that holds a list of them; and
MultiLabelCompat is MultiLabel, since nothing explained the "Compat".
- Two different things were both called Reflection: what a session needs
to reflect routes (export.rs, still Reflection) and the identifiers a
reflected route must not carry back to us (policy.rs, now
ReflectorIds).
- well_known_allow's peer_is_internal is ibgp, as everywhere else.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Five functions did several things each in one body of 120 to 190 lines.
Each is now a list of steps with a name and a doc comment of their own,
with nothing changed in what they do:
- run_session: hold_down_after, keep_for and release_routes.
- Session::establish: session_handle, validation and route_source. A
session keeps the peer's settings as one PeerSettings rather than
eight fields copied out of it one by one; the sweeps' two channel ends
are named fields, not .0 and .1; whether the peer can be asked to send
its routes again is a bool, not a copy of the families.
- Fsm::on_open: negotiate_role and negotiate_add_path.
- BgpServer::run: log_startup and spawn_tasks.
- Config::from_proto: parse_listen_endpoints, parse_peers, check_roles,
parse_originate and Ours::tell_peers, with the three per-peer defaults
in one PeerDefaults.
- Egress::outgoing: may_go, rewrite and next_hop_for.
- Exporter::follow: Learned::drain, judge and flush, with the three
things called "announced" told apart (own_announced, learned.announced,
batches) and the verdicts given a type.
The hold-downs are a module of their own, holddown.rs; policy.rs is
about policy again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Once every peer had sent all its routes, the kernel task let go of the
routes kept from the earlier run that the table did not have, at once.
But "the table has" and "the kernel task has read" are not the same
moment: a route a peer sent just before its End-of-RIB could still be
unread in the table's journal, in which case it was taken out of the
kernel and put back a moment later. The interop lab saw this now and
then as "N routes were taken out of the kernel over the restart".
The task now reads the table to the end before it decides what is
unconfirmed; follow() says whether the reader is caught up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
in the namespace rebuilt what kernel-tests.sh had just built. Each is
passed on only where set.
- The README's list of unsupported features still had route reflection
and graceful restart, both of which exist; the command-line reference
lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
unused variable; small things in the web crate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What the parsers and writers of messages, attributes and capabilities
had each written for themselves is one module (wire.rs): the shape of a
parser's result and its error, reading a list to the end, checking that
nothing was left over, a non-empty list of fixed-size records (the
community attributes and CLUSTER_LIST, which now fail alike), an AS
number at the session's width, an AFI/SAFI pair, and a length written
before what it counts. open.rs loses its fail! macro and attributes.rs
its own copies of the same.
Tests for what had none: AS4_AGGREGATOR kept opaque and discarded when
malformed; IPv4 routes with IPv6 next hops (RFC 8950) in the property
tests; and what a part-level parser accepts re-encodes to what it read,
but for the fields the decoders normalise.
Nlri::wire_len had no callers; the mask of a prefix's last octet is one
function; a test's name is spelled right; a comment is on the test it
describes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The RPKI client had one Timeout error whose text said "did not answer
within 60s" whatever had run out: the connect timeout, the time for a
whole answer, or the data's lifetime. These are what /api/rpki shows as
the last error; each is its own now. The protocol version shown is the
one the cache answered in, not the one being tried. The VRP set, hashed
by the hundred thousand with every answer, uses the quick hash the ASPA
table already did.
In the kernel installer the writer task's error was worked out in three
places, and the next-hop resolution checked for our own networks twice
on every cache miss. A refusal was logged twice, once with the status
lock held. The interfaces are listed in order of index by the structure
that holds them, from a function that can be tested on any OS, and is.
EIO is with the other errnos; Plan has no installed_prefixes (no
callers) and clears its records in one place, with the order of
Unresolved::ALL asserted at compile time since indexing depends on it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removing a peer's routes, sweeping its stale ones and sweeping the
attribute sets were each written out as their own spawn_blocking, and
each dealt with a failed task differently: the periodic sweep dropped a
panic on the floor, and a restarting peer whose kept routes could not be
removed was logged as "removed 0 routes". off_thread does it once, and
every caller logs a failure.
Also:
- the hard-reset wrapping of a NOTIFICATION is one function in the FSM;
- the two ways to close every session are one, and the registry has one
leave() rather than leave() and shutdown();
- the dialer says why a session ended with SessionEnd::reason, and no
longer turns a result into a result;
- as_path_prepend is parsed in one place, and the limits in error texts
come from the constants rather than being written out;
- router_id and cluster_id are parsed by one function;
- the exporter builds its Egress in one place;
- the fields a session keeps only to hand to its handle are fewer:
route_refresh is a bool, not a copy of the families;
- counting the stale routes removed cannot underflow;
- docs that had gone stale: the proto still said there was no route
reflection, the advertiser's module doc said it announced only our own
routes, from_server listed two of its five sources, and so on.
Tests for what had none: which session ends count as restarts, the Hard
Reset wrapping, the graceful restart settings, the dialer's jitter,
withdrawals in MP_UNREACH_NLRI, and AS_TRANS on a two-octet session.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A reader does not read past the earliest change still under way, which
may not have reached every shard. A change to every shard, as when a
peer's whole table goes, is under way for a while; changes others made
meanwhile are beyond it. A reader that found only those saw "more to
read" with nothing it could read, and asked again at once, round and
round, taking every shard's read lock each time, until the change ended:
every exporter and the kernel installer, spinning for as long as a full
table took to remove.
Now a pass that moves no cursor though there is more says it is caught
up, and waits. And a change wakes the readers when it ends if any other
change woke them while it was under way, since what they were woken for
is only readable now. The three per-shard change loops are one
(RibTable::change), which drops the change before waking, as only one of
them did.
Also: the add_peer doc was attached to peer(); some_routes had no
callers; count_stale is for tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Readability, no change in behaviour, but for one log line:
- The routes a peer holds from us were counted twice over, differently,
in the exporter: once per learned-route batch and once per originated
route. Learned::held_by_peer counts the learned ones in one place, and
ExportLimits::exceeded_by returns the limit that was passed instead of a
bool, so that ending the session no longer has to dig it out again with
a default that could not happen.
- The two hold-downs for a limit passed, ours or the peer's, were two
near-identical blocks; they are one match.
- How a session ended is put into words in one place
(SessionEnd::reason), for the dialer's backoff and the passive peer's
state alike.
- parse_export_limits parses one block; what a missing block means (no
limits at the top, the top's limits for a peer) is said where it
differs, instead of Option in and Option out.
- The startup warnings about export limits are a function of their own,
and the peers without limits are named in one line rather than one
line each: a router with eighty sessions does not need eighty warnings.
The warning about a peer without a role is gone: the configuration now
refuses one.
- Hex formatting of attribute octets is one function; the text for an
unknown attribute is built in three plain steps.
- The exporter's test fixture returns a named struct, not a tuple.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A session could only be configured for a peer we would then dial, every
few seconds to two minutes, for as long as it was not up. A new peer
whose side was not ready yet filled the log with failed attempts, and a
peer that opens its sessions itself was dialled for nothing.
A peer with `passive: true` is not dialled. A connection from its address
is taken as for any configured peer, and the session comes up with all
that is configured for it: role, import policy, export limits,
hold-downs. Until then the status page and /api/peers show it as
`Passive`, waiting for the peer to connect; after a session, with how
that session ended, which nothing else would say since nobody dials it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What other networks were told of rested on one thing: the Only to
Customer mark that routes from providers, peers and route servers get as
they come in. A route without it, from a session without a role or by a
mistake anywhere along the way, went to every peer, from every router of
ours, and nothing of ours stood in the way.
export_limits are a second check, independent of the roles, the marks
and the export policy. For peers above or beside us (PROVIDER, PEER,
ROUTE_SERVER, and any in another AS without a role):
- prefixes: only routes for these go out, learned or our own. With our
and our customers' space listed, a leak of anyone else's routes goes
nowhere above us.
- max_prefixes: if more routes were to go out than this, none of them
does. The session ends with a Cease / Administrative Shutdown whose
Shutdown Communication says why (RFC 9003), and the peer is held down
for its import.hold_down_seconds, so that it is told nothing rather
than too much. The whole table is counted before any of it is sent.
They are set at the top and hold for every such peer that has none of its
own; unlike `export`, a peer's own export policy does not drop them. A
peer's own export_limits take their place, and an empty one lifts them.
Customers, route server clients and our own routers are not limited.
A configuration in which learned routes are passed on and a peer in
another AS has no role is now refused, as is accept_unconfigured_peers
with learned routes: without a role, a peer's routes go to everyone and
it is told of everything. This used to be a warning.
At startup, a warning for each peer the limits apply to that has none,
and for each originated route the limits leave out. A metric,
bgp_export_limit_exceeded_total, counts the sessions ended for it.
The interop lab has a phase in which a customer leaks: a route outside
its space never reaches BIRD, and when more were to go than the limit,
BIRD's session ends and its log has our reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel section was one run-on line of interfaces and addresses. It is
now a small table of the installation state, and a table of interfaces
with their IPv4, IPv6 and link-local addresses one per line; ports that
are down and have no addresses share one line at the bottom.
Communities ran together into one block that wrapped anywhere. Each AS's
communities are now a line of their own, kept whole so that they copy,
with a hanging indent where a line wraps; pointing at a line names the AS.
Where control communities are on (now in /api/status), those of ours that
ask for something say what as a tooltip.
A rejected route's details were a fold-out that repeated the summary and
then dumped the decoded attributes as JSON. Rejected routes now have the
same columns as received ones, and both have an "other attributes" column
for what the summary had no field for, as text: aggregator, extended
communities (route targets and origins by name), AS4_PATH with its sets in
braces, originator ID, cluster list, and unknown attributes in hex. The
API has them as summary.other; the JSON stays in `attributes`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A route learned from one router of our AS went to no other (RFC 4271
§9.2.1), so every router needed a session with every other. A router at
the end of a single link, with one session to its neighbour inside the AS,
heard only of what that neighbour learned itself, and nothing of what the
neighbour had from the routers behind it.
A peer in our own AS can now be made a route reflector client
(`route_reflector_client`, RFC 4456). What a client sends goes to all our
routers; what the others send goes to the clients. A reflected route gets
ORIGINATOR_ID (where it started, if it does not say) and our cluster ID at
the front of CLUSTER_LIST; it is not sent back to the router it started
at, nor round to a cluster it has been through, and one that comes back
anyway is refused as "route reflected back to us". The cluster ID is the
router ID unless `cluster_id` says otherwise. The two attributes are now
decoded as such rather than kept as bytes, and are never sent to other
networks.
Unlike a standard reflector, this one is the next hop of what it reflects,
as it is of everything it announces: there is no resolving a next hop
that is not a neighbour, so a client reaches what lies behind another
router through the reflector.
Only peers in our own AS can be clients; the configuration refuses the
setting for any other. Nothing changes in what other networks are told:
Only to Customer marks go along with reflected routes, and every
receiving session keeps its own max_prefixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The import policy did the same to every route of a session. Now `rules`
can do more to some of them: those with certain communities, those for
certain prefixes, or both. Each rule can set the local preference and put
communities on; each route takes the first rule that holds for it.
Rules see a route as the session's own statements leave it, with our
communities taken off and the session's marks put on. So a router can act
on where another router of ours learned a route, by the mark that router
put on it, and a customer's single prefix can be marked for a prepend.
Routes of one UPDATE that different rules hold for go into the table with
attribute sets of their own. An UPDATE that no rule touches is taken as
before, without its attributes being written out again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:
L:11:P .. L:13:P our AS once to three times more towards AS P
(P 0: towards every peer in another AS)
L:20:G / L:21:G not to / only to the sessions of group G
L:31:G .. L:33:G our AS once to three times more towards group G
A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.
They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.
And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.
This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
