The API, the metrics, the status page and the documentation say a route
is "rejected" (/api/routes/rejected, rejected_routes,
bgp_routes_rejected_total, "Rejected routes"), and the reason type is
Rejection. The code that keeps the record said "refused": RefusalLog,
PeerRefusals, RefusedRoute, RefusalsView, RefusedRouteView, the refusals
module. Those are RejectionLog, PeerRejections, RejectedRoute,
RejectionsView, RejectedRouteView and rejections now, and the log line
says "Rejected a route". Nothing the API or the metrics expose changes.
Refusing stays what it was: a connection turned away at the door
(server::Refusal), a kernel that would not take a route, an OPEN we
would not have.
The two aliases of SendReceive, AddPathSendRecv and OrfSendRecv, are
gone; it is one type.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The test files had grown their own copies of the same helpers: prefix()
twelve times, established() seven, an UPDATE builder six, a seventeen
field PeerConfig literal in fifteen places, a timeout constant in twenty
one. Each is in crates/integration-tests/src once now: peer_config() and
TestServer::with_peer for a configured peer, Announcement for building
UPDATEs, established(), the waits (wait_until, wait_for, wait_for_routes,
wait_for_sessions), get_json, and TIMEOUT. A new field on PeerConfig is
one line to add, not fifteen.
TestPeer::recv has a deadline, so a test that gets nothing fails instead
of hanging; is_closed, which read one byte and took a KEEPALIVE for an
open connection, is closed_within and closed_unanswered.
Net 788 lines fewer; every test checks what it did.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- PathAttribute::UnknownPathAttribute and Capability::UnknownCapability
are PathAttribute::Unknown and Capability::Unknown: the enum already
says what kind of thing is unknown.
- The capability entry types are all named XxxEntry now
(ExtendedNextHopEntry, LongLivedGracefulRestartEntry), rather than two
of them sharing a name with the variant that holds a list of them; and
MultiLabelCompat is MultiLabel, since nothing explained the "Compat".
- Two different things were both called Reflection: what a session needs
to reflect routes (export.rs, still Reflection) and the identifiers a
reflected route must not carry back to us (policy.rs, now
ReflectorIds).
- well_known_allow's peer_is_internal is ibgp, as everywhere else.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Five functions did several things each in one body of 120 to 190 lines.
Each is now a list of steps with a name and a doc comment of their own,
with nothing changed in what they do:
- run_session: hold_down_after, keep_for and release_routes.
- Session::establish: session_handle, validation and route_source. A
session keeps the peer's settings as one PeerSettings rather than
eight fields copied out of it one by one; the sweeps' two channel ends
are named fields, not .0 and .1; whether the peer can be asked to send
its routes again is a bool, not a copy of the families.
- Fsm::on_open: negotiate_role and negotiate_add_path.
- BgpServer::run: log_startup and spawn_tasks.
- Config::from_proto: parse_listen_endpoints, parse_peers, check_roles,
parse_originate and Ours::tell_peers, with the three per-peer defaults
in one PeerDefaults.
- Egress::outgoing: may_go, rewrite and next_hop_for.
- Exporter::follow: Learned::drain, judge and flush, with the three
things called "announced" told apart (own_announced, learned.announced,
batches) and the verdicts given a type.
The hold-downs are a module of their own, holddown.rs; policy.rs is
about policy again.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Once every peer had sent all its routes, the kernel task let go of the
routes kept from the earlier run that the table did not have, at once.
But "the table has" and "the kernel task has read" are not the same
moment: a route a peer sent just before its End-of-RIB could still be
unread in the table's journal, in which case it was taken out of the
kernel and put back a moment later. The interop lab saw this now and
then as "N routes were taken out of the kernel over the restart".
The task now reads the table to the end before it decides what is
unconfirmed; follow() says whether the reader is caught up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
in the namespace rebuilt what kernel-tests.sh had just built. Each is
passed on only where set.
- The README's list of unsupported features still had route reflection
and graceful restart, both of which exist; the command-line reference
lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
unused variable; small things in the web crate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What the parsers and writers of messages, attributes and capabilities
had each written for themselves is one module (wire.rs): the shape of a
parser's result and its error, reading a list to the end, checking that
nothing was left over, a non-empty list of fixed-size records (the
community attributes and CLUSTER_LIST, which now fail alike), an AS
number at the session's width, an AFI/SAFI pair, and a length written
before what it counts. open.rs loses its fail! macro and attributes.rs
its own copies of the same.
Tests for what had none: AS4_AGGREGATOR kept opaque and discarded when
malformed; IPv4 routes with IPv6 next hops (RFC 8950) in the property
tests; and what a part-level parser accepts re-encodes to what it read,
but for the fields the decoders normalise.
Nlri::wire_len had no callers; the mask of a prefix's last octet is one
function; a test's name is spelled right; a comment is on the test it
describes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The RPKI client had one Timeout error whose text said "did not answer
within 60s" whatever had run out: the connect timeout, the time for a
whole answer, or the data's lifetime. These are what /api/rpki shows as
the last error; each is its own now. The protocol version shown is the
one the cache answered in, not the one being tried. The VRP set, hashed
by the hundred thousand with every answer, uses the quick hash the ASPA
table already did.
In the kernel installer the writer task's error was worked out in three
places, and the next-hop resolution checked for our own networks twice
on every cache miss. A refusal was logged twice, once with the status
lock held. The interfaces are listed in order of index by the structure
that holds them, from a function that can be tested on any OS, and is.
EIO is with the other errnos; Plan has no installed_prefixes (no
callers) and clears its records in one place, with the order of
Unresolved::ALL asserted at compile time since indexing depends on it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Removing a peer's routes, sweeping its stale ones and sweeping the
attribute sets were each written out as their own spawn_blocking, and
each dealt with a failed task differently: the periodic sweep dropped a
panic on the floor, and a restarting peer whose kept routes could not be
removed was logged as "removed 0 routes". off_thread does it once, and
every caller logs a failure.
Also:
- the hard-reset wrapping of a NOTIFICATION is one function in the FSM;
- the two ways to close every session are one, and the registry has one
leave() rather than leave() and shutdown();
- the dialer says why a session ended with SessionEnd::reason, and no
longer turns a result into a result;
- as_path_prepend is parsed in one place, and the limits in error texts
come from the constants rather than being written out;
- router_id and cluster_id are parsed by one function;
- the exporter builds its Egress in one place;
- the fields a session keeps only to hand to its handle are fewer:
route_refresh is a bool, not a copy of the families;
- counting the stale routes removed cannot underflow;
- docs that had gone stale: the proto still said there was no route
reflection, the advertiser's module doc said it announced only our own
routes, from_server listed two of its five sources, and so on.
Tests for what had none: which session ends count as restarts, the Hard
Reset wrapping, the graceful restart settings, the dialer's jitter,
withdrawals in MP_UNREACH_NLRI, and AS_TRANS on a two-octet session.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A reader does not read past the earliest change still under way, which
may not have reached every shard. A change to every shard, as when a
peer's whole table goes, is under way for a while; changes others made
meanwhile are beyond it. A reader that found only those saw "more to
read" with nothing it could read, and asked again at once, round and
round, taking every shard's read lock each time, until the change ended:
every exporter and the kernel installer, spinning for as long as a full
table took to remove.
Now a pass that moves no cursor though there is more says it is caught
up, and waits. And a change wakes the readers when it ends if any other
change woke them while it was under way, since what they were woken for
is only readable now. The three per-shard change loops are one
(RibTable::change), which drops the change before waking, as only one of
them did.
Also: the add_peer doc was attached to peer(); some_routes had no
callers; count_stale is for tests.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Readability, no change in behaviour, but for one log line:
- The routes a peer holds from us were counted twice over, differently,
in the exporter: once per learned-route batch and once per originated
route. Learned::held_by_peer counts the learned ones in one place, and
ExportLimits::exceeded_by returns the limit that was passed instead of a
bool, so that ending the session no longer has to dig it out again with
a default that could not happen.
- The two hold-downs for a limit passed, ours or the peer's, were two
near-identical blocks; they are one match.
- How a session ended is put into words in one place
(SessionEnd::reason), for the dialer's backoff and the passive peer's
state alike.
- parse_export_limits parses one block; what a missing block means (no
limits at the top, the top's limits for a peer) is said where it
differs, instead of Option in and Option out.
- The startup warnings about export limits are a function of their own,
and the peers without limits are named in one line rather than one
line each: a router with eighty sessions does not need eighty warnings.
The warning about a peer without a role is gone: the configuration now
refuses one.
- Hex formatting of attribute octets is one function; the text for an
unknown attribute is built in three plain steps.
- The exporter's test fixture returns a named struct, not a tuple.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A session could only be configured for a peer we would then dial, every
few seconds to two minutes, for as long as it was not up. A new peer
whose side was not ready yet filled the log with failed attempts, and a
peer that opens its sessions itself was dialled for nothing.
A peer with `passive: true` is not dialled. A connection from its address
is taken as for any configured peer, and the session comes up with all
that is configured for it: role, import policy, export limits,
hold-downs. Until then the status page and /api/peers show it as
`Passive`, waiting for the peer to connect; after a session, with how
that session ended, which nothing else would say since nobody dials it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What other networks were told of rested on one thing: the Only to
Customer mark that routes from providers, peers and route servers get as
they come in. A route without it, from a session without a role or by a
mistake anywhere along the way, went to every peer, from every router of
ours, and nothing of ours stood in the way.
export_limits are a second check, independent of the roles, the marks
and the export policy. For peers above or beside us (PROVIDER, PEER,
ROUTE_SERVER, and any in another AS without a role):
- prefixes: only routes for these go out, learned or our own. With our
and our customers' space listed, a leak of anyone else's routes goes
nowhere above us.
- max_prefixes: if more routes were to go out than this, none of them
does. The session ends with a Cease / Administrative Shutdown whose
Shutdown Communication says why (RFC 9003), and the peer is held down
for its import.hold_down_seconds, so that it is told nothing rather
than too much. The whole table is counted before any of it is sent.
They are set at the top and hold for every such peer that has none of its
own; unlike `export`, a peer's own export policy does not drop them. A
peer's own export_limits take their place, and an empty one lifts them.
Customers, route server clients and our own routers are not limited.
A configuration in which learned routes are passed on and a peer in
another AS has no role is now refused, as is accept_unconfigured_peers
with learned routes: without a role, a peer's routes go to everyone and
it is told of everything. This used to be a warning.
At startup, a warning for each peer the limits apply to that has none,
and for each originated route the limits leave out. A metric,
bgp_export_limit_exceeded_total, counts the sessions ended for it.
The interop lab has a phase in which a customer leaks: a route outside
its space never reaches BIRD, and when more were to go than the limit,
BIRD's session ends and its log has our reason.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The kernel section was one run-on line of interfaces and addresses. It is
now a small table of the installation state, and a table of interfaces
with their IPv4, IPv6 and link-local addresses one per line; ports that
are down and have no addresses share one line at the bottom.
Communities ran together into one block that wrapped anywhere. Each AS's
communities are now a line of their own, kept whole so that they copy,
with a hanging indent where a line wraps; pointing at a line names the AS.
Where control communities are on (now in /api/status), those of ours that
ask for something say what as a tooltip.
A rejected route's details were a fold-out that repeated the summary and
then dumped the decoded attributes as JSON. Rejected routes now have the
same columns as received ones, and both have an "other attributes" column
for what the summary had no field for, as text: aggregator, extended
communities (route targets and origins by name), AS4_PATH with its sets in
braces, originator ID, cluster list, and unknown attributes in hex. The
API has them as summary.other; the JSON stays in `attributes`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A route learned from one router of our AS went to no other (RFC 4271
§9.2.1), so every router needed a session with every other. A router at
the end of a single link, with one session to its neighbour inside the AS,
heard only of what that neighbour learned itself, and nothing of what the
neighbour had from the routers behind it.
A peer in our own AS can now be made a route reflector client
(`route_reflector_client`, RFC 4456). What a client sends goes to all our
routers; what the others send goes to the clients. A reflected route gets
ORIGINATOR_ID (where it started, if it does not say) and our cluster ID at
the front of CLUSTER_LIST; it is not sent back to the router it started
at, nor round to a cluster it has been through, and one that comes back
anyway is refused as "route reflected back to us". The cluster ID is the
router ID unless `cluster_id` says otherwise. The two attributes are now
decoded as such rather than kept as bytes, and are never sent to other
networks.
Unlike a standard reflector, this one is the next hop of what it reflects,
as it is of everything it announces: there is no resolving a next hop
that is not a neighbour, so a client reaches what lies behind another
router through the reflector.
Only peers in our own AS can be clients; the configuration refuses the
setting for any other. Nothing changes in what other networks are told:
Only to Customer marks go along with reflected routes, and every
receiving session keeps its own max_prefixes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The import policy did the same to every route of a session. Now `rules`
can do more to some of them: those with certain communities, those for
certain prefixes, or both. Each rule can set the local preference and put
communities on; each route takes the first rule that holds for it.
Rules see a route as the session's own statements leave it, with our
communities taken off and the session's marks put on. So a router can act
on where another router of ours learned a route, by the mark that router
put on it, and a customer's single prefix can be marked for a prepend.
Routes of one UPDATE that different rules hold for go into the table with
attribute sets of their own. An UPDATE that no rule touches is taken as
before, without its attributes being written out again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:
L:11:P .. L:13:P our AS once to three times more towards AS P
(P 0: towards every peer in another AS)
L:20:G / L:21:G not to / only to the sessions of group G
L:31:G .. L:33:G our AS once to three times more towards group G
A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.
They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.
And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.
This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two routers of an AS that both originate a prefix (anycast) each
announce it to the other over their internal session. Each took the
other's route, could not install it in the kernel, since the prefix is
on one of its own interfaces, and so held it, counted as rejected with
"next hop unreachable": a permanent rejection on a healthy session, for
a next hop that was in fact fine.
A route from a peer in our own AS for a prefix this router originates
is now ignored: not taken, not held, not counted. Only that prefix
itself; what is inside it is judged as usual. From other networks such
routes are refused as our own prefixes, as before.
A route that is held because its prefix is a network of this machine,
or inside one (an exchange's LAN prefix, say), is now counted as "prefix
is a network of this router", apart from routes whose next hop cannot
be reached.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A validator that is still running its first validation answers a query
with No Data Available (error code 2). That ended the session, and the
next attempt came after the retry interval: 600 seconds when the cache
had not yet said otherwise, which it cannot have before it has data.
For those ten minutes after a validator started, nothing was validated.
No Data Available is now asked again after 30 seconds, or after
retry_seconds if that is shorter, and logged as information rather than
as a warning. Other failures wait as before, after trying the next
cache if there is one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.
When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.
When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.
The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "refused" column counted rejected announcements since the start, not
routes, which made two peers that send the same routes look different
when one of them re-sends more often. It is now "rejected announcements",
with the held routes, the number that compares, beside it as "held now".
The page says "rejected" throughout, as the documentation and metrics do.
Rows are highlighted under the pointer, and pointing at an AS number
shows its holder's name, which the browser asks RIPEstat for. The daemon
looks nothing up; the monitoring page says what the browser sends where.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A route with the BLACKHOLE community (RFC 7999) asks for the traffic to
its prefix to be dropped. Exchange route servers pass these on as host
routes with the exchange's blackhole address as next hop, and they were
rejected as "prefix too specific", which hid what they are.
Routes from EBGP neighbors with BLACKHOLE are now rejected as "blackhole
request", whatever their length. One short enough to pass would have
been taken as an ordinary route, sending the traffic to the blackhole
next hop: the request honoured without anyone having chosen to. Acting
on blackhole requests is not supported; the import policy page and the
standards list say so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rejected route was recorded with its prefix, reason and AS path, which
is not enough to tell why a neighbor sent it: whether a /32 is a
blackhole request, which communities a leaked route carries, which
network marked it Only to Customer.
Each recorded rejection now keeps the attributes of the UPDATE it came
in, without the NLRI of MP_REACH_NLRI and MP_UNREACH_NLRI, and its next
hop. The attributes are copied once per UPDATE, only when a route of it
is rejected, and shared by its rejected routes. A route rejected later,
when the RPKI data changes, has the attributes it was stored with.
/api/routes/rejected gives them as `summary` and `attributes`, as for
received routes, and the status page shows them under "show" in each
row of the refused routes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/build-static.sh builds the daemon for x86_64 Linux against
musl, as one executable without dependencies to copy to a router. It
finds a musl C compiler for jemalloc (musl-cross on macOS, musl-gcc on
Linux), and uses its ar and ranlib too: with the host's, a build from
macOS gets an empty libjemalloc.a and fails to link. It prints the
revision and the SHA-256 of the result.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full table from several neighbors means millions of small, long-lived
allocations that are replaced as routes change. jemalloc keeps the
resident size of such a workload closer to what is in use than the
system allocator does, glibc's in particular, and does not depend on
the C library the binary is linked with. Not on MSVC, which jemalloc
does not support.
Building now needs a C compiler and make; the installation page says
so.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The blue two-colour treatment is replaced by the photograph's own
colours, slightly darkened, under a neutral shade that keeps the text
legible.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The documentation moves from READMEs into a site in docs/: getting
started, a user guide, operations, a reference, and development pages,
written in the style of network vendors' documentation (docs/STYLE.md).
bin/bgp/README.md and docs/RFCs.md are folded into it.
- A theme of its own in the International Typographic Style, in blues,
with self-hosted Inter and JetBrains Mono, search over Zola's index,
dark mode, and no resources loaded from other hosts.
- Example configurations in docs/examples/, shown on the pages and
checked with `bgp --check` in CI.
- A test that fails when a field of config.proto is missing from the
configuration reference.
- A CI job that builds the site; Zola fails on broken links and anchors.
- A hardened systemd unit, tried under systemd with kernel routes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Under systemd the log goes to the journal, where ANSI escape codes are
noise. Colours are kept for a person at a terminal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What was to be scripted is configuration, done natively:
- import { add_communities, add_large_communities, remove_communities,
local_pref }: every route taken from a peer is marked (where it was
learned, say), with nobody else's word for our marks taken, and given
its preference, which selection goes by.
- export { add_communities, add_large_communities, remove_communities,
as_path_prepend, learned_routes }, for a peer or for all that have none
of their own.
- control_communities: the communities by which routes say which peers
they are passed on to, as exchanges' route servers have them (0:P, L:P,
0:L, and the large forms L:0:P, L:1:P, L:0:0), taken off as the route
goes on.
And what the last of these is for: with export.learned_routes a peer is
told the best route of every prefix the others announced. The rules are
BGP's and cannot be turned off: not back where it came from, not to a
peer whose AS is on the path, not between two internal peers, not past
NO_EXPORT or NO_ADVERTISE; our AS on the path and ourselves as next hop
to another network, LOCAL_PREF to our own; to a route server's client
the path, MED and next hop as they came (RFC 7947). Which routes go where
follows from the roles of RFC 9234: a route that came down or across goes
on down only, so that with roles on every peer this makes us transit for
our customers and nothing else. A peer without a role is told everything,
and the daemon says so at startup.
Each session follows the table's best routes with a reader and keeps the
prefixes it has announced, so that it can withdraw them. Routes that go
out alike go in one UPDATE: changes are gathered for 50 ms first, and a
table that is given out anew is gathered whole (at most four sessions at
a time; it is tens of megabytes). For that to work the table's journal
had to give out together what was written together: every change now
carries a time on a table-wide clock, a reader's pass reads every shard
up to one time, no further than the changes that are complete in every
shard, and a resync takes its places in the journals as of one time.
One peer sending 1.23 M routes over a session, another told of them:
in our table after 0.47 s, all heard after 2.1 s, four routes an UPDATE.
The lab has a phase, transit: FRR as our customer and BIRD as our
provider, routes each way with the right paths and marks, one kept back
by a community, and pings between their networks through us.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rhai made a decision about a route cost one to three microseconds,
some two hundred times what the same costs in Rust. That was bearable
where a function is asked once for an UPDATE, and is not where every
route is weighed against every peer, as it is when routes are passed
on. What is wanted of a policy (mark routes by where they came from,
mark what is announced, let communities say who is told what) needs no
language; it is to be configuration, done natively.
Kept from that work: route selection goes by a LOCAL_PREF that policy
set on a route from an external peer, and the lab's configuration can
be added to by a phase.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new crate, bgp-policy, runs functions of a script on routes: on their
way in, after the general checks of a peer's import and before the
checks that can change (RPKI, next hop reachability); and on their way
out to a peer. A function is given the route and the peer, may refuse
the route, and may change its communities, large communities,
LOCAL_PREF and MED, and on the way out prepend our AS:
fn from_peer(route, peer) {
if route.has_community("65535:666") { return false; }
route.add_community(LEARNED_FROM[peer.role]);
route.local_pref = PREFERENCE[peer.role];
true
}
Configuration: policy { path | source, import_function, export_function }
and, for a peer, import { function } and export { function }. The script
is compiled at startup; `bgp --check` reads it and says which function
is whose. A function that fails on a route has refused it. Scripts have
no files, network or clock, and are stopped after 200,000 steps.
Route selection now goes by a LOCAL_PREF on a route from an external
peer: the decoder never lets a peer's own through, so one that is there
was set by policy.
What it costs was measured before it was built on (benches/policy.rs):
a call that tests a few communities and changes two takes 1 to 3
microseconds, some two hundred times what Rust would. So a function is
asked once for the routes of an UPDATE, which share their attributes,
unless it asks for the prefix. One peer sending 1.23 M routes over a
session: 0.37 s without a function, 0.80 s with one that marks and
ranks every route, 3.4 s if it also looks at the prefix.
Export applies to what is announced, which today is what the daemon
originates; learned routes are not passed on yet.
Tested by unit tests of the crate, sessions end to end, a fuzz target on
the example script (testdata/policy.rhai), and a lab phase in which a
script changes which of FRR's and BIRD's route is chosen and what each
of them is told.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Where routes go into the kernel, a route the kernel could not be given
is no longer taken into the table: its next hop is no neighbor on any
interface, or its prefix is a network the machine is itself on. Such a
route used to be able to win selection and then leave its prefix without
a route in the kernel, though another peer had one that worked.
It is held aside the way an RPKI-invalid route is, and by the same
means: judged on arrival against the interfaces as the kernel last
described them, looked for in the table when they change, and taken
from what is held when its next hop can be reached. The fib publishes
the interfaces for this; the next hop is resolved once for an UPDATE
and not for each of its routes.
The lab has a phase, nexthops: FRR's better route with a next hop that
is nowhere is held and BIRD's is installed, and they change places as
an address comes and goes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What the code-health review found, and what a fuzz target written for it
found in its first seconds.
- A set of attributes with the same attribute twice is refused by the
routing table. Its readers (the selection key, the path, the next hop)
could each take a different one of the two. The decoder never lets one
through; the new fuzz target `attrs` found that the table would.
- bgp_peer_up{peer}, to alert on a peer going away, and bgp_build_info.
Every metric is there from the first scrape. Alerts for a peer down,
hijack attempts and kernel errors are in the README.
- `bgp --check config` validates a configuration without starting, and
`--version` says which build it is. CI runs the first on the example
configuration, which a test now also parses with its commented fields
put in; the example had to learn about the next hop check.
- Path attributes have a module of their own, packet::attributes,
re-exported from packet::message as before.
- policy::From, which shadowed the trait, is RouteSource. The server's
connection, dialer and fsm modules are private. Logging arguments live
in the binary; the server library no longer depends on clap.
- One is_unicast_link_local (the standard library's) instead of two of
our own; SelectionKey::origin(); one place where an Import is made.
- READMEs say what the daemon does today rather than what it did not do
a year ago. Documentation links are checked in CI. Unused dependencies
and bgp_web::serve are gone; the Docker build is --locked and does not
ship the fuzz corpus.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three independent reviews (hijack resistance, reliability, code health)
of the whole tree. This is what they found that could let a bad route in
or take the daemon down.
Hijacks:
- A route had two paths on a two-octet session, AS_PATH and AS4_PATH,
read in two places by two rules. A peer could make the origin checked
on import differ from the one checked at revalidation, and get an
RPKI-invalid route into the table. AS4_PATH is now merged into AS_PATH
once, on receipt (RFC 6793 §4.2.3), and nothing else knows of it.
- The next hop is judged on import: it must be usable, agreed on
(RFC 8950), and by default the peer itself.
- Our own originated prefixes are not taken from other networks.
- A route for a network the machine is on is never put in the kernel.
- Paths with an AS_SET are refused (RFC 9774).
- A configured peer gets one session, not one per BGP Identifier.
- More IPv6 special-purpose prefixes are bogons.
Reliability:
- A batch of requests the kernel refused could overflow the netlink
socket and end the daemon. The socket has room now, a lost answer or a
silent kernel means asking what is there, and neither is fatal.
- Working the routes out again within the grace period after a restart
no longer drops the routes being kept.
- That work can no longer be put off for ever by interface changes.
- Peers whose sessions keep failing are slowed down.
- Unused attribute sets are swept when they pile up, not only each
minute.
- Log lines a peer can cause are said once in a while.
- An RTR answer is bounded in size and time; data cannot expire between
two refreshes.
- The shard of a key is taken from a stirred hash: neighbouring bits of
the hash proved alike for prefixes in a row. The test that showed it,
failing one run in sixteen, is deterministic now.
The lab has a phase, hijacks, in which FRR tries the first three.
route::apply is a struct with methods instead of nested closures, and
the kernel task's loop is shorter.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's endpoint may be a link-local IPv6 address with the interface it
is on. Such a peer is the address and the interface together, wherever
peers are told apart: admission, collisions, hold-downs, the refusal log,
the status page. The interface is looked up by name each time it is
needed, so one that appears later or is made anew is found.
Our routes go out with our link-local address in both places of
MP_REACH_NLRI's next hop, as FRR sends its own. A peer's link-local next
hop is taken alone, twice, or after "::" (BIRD), and resolved in the
kernel on the interface of the session rather than by looking fe80::/64
up, which every interface has.
The lab has a new phase, linklocal: sessions with FRR (by interface, with
radvd beside the daemon) and BIRD over a LAN with link-local addresses
only, both families, pings across, the interface down and up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
For a link that has only IPv6 addresses.
- The Extended Next Hop capability is offered on every session over IPv6
whose families include IPv4 unicast, and used if the peer offers it too.
Nothing to configure.
- On such a session our IPv4 routes go out in MP_REACH_NLRI with the
session's IPv6 address as next hop, and are withdrawn in
MP_UNREACH_NLRI. Without the agreement they are not sent, as before.
- IPv4 routes are read with an IPv6 next hop, global or global with
link-local.
- The kernel installer resolves an IPv4 route's IPv6 next hop like any
other and installs it: by way of an IPv6 nexthop object, or with RTA_VIA
where routes carry their own gateway.
Tested against a real kernel in both modes, with a link-local next hop
too, and in the lab against FRR and BIRD: the new `v6only` phase takes
the IPv4 addresses off the exchange LAN, runs one IPv6 session with each
router for both families, and pings IPv4 across.
Sessions between link-local addresses are not supported yet.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
scripts/interop/lab.sh runs the daemon against FRR, BIRD and Routinator,
each in a network namespace on one exchange LAN, and checks what every
side ends up with: sessions and routes over IPv4 and IPv6 with their
attributes, four-octet AS numbers, ADD-PATH, RFC 9234 roles and Only to
Customer, RTR version 2 and invalid routes, route refresh, session resets,
killed peers, a cut wire, the daemon stopped and killed, a table of a
million routes and a restart under it, role mismatch and the prefix limit.
All of it passes against FRR 10.5.1, BIRD 2.18 and Routinator 0.15.2.
scripts/interop/real-rpki.sh checks the RTR client against a Routinator
with the real RPKI: 713,258 IPv4 and 187,479 IPv6 VRPs and 3,379 ASPAs,
the same numbers on both sides.
What the lab found, fixed here:
- A daemon restarted with its routes in the kernel removed them all at
once, its table being empty until the peers had sent theirs again, and
then put them back: no forwarding meanwhile. Routes of an earlier run
are now kept for `stale_seconds` (120 by default) and only those nobody
announced again are removed. A restart under a million routes takes
1.1 s to be in step, with nothing removed.
- FRR ends a session it loses to a collision with a state machine error
without a subcode, which was shown as "error code 5 subcode 0".
CI gets a job that runs the lab.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A million IPv4 routes took 1.6 s to install (release build); they take
1.1 s now, and 0.5 s to remove instead of 1.15 s. Most of what is left is
the kernel's own work on each route.
- Nexthop objects (Linux 5.3 and later): a gateway is put in the kernel
once and routes name it, so the kernel checks it once and not for every
route. The daemon makes its own objects, uses nobody else's, is told
when one of its own is removed and puts it back, and removes them when
it stops, which takes all the routes that name them along at once.
`without_nexthop_objects` turns this off; a kernel without them is
noticed and served as before.
- Writing to the kernel is a task of its own, fed through a short queue,
so the next batch is worked out while the kernel is busy with the last.
"In step" is said when the kernel has done the batch that completes the
table, not when it was handed over.
- What a next hop resolves to is kept, by next hop: it was looked up among
the machine's addresses for every route.
- A quicker hash for the plan's maps, and no lookups in the ones that are
empty.
Fixes a hole found on the way: when an interface went down and came back
within the second it takes us to react, the kernel had dropped our routes
and we did not put them back, since nothing about them had changed for
us. After such a change the kernel is now asked what it actually has.
scripts/in-namespace.sh runs one command in a scratch namespace;
FERRIX_ROUTES sets the size of the timing test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.
- netlink: the rtnetlink messages used (routes, addresses, links, errors),
as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
routes of ours found at startup are kept if still right and removed if
not; a change of interfaces has every route worked out again; a clean
stop removes our routes.
Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.
Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.
/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.
Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
/api/routes/received now returns one page: {routes, next_after,
prev_before}. Pages are in prefix order and begin at a prefix (after= or
before=), not at a count, so a table that changes between two pages does
not make them overlap or leave a gap. A page holds whole prefixes, about
`limit` routes' worth.
The table keeps prefixes in no order, so a page is found by looking at
every prefix once and keeping the nearest few; the routes of the others
are not touched (RibTable::for_each_prefix).
The status page shows about 100 routes with first / previous / next
links; the place is kept in the page's address with the lookup.
The response of /api/routes/received was an array before: this changes
its shape.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's configured role now does three more things:
- Our OPEN carries the BGP Role capability with what we are to the peer.
A peer whose own says it is something that does not pair with that, or
that sends two different roles, is refused with OPEN Message Error /
Role Mismatch. A peer that says nothing is accepted unless require_role
is set. A role on a peer in our own AS is a configuration error.
- Routes from a provider, a peer or a route server are marked Only to
Customer on the way in if they are not already, and our routes are
marked on the way out to a customer, a peer or a route server client.
- A marked route from a customer or route server client, or from a peer
with someone else's mark, is a leak and is refused. No policy setting
turns that off.
The packet crate gains the Only to Customer attribute (malformed:
treat-as-withdraw), and the Role Mismatch subcode is corrected from 8 to
11.
The status page shows whether the peer confirmed its role.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Origin validation says who may originate a prefix and nothing about the
rest of the path: it does not catch a route leak, nor a hijack that keeps
the right origin at the end of a made-up path. ASPAs, in which a network
lists its providers, do.
bgp-rpki:
- aspa: the ASPAs and the upstream and downstream verification procedures
of draft-ietf-sidrops-aspa-verification-24, tested against the draft's
wording as written.
- rtr: protocol version 2 (draft-ietf-sidrops-8210bis-23) with the ASPA
PDU and the new error codes.
- client: asks for version 2 and falls back to whatever older version the
cache speaks; keeps ASPAs with the VRPs, applied whole or not at all;
publishes which customers' ASPAs changed; tells the cache why it is
being left (Error Report); does not ask one version's serial of another.
Server:
- A peer has a role (provider, customer, peer, route server, route server
client), which decides how its paths are verified. Without one they are
not.
- A route whose path is invalid is refused ("ASPA invalid") and held, like
one with an invalid origin; accept_aspa_invalid exempts a peer.
- When ASPAs change, routes in the table whose path has a changed customer
on it are verified again, and held routes are released when they pass
both checks.
- The revalidator finds a route's peer by index instead of searching a
list for every route.
The status page and API show each peer's role, each route's path validity,
the number of ASPAs and the RTR version in use; bgp_rpki_aspas is on
/metricz.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With rpki { caches { ... } } in the configuration the server keeps the
validated ROA payloads of a validator and validates every route from a
peer outside its own AS (RFC 6811). An invalid route is refused, unless
its peer has accept_rpki_invalid. Valid and not-found routes are taken;
validity does not otherwise enter route selection.
Validity is not settled when a route arrives: the data changes every few
minutes. The routes follow, without the peer sending anything again.
- A route refused as invalid is held by its session, out of the table.
When the data changes the session looks at what it holds, and what is
no longer invalid goes into the table. A held route is the peer's to
withdraw or replace like any other.
- A route in the table that becomes invalid is found by a task that
looks through the table when the data changes, at the prefixes the
changed VRPs cover, and is taken out and held by its session.
- If the data expires because no cache confirms it, nothing is invalid
any more and everything held is taken.
- A session holds at most 200,000 routes; past that it asks the peer for
a route refresh when the data changes.
So that the table can be looked through without parsing attributes, an
attribute set's key now has the route's origin AS. Memory per route is
unchanged at 52 bytes on the real table.
Shown: /api/rpki and a section of the status page with the cache, its
state and the age of the data; each route's validity; each peer's count
of held routes. Refusals are recorded under "RPKI invalid". Metrics:
bgp_rpki_vrps and bgp_rpki_updates_total.
Also: Negotiated says whether the peer supports route refresh, and
docs/RFCs.md is brought up to date.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
