Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:
L:11:P .. L:13:P our AS once to three times more towards AS P
(P 0: towards every peer in another AS)
L:20:G / L:21:G not to / only to the sessions of group G
L:31:G .. L:33:G our AS once to three times more towards group G
A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.
They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.
And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.
This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.
When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.
When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.
The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What was to be scripted is configuration, done natively:
- import { add_communities, add_large_communities, remove_communities,
local_pref }: every route taken from a peer is marked (where it was
learned, say), with nobody else's word for our marks taken, and given
its preference, which selection goes by.
- export { add_communities, add_large_communities, remove_communities,
as_path_prepend, learned_routes }, for a peer or for all that have none
of their own.
- control_communities: the communities by which routes say which peers
they are passed on to, as exchanges' route servers have them (0:P, L:P,
0:L, and the large forms L:0:P, L:1:P, L:0:0), taken off as the route
goes on.
And what the last of these is for: with export.learned_routes a peer is
told the best route of every prefix the others announced. The rules are
BGP's and cannot be turned off: not back where it came from, not to a
peer whose AS is on the path, not between two internal peers, not past
NO_EXPORT or NO_ADVERTISE; our AS on the path and ourselves as next hop
to another network, LOCAL_PREF to our own; to a route server's client
the path, MED and next hop as they came (RFC 7947). Which routes go where
follows from the roles of RFC 9234: a route that came down or across goes
on down only, so that with roles on every peer this makes us transit for
our customers and nothing else. A peer without a role is told everything,
and the daemon says so at startup.
Each session follows the table's best routes with a reader and keeps the
prefixes it has announced, so that it can withdraw them. Routes that go
out alike go in one UPDATE: changes are gathered for 50 ms first, and a
table that is given out anew is gathered whole (at most four sessions at
a time; it is tens of megabytes). For that to work the table's journal
had to give out together what was written together: every change now
carries a time on a table-wide clock, a reader's pass reads every shard
up to one time, no further than the changes that are complete in every
shard, and a resync takes its places in the journals as of one time.
One peer sending 1.23 M routes over a session, another told of them:
in our table after 0.47 s, all heard after 2.1 s, four routes an UPDATE.
The lab has a phase, transit: FRR as our customer and BIRD as our
provider, routes each way with the right paths and marks, one kept back
by a community, and pings between their networks through us.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rhai made a decision about a route cost one to three microseconds,
some two hundred times what the same costs in Rust. That was bearable
where a function is asked once for an UPDATE, and is not where every
route is weighed against every peer, as it is when routes are passed
on. What is wanted of a policy (mark routes by where they came from,
mark what is announced, let communities say who is told what) needs no
language; it is to be configuration, done natively.
Kept from that work: route selection goes by a LOCAL_PREF that policy
set on a route from an external peer, and the lab's configuration can
be added to by a phase.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new crate, bgp-policy, runs functions of a script on routes: on their
way in, after the general checks of a peer's import and before the
checks that can change (RPKI, next hop reachability); and on their way
out to a peer. A function is given the route and the peer, may refuse
the route, and may change its communities, large communities,
LOCAL_PREF and MED, and on the way out prepend our AS:
fn from_peer(route, peer) {
if route.has_community("65535:666") { return false; }
route.add_community(LEARNED_FROM[peer.role]);
route.local_pref = PREFERENCE[peer.role];
true
}
Configuration: policy { path | source, import_function, export_function }
and, for a peer, import { function } and export { function }. The script
is compiled at startup; `bgp --check` reads it and says which function
is whose. A function that fails on a route has refused it. Scripts have
no files, network or clock, and are stopped after 200,000 steps.
Route selection now goes by a LOCAL_PREF on a route from an external
peer: the decoder never lets a peer's own through, so one that is there
was set by policy.
What it costs was measured before it was built on (benches/policy.rs):
a call that tests a few communities and changes two takes 1 to 3
microseconds, some two hundred times what Rust would. So a function is
asked once for the routes of an UPDATE, which share their attributes,
unless it asks for the prefix. One peer sending 1.23 M routes over a
session: 0.37 s without a function, 0.80 s with one that marks and
ranks every route, 3.4 s if it also looks at the prefix.
Export applies to what is announced, which today is what the daemon
originates; learned routes are not passed on yet.
Tested by unit tests of the crate, sessions end to end, a fuzz target on
the example script (testdata/policy.rhai), and a lab phase in which a
script changes which of FRR's and BIRD's route is chosen and what each
of them is told.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three independent reviews (hijack resistance, reliability, code health)
of the whole tree. This is what they found that could let a bad route in
or take the daemon down.
Hijacks:
- A route had two paths on a two-octet session, AS_PATH and AS4_PATH,
read in two places by two rules. A peer could make the origin checked
on import differ from the one checked at revalidation, and get an
RPKI-invalid route into the table. AS4_PATH is now merged into AS_PATH
once, on receipt (RFC 6793 §4.2.3), and nothing else knows of it.
- The next hop is judged on import: it must be usable, agreed on
(RFC 8950), and by default the peer itself.
- Our own originated prefixes are not taken from other networks.
- A route for a network the machine is on is never put in the kernel.
- Paths with an AS_SET are refused (RFC 9774).
- A configured peer gets one session, not one per BGP Identifier.
- More IPv6 special-purpose prefixes are bogons.
Reliability:
- A batch of requests the kernel refused could overflow the netlink
socket and end the daemon. The socket has room now, a lost answer or a
silent kernel means asking what is there, and neither is fatal.
- Working the routes out again within the grace period after a restart
no longer drops the routes being kept.
- That work can no longer be put off for ever by interface changes.
- Peers whose sessions keep failing are slowed down.
- Unused attribute sets are swept when they pile up, not only each
minute.
- Log lines a peer can cause are said once in a while.
- An RTR answer is bounded in size and time; data cannot expire between
two refreshes.
- The shard of a key is taken from a stirred hash: neighbouring bits of
the hash proved alike for prefixes in a row. The test that showed it,
failing one run in sixteen, is deterministic now.
The lab has a phase, hijacks, in which FRR tries the first three.
route::apply is a struct with methods instead of nested closures, and
the kernel task's loop is shorter.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's endpoint may be a link-local IPv6 address with the interface it
is on. Such a peer is the address and the interface together, wherever
peers are told apart: admission, collisions, hold-downs, the refusal log,
the status page. The interface is looked up by name each time it is
needed, so one that appears later or is made anew is found.
Our routes go out with our link-local address in both places of
MP_REACH_NLRI's next hop, as FRR sends its own. A peer's link-local next
hop is taken alone, twice, or after "::" (BIRD), and resolved in the
kernel on the interface of the session rather than by looking fe80::/64
up, which every interface has.
The lab has a new phase, linklocal: sessions with FRR (by interface, with
radvd beside the daemon) and BIRD over a LAN with link-local addresses
only, both families, pings across, the interface down and up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
scripts/interop/lab.sh runs the daemon against FRR, BIRD and Routinator,
each in a network namespace on one exchange LAN, and checks what every
side ends up with: sessions and routes over IPv4 and IPv6 with their
attributes, four-octet AS numbers, ADD-PATH, RFC 9234 roles and Only to
Customer, RTR version 2 and invalid routes, route refresh, session resets,
killed peers, a cut wire, the daemon stopped and killed, a table of a
million routes and a restart under it, role mismatch and the prefix limit.
All of it passes against FRR 10.5.1, BIRD 2.18 and Routinator 0.15.2.
scripts/interop/real-rpki.sh checks the RTR client against a Routinator
with the real RPKI: 713,258 IPv4 and 187,479 IPv6 VRPs and 3,379 ASPAs,
the same numbers on both sides.
What the lab found, fixed here:
- A daemon restarted with its routes in the kernel removed them all at
once, its table being empty until the peers had sent theirs again, and
then put them back: no forwarding meanwhile. Routes of an earlier run
are now kept for `stale_seconds` (120 by default) and only those nobody
announced again are removed. A restart under a million routes takes
1.1 s to be in step, with nothing removed.
- FRR ends a session it loses to a collision with a state machine error
without a subcode, which was shown as "error code 5 subcode 0".
CI gets a job that runs the lab.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A million IPv4 routes took 1.6 s to install (release build); they take
1.1 s now, and 0.5 s to remove instead of 1.15 s. Most of what is left is
the kernel's own work on each route.
- Nexthop objects (Linux 5.3 and later): a gateway is put in the kernel
once and routes name it, so the kernel checks it once and not for every
route. The daemon makes its own objects, uses nobody else's, is told
when one of its own is removed and puts it back, and removes them when
it stops, which takes all the routes that name them along at once.
`without_nexthop_objects` turns this off; a kernel without them is
noticed and served as before.
- Writing to the kernel is a task of its own, fed through a short queue,
so the next batch is worked out while the kernel is busy with the last.
"In step" is said when the kernel has done the batch that completes the
table, not when it was handed over.
- What a next hop resolves to is kept, by next hop: it was looked up among
the machine's addresses for every route.
- A quicker hash for the plan's maps, and no lookups in the ones that are
empty.
Fixes a hole found on the way: when an interface went down and came back
within the second it takes us to react, the kernel had dropped our routes
and we did not put them back, since nothing about them had changed for
us. After such a change the kernel is now asked what it actually has.
scripts/in-namespace.sh runs one command in a scratch namespace;
FERRIX_ROUTES sets the size of the timing test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.
- netlink: the rtnetlink messages used (routes, addresses, links, errors),
as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
routes of ours found at startup are kept if still right and removed if
not; a change of interfaces has every route worked out again; a clean
stop removes our routes.
Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.
Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.
/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.
Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's configured role now does three more things:
- Our OPEN carries the BGP Role capability with what we are to the peer.
A peer whose own says it is something that does not pair with that, or
that sends two different roles, is refused with OPEN Message Error /
Role Mismatch. A peer that says nothing is accepted unless require_role
is set. A role on a peer in our own AS is a configuration error.
- Routes from a provider, a peer or a route server are marked Only to
Customer on the way in if they are not already, and our routes are
marked on the way out to a customer, a peer or a route server client.
- A marked route from a customer or route server client, or from a peer
with someone else's mark, is a leak and is refused. No policy setting
turns that off.
The packet crate gains the Only to Customer attribute (malformed:
treat-as-withdraw), and the Role Mismatch subcode is corrected from 8 to
11.
The status page shows whether the peer confirmed its role.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Origin validation says who may originate a prefix and nothing about the
rest of the path: it does not catch a route leak, nor a hijack that keeps
the right origin at the end of a made-up path. ASPAs, in which a network
lists its providers, do.
bgp-rpki:
- aspa: the ASPAs and the upstream and downstream verification procedures
of draft-ietf-sidrops-aspa-verification-24, tested against the draft's
wording as written.
- rtr: protocol version 2 (draft-ietf-sidrops-8210bis-23) with the ASPA
PDU and the new error codes.
- client: asks for version 2 and falls back to whatever older version the
cache speaks; keeps ASPAs with the VRPs, applied whole or not at all;
publishes which customers' ASPAs changed; tells the cache why it is
being left (Error Report); does not ask one version's serial of another.
Server:
- A peer has a role (provider, customer, peer, route server, route server
client), which decides how its paths are verified. Without one they are
not.
- A route whose path is invalid is refused ("ASPA invalid") and held, like
one with an invalid origin; accept_aspa_invalid exempts a peer.
- When ASPAs change, routes in the table whose path has a changed customer
on it are verified again, and held routes are released when they pass
both checks.
- The revalidator finds a route's peer by index instead of searching a
list for every route.
The status page and API show each peer's role, each route's path validity,
the number of ASPAs and the RTR version in use; bgp_rpki_aspas is on
/metricz.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
With rpki { caches { ... } } in the configuration the server keeps the
validated ROA payloads of a validator and validates every route from a
peer outside its own AS (RFC 6811). An invalid route is refused, unless
its peer has accept_rpki_invalid. Valid and not-found routes are taken;
validity does not otherwise enter route selection.
Validity is not settled when a route arrives: the data changes every few
minutes. The routes follow, without the peer sending anything again.
- A route refused as invalid is held by its session, out of the table.
When the data changes the session looks at what it holds, and what is
no longer invalid goes into the table. A held route is the peer's to
withdraw or replace like any other.
- A route in the table that becomes invalid is found by a task that
looks through the table when the data changes, at the prefixes the
changed VRPs cover, and is taken out and held by its session.
- If the data expires because no cache confirms it, nothing is invalid
any more and everything held is taken.
- A session holds at most 200,000 routes; past that it asks the peer for
a route refresh when the data changes.
So that the table can be looked through without parsing attributes, an
attribute set's key now has the route's origin AS. Memory per route is
unchanged at 52 bytes on the real table.
Shown: /api/rpki and a section of the status page with the cache, its
state and the age of the data; each route's validity; each peer's count
of held routes. Refusals are recorded under "RPKI invalid". Metrics:
bgp_rpki_vrps and bgp_rpki_updates_total.
Also: Negotiated says whether the peer supports route refresh, and
docs/RFCs.md is brought up to date.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A configured peer could announce any prefix with any path, in any
number, and all of it was stored and could be chosen as best. Each
peer now has an import policy, and every route goes through it before
it reaches the routing table.
Checks that hold for any peer unless turned off for it:
- no prefixes that are not routed on the Internet (private, loopback,
link-local, documentation, multicast, reserved; for IPv6, anything
outside global unicast);
- nothing longer than /24 or /48, no default route, nothing shorter
than /8 or /16;
- no reserved, private or documentation AS number in the path;
- a path that begins with the peer's own AS (turned off for a route
server), and of at most 64 AS numbers.
Always: a route with our own AS number in its path is refused. RFC 4271
requires this loop check and it was missing.
Per peer, if configured: the prefixes it may announce, each with an
optional longest more-specific, and the AS numbers its routes may
originate from. This is the list generated from the routing registries.
max_prefixes limits how many routes a peer may have. One more ends the
session with Cease / Maximum Number of Prefixes Reached, and the peer
is kept down for hold_down_seconds (ten minutes unless set): its
connections are not taken and it is not dialled. Refused routes do not
count towards the limit.
A refused route is withdrawn rather than ignored, so that it replaces
whatever the peer had announced for it before.
On a session with two-octet AS numbers the path is judged with its
AS4_PATH merged in, so that the real AS numbers are what is checked.
Peers that are not configured (accept_unconfigured_peers, for test
networks) get a policy that takes everything but loops.
Refusals are counted in bgp_routes_rejected_total by reason. The status
page and /api/peers show, for each session, how many routes the peer
has and how many were refused. The server warns at startup about peers
with no limit.
The prefix check costs about 4 ns a route with the default policy; the
end-to-end benchmark is unchanged.
Also: the benchmark's server AS is one that is in none of its generated
paths, and a table test that depended on the hasher's seed is fixed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The work of the speaker-cleanup branch as one commit; the branch keeps
the step-by-step history.
Sessions
- The state machine is a pure function from events to actions, driven by
one task per connection. Idle, Connect and Active belong to a dialer
that keeps configured peers up with backoff.
- Capabilities are negotiated: address families, four-octet AS numbers
with AS_TRANS and AS4_PATH, route refresh, ADD-PATH per family and
direction.
- Collision detection per RFC 4271 section 6.8, by address and
identifier.
- Connections are taken only from configured peers, a few per address.
- A panic in a session is contained and answered with a Cease; a core
task that stops shuts the server down with a Cease to every peer.
- Writes that make no progress are given up (RFC 9687).
Codec
- Malformed UPDATEs are handled as RFC 7606 says: treat-as-withdraw,
attribute discard, or session reset, with the NOTIFICATION to send.
- Address families are open types; any can be named and ignored.
- Prefixes are fixed-size values, normalised, with path identifiers.
- Lengths are filled in after writing; one definition of each type code.
- Property tests and fuzz targets for every decoder.
Routing table
- Adj-RIBs-In and Loc-RIB in one sharded table that sessions write to
directly. Routes are sixteen bytes; attribute sets are interned in
wire form. Best-path selection per RFC 4271 section 9.1.2.2 with
deterministic MED.
- A journal per shard lets followers read changes at their own pace;
one that falls behind is resynchronised.
- On a RIPE RIS dump of 148 peers and 39.3M routes: loaded in 2.0 s on
12 cores, about 52 bytes a route.
Announcing
- Originated routes come from the configuration. Each session has its
own exporter with an Adj-RIB-Out, so a slow peer delays only itself.
- Routes to a peer in our own AS have an empty path and a LOCAL_PREF.
Operations
- HTTP status page, JSON API, event stream and Prometheus metrics.
- Benchmarks on synthetic and real tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- Implement passive-side BGP FSM (OpenSent → OpenConfirm → Established)
with hold timer, keepalive timer, and NOTIFICATION on expiry
- Add bgp-server crate: config validation, TCP listener accept loop
(BgpServerBuilder → BgpServer), connection dispatch
- Add bgp-rib crate: Adj-RIB-In actor owning route state with event
broadcast for streaming consumers and snapshot for introspection
- Wire Adj-RIB-In into FSM: announce/withdraw on UPDATE for both legacy
IPv4 NLRI fields and MP_REACH/MP_UNREACH path attributes
- Add bgp-integration-tests crate with 15 end-to-end tests covering
session establishment, timers, UPDATE handling, and RIB introspection
- Fix require_empty! macro bug in open.rs (checked wrong buffer)
- Fix FourByteAsn capability parser returning consumed buffer as remainder
- Fix tracing subscriber not being installed in TracingArgs::install
- Expand docs/RFCs.md with full implementation status across 15 RFCs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
