Three independent reviews (hijack resistance, reliability, code health)
of the whole tree. This is what they found that could let a bad route in
or take the daemon down.
Hijacks:
- A route had two paths on a two-octet session, AS_PATH and AS4_PATH,
read in two places by two rules. A peer could make the origin checked
on import differ from the one checked at revalidation, and get an
RPKI-invalid route into the table. AS4_PATH is now merged into AS_PATH
once, on receipt (RFC 6793 §4.2.3), and nothing else knows of it.
- The next hop is judged on import: it must be usable, agreed on
(RFC 8950), and by default the peer itself.
- Our own originated prefixes are not taken from other networks.
- A route for a network the machine is on is never put in the kernel.
- Paths with an AS_SET are refused (RFC 9774).
- A configured peer gets one session, not one per BGP Identifier.
- More IPv6 special-purpose prefixes are bogons.
Reliability:
- A batch of requests the kernel refused could overflow the netlink
socket and end the daemon. The socket has room now, a lost answer or a
silent kernel means asking what is there, and neither is fatal.
- Working the routes out again within the grace period after a restart
no longer drops the routes being kept.
- That work can no longer be put off for ever by interface changes.
- Peers whose sessions keep failing are slowed down.
- Unused attribute sets are swept when they pile up, not only each
minute.
- Log lines a peer can cause are said once in a while.
- An RTR answer is bounded in size and time; data cannot expire between
two refreshes.
- The shard of a key is taken from a stirred hash: neighbouring bits of
the hash proved alike for prefixes in a row. The test that showed it,
failing one run in sixteen, is deterministic now.
The lab has a phase, hijacks, in which FRR tries the first three.
route::apply is a struct with methods instead of nested closures, and
the kernel task's loop is shorter.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A peer's endpoint may be a link-local IPv6 address with the interface it
is on. Such a peer is the address and the interface together, wherever
peers are told apart: admission, collisions, hold-downs, the refusal log,
the status page. The interface is looked up by name each time it is
needed, so one that appears later or is made anew is found.
Our routes go out with our link-local address in both places of
MP_REACH_NLRI's next hop, as FRR sends its own. A peer's link-local next
hop is taken alone, twice, or after "::" (BIRD), and resolved in the
kernel on the interface of the session rather than by looking fe80::/64
up, which every interface has.
The lab has a new phase, linklocal: sessions with FRR (by interface, with
radvd beside the daemon) and BIRD over a LAN with link-local addresses
only, both families, pings across, the interface down and up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
For a link that has only IPv6 addresses.
- The Extended Next Hop capability is offered on every session over IPv6
whose families include IPv4 unicast, and used if the peer offers it too.
Nothing to configure.
- On such a session our IPv4 routes go out in MP_REACH_NLRI with the
session's IPv6 address as next hop, and are withdrawn in
MP_UNREACH_NLRI. Without the agreement they are not sent, as before.
- IPv4 routes are read with an IPv6 next hop, global or global with
link-local.
- The kernel installer resolves an IPv4 route's IPv6 next hop like any
other and installs it: by way of an IPv6 nexthop object, or with RTA_VIA
where routes carry their own gateway.
Tested against a real kernel in both modes, with a link-local next hop
too, and in the lab against FRR and BIRD: the new `v6only` phase takes
the IPv4 addresses off the exchange LAN, runs one IPv6 session with each
router for both families, and pings IPv4 across.
Sessions between link-local addresses are not supported yet.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
scripts/interop/lab.sh runs the daemon against FRR, BIRD and Routinator,
each in a network namespace on one exchange LAN, and checks what every
side ends up with: sessions and routes over IPv4 and IPv6 with their
attributes, four-octet AS numbers, ADD-PATH, RFC 9234 roles and Only to
Customer, RTR version 2 and invalid routes, route refresh, session resets,
killed peers, a cut wire, the daemon stopped and killed, a table of a
million routes and a restart under it, role mismatch and the prefix limit.
All of it passes against FRR 10.5.1, BIRD 2.18 and Routinator 0.15.2.
scripts/interop/real-rpki.sh checks the RTR client against a Routinator
with the real RPKI: 713,258 IPv4 and 187,479 IPv6 VRPs and 3,379 ASPAs,
the same numbers on both sides.
What the lab found, fixed here:
- A daemon restarted with its routes in the kernel removed them all at
once, its table being empty until the peers had sent theirs again, and
then put them back: no forwarding meanwhile. Routes of an earlier run
are now kept for `stale_seconds` (120 by default) and only those nobody
announced again are removed. A restart under a million routes takes
1.1 s to be in step, with nothing removed.
- FRR ends a session it loses to a collision with a state machine error
without a subcode, which was shown as "error code 5 subcode 0".
CI gets a job that runs the lab.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A million IPv4 routes took 1.6 s to install (release build); they take
1.1 s now, and 0.5 s to remove instead of 1.15 s. Most of what is left is
the kernel's own work on each route.
- Nexthop objects (Linux 5.3 and later): a gateway is put in the kernel
once and routes name it, so the kernel checks it once and not for every
route. The daemon makes its own objects, uses nobody else's, is told
when one of its own is removed and puts it back, and removes them when
it stops, which takes all the routes that name them along at once.
`without_nexthop_objects` turns this off; a kernel without them is
noticed and served as before.
- Writing to the kernel is a task of its own, fed through a short queue,
so the next batch is worked out while the kernel is busy with the last.
"In step" is said when the kernel has done the batch that completes the
table, not when it was handed over.
- What a next hop resolves to is kept, by next hop: it was looked up among
the machine's addresses for every route.
- A quicker hash for the plan's maps, and no lookups in the ones that are
empty.
Fixes a hole found on the way: when an interface went down and came back
within the second it takes us to react, the kernel had dropped our routes
and we did not put them back, since nothing about them had changed for
us. After such a change the kernel is now asked what it actually has.
scripts/in-namespace.sh runs one command in a scratch namespace;
FERRIX_ROUTES sets the size of the timing test.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.
- netlink: the rtnetlink messages used (routes, addresses, links, errors),
as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
routes of ours found at startup are kept if still right and removed if
not; a change of interfaces has every route worked out again; a clean
stop removes our routes.
Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.
Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.
/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.
Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
