Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.
Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:
* The route is looked up in one kernel table, the "resolution table"
(`kernel { resolution_table: N }`, default 254 = main). The longest
prefix that contains the next hop wins; of several routes for that
prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
route is installed with the gateway and interface of the kernel route
that covers the next hop. A kernel route without a gateway (a route
onto a point-to-point or tunnel interface) makes the next hop itself
the gateway on that interface. A kernel route that names a nexthop
object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
own or another daemon's. A next hop that is reachable only through a
BGP route depends on another BGP next hop, and such chains are not
predictable.
* The default route does not resolve a next hop unless
`kernel { resolve_via_default_route: true }` is set. On a router the
default route points at a transit provider, and a next hop that is
reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
unresolvable, as does a route whose interface is down.
The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.
Implementation:
* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
and the kernel's nexthop objects into one value. The kernel task
publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
kernels get a dump of every table, filtered in user space). It asks
again, debounced, whenever a dedicated netlink socket that is a member
of the route multicast groups receives anything. That socket is given a
64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
installs on it too, up to a million at a time, and an ENOBUFS overflow
is treated as the same hint as a message. Its contents are never
parsed. An earlier attempt to receive route notifications on the
socket that writes routes failed, because the kernel only excludes the
requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
`Discarded`, which appear as the reasons in `/api/kernel` and on the
status page.
Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.
Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The documentation sources are Zola Markdown: front matter between +++
lines, shortcodes that insert the example configurations, and @/ internal
links. A code forge shows all three unrendered, which the README used to
deny. A docs/README.md now explains each, and the README points to it.
"Configuring BGP Neighbors" becomes "BGP Neighbors", as the other feature
pages are titled; "When a Neighbor Restarts" becomes "Neighbor Restart", a
noun phrase as the style guide asks.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What Ferrix is and does, that it routes AS210036, how to build it and try
it without a router, the security points an operator must know, and a note
that the code is written with an AI coding agent under the maintainer's
review.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
in the namespace rebuilt what kernel-tests.sh had just built. Each is
passed on only where set.
- The README's list of unsupported features still had route reflection
and graceful restart, both of which exist; the command-line reference
lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
unused variable; small things in the web crate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The documentation moves from READMEs into a site in docs/: getting
started, a user guide, operations, a reference, and development pages,
written in the style of network vendors' documentation (docs/STYLE.md).
bin/bgp/README.md and docs/RFCs.md are folded into it.
- A theme of its own in the International Typographic Style, in blues,
with self-hosted Inter and JetBrains Mono, search over Zola's index,
dark mode, and no resources loaded from other hosts.
- Example configurations in docs/examples/, shown on the pages and
checked with `bgp --check` in CI.
- A test that fails when a field of config.proto is missing from the
configuration reference.
- A CI job that builds the site; Zola fails on broken links and anchors.
- A hardened systemd unit, tried under systemd with kernel routes on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What was to be scripted is configuration, done natively:
- import { add_communities, add_large_communities, remove_communities,
local_pref }: every route taken from a peer is marked (where it was
learned, say), with nobody else's word for our marks taken, and given
its preference, which selection goes by.
- export { add_communities, add_large_communities, remove_communities,
as_path_prepend, learned_routes }, for a peer or for all that have none
of their own.
- control_communities: the communities by which routes say which peers
they are passed on to, as exchanges' route servers have them (0:P, L:P,
0:L, and the large forms L:0:P, L:1:P, L:0:0), taken off as the route
goes on.
And what the last of these is for: with export.learned_routes a peer is
told the best route of every prefix the others announced. The rules are
BGP's and cannot be turned off: not back where it came from, not to a
peer whose AS is on the path, not between two internal peers, not past
NO_EXPORT or NO_ADVERTISE; our AS on the path and ourselves as next hop
to another network, LOCAL_PREF to our own; to a route server's client
the path, MED and next hop as they came (RFC 7947). Which routes go where
follows from the roles of RFC 9234: a route that came down or across goes
on down only, so that with roles on every peer this makes us transit for
our customers and nothing else. A peer without a role is told everything,
and the daemon says so at startup.
Each session follows the table's best routes with a reader and keeps the
prefixes it has announced, so that it can withdraw them. Routes that go
out alike go in one UPDATE: changes are gathered for 50 ms first, and a
table that is given out anew is gathered whole (at most four sessions at
a time; it is tens of megabytes). For that to work the table's journal
had to give out together what was written together: every change now
carries a time on a table-wide clock, a reader's pass reads every shard
up to one time, no further than the changes that are complete in every
shard, and a resync takes its places in the journals as of one time.
One peer sending 1.23 M routes over a session, another told of them:
in our table after 0.47 s, all heard after 2.1 s, four routes an UPDATE.
The lab has a phase, transit: FRR as our customer and BIRD as our
provider, routes each way with the right paths and marks, one kept back
by a community, and pings between their networks through us.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rhai made a decision about a route cost one to three microseconds,
some two hundred times what the same costs in Rust. That was bearable
where a function is asked once for an UPDATE, and is not where every
route is weighed against every peer, as it is when routes are passed
on. What is wanted of a policy (mark routes by where they came from,
mark what is announced, let communities say who is told what) needs no
language; it is to be configuration, done natively.
Kept from that work: route selection goes by a LOCAL_PREF that policy
set on a route from an external peer, and the lab's configuration can
be added to by a phase.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A new crate, bgp-policy, runs functions of a script on routes: on their
way in, after the general checks of a peer's import and before the
checks that can change (RPKI, next hop reachability); and on their way
out to a peer. A function is given the route and the peer, may refuse
the route, and may change its communities, large communities,
LOCAL_PREF and MED, and on the way out prepend our AS:
fn from_peer(route, peer) {
if route.has_community("65535:666") { return false; }
route.add_community(LEARNED_FROM[peer.role]);
route.local_pref = PREFERENCE[peer.role];
true
}
Configuration: policy { path | source, import_function, export_function }
and, for a peer, import { function } and export { function }. The script
is compiled at startup; `bgp --check` reads it and says which function
is whose. A function that fails on a route has refused it. Scripts have
no files, network or clock, and are stopped after 200,000 steps.
Route selection now goes by a LOCAL_PREF on a route from an external
peer: the decoder never lets a peer's own through, so one that is there
was set by policy.
What it costs was measured before it was built on (benches/policy.rs):
a call that tests a few communities and changes two takes 1 to 3
microseconds, some two hundred times what Rust would. So a function is
asked once for the routes of an UPDATE, which share their attributes,
unless it asks for the prefix. One peer sending 1.23 M routes over a
session: 0.37 s without a function, 0.80 s with one that marks and
ranks every route, 3.4 s if it also looks at the prefix.
Export applies to what is announced, which today is what the daemon
originates; learned routes are not passed on yet.
Tested by unit tests of the crate, sessions end to end, a fuzz target on
the example script (testdata/policy.rhai), and a lab phase in which a
script changes which of FRR's and BIRD's route is chosen and what each
of them is told.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
What the code-health review found, and what a fuzz target written for it
found in its first seconds.
- A set of attributes with the same attribute twice is refused by the
routing table. Its readers (the selection key, the path, the next hop)
could each take a different one of the two. The decoder never lets one
through; the new fuzz target `attrs` found that the table would.
- bgp_peer_up{peer}, to alert on a peer going away, and bgp_build_info.
Every metric is there from the first scrape. Alerts for a peer down,
hijack attempts and kernel errors are in the README.
- `bgp --check config` validates a configuration without starting, and
`--version` says which build it is. CI runs the first on the example
configuration, which a test now also parses with its commented fields
put in; the example had to learn about the next hop check.
- Path attributes have a module of their own, packet::attributes,
re-exported from packet::message as before.
- policy::From, which shadowed the trait, is RouteSource. The server's
connection, dialer and fsm modules are private. Logging arguments live
in the binary; the server library no longer depends on clap.
- One is_unicast_link_local (the standard library's) instead of two of
our own; SelectionKey::origin(); one place where an Import is made.
- READMEs say what the daemon does today rather than what it did not do
a year ago. Documentation links are checked in CI. Unused dependencies
and bgp_web::serve are gone; the Docker build is --locked and does not
ship the fuzz corpus.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The work of the speaker-cleanup branch as one commit; the branch keeps
the step-by-step history.
Sessions
- The state machine is a pure function from events to actions, driven by
one task per connection. Idle, Connect and Active belong to a dialer
that keeps configured peers up with backoff.
- Capabilities are negotiated: address families, four-octet AS numbers
with AS_TRANS and AS4_PATH, route refresh, ADD-PATH per family and
direction.
- Collision detection per RFC 4271 section 6.8, by address and
identifier.
- Connections are taken only from configured peers, a few per address.
- A panic in a session is contained and answered with a Cease; a core
task that stops shuts the server down with a Cease to every peer.
- Writes that make no progress are given up (RFC 9687).
Codec
- Malformed UPDATEs are handled as RFC 7606 says: treat-as-withdraw,
attribute discard, or session reset, with the NOTIFICATION to send.
- Address families are open types; any can be named and ignored.
- Prefixes are fixed-size values, normalised, with path identifiers.
- Lengths are filled in after writing; one definition of each type code.
- Property tests and fuzz targets for every decoder.
Routing table
- Adj-RIBs-In and Loc-RIB in one sharded table that sessions write to
directly. Routes are sixteen bytes; attribute sets are interned in
wire form. Best-path selection per RFC 4271 section 9.1.2.2 with
deterministic MED.
- A journal per shard lets followers read changes at their own pace;
one that falls behind is resynchronised.
- On a RIPE RIS dump of 148 peers and 39.3M routes: loaded in 2.0 s on
12 cores, about 52 bytes a route.
Announcing
- Originated routes come from the configuration. Each session has its
own exporter with an Adj-RIB-Out, so a slow peer delays only itself.
- Routes to a peer in our own AS have an empty path and a LOCAL_PREF.
Operations
- HTTP status page, JSON API, event stream and Prometheus metrics.
- Benchmarks on synthetic and real tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
