ferrix/docs/content/docs/development/architecture.md at 48da8cc7d253a60fdf98f9a69eaa6e679ed32016

0 Fork 0

Files

T

Rayhaan JaufeerallyandClaude Fable 5.1 3608897245 fib: resolve next hops through the kernel's routing table

Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.

Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:

* The route is looked up in one kernel table, the "resolution table"
  (`kernel { resolution_table: N }`, default 254 = main). The longest
  prefix that contains the next hop wins; of several routes for that
  prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
  route is installed with the gateway and interface of the kernel route
  that covers the next hop. A kernel route without a gateway (a route
  onto a point-to-point or tunnel interface) makes the next hop itself
  the gateway on that interface. A kernel route that names a nexthop
  object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
  own or another daemon's. A next hop that is reachable only through a
  BGP route depends on another BGP next hop, and such chains are not
  predictable.
* The default route does not resolve a next hop unless
  `kernel { resolve_via_default_route: true }` is set. On a router the
  default route points at a transit provider, and a next hop that is
  reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
  unresolvable, as does a route whose interface is down.

The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.

Implementation:

* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
  table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
  and the kernel's nexthop objects into one value. The kernel task
  publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
  and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
  alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
  kernels get a dump of every table, filtered in user space). It asks
  again, debounced, whenever a dedicated netlink socket that is a member
  of the route multicast groups receives anything. That socket is given a
  64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
  installs on it too, up to a million at a time, and an ENOBUFS overflow
  is treated as the same hint as a message. Its contents are never
  parsed. An earlier attempt to receive route notifications on the
  socket that writes routes failed, because the kernel only excludes the
  requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
  `Discarded`, which appear as the reasons in `/api/kernel` and on the
  status page.

Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.

Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-10-08 22:23:27 +02:00

9.7 KiB

Raw Blame History

+++ title = "Architecture Overview" description = "Describes the internal components of Ferrix and how a route is processed from reception to kernel installation and advertisement." weight = 1

[extra] nav = "Architecture" +++

This topic describes the internal components of Ferrix and how they process a route. It is intended for developers who work on the source code. Figure 1 shows the components and the path of a route through them.

Figure 1: Route processing in Ferrix

Design Principles

Ferrix is built on the following principles:

Memory safety. The workspace lints forbid unsafe code in every crate.
No central routing table task. Each session task applies the routes of an UPDATE message to the routing table directly, and locks only the shards that contain the affected prefixes. Sessions that update different shards do not wait for each other. A session that must wait is slowed down, which provides back-pressure without an unbounded queue in front of the routing table.
Independent consumers. The components that act on best path changes (the export task of each session, kernel route installation, and the status page) read the changes from a journal at their own rate. A slow consumer does not delay the routing table or other consumers.
Fault isolation. A panic ends only the task in which it occurs. If a session task panics, Ferrix closes the session, notifies the neighbor, and removes the neighbor's routes. Shared tables use locks that are not poisoned by a panic. The daemon exits only if it loses a component that it cannot operate without.
Resistance to neighbor-triggered load. Ferrix rate-limits log messages that a neighbor can trigger, closes connections from unconfigured addresses before reading from them, and places flapping neighbors in hold-down.

Inbound Route Processing

Connection Management

The dialer (server::dialer) initiates a connection to each configured neighbor and retries with exponential backoff. The listener accepts connections from the addresses of configured neighbors only. The collision module (server::collision) retains one connection for each neighbor when both speakers initiate a connection (RFC 4271 Section 6.8).

BGP Finite State Machine

The BGP finite state machine (server::fsm) implements RFC 4271 Section 8 without input or output. It receives events and returns the actions to perform, so that each state transition can be tested with a function call. The connection driver (server::connection) converts received bytes and timer expirations into events, and performs the returned actions.

Message Decoding

The packet crate decodes and encodes BGP messages. Errors in UPDATE messages are handled as specified in RFC 7606: depending on the error, the attribute is discarded or the routes are treated as withdrawn. All parsers are fuzz-tested.

Import Processing

The server::routes module applies the routes of an UPDATE message to the routing table, according to the neighbor's import policy:

server::policy evaluates each route against the import policy.
server::rpki validates the route against the data that the RTR client (rpki crate) receives.
server::reach determines whether the next hop is resolvable.

A route that is rejected for a reason that can change without a new UPDATE (RPKI invalid, ASPA invalid, or an unresolvable next hop) is stored as a held route. Ferrix reevaluates held routes when the RPKI data or the interfaces change.

Routing Table

The routing table (rib::table) combines the Adj-RIBs-In and the Loc-RIB (RFC 4271 Section 3.2) in one structure, organized by prefix. The entry for a prefix contains the routes from all neighbors and identifies the best path. After a change, best path selection examines only the entry for the affected prefix. The routing table is designed for hundreds of full routing tables:

A route occupies 16 bytes: a pointer to its path attribute set, and one word that encodes the neighbor and the first steps of best path selection as a single comparable value.
Path attribute sets are stored once and shared by all routes with identical attributes. Unused sets are removed periodically.
The table is divided into shards by a hash of the prefix. Each shard has its own lock.

Outbound Route Processing

Change Journal

Each shard of the routing table maintains a journal of the prefixes whose best path changed. Each consumer reads the journal through its own reader. The journal records that the best path of a prefix changed, not the new best path; the reader retrieves the current best path when it processes the entry. A reader that falls too far behind compares its state with the routing table instead. Changes made together carry the same table-wide sequence stamp, so that a reader processes them together and they are advertised in the same UPDATE messages.

Route Advertisement

The server::advertise module runs an export task for each session. The task advertises the originated routes and, if the export policy enables learned_routes, the best paths learned from other neighbors, as the server::export module permits for each route. The task records the routes that it has advertised, groups routes with identical attributes into one UPDATE message, and waits when the session's output queue is full.

Kernel Route Installation

The fib crate reads best path changes and maintains the kernel routing table over rtnetlink, with one route for each prefix through the resolved next hop of its best path. It also receives the router's interfaces, addresses and the routes of the resolution table from the kernel, which determine whether and how a next hop is resolvable; the import policy consults the same view, so that a route with an unresolvable next hop is held rather than selected.

HTTP Interface

The web crate provides read-only access to the state owned by the server. It does not store routing state.

Crates

Table 1 lists the crates of the workspace.

Crate	Contents
`crates/primitives`	Prefixes, address families, and parsing utilities
`crates/packet`	BGP message encoding and decoding
`crates/rib`	The routing table, best path selection, and the change journal
`crates/rpki`	The RTR client, route origin validation, and ASPA verification
`crates/fib`	Kernel route installation over netlink
`crates/server`	Sessions, the finite state machine, policy, export, and configuration
`crates/web`	The status page, HTTP API, and metrics
`crates/integration-tests`	Tests that run the server against in-process BGP peers
`crates/bench`	Benchmarks, with a synthetic Internet routing table
`bin/bgp`	The daemon

The API documentation of each crate (cargo doc --open) describes its modules. Module-level comments document the reasons for each design.

9.7 KiB Raw Blame History