Rayhaan JaufeerallyandClaude Fable 5.1
6d8f89e509
Before this change, a route carrying the BLACKHOLE community (65535:666,
RFC 7999) from an EBGP neighbor was always rejected with the reason
"blackhole request", and nothing discarded traffic. A customer under a
denial-of-service attack had no way to ask this router to drop traffic
to an address.
This change adds `accept_blackholes` to the import policy. When it is
set, a route with the BLACKHOLE community is accepted if:
* its prefix is a single address, a /32 or a /128 (otherwise it is
rejected with the new reason "blackhole for more than one address");
the `max_ipv4_prefix_length` and `max_ipv6_prefix_length` limits do
not apply, as RFC 7999 section 3.2 recommends;
* the address is inside a prefix the neighbor may announce, if
`prefixes` is configured, regardless of the entry's `max_length`
(new `PrefixSet::covers`); without a prefix list, an address inside
the local AS's own prefixes is rejected as before ("prefix is our
own");
* the AS path, origin and RPKI checks that apply to every route pass.
The next hop of an accepted blackhole route is not checked for
third-party addresses and is not resolved, because traffic is never
forwarded to it. The route takes part in best path selection as usual;
a host route is normally the active route for its address.
The kernel task installs the active route as a kernel blackhole route
(`RTN_BLACKHOLE`, shown by `ip route` as `blackhole PREFIX proto bgp`).
Blackhole routes left by an earlier run are adopted at startup like
other routes of ours. The route is advertised to other neighbors as the
export policy and export limits permit, with the community intact, so a
provider that honors BLACKHOLE can discard the traffic upstream.
Implementation:
* `plan::Target` becomes an enum: `Via { gateway, interface }` or
`Discard`. `plan::resolve` takes a `blackhole` flag and returns
`Discard` for such routes without looking at the next hop (the
own-network check still applies). No nexthop object is made for a
discard target.
* `AttrSet::is_blackhole` reads the community off the stored attribute
bytes, like `next_hop` and `as_path`, so the kernel task, the
revalidator and the RPKI revalidation can ask it of every route.
* `ImportPolicy::is_blackhole_request` and `check_blackhole_prefix`
carry the import side; the import flow (`routes.rs`) computes once per
UPDATE whether its routes are an accepted blackhole request and skips
the next hop checks for them.
* `Status::blackholes` and the `blackholes` field of `/api/kernel` count
the installed blackhole routes; the status page shows the count next
to the installed routes.
Not covered: a prefix this router originates with the BLACKHOLE
community is advertised but not installed as a blackhole route. The
documentation says to add a kernel blackhole route by hand for that
case.
Documentation: the "Blackhole Routes" section of the import policy
guide replaces "Blackhole Requests", the kernel guide gains a "Blackhole
Routes" section, the configuration reference describes
`accept_blackholes`, the HTTP API reference describes `blackholes`, and
the RFC 7999 rows of the standards reference are updated.
Testing: unit tests for the attribute check, the prefix checks and the
plan; integration tests that a blackhole request is refused by default
and, with `accept_blackholes`, taken for one address inside the peer's
space and refused otherwise, with the recorded reasons; a kernel test
that a blackhole request becomes a kernel blackhole route whatever its
next hop, is removed on withdrawal and is adopted after a restart. The
existing kernel tests, the interoperability lab and the workspace tests
pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
10 KiB
+++ title = "Kernel Route Installation" description = "Describes how Ferrix installs active routes in the Linux kernel routing table over netlink, and how it handles next hop resolution, restarts, and kernel errors." weight = 5
[extra] nav = "Kernel Route Installation" +++
Ferrix can install the active route for each prefix in the Linux kernel routing table, so that the router forwards traffic according to BGP. Ferrix communicates with the kernel over rtnetlink. Kernel route installation is supported on Linux only.
If the kernel statement is not configured, Ferrix operates as a BGP
speaker only and does not modify the kernel routing table.
Configuring Kernel Route Installation
The following example enables kernel route installation with the default values:
kernel {
table: 254
metric: 20
}
| Statement | Description | Default |
|---|---|---|
table |
Kernel routing table in which routes are installed | 254 (main) |
metric |
Metric of the installed routes. The kernel prefers the route with the lowest metric. | 20 |
without_nexthop_objects |
Installs each route with its own gateway instead of a nexthop object | Disabled |
stale_seconds |
Time for which routes from a previous run are retained at startup | 120 seconds |
resolution_table |
Kernel routing table whose routes resolve next hops that are not directly connected | 254 (main) |
resolve_via_default_route |
Permits the default route to resolve a next hop | Disabled |
Ferrix requires the CAP_NET_ADMIN capability to modify the kernel routing
table. If the kernel rejects a request for lack of privileges, the daemon
exits. See Granting Privileges.
Note
Ferrix does not enable IP forwarding. Set
net.ipv4.ip_forward=1andnet.ipv6.conf.all.forwarding=1to forward transit traffic.
Next Hop Resolution
Ferrix resolves the next hop of each active route to a gateway and an outgoing interface before it installs the route:
- A next hop that is directly connected, that is, an address in a subnet of an interface that is up, is the gateway, and the interface is that interface. For an IPv6 route with a link-local next hop, the outgoing interface is the interface on which the neighbor's session is established.
- A next hop that is not directly connected is resolved through the
routes of the resolution table (
resolution_table, by default themaintable). Ferrix selects the route with the longest prefix that contains the next hop, and of several routes for that prefix the one with the lowest metric. If the route has a gateway, the gateway and the route's interface are used. If the route has no gateway, as a route to a point-to-point interface or a tunnel does not, the next hop itself is the gateway and the route's interface is used. A route that names a kernel nexthop object is resolved through that object.
The Linux kernel does not resolve gateways recursively, so Ferrix installs
the route with the resolved gateway and interface rather than with the
BGP next hop. For example, a route from an IBGP neighbor with the next hop
192.0.2.9, when the kernel has the static route
192.0.2.0/24 via 10.0.0.2 dev wg0, is installed as a route
via 10.0.0.2 dev wg0. When the static route changes or is removed,
Ferrix resolves the next hop again and updates or removes the installed
route.
The following routes do not resolve a next hop:
- Routes with protocol
bgp, whether installed by Ferrix or by another BGP daemon. A next hop that is reachable only through a BGP route would depend on another next hop, and such dependencies between routes are not predictable. - The default route, unless
resolve_via_default_routeistrue. On a router, the default route typically points to a transit provider, and a next hop that is reachable only through it is not a usable next hop. - Blackhole, unreachable and prohibit routes. A next hop covered by such a route is unresolvable.
- Routes whose interface is down.
Ferrix learns the routes of the resolution table from the kernel at startup and is notified of changes. Changes to the routes take effect within a few seconds.
Note
Routes from an IBGP neighbor that does not set itself as the next hop, such as a route reflector that preserves the next hop, require a route to the next hop in the resolution table: a static route, a route to the tunnel on which the next hop is reached, or a route from an IGP.
Exclusion of Unresolvable Routes
If the kernel statement is configured, Ferrix excludes routes with an
unresolvable next hop from best path selection (RFC 4271 Section 9.1.2.1).
This ensures that such a route is not selected while a usable route for the
same prefix exists.
A route with an unresolvable next hop is a held route. Ferrix counts it as
rejected with the reason next hop unreachable, and displays it as held on
the status page. When an interface or address change makes the next hop
resolvable, Ferrix accepts the route. A route in the routing table whose
next hop becomes unresolvable is also moved to the held state.
If the kernel statement is not configured, Ferrix has no information about
next hop reachability and accepts all routes.
Blackhole Routes
Ferrix installs the active route for a prefix as a kernel blackhole route
(ip route displays it as blackhole PREFIX proto bgp) if the route
carries the BLACKHOLE community (RFC 7999) and the import policy accepted
it with accept_blackholes. The kernel discards traffic to the prefix.
The next hop of the route is not resolved, and the route is installed
even if the next hop is unreachable. The kernel section of the status
page and /api/kernel report the number of installed blackhole routes.
See Blackhole Routes
in the import policy guide.
Connected Networks
Ferrix never installs a route for a subnet of a local interface, or for a prefix within such a subnet, even if the import policy accepts it. The kernel reaches these destinations directly. Without this rule, a neighbor on a shared LAN that advertises part of the LAN could attract the traffic for other hosts on the LAN, including the BGP sessions with them.
Ferrix holds such a route, and counts it as rejected with the reason
prefix is a network of this router. If the subnet is removed from the
interface, Ferrix accepts the route.
Route Ownership
Ferrix installs routes with the protocol bgp (proto bgp). Ferrix
considers every route with this protocol, in the configured table and with
the configured metric, to be its own. It does not modify or remove any
other route.
Warning
Do not run two routing daemons that install BGP routes in the same table with the same metric. Each daemon would remove the other's routes.
Nexthop Objects
On Linux 5.3 and later, Ferrix installs each gateway once as a nexthop object, and installs routes that reference the object. The kernel then validates each gateway once instead of once for each route. This reduces the time to install a full routing table by approximately one third, and allows all routes to be removed at once when the daemon stops.
The command ip route displays these routes with an nhid attribute, and
the command ip nexthop displays the objects. Ferrix uses only the nexthop
objects that it creates, recreates them if they are removed, and removes
them when it stops.
To install each route with its own gateway, as on older kernels, configure
without_nexthop_objects: true.
Startup and Shutdown Behavior
Ferrix handles its kernel routes as follows:
- At startup, Ferrix retains the routes installed by a previous run, so
that the router continues to forward traffic while the neighbors
readvertise their routes. When every configured neighbor has sent an
End-of-RIB marker for each address family, or after
stale_seconds, whichever is first, Ferrix removes the routes that no neighbor has readvertised. - On SIGINT or SIGTERM, Ferrix closes its BGP sessions and removes its routes from the kernel.
- On SIGUSR1, Ferrix leaves its routes in the kernel and exits, to be started again (graceful restart). See Graceful Restart.
- After a crash, the routes remain in the kernel until the next start.
Interface Changes
Ferrix receives the router's interfaces and addresses from the kernel, and receives a notification when they change. Ferrix reevaluates all routes 1 second after the last change, and at most 5 seconds after the first change, because a change can affect the resolution of any next hop.
Kernel Error Handling
The kernel reports each request that it rejects. If the kernel's notifications exceed the socket buffer, or if the kernel does not respond within 10 seconds, Ferrix reads the kernel routing table and corrects any difference from the expected state.
Verifying Installed Routes
To display the installed routes, use the following commands:
ip route show proto bgp
ip -6 route show proto bgp
ip nexthop show
The status page and the HTTP API endpoint /api/kernel display the number
of installed routes, the number of routes not installed and the reason, the
last error reported by the kernel, and the interfaces. Table 2 lists the
related metrics.
| Metric | Description |
|---|---|
bgp_kernel_routes |
Number of routes installed by Ferrix |
bgp_kernel_routes_unresolved |
Number of prefixes whose active route is not installed because its next hop is unresolvable |
bgp_kernel_routes_refused |
Number of prefixes whose active route the kernel rejected |
bgp_kernel_route_changes_total{change} |
Number of route installations and removals requested |
bgp_kernel_errors_total |
Number of requests that the kernel rejected |
Performance
In a virtual machine on an Apple M-series computer, with a release build, Ferrix installs 1 million IPv4 routes in slightly more than 1 second and removes them in approximately 0.5 seconds. Approximately three quarters of this time is spent in the kernel. Ferrix reduces its own share as follows:
- It sends hundreds of requests in each write to the netlink socket, and requests one acknowledgment for each write instead of each route.
- A dedicated task writes to the socket, so that the next batch is prepared while the kernel processes the previous one.
- It resolves each next hop once, not once for each route.