Install best routes in the Linux kernel over netlink
New crate bgp-fib. It follows the routing table's best routes and keeps
the kernel's routing table the same, and learns the machine's interfaces
and addresses from the kernel, which is what says whether a next hop can
be reached.
- netlink: the rtnetlink messages used (routes, addresses, links, errors),
as bytes and back. No I/O; property tested and fuzzed.
- interfaces: what is directly reachable, and from which interface.
- plan: what to ask the kernel for, given the best routes and what the
kernel has. No I/O; tested against a model kernel.
- kernel (Linux only): the sockets and the task. Requests go in batches;
routes of ours found at startup are kept if still right and removed if
not; a change of interfaces has every route worked out again; a clean
stop removes our routes.
Only routes marked proto bgp with the configured table and metric are
ever touched. A best route is installed if its next hop is a neighbor;
next hops further away are not resolved through other routes yet, and
such prefixes are counted and shown instead.
Configured with `kernel { table metric }`; off by default. The daemon
needs CAP_NET_ADMIN and stops if the kernel refuses it. On shutdown it
closes sessions and then removes its routes.
/api/kernel and the status page show installed, unresolved and refused
counts, the last error and the interfaces; five bgp_kernel_* metrics.
Testing: scripts/kernel-tests.sh runs, in network namespaces of their
own and with only the capabilities the daemon needs, the installer
against `ip route`, the daemon with a peer, and two daemons that route
for each other with pings across over IPv4 and IPv6.
scripts/linux-vm.sh runs that, or anything else, in a Lima VM from
macOS. CI runs the kernel tests and fuzzes the netlink decoder.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f84235e87
commit
1fc44798b7
38 files changed
+3745
-16
No files matched your search
+67
-3
@@ -24,6 +24,7 @@ When `http_endpoint` is set the binary also serves:
|
||||
| `GET /api/routes/received` | Routes learned from peers; see "Finding a route" |
|
||||
| `GET /api/routes/rejected` | Routes peers announced that were refused, by peer |
|
||||
| `GET /api/rpki` | Validation with the RPKI: the cache, the data and its age |
|
||||
| `GET /api/kernel` | The routes in the kernel: how many, and why the others are not; the machine's interfaces |
|
||||
| `GET /api/events` | Server-sent events for session and route changes |
|
||||
| `GET /metricz` | Prometheus metrics |
|
||||
|
||||
@@ -34,9 +35,11 @@ put it behind a reverse proxy.
|
||||
|
||||
It establishes sessions (both dialling configured peers and accepting
|
||||
connections, with RFC 4271 collision detection), announces its originated
|
||||
routes to every peer with itself as next hop, and records what peers send in
|
||||
an Adj-RIB-In. It does not select best paths, re-advertise received routes or
|
||||
program a forwarding table, and it has no import or export policy.
|
||||
routes to every peer with itself as next hop, judges what peers send by each
|
||||
peer's import policy, keeps what passes, chooses a best route for each
|
||||
prefix, and on Linux can put those best routes in the kernel so that the
|
||||
machine forwards by them. It does not yet pass routes learned from one peer
|
||||
on to another, and has no export policy.
|
||||
|
||||
## Who may connect
|
||||
|
||||
@@ -165,6 +168,67 @@ peers {
|
||||
}
|
||||
```
|
||||
|
||||
## Forwarding: routes in the kernel (Linux)
|
||||
|
||||
With `kernel {}` in the configuration the daemon puts the best route of every
|
||||
prefix in the kernel's routing table, over netlink, and keeps it there as the
|
||||
best routes change. Without it the daemon only speaks BGP and the machine's
|
||||
forwarding is not touched.
|
||||
|
||||
```
|
||||
kernel {
|
||||
table: 254 # the main table; the default
|
||||
metric: 20 # the default
|
||||
}
|
||||
```
|
||||
|
||||
- **What is installed.** A best route whose next hop is a neighbor: an
|
||||
address in the network of one of the machine's interfaces that is up. For
|
||||
an IPv6 route with a link-local next hop, the interface is the one the
|
||||
peer is a neighbor on. A route whose next hop is further away (a route
|
||||
from a peer in our own AS that did not set itself as next hop, say) is not
|
||||
installed, and neither is whatever it replaced: looking up the way to such
|
||||
a next hop through other routes is not done yet. The status page counts
|
||||
these, by reason.
|
||||
- **What is ours.** Routes are marked `proto bgp`. Every route with that
|
||||
mark in the configured table with the configured metric is taken to be
|
||||
this daemon's, and no other route is ever changed or removed. So do not
|
||||
run two daemons that install BGP routes into the same table with the same
|
||||
metric.
|
||||
- **Starting and stopping.** At startup, routes of ours left by an earlier
|
||||
run are kept if they are still right and removed if not. On SIGINT or
|
||||
SIGTERM the daemon closes its sessions and then removes its routes. After
|
||||
a crash they stay until the next start.
|
||||
- **Interfaces.** The daemon learns the machine's interfaces and addresses
|
||||
from the kernel and is told when they change; a second later it works all
|
||||
routes out again, since any next hop may resolve differently.
|
||||
- **Privileges.** Changing routes needs the `CAP_NET_ADMIN` capability, and
|
||||
listening on port 179 needs `CAP_NET_BIND_SERVICE`; neither needs root.
|
||||
Without the first the daemon stops as soon as the kernel refuses it.
|
||||
|
||||
```
|
||||
setcap cap_net_admin,cap_net_bind_service+ep /usr/local/bin/bgp
|
||||
```
|
||||
|
||||
`ip route show proto bgp` shows what is there. `/api/kernel` and the status
|
||||
page show how many routes are installed, how many are not and why, the last
|
||||
refusal by the kernel, and the interfaces; `/metricz` has `bgp_kernel_routes`,
|
||||
`bgp_kernel_routes_unresolved`, `bgp_kernel_routes_refused`,
|
||||
`bgp_kernel_route_changes_total{change}` and `bgp_kernel_errors_total`.
|
||||
|
||||
To forward packets for others the machine also needs
|
||||
`net.ipv4.ip_forward=1` and `net.ipv6.conf.all.forwarding=1`; the daemon does
|
||||
not set them.
|
||||
|
||||
### Testing against a real kernel
|
||||
|
||||
The tests that change a kernel's routes are not run by `cargo test`.
|
||||
`scripts/kernel-tests.sh` runs them on Linux, each in a network namespace
|
||||
made for it: the route installer against `ip route`, the daemon with a peer,
|
||||
and two daemons that route for each other with pings across. From macOS,
|
||||
`scripts/linux-vm.sh` runs it (or any command, such as `cargo test`) in a
|
||||
Lima virtual machine that it creates on first use.
|
||||
|
||||
## Finding a route
|
||||
|
||||
The status page has a look-up box over the received routes, and
|
||||
|
||||
Reference in new issue
Block a user