rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fib: resolve next hops through the kernel's routing table

Browse Source 
Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.

Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:

* The route is looked up in one kernel table, the "resolution table"
  (`kernel { resolution_table: N }`, default 254 = main). The longest
  prefix that contains the next hop wins; of several routes for that
  prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
  route is installed with the gateway and interface of the kernel route
  that covers the next hop. A kernel route without a gateway (a route
  onto a point-to-point or tunnel interface) makes the next hop itself
  the gateway on that interface. A kernel route that names a nexthop
  object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
  own or another daemon's. A next hop that is reachable only through a
  BGP route depends on another BGP next hop, and such chains are not
  predictable.
* The default route does not resolve a next hop unless
  `kernel { resolve_via_default_route: true }` is set. On a router the
  default route points at a transit provider, and a next hop that is
  reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
  unresolvable, as does a route whose interface is down.

The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.

Implementation:

* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
  table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
  and the kernel's nexthop objects into one value. The kernel task
  publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
  and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
  alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
  kernels get a dump of every table, filtered in user space). It asks
  again, debounced, whenever a dedicated netlink socket that is a member
  of the route multicast groups receives anything. That socket is given a
  64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
  installs on it too, up to a million at a time, and an ENOBUFS overflow
  is treated as the same hint as a message. Its contents are never
  parsed. An earlier attempt to receive route notifications on the
  socket that writes routes failed, because the kernel only excludes the
  requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
  `Discarded`, which appear as the reasons in `/api/kernel` and on the
  status page.

Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.

Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Rayhaan JaufeerallyandClaude Fable 5.1 committed 2026-10-08 22:23:27 +02:00
1 parent 430addcf20
commit 3608897245
19 files changed
+1142 -173
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

crates/fib/src/plan.rs
+273 -35
View File
@@ -9,7 +9,8 @@ use bgp_packet::message::NlriNextHop;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::interfaces::Interfaces;
use crate::netlink::KIND_UNICAST;
use crate::topology::Topology;
// Asked for every route: a quick hash matters here.
pub(crate) type HashMap<K, V> = std::collections::HashMap<K, V, foldhash::fast::RandomState>;
@@ -32,10 +33,17 @@ pub enum Unresolved {
OtherFamily,
/// The next hop is one of this machine's own addresses.
Ourselves,
/// The next hop is not in the network of any interface that is up.
/// Finding the way to such a next hop through other routes is not
/// done.
NotConnected,
/// The next hop is not in the network of any interface that is up,
/// and none of the kernel's routes that count says the way to it
/// (see `crate::routes`).
NoRoute,
/// The only route to the next hop is the default route, which is not
/// taken as the way to a next hop unless asked for
/// (`Topology::via_default`).
OnlyDefault,
/// The route to the next hop is one that discards packets: a
/// blackhole, unreachable or prohibit route.
Discarded,
/// The route is for a network this machine is on itself, or for part
/// of one. The kernel reaches that directly; a route from a peer that
/// said otherwise would send the neighbors' traffic, and the sessions
@@ -44,11 +52,13 @@ pub enum Unresolved {
}
impl Unresolved {
pub const ALL: [Self; 5] = [
pub const ALL: [Self; 7] = [
Self::Missing,
Self::OtherFamily,
Self::Ourselves,
Self::NotConnected,
Self::NoRoute,
Self::OnlyDefault,
Self::Discarded,
Self::OwnNetwork,
];
@@ -57,7 +67,9 @@ impl Unresolved {
Self::Missing => "no next hop",
Self::OtherFamily => "IPv4 next hop for an IPv6 route",
Self::Ourselves => "next hop is this machine",
Self::NotConnected => "next hop not directly connected",
Self::NoRoute => "no route to the next hop",
Self::OnlyDefault => "next hop reachable only by the default route",
Self::Discarded => "the route to the next hop discards packets",
Self::OwnNetwork => "inside a network of this machine",
}
}
@@ -95,19 +107,68 @@ impl From<IpAddr> for Neighbor {
/// Works out where the kernel is to send packets for a route to `prefix`
/// with the next hop `next_hop`, learned from `peer`.
///
/// Only a next hop that is a neighbor on one of our interfaces is
/// resolved. A link-local next hop means something only on one link, and
/// that is taken to be the link the peer itself is on.
/// A next hop that is a neighbor on one of our interfaces is where the
/// packets go. One that is not is looked up in the kernel's own routes,
/// and the packets go where the route that covers it sends them: to its
/// gateway, or onto its interface. A link-local next hop means something
/// only on one link, and that is taken to be the link the peer itself is
/// on.
pub fn resolve(
interfaces: &Interfaces,
topology: &Topology,
prefix: &IpPrefix,
next_hop: Option<&NlriNextHop>,
peer: Option<Neighbor>,
) -> Result<Target, Unresolved> {
if interfaces.has_network_around(prefix) {
if topology.interfaces.has_network_around(prefix) {
return Err(Unresolved::OwnNetwork);
}
resolve_next_hop(interfaces, prefix.address_family(), next_hop, peer)
resolve_next_hop(topology, prefix.address_family(), next_hop, peer)
}
/// The way to `gateway`, which is not a neighbor, by the kernel's routes
/// that count: where the route that covers it sends packets. One step
/// only: the kernel sees to it that a route's own gateway is a neighbor.
fn through_routes(
topology: &Topology,
routes: IpVersion,
gateway: IpAddr,
) -> Result<Target, Unresolved> {
let route = topology.routes.lookup(gateway).ok_or(Unresolved::NoRoute)?;
if route.prefix.length() == 0 && !topology.via_default {
return Err(Unresolved::OnlyDefault);
}
if route.kind != KIND_UNICAST {
return Err(Unresolved::Discarded);
}
let (hop, interface) = match route.nexthop {
Some(id) => {
// A group of nexthops has no gateway or interface of its own;
// which of its members packets take is the kernel's to say,
// and nothing to give it as a route.
let nexthop = topology.nexthops.get(&id).ok_or(Unresolved::NoRoute)?;
(
nexthop.gateway,
nexthop.interface.ok_or(Unresolved::NoRoute)?,
)
}
None => (route.gateway, route.interface.ok_or(Unresolved::NoRoute)?),
};
if !topology
.interfaces
.link(interface)
.is_some_and(|link| link.up)
{
return Err(Unresolved::NoRoute);
}
// A route without a gateway puts the destination on its interface:
// the next hop itself is the neighbor there.
let gateway = hop.unwrap_or(gateway);
// An IPv4 route may go by an IPv6 gateway (RFC 8950, and the kernel
// allows it); an IPv6 route by an IPv4 gateway is nothing.
if routes == IpVersion::Ipv6 && gateway.is_ipv4() {
return Err(Unresolved::NoRoute);
}
Ok(Target { gateway, interface })
}
/// The part of [`resolve`] that is the same for every route of IP version
@@ -115,11 +176,12 @@ pub fn resolve(
/// those who ask it of many such routes, and ask of each only whether it
/// is for one of the machine's own networks.
pub fn resolve_next_hop(
interfaces: &Interfaces,
topology: &Topology,
routes: IpVersion,
next_hop: Option<&NlriNextHop>,
peer: Option<Neighbor>,
) -> Result<Target, Unresolved> {
let interfaces = &topology.interfaces;
let next_hop = next_hop.ok_or(Unresolved::Missing)?;
let version = match next_hop {
NlriNextHop::Ipv4(_) => IpVersion::Ipv4,
@@ -135,17 +197,17 @@ pub fn resolve_next_hop(
if interfaces.is_local(gateway) {
return Err(Unresolved::Ourselves);
}
interfaces
.connected(gateway)
.map(|connected| Target {
match interfaces.connected(gateway) {
Some(connected) => Ok(Target {
gateway,
interface: connected.interface,
})
.ok_or(Unresolved::NotConnected)
}),
None => through_routes(topology, routes, gateway),
}
};
// The interface the peer is a neighbor on, for a link-local gateway.
let on_peers_link = |gateway: Ipv6Addr| {
let interface = match peer.ok_or(Unresolved::NotConnected)? {
let interface = match peer.ok_or(Unresolved::NoRoute)? {
// The session is on a link-local address itself, and so says
// which link: every interface has fe80::/64, and looking the
// peer up there would find any of them.
@@ -157,7 +219,7 @@ pub fn resolve_next_hop(
.connected(address)
.map(|connected| connected.interface),
}
.ok_or(Unresolved::NotConnected)?;
.ok_or(Unresolved::NoRoute)?;
if interfaces
.addresses_of(interface)
.any(|ours| ours.local == IpAddr::V6(gateway))
@@ -182,8 +244,11 @@ pub fn resolve_next_hop(
{
on_peers_link(link_local)
}
// Where there is no way to the global one, the link-local one is
// tried; a way that is there and is bad (ourselves, a blackhole)
// is an answer as it stands.
NlriNextHop::Ipv6WithLl { global, link_local } => match direct(global.into()) {
Err(Unresolved::NotConnected) => on_peers_link(link_local),
Err(Unresolved::NoRoute | Unresolved::OnlyDefault) => on_peers_link(link_local),
resolved => resolved,
},
}
@@ -435,7 +500,7 @@ mod tests {
use proptest::prelude::*;
use super::*;
use crate::netlink::{Address, Link, Message};
use crate::netlink::{Address, Link, Message, Nexthop, PROTOCOL_BGP, Route, TABLE_MAIN};
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
@@ -448,14 +513,14 @@ mod tests {
}
}
fn interfaces() -> Interfaces {
let mut interfaces = Interfaces::default();
fn interfaces() -> Topology {
let mut topology = Topology::new(TABLE_MAIN, false);
for (index, local, network) in [
(2, "192.0.2.1", "192.0.2.0/24"),
(2, "2001:db8::1", "2001:db8::/64"),
(3, "198.51.100.1", "198.51.100.0/24"),
] {
interfaces.apply(&Message::Link {
topology.apply(&Message::Link {
present: true,
link: Link {
index,
@@ -463,7 +528,7 @@ mod tests {
up: true,
},
});
interfaces.apply(&Message::Address {
topology.apply(&Message::Address {
present: true,
address: Address {
interface: index,
@@ -472,7 +537,28 @@ mod tests {
},
});
}
interfaces
topology
}
/// A route of the kernel's main table, as a static route is.
fn kernel_route(prefix_text: &str, gateway: Option<&str>, interface: Option<u32>) -> Route {
Route {
prefix: prefix(prefix_text),
table: TABLE_MAIN,
protocol: 4,
metric: 0,
gateway: gateway.map(|g| g.parse().unwrap()),
interface,
nexthop: None,
kind: KIND_UNICAST,
}
}
fn with_route(topology: &mut Topology, route: Route) {
assert!(topology.apply(&Message::Route {
present: true,
route,
}));
}
fn neighbor(address: &str) -> Neighbor {
@@ -502,7 +588,7 @@ mod tests {
Ok(target("2001:db8::2", 2))
);
for (hop, why) in [
(Some(v4_hop("10.9.9.9")), Unresolved::NotConnected),
(Some(v4_hop("10.9.9.9")), Unresolved::NoRoute),
(Some(v4_hop("192.0.2.1")), Unresolved::Ourselves),
(None, Unresolved::Missing),
] {
@@ -588,7 +674,7 @@ mod tests {
let far = Some(neighbor("2001:db8:ffff::9"));
assert_eq!(
resolve(&interfaces, &v6, Some(&NlriNextHop::Ipv6(link_local)), far),
Err(Unresolved::NotConnected)
Err(Unresolved::NoRoute)
);
assert_eq!(
resolve(&interfaces, &v6, Some(&both("2001:db8::1")), peer),
@@ -645,7 +731,7 @@ mod tests {
// An interface that is not there, or is down.
assert_eq!(
resolve(&interfaces, &v6, Some(&next_hop), on(9)),
Err(Unresolved::NotConnected)
Err(Unresolved::NoRoute)
);
}
// Our own address on that link, which on another is a neighbor's.
@@ -666,6 +752,158 @@ mod tests {
);
}
/// A next hop that is no neighbor is reached the way the kernel's
/// own routes say, and the route goes into the kernel with that way's
/// gateway and interface.
#[test]
fn a_far_next_hop_resolves_by_the_kernels_routes() {
let mut topology = interfaces();
let v4 = prefix("203.0.113.0/24");
let v6 = prefix("2001:db8:100::/48");
let peer = Some(neighbor("192.0.2.2"));
let v4_hop = |s: &str| NlriNextHop::Ipv4(s.parse().unwrap());
let v6_hop = |s: &str| NlriNextHop::Ipv6(s.parse().unwrap());
let far = v4_hop("10.9.9.9");
// Nothing says the way.
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Err(Unresolved::NoRoute)
);
// The default route does not count unless asked for.
with_route(
&mut topology,
kernel_route("0.0.0.0/0", Some("192.0.2.254"), Some(2)),
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Err(Unresolved::OnlyDefault)
);
topology.via_default = true;
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Ok(target("192.0.2.254", 2))
);
topology.via_default = false;
// A static route with a gateway: the gateway is where packets go.
with_route(
&mut topology,
kernel_route("10.9.0.0/16", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Ok(target("198.51.100.9", 3))
);
// A more specific route onto an interface, as a tunnel's is: the
// next hop itself is the neighbor there.
topology.apply(&Message::Link {
present: true,
link: Link {
index: 7,
name: "wg0".into(),
up: true,
},
});
with_route(&mut topology, kernel_route("10.9.9.0/24", None, Some(7)));
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Ok(target("10.9.9.9", 7))
);
// The tunnel goes down: the kernel keeps such routes, we do not
// use them.
topology.apply(&Message::Link {
present: true,
link: Link {
index: 7,
name: "wg0".into(),
up: false,
},
});
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Err(Unresolved::NoRoute)
);
// A route that names a nexthop object goes where the object says.
with_route(
&mut topology,
Route {
nexthop: Some(77),
..kernel_route("10.9.9.9/32", None, None)
},
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Err(Unresolved::NoRoute),
"an object the kernel has not told of"
);
topology.apply(&Message::Nexthop {
present: true,
nexthop: Nexthop {
id: 77,
protocol: 4,
gateway: Some("192.0.2.77".parse().unwrap()),
interface: Some(2),
},
});
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Ok(target("192.0.2.77", 2))
);
// A route that discards packets is a way to nowhere.
with_route(
&mut topology,
Route {
kind: 6, // blackhole
..kernel_route("10.9.9.9/32", None, None)
},
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer),
Err(Unresolved::Discarded)
);
// Routes of BGP's never say the way, be the prefix ever so long.
assert!(!topology.apply(&Message::Route {
present: true,
route: Route {
protocol: PROTOCOL_BGP,
metric: 20,
..kernel_route("10.9.9.9/32", Some("198.51.100.9"), Some(3))
},
}));
// An IPv6 next hop by an IPv4 gateway is no way for an IPv6 route,
// but is one for an IPv4 route (RFC 8950 the other way about is
// what the kernel allows).
with_route(
&mut topology,
kernel_route("2001:db8:9::/48", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v6, Some(&v6_hop("2001:db8:9::9")), peer),
Err(Unresolved::NoRoute)
);
with_route(
&mut topology,
kernel_route("2001:db8:9::/48", Some("2001:db8::9"), Some(2)),
);
assert_eq!(
resolve(&topology, &v6, Some(&v6_hop("2001:db8:9::9")), peer),
Ok(target("2001:db8::9", 2))
);
assert_eq!(
resolve(&topology, &v4, Some(&v6_hop("2001:db8:9::9")), peer),
Ok(target("2001:db8::9", 2))
);
// A neighbor is a neighbor, whatever the routes say.
with_route(
&mut topology,
kernel_route("192.0.2.2/32", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v4, Some(&v4_hop("192.0.2.2")), peer),
Ok(target("192.0.2.2", 2))
);
}
#[test]
fn the_kernel_is_asked_only_for_what_changes() {
let mut plan = Plan::default();
@@ -692,13 +930,13 @@ mod tests {
assert_eq!(plan.installed(), 1);
// A best route that cannot be installed takes the old one out.
assert_eq!(
plan.best(p, Err(Unresolved::NotConnected)),
plan.best(p, Err(Unresolved::NoRoute)),
Some(Op::Remove { prefix: p })
);
assert_eq!((plan.installed(), plan.unresolved()), (0, 1));
assert_eq!(
plan.unresolved_by_reason().collect::<Vec<_>>(),
[(Unresolved::NotConnected, 1)]
[(Unresolved::NoRoute, 1)]
);
assert_eq!(plan.withdrawn(p), None);
assert_eq!(plan.unresolved_by_reason().count(), 0);
@@ -866,7 +1104,7 @@ mod tests {
Some(Op::Remove { prefix }) => { kernel.remove(&prefix); }
None => {}
};
let resolved = |gateway: Option<u8>| gateway.map(t).ok_or(Unresolved::NotConnected);
let resolved = |gateway: Option<u8>| gateway.map(t).ok_or(Unresolved::NoRoute);
// A new reader is first given the whole table.
let steps = std::iter::once(Step::Resync).chain(steps);
for step in steps {