ferrix/bin/bgp/README.md

BGP

This crate provides the bgp binary, which runs the Border Gateway Protocol daemon together with its read-only HTTP status interface.

cargo run -- path/to/config.textpb

The configuration format is defined in crates/server/proto/config.proto; testdata/server.textpb is a worked example. The daemon is configured entirely through that file: peers, and the routes it originates (originate { prefix: ... }). There is no way to change either at runtime.

When http_endpoint is set the binary also serves:

Path	Purpose
GET /statusz	Status page: configuration, peers and session state, originated and received routes, live event log
GET /api/status	Router ID, ASN, uptime and table sizes
GET /api/config	The configuration file as read at startup
GET /api/peers	Configured peers and established sessions
GET /api/routes/local	Originated routes
GET /api/routes/received	Routes learned from peers; see "Finding a route"
GET /api/routes/rejected	Routes peers announced that were refused, by peer
GET /api/rpki	Validation with the RPKI: the cache, the data and its age
GET /api/kernel	The routes in the kernel: how many, and why the others are not; the machine's interfaces
GET /api/events	Server-sent events for session and route changes
GET /metricz	Prometheus metrics

Everything under it is read-only and unauthenticated. Bind it to loopback or put it behind a reverse proxy.

What it does today

It establishes sessions (both dialling configured peers and accepting connections, with RFC 4271 collision detection), announces its originated routes to every peer with itself as next hop, judges what peers send by each peer's import policy, keeps what passes, chooses a best route for each prefix, and on Linux can put those best routes in the kernel so that the machine forwards by them. It does not yet pass routes learned from one peer on to another, and has no export policy.

Who may connect

Connections are taken only from the addresses of configured peers, and from each of those only a few at a time. Anything else is closed unread and counted in bgp_connections_refused_total. accept_unconfigured_peers: true lets any host open a session as any AS; it exists for tests on a closed network.

What is taken from a peer

Every route a peer announces is judged before it goes into the routing table (import in a peer's configuration; see config.proto):

• Refused for any peer unless turned off: prefixes that are not routed on the Internet (private, documentation, multicast, reserved), prefixes longer than /24 or /48, a default route, paths with a private or reserved AS number, and paths that do not begin with the peer's own AS.
• Always refused: a path with our own AS number in it.
• If configured: a list of the prefixes the peer may announce and of the AS numbers its routes may originate from.
• max_prefixes: one route more ends the session with Cease / Maximum Number of Prefixes Reached and keeps the peer down for ten minutes.

A refused route is dropped, along with the route it would have replaced. bgp_routes_rejected_total counts them by reason, and the status page shows how many routes each peer has and how many were refused.

Route origin validation (RPKI)

With rpki { caches { address: "..." port: 3323 } } in the configuration, the daemon keeps the validated ROA payloads of a validator (Routinator, rpki-client with StayRTR, FORT) over the RPKI-to-Router protocol, and validates every route from peers outside its own AS (RFC 6811):

• invalid (the holder of the addresses authorised another origin, or a shorter prefix): refused, like any route the import policy refuses, unless the peer has accept_rpki_invalid: true;
• valid and not found: taken. Validity does not otherwise change which route is chosen.

The data changes all the time, and the routes follow without the peer having to send them again: a route in the table that becomes invalid is taken out, and a route refused as invalid is kept aside ("held") and taken when it no longer is. If no cache confirms the data for the expire interval (two hours unless the cache or the configuration says otherwise) it is dropped, and nothing is invalid any more.

The status page shows the cache, how fresh the data is, each route's validity, and how many of each peer's routes are held. /api/rpki has the same as JSON; bgp_rpki_vrps{family}, bgp_rpki_aspas and bgp_rpki_updates_total are on /metricz. Refused routes are counted under the reasons RPKI invalid and ASPA invalid.

The connection to the cache is plain TCP, without authentication: run the validator on the same host or on a network you trust.

Roles and route leaks (RFC 9234)

role on a peer says what the peer is to us: PROVIDER, CUSTOMER, PEER, ROUTE_SERVER or ROUTE_SERVER_CLIENT. It is not for peers in our own AS. With it:

• The two ends have to agree. Our OPEN says what we are to the peer (its provider, if it is our customer). A peer whose OPEN says it is something else than configured is refused with OPEN Message Error / Role Mismatch: one side thinking "customer" and the other "peer" is how leaks happen. A peer that says nothing (it does not implement the RFC) is accepted, unless require_role: true. The status page shows the role, with a tick when the peer confirmed it.
• Routes are marked "only to customer" (the OTC attribute) when they come in from a provider, a peer or a route server, and when ours go out to a customer, a peer or a route server client. The mark stays on the route.
• A marked route coming the wrong way is a leak and is refused: from a customer or a route server client always, and from a peer unless the peer put the mark there itself. The reason shown is route leak (Only to Customer). No setting turns this off, as the RFC requires.

This catches leaks even where nobody on the path has an ASPA, as long as the networks on either side of the leaker mark their routes.

AS path verification (ASPA)

Origin validation says who may originate a prefix. It says nothing about the rest of the path, so it does not catch a route leak (a network passing its provider's or peer's routes on to another provider or peer), nor a hijack that keeps the right origin at the end of a made-up path. ASPAs do: in one, a network lists its providers, and a path that goes from that network to anyone else, where only a provider could come next, is invalid (draft-ietf-sidrops-aspa-verification).

Two things are needed for paths to be verified:

• a validator that hands out ASPAs, which takes version 2 of the RPKI-to-Router protocol (draft-ietf-sidrops-8210bis). The daemon asks for version 2 and falls back to 1 or 0 if that is what the validator speaks; the status page says which is in use, and warns when it is one without ASPAs;
• role on the peer, saying what the peer is to us, because what a path may look like depends on who hands it over:

role	The peer	Its paths
CUSTOMER	buys transit from us	must lead up through providers all the way
PEER	exchanges its own and its customers' routes with us	the same
ROUTE_SERVER	is an exchange's route server	the same; the path need not begin with the route server's AS
ROUTE_SERVER_CLIENT	is a client of ours, where we are the route server	the same
PROVIDER	sells us transit	may lead up, across one peering, and down

A peer without a role has its origins validated and its paths not.

A path is invalid (refused, unless the peer has accept_aspa_invalid: true), valid, or unknown, which is what most paths are while few networks have an ASPA; unknown ones are taken. A path with an AS_SET in it is invalid, and so is one that does not begin with the peer's AS (other than from a route server). As with origins, routes follow the data: held while invalid, taken when they no longer are.

rpki { caches { address: "127.0.0.1" port: 3323 } }
peers {
    name: "transit"
    endpoint { address: "192.0.2.1" }
    remote_asn: 64496
    role: PROVIDER
}

Forwarding: routes in the kernel (Linux)

With kernel {} in the configuration the daemon puts the best route of every prefix in the kernel's routing table, over netlink, and keeps it there as the best routes change. Without it the daemon only speaks BGP and the machine's forwarding is not touched.

kernel {
    table: 254     # the main table; the default
    metric: 20     # the default
}

• What is installed. A best route whose next hop is a neighbor: an address in the network of one of the machine's interfaces that is up. For an IPv6 route with a link-local next hop, the interface is the one the peer is a neighbor on. A route whose next hop is further away (a route from a peer in our own AS that did not set itself as next hop, say) is not installed, and neither is whatever it replaced: looking up the way to such a next hop through other routes is not done yet. The status page counts these, by reason.

• Nexthop objects. Where the kernel has them (Linux 5.3 and later) a gateway is put there once, as a nexthop object, and routes name it; ip route shows such routes with nhid, and ip nexthop the objects. The kernel then checks the gateway once and not for every route, which makes a full table about a third faster to install, and lets all of them be removed at once when the daemon stops. The objects the daemon makes are its own: it does not use anyone else's, puts its own back if one is removed, and removes them when it stops. without_nexthop_objects: true has every route carry its own gateway instead, as on older kernels.

• What is ours. Routes are marked proto bgp. Every route with that mark in the configured table with the configured metric is taken to be this daemon's, and no other route is ever changed or removed. So do not run two daemons that install BGP routes into the same table with the same metric.

• Starting and stopping. At startup, routes of ours left by an earlier run are kept, so that the machine goes on forwarding while the peers send their tables again; those that no peer has announced again after stale_seconds (120 unless set) are removed. On SIGINT or SIGTERM the daemon closes its sessions and then removes its routes. After a crash they stay until the next start.

• Interfaces. The daemon learns the machine's interfaces and addresses from the kernel and is told when they change; a second later it works all routes out again, since any next hop may resolve differently.

• Privileges. Changing routes needs the CAP_NET_ADMIN capability, and listening on port 179 needs CAP_NET_BIND_SERVICE; neither needs root. Without the first the daemon stops as soon as the kernel refuses it.

  setcap cap_net_admin,cap_net_bind_service+ep /usr/local/bin/bgp
  

ip route show proto bgp shows what is there. /api/kernel and the status page show how many routes are installed, how many are not and why, the last refusal by the kernel, and the interfaces; /metricz has bgp_kernel_routes, bgp_kernel_routes_unresolved, bgp_kernel_routes_refused, bgp_kernel_route_changes_total{change} and bgp_kernel_errors_total.

To forward packets for others the machine also needs net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding=1; the daemon does not set them.

How long it takes

A million IPv4 routes, about the size of the Internet's table, go into the kernel in a little over a second and come out in half a second (release build, in a virtual machine on an Apple M-series laptop; the test many_routes_are_installed_and_removed with FERRIX_ROUTES=1000000). About three quarters of that is the kernel's own work on each route. What keeps the rest small: requests go to the kernel hundreds to a write, with one acknowledgement for each write and not for each route; a task of its own does the writing, so the next batch is worked out while the kernel is busy with the last; and what a next hop resolves to is worked out once for each next hop, not once for each route.

Testing against a real kernel

The tests that change a kernel's routes are not run by cargo test. scripts/kernel-tests.sh runs them on Linux, each in a network namespace made for it (scripts/in-namespace.sh runs any one command that way): the route installer against ip route, the daemon with a peer, and two daemons that route for each other with pings across. From macOS, scripts/linux-vm.sh runs it (or any command, such as cargo test) in a Lima virtual machine that it creates on first use.

Interoperability

scripts/interop/lab.sh runs the daemon against FRR, BIRD and Routinator, each in a network namespace of its own on one exchange LAN, and checks what every side ends up with. It needs Linux (scripts/linux-vm.sh scripts/interop/lab.sh from macOS; scripts/interop/install.sh installs the others on Ubuntu). Last run against FRR 10.5.1, BIRD 2.18 and Routinator 0.15.2. What it covers:

Phase	What is checked
basics	Sessions over IPv4 and IPv6; routes both ways with path, next hop, communities and large communities intact; four-octet AS numbers; ADD-PATH from BIRD; pings across by the installed routes
roles	RFC 9234 roles agree with FRR (peer) and BIRD (provider); Only to Customer is set where it should be, in both directions
rpki	RTR version 2 with Routinator; an invalid route from FRR is refused and held, and taken after Routinator restarts with data that allows it
refresh	Routes added and withdrawn by FRR; FRR's ROUTE-REFRESH answered; BIRD re-sending
flaps	Five session resets; FRR and BIRD killed and restarted; a cut wire noticed by the hold timer
restart	The daemon stopped in good order (kernel emptied, peers told) and killed (routes kept, stale one cleared by the next start)
scale	A table from BIRD (SCALE_ROUTES, a million for the Internet's size) into the table and the kernel; the daemon killed and restarted under it; the table withdrawn
limits	A peer whose role does not fit gets no session; a peer over its prefix limit gets Cease / Maximum Number of Prefixes and is held down

scripts/interop/real-rpki.sh checks the RTR client against a Routinator that has the real RPKI, ASPAs included: the daemon has to end up with as many VRPs and ASPAs as the validator says it has.

Not covered: routers other than FRR and BIRD; IPv6 routes over an IPv4 session (the daemon does not send them); long runs.

Finding a route

The status page has a look-up box over the received routes, and /api/routes/received takes the same questions:

Query	Answer
?prefix=193.0.0.0/21	Every route for that prefix, the best first
?address=193.0.0.1	Every prefix that covers the address, most specific first
?within=193.0.0.0/16	That prefix and everything inside it
&peer=NAME	Only routes from that peer (its name or its address)
&best=true	Only the route chosen for each prefix
&limit=N	At most N routes (1000 unless given, 100000 at most)

within, peer and best on their own look through the whole table, which on a full table takes a moment.

Seeing and alerting on refused routes

A refused route is not in the table. What is kept instead, for each peer, is how many were refused for each reason and the latest 200 of them with their AS paths. The status page lists them under "Refused routes"; /api/routes/rejected[?peer=NAME] has them as JSON. The record is kept when the session ends, so a peer that was cut off can still be looked into. The log has a line for a peer's first refused route and then at most one a minute.

For alerting, /metricz has:

Metric	Meaning
bgp_routes_rejected_total{peer,reason}	Routes refused, by peer and why
bgp_peer_routes{peer}	Routes an established peer has in the table
bgp_peer_max_prefixes{peer}	The peer's limit, where it has one
bgp_prefix_limit_exceeded_total{peer}	Sessions ended for going over the limit
bgp_connections_refused_total{reason}	Connections closed without a session

Rules to start from:

groups:
  - name: bgp
    rules:
      # A peer is announcing what its policy does not allow: a leak or a
      # hijack on its side, or a list on ours that is out of date.
      - alert: BgpRoutesRefused
        expr: sum by (peer, reason) (increase(bgp_routes_rejected_total[10m])) > 0
        labels: { severity: warning }
      # A route that has been through our own AS came back: a loop, or
      # someone announcing our AS number.
      - alert: BgpOwnAsInPath
        expr: increase(bgp_routes_rejected_total{reason="own AS in path"}[10m]) > 0
        labels: { severity: page }
      # A peer is close to its limit; the limit may need raising before the
      # session is cut.
      - alert: BgpPeerNearPrefixLimit
        expr: bgp_peer_routes / on (peer) bgp_peer_max_prefixes > 0.8
        for: 10m
        labels: { severity: warning }
      - alert: BgpPrefixLimitExceeded
        expr: increase(bgp_prefix_limit_exceeded_total[1h]) > 0
        labels: { severity: page }

Trying it without a router

fake_peer is a minimal BGP speaker that connects to the daemon, announces a prefix, prints every UPDATE it receives and, with --echo, re-announces each received prefix back with its own ASN prepended (dropping NO_EXPORT routes). The prefix it announces shows up in the received table. The ones it echoes have been through our AS already, so they are refused as loops and counted in the peer's refused routes:

cargo run -- testdata/server.textpb
cargo run -p bgp-integration-tests --example fake_peer -- \
    --server 127.0.0.1:179 --asn 65001 --announce 203.0.113.0/24 --echo

Please refer to the other implementation crates in this repository where most of the actual implementation lives.

Warning: This is still very much a work in progress and is not yet ready for production use.