Rayhaan JaufeerallyandClaude Fable 5.1
bdaf226a16
With rpki { caches { ... } } in the configuration the server keeps the
validated ROA payloads of a validator and validates every route from a
peer outside its own AS (RFC 6811). An invalid route is refused, unless
its peer has accept_rpki_invalid. Valid and not-found routes are taken;
validity does not otherwise enter route selection.
Validity is not settled when a route arrives: the data changes every few
minutes. The routes follow, without the peer sending anything again.
- A route refused as invalid is held by its session, out of the table.
When the data changes the session looks at what it holds, and what is
no longer invalid goes into the table. A held route is the peer's to
withdraw or replace like any other.
- A route in the table that becomes invalid is found by a task that
looks through the table when the data changes, at the prefixes the
changed VRPs cover, and is taken out and held by its session.
- If the data expires because no cache confirms it, nothing is invalid
any more and everything held is taken.
- A session holds at most 200,000 routes; past that it asks the peer for
a route refresh when the data changes.
So that the table can be looked through without parsing attributes, an
attribute set's key now has the route's origin AS. Memory per route is
unchanged at 52 bytes on the real table.
Shown: /api/rpki and a section of the status page with the cache, its
state and the age of the data; each route's validity; each peer's count
of held routes. Refusals are recorded under "RPKI invalid". Metrics:
bgp_rpki_vrps and bgp_rpki_updates_total.
Also: Negotiated says whether the peer supports route refresh, and
docs/RFCs.md is brought up to date.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
7.8 KiB
BGP
This crate provides the bgp binary, which runs the Border Gateway Protocol
daemon together with its read-only HTTP status interface.
cargo run -- path/to/config.textpb
The configuration format is defined in crates/server/proto/config.proto;
testdata/server.textpb is a worked example. The daemon is configured
entirely through that file: peers, and the routes it originates
(originate { prefix: ... }). There is no way to change either at runtime.
When http_endpoint is set the binary also serves:
| Path | Purpose |
|---|---|
GET /statusz |
Status page: configuration, peers and session state, originated and received routes, live event log |
GET /api/status |
Router ID, ASN, uptime and table sizes |
GET /api/config |
The configuration file as read at startup |
GET /api/peers |
Configured peers and established sessions |
GET /api/routes/local |
Originated routes |
GET /api/routes/received |
Routes learned from peers; see "Finding a route" |
GET /api/routes/rejected |
Routes peers announced that were refused, by peer |
GET /api/rpki |
Route origin validation: the cache, the data and its age |
GET /api/events |
Server-sent events for session and route changes |
GET /metricz |
Prometheus metrics |
Everything under it is read-only and unauthenticated. Bind it to loopback or put it behind a reverse proxy.
What it does today
It establishes sessions (both dialling configured peers and accepting connections, with RFC 4271 collision detection), announces its originated routes to every peer with itself as next hop, and records what peers send in an Adj-RIB-In. It does not select best paths, re-advertise received routes or program a forwarding table, and it has no import or export policy.
Who may connect
Connections are taken only from the addresses of configured peers, and from
each of those only a few at a time. Anything else is closed unread and
counted in bgp_connections_refused_total. accept_unconfigured_peers: true
lets any host open a session as any AS; it exists for tests on a closed
network.
What is taken from a peer
Every route a peer announces is judged before it goes into the routing
table (import in a peer's configuration; see config.proto):
- Refused for any peer unless turned off: prefixes that are not routed on the Internet (private, documentation, multicast, reserved), prefixes longer than /24 or /48, a default route, paths with a private or reserved AS number, and paths that do not begin with the peer's own AS.
- Always refused: a path with our own AS number in it.
- If configured: a list of the prefixes the peer may announce and of the AS numbers its routes may originate from.
max_prefixes: one route more ends the session with Cease / Maximum Number of Prefixes Reached and keeps the peer down for ten minutes.
A refused route is dropped, along with the route it would have replaced.
bgp_routes_rejected_total counts them by reason, and the status page shows
how many routes each peer has and how many were refused.
Route origin validation (RPKI)
With rpki { caches { address: "..." port: 3323 } } in the configuration,
the daemon keeps the validated ROA payloads of a validator (Routinator,
rpki-client with StayRTR, FORT) over the RPKI-to-Router protocol, and
validates every route from peers outside its own AS (RFC 6811):
- invalid (the holder of the addresses authorised another origin, or a
shorter prefix): refused, like any route the import policy refuses, unless
the peer has
accept_rpki_invalid: true; - valid and not found: taken. Validity does not otherwise change which route is chosen.
The data changes all the time, and the routes follow without the peer having to send them again: a route in the table that becomes invalid is taken out, and a route refused as invalid is kept aside ("held") and taken when it no longer is. If no cache confirms the data for the expire interval (two hours unless the cache or the configuration says otherwise) it is dropped, and nothing is invalid any more.
The status page shows the cache, how fresh the data is, each route's
validity, and how many of each peer's routes are held. /api/rpki has the
same as JSON; bgp_rpki_vrps{family} and bgp_rpki_updates_total are on
/metricz. Refused routes are counted under the reason RPKI invalid.
The connection to the cache is plain TCP, without authentication: run the validator on the same host or on a network you trust.
Finding a route
The status page has a look-up box over the received routes, and
/api/routes/received takes the same questions:
| Query | Answer |
|---|---|
?prefix=193.0.0.0/21 |
Every route for that prefix, the best first |
?address=193.0.0.1 |
Every prefix that covers the address, most specific first |
?within=193.0.0.0/16 |
That prefix and everything inside it |
&peer=NAME |
Only routes from that peer (its name or its address) |
&best=true |
Only the route chosen for each prefix |
&limit=N |
At most N routes (1000 unless given, 100000 at most) |
within, peer and best on their own look through the whole table, which
on a full table takes a moment.
Seeing and alerting on refused routes
A refused route is not in the table. What is kept instead, for each peer, is
how many were refused for each reason and the latest 200 of them with their
AS paths. The status page lists them under "Refused routes";
/api/routes/rejected[?peer=NAME] has them as JSON. The record is kept when
the session ends, so a peer that was cut off can still be looked into. The
log has a line for a peer's first refused route and then at most one a
minute.
For alerting, /metricz has:
| Metric | Meaning |
|---|---|
bgp_routes_rejected_total{peer,reason} |
Routes refused, by peer and why |
bgp_peer_routes{peer} |
Routes an established peer has in the table |
bgp_peer_max_prefixes{peer} |
The peer's limit, where it has one |
bgp_prefix_limit_exceeded_total{peer} |
Sessions ended for going over the limit |
bgp_connections_refused_total{reason} |
Connections closed without a session |
Rules to start from:
groups:
- name: bgp
rules:
# A peer is announcing what its policy does not allow: a leak or a
# hijack on its side, or a list on ours that is out of date.
- alert: BgpRoutesRefused
expr: sum by (peer, reason) (increase(bgp_routes_rejected_total[10m])) > 0
labels: { severity: warning }
# A route that has been through our own AS came back: a loop, or
# someone announcing our AS number.
- alert: BgpOwnAsInPath
expr: increase(bgp_routes_rejected_total{reason="own AS in path"}[10m]) > 0
labels: { severity: page }
# A peer is close to its limit; the limit may need raising before the
# session is cut.
- alert: BgpPeerNearPrefixLimit
expr: bgp_peer_routes / on (peer) bgp_peer_max_prefixes > 0.8
for: 10m
labels: { severity: warning }
- alert: BgpPrefixLimitExceeded
expr: increase(bgp_prefix_limit_exceeded_total[1h]) > 0
labels: { severity: page }
Trying it without a router
fake_peer is a minimal BGP speaker that connects to the daemon, announces a
prefix, prints every UPDATE it receives and, with --echo, re-announces each
received prefix back with its own ASN prepended (dropping NO_EXPORT routes).
The prefix it announces shows up in the received table. The ones it echoes
have been through our AS already, so they are refused as loops and counted
in the peer's refused routes:
cargo run -- testdata/server.textpb
cargo run -p bgp-integration-tests --example fake_peer -- \
--server 127.0.0.1:179 --asn 65001 --announce 203.0.113.0/24 --echo
Please refer to the other implementation crates in this repository where most of the actual implementation lives.
Warning: This is still very much a work in progress and is not yet ready for production use.