rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: rayhaan/ferrix:3a1e42e594239ce8403210836641a52cd5a1823f
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
...
compare: rayhaan/ferrix:speaker-cleanup
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
65 Commits
3a1e42e594 ... speaker-cleanup
Author SHA1 Message Date
Rayhaan JaufeerallyandClaude Fable 5.1 9caaf5018e Review pass on the server: bugs fixed, session setup simplified
ci / check (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
A review of the session driver, the server loop, the dialer, the
advertiser and the HTTP API, with the fixes.

Bugs:

- A peer that stopped reading held its session, and our routes, for as
  long as TCP kept retrying: a blocked write was awaited with no timer
  running. Writes now give up after the Send Hold Time of RFC 9687.
- Routes we originate were sent to a peer in our own AS with our AS
  number on the path and no LOCAL_PREF, which such a peer refuses. They
  now go with an empty path and a LOCAL_PREF.
- Collision detection knew a peer by its BGP Identifier alone. A
  neighbour with one session over IPv4 and one over IPv6 had the second
  closed as a collision, and any host that could connect could displace
  a peer's connection by claiming its identifier. A peer is now its
  address and identifier together.
- The dialer reset its backoff whenever TCP connected, so a peer that
  accepts the connection and refuses the session was dialled every few
  seconds for ever. It resets only after a session was established.
- A failure on our side (routes that cannot be stored) dropped the
  connection without a NOTIFICATION. It goes through the state machine
  and the peer is sent Cease. A queued message that cannot be encoded
  is logged and skipped instead of ending the session.
- The accept loop retried at once after an error, spinning when the
  process was out of file descriptors.
- Attribute sets no route used any more were dropped only when a peer
  went away. A task now does it every minute.
- The advertiser subscribed to session events inside its task, after the
  dialers were started, and could miss the first session.
- Shutdown waited without limit on a session whose queue was full.
- /api/routes/received took any limit and built the reply on a runtime
  thread. The limit is capped and the work is done off those threads.
- The default log level was TRACE, which logs every UPDATE.
- IPv6 listeners are IPv6-only: listening on 0.0.0.0 and [::] fails on
  systems where the second would take IPv4 too, and IPv4 peers arriving
  under mapped addresses matched no configured peer.
- A connection closed after a NOTIFICATION is kept for a moment so that
  the peer can read it.
- The configuration accepted two peers with one name or one address,
  and a router id of 0.0.0.0.
- An HTTP address that cannot be bound now stops the daemon at startup.

Simplifications:

- SessionConfig mixed what a server's sessions share with what is
  configured for one peer, and was cloned and patched field by field in
  two places. It is ServerContext, shared, and PeerSettings, made from a
  PeerConfig in one place.
- The attributes of an UPDATE as they were on the wire travelled from
  the reader to the table through a field of the session, around the
  state machine. They go through it, with the message.
- run_session returns how the session ended and whether it had been
  established; Ok no longer reads as "closed by peer".
- Session is not generic over its reader.
- BgpServerBuilder had nothing to build: BgpServer::bind takes the
  configuration.
- One shutdown path, close_sessions, for the signal handler and for a
  failed server.
- The advertiser's three copies of "send every route" are one.
- PeerId moved from the rib crate, which did not use it, to the server.
- Unused API removed or made private to the crate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 10:46:09 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a8aba52562 Review pass on the codec and the table: bugs fixed, encoders simplified 
A review of the wire codec and the routing table, with the fixes.

Bugs in the codec:

- A NOTIFICATION with a code or subcode we have no name for (subcode 0,
  "unspecific", is common) failed to decode and was answered as a bad
  message. Every pair now reads, as BgpError::Other.
- Optional parameters in the extended form of RFC 9072 were read and
  written with one-octet parameter lengths; they are two.
- An ORF capability listing more than one address family made the whole
  OPEN malformed.
- Bytes after an OPEN's optional parameters were accepted.
- Extended communities were split into fields by rules that were wrong
  for the common types. They are kept as their eight octets.
- Several length fields were computed in u16 and could wrap on encode.

Simplifications in the codec:

- Lengths are filled in after the value is written, for messages,
  attributes, capabilities and the UPDATE's two fields. The parallel
  wire_len functions, which had to be kept in step by hand, are gone.
  Message::to_wire no longer encodes into a second buffer.
- Type codes for messages, attributes and capabilities are each defined
  once. The strum discriminant enums and their hand-written conversions
  are removed.
- One function splits an attribute off a buffer; the UPDATE decoder, the
  plain attribute parser and the table's key reader all use it.
- The UPDATE decoder's 185-line loop is an AttributeReader with one
  method that decides what becomes of an attribute. Duplicate detection
  is a bitset, not a HashSet allocated per UPDATE.
- ParserContext::four_octet_asn is a bool. The "context not set" error
  path, which treated a programming error as a malformed route, is gone.
- Codec functions take a context only if they read it, and return a
  Result only if they can fail.
- MP_REACH_NLRI and MP_UNREACH_NLRI hold an AfiSafi.
- AddressFamilyId is IpVersion, which is what it was.
- Deserialize is no longer derived on codec types: nothing used it, and
  it bypassed IpPrefix's invariants. IpPrefix::to_wire, which was
  broken and unused, is removed.

Bugs in the table:

- The index of a row's best route was sixteen bits. A peer using
  ADD-PATH could send more routes for a prefix than that counts, and
  the index wrapped. It is 32 bits, at no cost in row size.
- Shards and attribute-store stripes were picked from the top bits of
  the hash, which are the bits hashbrown keeps as the tag it compares
  first. With 64 shards each shard's table had 2 of 128 tag values.
  They are now picked from bits the table does not use.
- remove_peer held the peer list's lock for the whole scan of the
  table, so no session could establish while a peer was removed.
- When more prefixes went at once than were left, the forced trim
  dropped a fixed few entries, reset every reader, and re-ran the tidy
  on every later write. It drops the older half.
- Readers were woken by every change, including a repeated announcement
  and changes their journal does not record.
- Reader::poll with a budget of zero never made progress.

Simplifications in the table:

- The two journals of a shard are one Journals value written through
  three calls; the five places that kept them in step by hand are gone.
  Whether an entry is still live is decided in one place.
- The shortcuts of route selection moved to select.rs, next to the
  invariant they depend on, which is now written down.
- announce and withdraw take routes with or without path identifiers;
  the _paths variants are removed.
- The selection key is read off the wire form only; the second
  implementation over parsed attributes is removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 10:32:27 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d49bab58a5 Add ADD-PATH (RFC 7911): several routes for a prefix from one peer 
A peer can now send several routes for one prefix, told apart by a path
identifier, and each is a route of its own in the table.

Wire:
- The ADD-PATH capability lists several address families. It was read as
  exactly one, so the OPEN of a router offering it for IPv4 and IPv6 was
  malformed and the session never came up.
- Prefixes in UPDATEs are `Nlri`: a prefix and a path identifier. The
  identifier is on the wire, in the body and in MP_REACH/MP_UNREACH, for
  the families the session uses ADD-PATH for in that direction. Sending
  and receiving are settled separately, so a session has one codec context
  for each.

Negotiation:
- `add_path` on a peer in the configuration: off (the default), receive,
  send or both, offered for every family offered to the peer.
- In use for a family and a direction when one side offered to send and
  the other to receive. Shown on /statusz and in /api/peers.

Table:
- A route is known by its peer and path identifier together. Identifiers
  are stored only in rows that have a non-zero one, so routes of peers
  that send one route each cost nothing more.
- Equal routes from one peer are chosen between by the lower identifier.
- /api/routes/received gives each route's path_id.

Journal:
- A second journal in each shard records every change to any route, for
  followers that want all the routes of a prefix and not only the best:
  RibTable::reader_of_all_routes and Event::Routes. This is what sending
  several routes to a peer will read. It costs nothing while nobody
  follows it.

Sending: prefixes go out with a path identifier where that was agreed, but
there is still one route for each prefix to send: only originated routes
are announced.

Tested with the table and journal model tests extended to path identifiers
and to followers of all routes, the codec property tests and fuzz targets
extended to ADD-PATH sessions, and end to end in both directions and for
the case where only one side offers it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 00:57:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 31aa3a4563 Use the sharded table in the server: sessions choose best routes 
The server now keeps what it learns in table::RibTable. The Adj-RIB-In
actor is gone, and with it the queue every UPDATE from every session went
through.

- A session takes a number in the table when it is established and writes
  its UPDATEs into the table itself. Its routes are removed when it ends,
  on a blocking thread since that looks at every prefix; a session torn
  down without cleaning up still has them removed.
- For every prefix the best route is chosen (RFC 4271 §9.1.2.2) as routes
  come and go. Nothing acts on the choice yet beyond showing it.
- The UPDATE decoder hands over the bytes of the attributes it kept, so the
  table finds the attribute set from them without writing the attributes
  out again. On a session with two-octet AS numbers the bytes are not in
  the table's form, and the parsed attributes are used.
- /api/routes/received marks the best route of each prefix and returns at
  most `limit` routes (default 1000): the table can hold hundreds of
  millions. /api/status has the totals, and the status page says how many
  it is showing. The time a route was received is no longer kept or shown.
- The event stream reports changes of best route (best_changed,
  best_withdrawn) in place of every route received. Each client follows
  the table's journal with a reader of its own, from the moment it
  connects: table::RibTable::reader_from_now.
- Metrics: bgp_rib_routes, bgp_rib_prefixes, bgp_rib_attribute_sets and
  bgp_rib_journal_entries replace bgp_adj_rib_in_routes.
- A line is no longer logged at INFO for every UPDATE sent or received; a
  full table is a million of them. They are at DEBUG.
- A table with no room for another peer ends the session with Cease / Out
  of Resources.

Four peers sending a synthetic full table each over TCP, 4.9 million
routes: 0.80 s, against 4.76 s before. One peer: 0.45 s against 0.69 s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 00:31:28 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3b20576971 Add the journal: following the table's best routes at one's own pace 
Whatever acts on the best routes (export to peers, the forwarding table,
the status page) reads their changes from a per-shard journal through a
table::Reader. Nothing uses it yet.

- The journal records that a prefix's best route changed, not what to. A
  reader that comes to an entry is given the route as it is then. A prefix
  that changes a thousand times between reads is one event, and the
  journal never holds more live entries than the table has prefixes.
- Each row keeps the sequence number of its latest change; an entry with
  another number is dead and is dropped when the journal is tidied.
- Entries go once every reader has passed them. A reader that does not
  read cannot hold them back for ever: past a limit the oldest half is
  dropped, and that reader is given the whole table again (Reset, every
  best route, Synced). A new reader starts the same way.
- With no reader nothing is recorded.
- Reader::changed waits for there to be something to read.

Tested against a mirror kept from the events: after any interleaving of
announcements, withdrawals, session flaps, reads with small budgets and
readers coming and going, with journals small enough to be tidied and
overrun constantly, a reader that has caught up holds exactly the table's
best routes. Also with writers on four threads and a reader on a fifth.
The property test found one fault (a pass that resumed part-way claimed
to have caught up) and catches three more put in on purpose.

On the AMS-IX dump, 12 threads: loading takes 2.29 s with a follower
reading throughout against 2.17 s with none, the journal peaking at
170,000 entries for 1.39 million prefixes. With a follower that does not
read, the journal ends at 1.74 million entries, and the follower then
catches up in 0.08 s.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 00:14:20 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 eab891567b Add the sharded routing table: Adj-RIBs-In and Loc-RIB in one 
bgp_rib::table::RibTable holds every route from every peer and the best
one for each prefix. It is not used by the server yet; this adds it with
its tests and benchmarks.

- Organised by prefix. A row holds the prefix's routes from all peers in
  one array; choosing again after a change reads that row and nothing else.
- A route is sixteen bytes: a pointer to its attribute set and one word
  holding the peer and the first three steps of selection (LOCAL_PREF, AS
  path length, ORIGIN) as a single number. That number alone settles most
  changes; a tie goes on to MED, compared only within a neighbouring AS
  and in a way that does not depend on arrival order, and the tie-breaks.
- Attribute sets are stored once, in wire form, with what selection needs
  from them worked out when they are stored.
- Sharded by hash of the prefix, a lock per shard. A session applies its
  own UPDATEs; there is no queue or owning thread in front of the table.
- Readers take one shard at a time for reading and look at rows in place.

Route selection is checked against a plain implementation with no
shortcuts by a property test over random announcements, withdrawals and
session flaps, for both the chosen route and the count of changes.

On a RIPE RIS dump from AMS-IX (148 peers, 39.3 million routes), against
the Adj-RIB-In actor, which does no selection:

                          actor       table
  memory                  11.0 GiB    1.8 GiB   (279 -> 48 bytes a route)
  load, 1 thread          10.2 s      12.3 s
  load, 12 threads        (no gain)   2.1 s     (19 million routes a second)
  read every route        1.6 s, copying 8 GiB, table stopped
                                      0.2 s, in place, table running

The benchmark crate counts memory only with --features count-memory: the
counters are shared between threads and distort the timings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-02 00:01:50 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c1cf15c792 Benchmark on a real table: read MRT dumps from route collectors 
crates/bench reads TABLE_DUMP_V2 dumps (RFC 6396) as published by RIPE RIS
and RouteViews, and `--bench real_table` runs the Adj-RIB-In on one. The
dumps are not checked in; crates/bench/data/README.md says how to fetch
one. Without a dump the benchmark says so and does nothing.

It also prints what in the table decides the cost of a RIB: routes per
prefix, distinct attribute sets, how many routes share a set, and how much
the sets are shared between peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:43:22 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 7e488e31aa Make IpPrefix a fixed-size value with no allocation behind it 
A prefix was a Vec of its bytes: a heap allocation for every prefix parsed,
stored or copied, and a variable-length key to hash. It is now twenty
bytes, Copy, with private fields and one representation; `bytes()` gives
the wire form.

On the full-table benchmark this alone takes decoding from 9.3 to 10.7
million routes a second and inserting into the Adj-RIB-In from 2.9 to 3.6.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:38:39 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 64df419b3a Add a benchmark of taking in the Internet routing table 
crates/bench makes a synthetic full table (a million IPv4 and 230,000 IPv6
prefixes, announced by several peers in UPDATEs shaped like a router's)
and times the steps of taking it in: decoding, the Adj-RIB-In, snapshot,
peer removal, and the whole path over TCP. It also counts the memory the
table holds, and fills a plain packed hash map with the same routes as a
point of comparison.

Run with `cargo bench -p bgp-bench`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:30:29 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a772322d9e Unwind panics safely: say goodbye to the peer, keep the other sessions 
A panic is a bug, and it should cost as little as it can. The server
already closed every session with a Cease and exited when a RIB or a
background task was lost. This fills in the rest:

- A panic inside a session is caught. The peer is sent Cease /
  Administrative Reset, the session's routes are removed in the usual
  order, and the dialer reconnects. Before, the peer was left with a TCP
  connection closing under it and no reason.
- The tables every session shares (the session registry and the collision
  table) no longer refuse to be locked after a panic. Lock poisoning would
  have made each remaining session panic in turn the next time it touched
  one, so one bug in one session took all of them down.
- The server's watcher reports a task that panicked with the panic's
  message, and is now tested for both a panicking task and a stopped RIB.
- The daemon sends panics to its log, with their location, instead of
  leaving them to the default report on stderr.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:24:00 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 41090966e4 Stop treating a dead RIB as an empty one 
The RIB handles swallowed every failure of their actor: an UPDATE applied
to a stopped Adj-RIB-In vanished, and a snapshot of it came back empty.
Sessions stayed up, peers' routes went nowhere, and the status page showed
a healthy router with no routes. An actor stops only by panicking, so this
is the state the server would be in after a bug.

- Every operation on AdjRibIn and LocalRib returns Result<_, RibStopped>.
  LocalRibError is replaced by it.
- The server waits on both actors, the advertiser and the dialers. If one
  stops it sends every peer Cease / Administrative Shutdown and returns an
  error, so the process exits and can be restarted. Before, the tasks'
  handles were held and never looked at.
- A session that cannot store an UPDATE ends with the same Cease.
- The HTTP API answers 503 with the reason for anything that needs a table
  it cannot read, including /metricz: a scrape that fails is noticed, last
  known sizes are not.

Also:
- The routes of a session are now removed even if its task is torn down
  without running its cleanup (a panic), instead of staying in the table
  as routes of a peer that is gone.
- Table sizes for the status page and metrics are asked for as a count
  instead of copying the whole table to take its length.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:18:59 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 6197a7ad74 Handle errors in UPDATE messages as RFC 7606 says, not by resetting 
Any malformed UPDATE used to reset the session, withdrawing every route
learned over it. One bad attribute, typically originated ASes away and
passed on by a peer that did not look at it, could take down a session
carrying a full table.

UpdateMessage::decode now applies RFC 7606 and returns the UPDATE as it is
to be acted on, plus a report of what was wrong:

- Treat-as-withdraw: a malformed or misflagged attribute, a missing
  mandatory one, or an attribute that overruns the attribute list. The
  routes the UPDATE announced (in the body and in MP_REACH_NLRI) are
  withdrawn.
- Attribute discard: malformed ATOMIC_AGGREGATE, AGGREGATOR, AS4_PATH and
  AS4_AGGREGATOR; an attribute sent twice (the first is kept); LOCAL_PREF,
  ORIGINATOR_ID and CLUSTER_LIST from an external peer; AS4_PATH between
  four-octet speakers.
- Session reset remains where the prefixes are in doubt: the lengths that
  frame the message, an unreadable prefix, a malformed or repeated
  MP_REACH_NLRI or MP_UNREACH_NLRI, an unrecognised well-known attribute.
  Each now sends its own subcode instead of always Malformed Attribute
  List.

The session logs each handled error and counts it in
bgp_update_errors_total{handling}. The parser context says whether the
peer is internal (replacing a field nothing read).

Stricter than before, because the checks did not exist:
- An announcement must carry ORIGIN and AS_PATH, NEXT_HOP for routes in the
  body, and LOCAL_PREF from an internal peer.
- An AS_PATH segment must be an AS_SET or AS_SEQUENCE and not empty.
- An attribute value must be used up entirely by its parser; a NEXT_HOP of
  six octets, say, was accepted before.

Bugs found on the way: we sent ATOMIC_AGGREGATE flagged optional and
AGGREGATOR flagged non-transitive. The flags now come from one table used
for both writing and checking.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 23:12:33 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 39632e3c38 Property tests and fuzz targets for the parsers, and the bugs they found 
Property tests (run by `cargo test`):
- crates/packet/tests/properties.rs: every kind of message, generated,
  must be written, be as long as wire_len says, and read back equal; and
  noise, noise behind a valid header, and valid messages with bytes damaged
  must never do more than return an error. Anything the decoder accepts
  must be showable and survive being written and read again.
- IpPrefix: text round trip, noise as text, `contains` against a bitwise
  model.
- The session's framing: a stream of messages yields the same messages
  however it is cut into reads.

Fuzz targets (fuzz/, cargo-fuzz on nightly, a minute each in CI): raw
messages, message bodies behind a valid header, prefix text, the
configuration file.

Bugs found, each with a named regression test:
- A prefix longer than its address family (e.g. an IPv4 NLRI of length 99)
  was accepted, and showing it panicked. The session logs every route, so
  one UPDATE from a peer ended the session task. Such a prefix is now
  malformed, and showing a prefix cannot fail.
- Bits past a prefix's length were kept, so the same prefix sent with stray
  bits was a different key in the RIB. Prefixes now have one form.
- An opaque extended community was read as seven octets instead of eight,
  putting every community after it out of step.
- AGGREGATOR could be neither read from nor written to a peer with
  two-octet AS numbers.
- The Outbound Route Filter and Long-Lived Graceful Restart capabilities
  were written differently from how they are read.
- A capability value or unknown attribute too long for its length field
  was written with the length wrapped around; it is now an error, or for
  the attribute the two-octet length.

Also: ROUTE-REFRESH carries its AFI and SAFI instead of being modelled as
an empty message, and one with a body of the wrong length is refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 22:54:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b0762874e0 Make address families open values; keep "supported" a separate list 
AFI and SAFI were closed enums, so a family without a variant could not be
represented at all, and the set of negotiated families was a struct with
one bool per family. Supporting another family meant touching both.

- `Afi`, `Safi` and `AfiSafi` in bgp-primitives are newtypes over the wire
  numbers, with constants and names for the common families. Capabilities
  and MP_REACH/MP_UNREACH use them, so any family a peer names is read as
  what it is. The special case that turned a capability for an unlisted
  family into an unknown capability is gone.
- `FamilySet` is a set of `AfiSafi`. What we can exchange routes for is the
  explicit `family::SUPPORTED` list; only those are offered in our OPEN.
- `AddressFamilyId` stays as the IP version of a prefix.
- NLRI is still a list of IP prefixes. MP_REACH/MP_UNREACH for a family
  whose NLRI is something else (labelled, VPN, EVPN, ...) is refused
  rather than misread as prefixes, which is what happened before for the
  SAFIs the old enum listed.

Bug found on the way: the reserved octet in MP_REACH_NLRI was skipped only
for IPv6, so an IPv4 MP_REACH_NLRI was parsed with an extra 0.0.0.0/0
route that the peer never sent.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 22:38:40 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 22d0099a46 Negotiate address families and AS number width instead of assuming them 
The OPEN used to offer IPv4 and IPv6 unicast unconditionally, and what the
peer offered was ignored apart from its four-octet AS number.

Address families (RFC 4760):
- A peer's `address_families` in the configuration says which to offer it;
  all of them by default.
- A session carries the families both OPENs offered. A peer that sends no
  Multiprotocol capability is a plain RFC 4271 speaker: IPv4 unicast.
- Routes of a family that was not negotiated are not sent, and are ignored
  when received; the rest of the UPDATE is applied and the session stays up.
- A session with no family in common is refused with Unsupported
  Capability, naming the capabilities we needed (RFC 5492).
- The negotiated families are shown on /statusz and in /api/peers.

Bugs found on the way:
- A capability naming an address family we have no constant for (EVPN,
  VPNv4, ...) failed the whole OPEN, so such a peer could never establish.
  It is now kept as an unknown capability and ignored.
- An UPDATE could not be encoded for a peer without four-octet AS support,
  which ended the session on the first route. AS_PATH is now written two
  octets wide with AS_TRANS where needed, and AS4_PATH carries the real
  numbers (RFC 6793).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 22:22:54 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a4a180e948 Make the session state machine explicit and free of I/O 
A session was one function that walked OpenSent, OpenConfirm and
Established with the socket, the timers, collision detection, the registry
and the RIB all in the same loop. There was no state or event type, and no
way to test a transition without a TCP connection.

It is now two pieces:

- fsm.rs is the state machine of RFC 4271 section 8 for a connection. It
  takes an Event and returns the Actions to carry out. It owns no socket,
  timer or task. What the OPENs negotiated lives in the states that have
  it, so there is nothing to unwrap. Thirteen tests cover the handshake,
  hold-time negotiation, refused OPENs, messages out of turn, timer expiry,
  collisions and shutdown by feeding it events.
- connection.rs is the driver: it turns the socket, the timers and the
  outbound queue into events and carries the actions out.

Behaviour changes:

- A message that arrived in pieces was lost if a timer fired between the
  pieces: read_exact was used inside select!, which discards what it has
  read when another branch wins. Messages are now framed from a buffer
  that survives cancellation.
- A NOTIFICATION from the peer during the handshake is no longer answered
  with an FSM error; RFC 4271 says to close without replying.
- A session the peer ended is reported as 'peer sent NOTIFICATION: ...'
  with its error, where the dialer used to record 'session closed by peer'.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:59:55 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 16a4e3b640 Typed errors in the library crates; send the NOTIFICATION RFC 4271 assigns 
The library crates returned eyre reports, so a failure was a string and the
session code could not tell a truncated UPDATE from a bad header. Whatever
went wrong, the peer got an FSM error or nothing.

Each crate now has its own error types:

- bgp-primitives: PrefixError, UnknownAfi and UnknownSafi, and EncodeError
  for every to_wire.
- bgp-packet: Message::decode returns a DecodeError naming the NOTIFICATION
  code, subcode and data for the failure (header errors per section 6.1,
  OPEN per 6.2, UPDATE per 6.3). InvalidValue and CommunityError replace
  the remaining strings. from_wire is now a thin wrapper around decode.
- bgp-server: SessionError, ServerError and ConfigError. Every abnormal end
  of a session goes through one function that sends the NOTIFICATION the
  error maps to. The message length is checked against the 4096-octet
  maximum before the body is read.
- bgp-web: serve returns io::Error.

eyre remains in the binary and in tests.

Bugs found and fixed on the way:

- An OPEN with no optional parameters, which is valid, failed to parse.
- An unsupported BGP version was reported as an unknown optional parameter.
  It now gets Unsupported Version Number with the version we speak as data.
- A SAFI above 255 was truncated to its low octet and could match a known
  one.
- Prefix text with a length beyond the address size (10.0.0.0/99) was
  accepted.
- Message::to_wire truncated the body length to 16 bits before checking it.
- A peer hold time above 65535 seconds in the configuration wrapped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:18:35 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 666471e38d Share path attributes between Adj-RIB-In entries 
Each received prefix kept its own copy of the UPDATE's whole attribute list,
and for multiprotocol routes that list included MP_REACH_NLRI with every
other prefix from the same UPDATE: n prefixes stored n copies of n prefixes.

An UPDATE is now turned into a ReceivedUpdate once. Its prefixes share one
attribute list, which leaves out MP_REACH_NLRI and MP_UNREACH_NLRI; the next
hop from MP_REACH_NLRI is kept on the entry. The RIB takes one command per
UPDATE instead of one per prefix, and its table is keyed by peer so that
dropping a peer's routes no longer scans every route.

The status API shows the multiprotocol next hop as an address instead of
its Debug form.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:08:14 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 2de0a14712 Add the Apache 2.0 licence text and CI 
Cargo.toml has declared Apache-2.0 without the licence being in the tree.

CI checks formatting, clippy and the tests on every push, with warnings as
errors. The workflow lives in .github/workflows, which Gitea Actions also
reads.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:01:26 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b7c7514d3e Fix all clippy warnings and enforce lints across the workspace 
Lints are declared once under [workspace.lints] and every crate opts in:
clippy's default set, plus dbg!, todo! and unimplemented!, and unsafe code
is forbidden. Every crate now inherits its version, licence, authors and
repository from the workspace, so they no longer disagree (four were at
0.0.0).

The poptrie module is unfinished and unused; it says so and expects the
dead-code lint instead of warning on every build. Its construction method
is renamed from new, which took &mut self, to build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:01:26 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 126368852a Parse the Multiple Labels capability instead of panicking 
The capability parser had a todo!() for capability code 8 (RFC 8277), and
the encoder two more. A peer advertising it took the session task down.
It is now parsed into its (AFI, SAFI, count) entries and round-trips.

Also carries clippy's mechanical fixes for this file.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 21:01:26 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 2ab6c1c38b Format the workspace with rustfmt 
Default settings. No change in behaviour; the tree was last formatted
before the lecture work and had drifted to a mix of line widths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:58:12 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d8e8924b4e Point the repository URL at ferrix 
The repository was renamed. Also drops workspace.package.name, which Cargo
ignores (a package name cannot be inherited) and warned about on every build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:53:26 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b7e5084f83 Tidy the remaining top-level files after the lecture 
The preview launcher pointed at the removed slides and editor state does
not belong in the tree. The Dockerfile and README no longer refer to the
lab.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:51:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 22b9eeacdc Count multiprotocol prefixes in the UPDATE log lines 
The 'Sending UPDATE' and 'Received UPDATE' lines counted only the IPv4
fields of the message body, so every IPv6 UPDATE logged announced=0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:51:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b800e3a322 Originate routes from the configuration; make the HTTP API read-only 
Routes were announced and withdrawn through an HTTP form, each tagged with a
free-text owner and checked against an allow-list and a per-owner limit.
That was the lab's interface, not a router's.

Routes now come from 'originate' entries in the configuration file and are
in the local RIB before any session starts. The local RIB holds one route
per prefix with no notion of owners or policy. local_route_policy is gone
from the configuration.

The HTTP API keeps its GET endpoints (status, config, peers, originated and
received routes, events, metrics). The announce, withdraw, wire-bytes and
well-known-community endpoints are removed, and the status page loses the
announce form, the community presets and the byte decoder. Community text
parsing moves from the web crate to bgp-packet so the configuration can use
it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:51:15 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c37cfd029a Remove the lecture material and the ping monitor 
The slides, handout, lab router configs, RIPE objects and load test were
for the Network Security 2026 lecture and are kept on the netsec-2026 tag.

The ping monitor sniffed ICMPv6 echo requests so students could watch their
prefix being used. It has no place in a BGP speaker: it needs CAP_NET_RAW,
adds the only unsafe code in the tree and a libc dependency.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:45:24 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b1b72fed39 lab: multi-line answer boxes, one sheet per student 
The fill-in fields put the label on its own line with dotted lines to write
on. handout.tex takes \studentnum and \studenthex; build-class.py compiles
one copy per student with the prefix filled in, checks each is two pages
and joins them for double-sided printing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:40:55 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 2910a2a800 slides: light theme, final edits, single-file bundle 
As presented. The palette is light on request of the reviewer; code blocks
and byte dumps keep a dark ground so the byte colours still work. The OPEN
walk lights the hovered bytes as an inverted block and its table cells are
dark text on a tint of their group colour.

build.py inlines reveal.js, the stylesheets, scripts and Markdown sections
into slides/dist/hands-on-bgp.html for uploading.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:40:55 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 8c6e8a80e1 chore: ignore .DS_Store and generated lecture output 
slides/dist/ (the single-file bundle) and docs/lab/handout/class.pdf (one
sheet per student) are build products.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 20:40:55 +02:00
Rayhaan Jaufeerally 597941a70a handout finalized 2026-09-24 10:33:45 +02:00
Rayhaan Jaufeerally b607058650 Ferrix web UI working for demo and slides mostly done 2026-09-23 15:38:06 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 f3bb8cc89e lab: exercise sheet cut to two pages, three simple steps 
One double-sided A4 sheet with 1in margins, still in the LNCS look.
1. Who announces the address you are using: find your current address and
   its prefix and origin AS, then whois it and your lab prefix.
2. Announce your prefix, read the bytes, find it from outside, ping it.
3. Announce with communities: only the SwissIX route servers, then only
   SWITCH, with reachability checkboxes, one discussion question, clean up.
Sections 1 and 2 fill page one; section 3 has page two. Dropped the ETH
lookup, the long registry comparison, NO_EXPORT, the byte-decoding task,
the reference appendix and most of the bibliography.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 22:07:23 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 83ae367b1c lab: printable exercise sheet for the Hands-on BGP lecture 
docs/lab/handout/handout.tex, typeset with Springer's LNCS class (title
block, abstract and keywords, figure, booktabs tables, RFC references), six
A4 pages so it prints double-sided on three sheets. The built PDF is
committed alongside for printing.

Three sections students work through with the slides:
1. Who holds which addresses: whois their own /48 and ethz.ch's prefix in
   the RIPE database, then compare with what is actually routed.
2. Announcing the prefix on the lab page, reading the UPDATE bytes, finding
   it from outside, and reaching it.
3. Steering with communities until the prefix is reachable only from the
   university network (AS559 via the SwissIX route servers), with
   reachability checkboxes for laptop and phone at each step.
Plus a one-page reference appendix: IPv6 notation and hex, the UPDATE
layout, path attributes, glossary.

The lab host, the lab address space and the prefix pattern are macros at
the top of the file. Expected answers were checked against the RIPE
database and RIPEstat (ETH: 2001:67c:10ec::/48, ASSIGNED PI, origin AS559;
lab: 2a0d:d741::/40, AGGREGATED-BY-LIR, route6 origin AS210036).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 22:02:32 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9d6996c6c1 slides: no step-by-step reveals; UPDATE example is 2a0d:d740::/48 at SwissIX 
- Removed every fragment and stepped code highlight: a slide now shows
  everything at once. Two code blocks keep a static highlight on one line.
- Dropped "What stands between you and an outage".
- The UPDATE walked through in section 3 is now 2a0d:d740::/48 as AS210036
  announces it to the SwissIX route servers: next hop 2001:7f8:24::c9
  (pr01's port on the peering LAN), AS path 210036, MED 0 and 210036:10:2,
  matching what pr01's export filter puts on routes to peers. Encoded with
  the packet crate and parsed back to the identical message; the slide
  generator asserted every quoted number, including the next hop, against
  the bytes. Adds a MULTI_EXIT_DISC slide to the walkthrough.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:48:12 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1399cbd39f slides: "Hands-on BGP", restructured into five sections 
- Title is now "Hands-on BGP"; the AS-path diagram is gone from the cover.
- Sections: IPv6 addresses and prefixes; where AS numbers and prefixes
  come from (IANA, RIRs, the RIPE database, IRR vs RPKI); BGP and the
  bytes of an UPDATE; IP transit vs Internet exchanges (the peering LAN as
  an Ethernet segment, route servers); the lab (ferrix, pr01.rue, SwissIX,
  NL-ix, France-IX, DE-CIX, transit).
- The UPDATE walked through in section 3 is a real 94-byte capture from
  ferrix for 2a0d:d741:1::/48. Vertical slides light up one part at a
  time with its decode; the generator asserted every quoted number
  against the capture. Session counts come from the tracked bird.conf,
  the route6 object from the RPSL file, SwissIX's LAN from PeeringDB.
- New components: focusable bytes, decode table, prefix bits, and SVG
  diagrams drawn in the theme's colours.
- Opening index.html straight from disk used to give a title and empty
  panels, because browsers block the module and the section files there.
  The page now explains that, and also reports a failed reveal.js load.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:41:31 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 106edd1df0 slides: reveal.js deck template for the BGP lecture 
Structured like Matt Godbolt's newer talks (hrt-theme.css in
what-happens-beneath-accu-2026): reveal.js with no stock theme but a custom
dark one, every slide on a rounded panel, IBM Plex Sans and Mono, monokai
code with glowing stepped lines, 16:9 with fade transitions, and the
content in one Markdown file per section.

Made our own: the palette is a handful of variables (with ?accent= presets
for comparing), and his ring-and-thread corner motif became a chain of
nodes and links, an AS path, which also opens the title slide. A wordmark
sits where his decks carry the employer logo.

Also: a small `bird` syntax highlighter for router config on slides, a
bytes component in the four groups of the /statusz bytes view, columns,
boxes, and a `bare` slide class. reveal.js 6 loads from a vendored copy
when slides/vendor-reveal.sh has been run (works offline, ignored by git)
and from the CDN otherwise. Section files carry a starter outline for the
lab that doubles as a tour of the slide types.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:30:41 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e35507d082 lab: track the applied router config and the RIPE objects 
- docs/lab/prod/pr01/bird.conf is now the config generated for per-session
  export control (every eBGP session passes its neighbour AS and location
  to the shared export function; customer sessions go through the same
  gate), with the ferrix stanza in its same-host form (::1, multihop).
- docs/lab/ripe/lab-objects.rpsl: the inet6num and the 128 route6 objects
  for the student /48s. Public registry data, no credentials.
- Ignore Python bytecode caches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:21:25 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c60bd8baf5 docs: use built-in Apache error documents so the login covers every path 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:13:56 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 36b73844af docs: keep Apache's /error/ pages out of the proxy so 401 prompts for a password 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 21:09:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9d644a212a tools: concurrent load test for the announce API 
tools/loadtest.py releases N simulated students through a barrier so they
announce at the same instant, then verifies behaviour rather than only
status codes: the table holds every route under the right owner, each
route became an UPDATE per established session, the event stream lost no
event, concurrent takeover attempts are all refused, withdrawal leaves
nothing behind, and the daemon's own counters show the matching number of
UPDATEs sent for announcements and withdrawals. Standard library only;
supports basic auth and repeated rounds.

Safe by default against a production router: every route carries
210036:0:0 (announce nowhere), leftovers from earlier runs are removed,
prefixes held by real owners are skipped, and everything is withdrawn even
if the run fails.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 19:49:19 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1ea2a12bd6 docs: tested Apache reverse proxy with basic auth for the status page 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 19:32:50 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 42d79354e3 docs: peer with BIRD over ::1 on the router; explain the multihop requirement 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 19:29:31 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 fd496d26bf feature: run on the router itself: protoc-free build, per-peer source address 
Preparing to run ferrix next to BIRD on the production router.

- build: the config proto is now parsed in pure Rust, so building needs no
  `protoc` binary. It previously worked only where one happened to be
  installed; the Dockerfile in this repo could not have built.
- peers gain `local_address`, the source IP for connections we open (and
  therefore the next hop we advertise). When both speakers share a host
  the OS would otherwise pick an arbitrary local address and the peer,
  which identifies us by address, would refuse the session. Address-family
  mismatches are rejected at config load; an address the host lacks shows
  up as the peer's backoff reason.
- docs/lab/prod/on-router: static musl cross-build instructions, config,
  hardened systemd unit (unprivileged, SIGTERM gives BIRD a Cease),
  loopback addresses for networkd, the BIRD stanza for a same-host
  multihop session, and reverse-proxy examples with TLS and a password,
  since the API has no authentication.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 18:15:09 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 af76ecb345 fix: leave sessions politely on shutdown; answer ROUTE-REFRESH 
Against a production BIRD router, restarting ferrix was followed by
several refused connection attempts before the session came back.

- The daemon had no signal handling: Ctrl-C killed the process and the
  peer only ever saw a dead TCP connection. On SIGINT or SIGTERM it now
  sends Cease / Administrative Shutdown (RFC 4486) to every established
  peer, waits up to two seconds for the sessions to close, and stops the
  dialers from reconnecting meanwhile. A NOTIFICATION queued for a session
  ends that session once sent.
- ROUTE-REFRESH was parsed and ignored. A peer sends it when its import
  policy changes (e.g. after `birdc configure`), expecting our routes
  again so the new policy can be applied. The session now reports the
  request and the advertiser re-sends every local route to that peer. The
  request also appears in the event stream and the status page log.

Tests: a refresh request in real wire format re-sends all routes and keeps
the session up; shutdown delivers the Cease to every peer and closes the
sessions. Verified with the real binary: SIGINT, peer logs the Cease.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 17:51:04 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 57099da73d feature: export control by neighbour AS and by location 
The production router has the same neighbour AS at several exchanges, so
per-AS control alone cannot express "only at SwissIX". Every session now
has two coordinates, neighbour AS and location, and the scheme gains two
functions:
  210036:0:<as> / 210036:2:<location>   never announce      (wins)
  210036:1:<as> / 210036:3:<location>   announce
  210036:0:0                            announce nowhere
  nothing                               announce everywhere
docs/lab/prod/bird-ferrix.conf documents the functions, the per-session
call (export where peers_out(<as>, <location>)) and the gate that sessions
exporting everything need as well.

Status page: location presets, and AS210036 communities are decoded from
their structure so any value reads well in the bytes hover.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 17:34:57 +02:00
Rayhaan Jaufeerally 98f25a5847 Add existing BIRD config 2026-09-20 17:29:01 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 77c406e7d1 docs: wire lab export control into a shared BIRD filter via a function and wrapper filters 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 17:21:36 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b28f59f30d feature: AS210036 export-control large communities for lab routes 
BIRD side (docs/lab/prod/bird-ferrix.conf): the ferrix import filter now
admits 210036:0:* and 210036:1:*, and lab_export_allowed(peer_as) decides
per eBGP session whether a lab route is announced, with the same shape and
precedence as the SwissIX route-server communities:
  1. 210036:0:<peer-as>  never   2. 210036:1:<peer-as>  announce
  3. 210036:0:0          nobody  4. default             everyone
The per-session export block applies only to routes learned from ferrix,
strips the private ASN and removes the action communities before sending.

Status page: the large-community picker is grouped into "AS210036" and
"SwissIX route servers" presets, including "announce to the SwissIX route
servers" and "send to SWITCH (AS559)"; the bytes hover names both families.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-20 17:02:42 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 eb77090d01 feature: config zippy on statusz; drop prepend options from the UI 
- The status page has a collapsible "Configuration" section showing the
  config file as read at startup, with its path. Served by the new
  GET /api/config; RouterInfo::with_config carries the text.
- Removed the AS-path prepends field and the SwissIX prepend presets
  (42476:101..103) from the form to keep the exercise focused. The API
  still accepts `prepend`.
- Docs follow: the handout loses its prepending exercise, the lab README
  its prepend row, and the BIRD config notes that the 42476 namespace is
  still admitted for hand-typed values.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 22:46:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3f04152f19 docs: simplify prod ferrix config, name the peer pr01.rue.rayhaan.net 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 22:44:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 dd838eec9a feature: decoded hover for UPDATE bytes; opt-in auto-refresh 
- Hovering (or focusing) a byte segment opens a popup that decodes it:
  header fields, the two length fields, and each path attribute with its
  flags spelled out and its value in readable form. AS_PATH shows segment
  type, ASNs, path length and ASN width; COMMUNITIES shows well-known
  names; LARGE_COMMUNITIES shows the SwissIX meaning of known values;
  MP_REACH/UNREACH show address family, next hop and prefixes; NLRI and
  withdrawn routes list prefixes. Decoding happens on hover from the bytes
  themselves. /api/routes/local/wire gains four_octet_asn so the AS_PATH
  width is known.
- The page is now a snapshot by default. ?refresh=<seconds> (1..3600)
  turns on reloading of all tables at that interval and connects the
  event log; the header offers 2s / 5s / 30s, "turn off" and "reload now".
  Events only feed the log, so the tables never change faster than asked.
  The reader's own announce / withdraw still shows its effect at once.
- Removed the "tables refresh on every event" footer text.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 22:20:37 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 20cad53b24 fix: statusz tables no longer flicker on refresh 
Every poll and event rebuilt each table from scratch, and an open bytes
view collapsed to "loading..." and was refetched every five seconds.

- Tables are rewritten only when their markup changes. Clock-dependent
  text (ages, retry countdowns, uptime) is rendered as placeholders and
  ticked in place once a second, so unchanged data means no DOM writes.
- The bytes view is cached per route and set of established sessions, and
  the previous bytes stay visible while new ones load.

Measured with a MutationObserver: zero table rebuilds across 13 s of
polling with a bytes view open; one announcement redraws only the local
and received tables once each, with no "loading..." flash.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 17:10:42 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 cc022c3b7b feature: restyle statusz after the classic Google status pages 
Modelled on Vitess's servenv status page and golang.org/x/net/trace, both
written by Google engineers to mirror the internal pages: plain sans-serif
HTML, default links and native form controls, dense bordered tables, and a
full-width centred light blue (#eef) heading bar per section. Header shows
"Status for ferrix_bgp" with start time, uptime, version, router ID and AS
on the left and host, "View:" links and live-update state on the right.
The dark theme is dropped on purpose.

Behaviour is unchanged (live updates, owner-gated withdraw, per-session
UPDATE bytes, looped-route highlight, peers polling). Additions:
- a SwissIX large-community picker next to the field; presets with a peer
  AS ask for the number
- event log lines include prepend and MED
- /api/status gains hostname and started_at_unix (RouterInfo::new)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 17:06:00 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 d23a578d72 docs: listen on 9179 in the prod ferrix config (179 needs root on macOS) 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 16:08:22 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 22384d0b07 docs: use the real WireGuard tunnel addresses in the prod peering configs 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 16:05:44 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e41776f253 docs: allow SwissIX large action communities through the ferrix import filter 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 16:00:42 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 160e28690a fix: capability list parsing; advertise IPv6 unicast; BIRD peering config 
Preparing to peer with a production BIRD router turned up two problems.

- packet: most Capability::from_wire arms returned the per-capability
  buffer (always empty by then) instead of the bytes following the
  capability. The list parser then saw an empty remainder and silently
  dropped every later capability. Any OPEN listing extended message, role,
  graceful restart, add-path, enhanced route refresh, LLGR, extended next
  hop or an unknown capability before the 4-octet ASN capability lost the
  ASN, so the peer appeared as AS_TRANS (23456) and a configured
  remote_asn was refused with Bad Peer AS. BIRD and FRR both do this.
  Single-capability round-trip tests could not catch it; new tests parse
  every capability with another following it, and a full BIRD capability
  list.
- server: advertise MultiProtocol IPv6 unicast alongside IPv4. Without it
  a peer will not accept the IPv6 routes we originate via MP_REACH_NLRI.
- Interop test: byte-for-byte BIRD 2 OPEN is accepted with the right ASN,
  and an IPv6 End-of-RIB neither drops the session nor creates a route.
- docs/lab/prod: BIRD config for an announce-only ferrix neighbour
  (strict /48 import filter, community scrubbing, unreachable next hop,
  import limit, export none, passive) and the matching ferrix config.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:57:23 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 87f96f9389 feature: RFC 4271 §6.8 connection collision detection 
When both ends dial each other, exactly one TCP connection must survive
and both ends must pick the same one. Previously ferrix relied on the
other router to sort this out and could briefly hold two sessions.

- collision: a shared map keyed by the peer's BGP Identifier holds the one
  connection currently preferred for that peer plus a kill signal. Each
  connection calls admit() on receiving the peer's OPEN. The connection
  initiated by the higher BGP Identifier wins; the loser gets Cease /
  Connection Collision Resolution, whether it is the new connection or an
  existing one in OpenConfirm running in another task.
- A collision with an Established session closes the new connection (RFC
  default). The optional CollisionDetectEstablishedState is available as
  collision_detect_established_state: a restarted peer's new connection
  then replaces the stale session at once instead of being refused until
  the hold timer expires. Enabled in the lab config.
- Two connections opened by the same side: the newer wins.
- OPEN validation rejects a peer using our own BGP Identifier or 0.0.0.0
  with Bad BGP Identifier; collision resolution needs distinct identifiers.
- OpenConfirm no longer times out instantly when the negotiated hold time
  is zero.
- Sessions record which side opened them; shown as "opened by" on the
  status page and as `direction` in /api/peers. New counter
  bgp_connection_collisions_total{closed}.
- Tests: unit tests for the rule (including that both ends agree), and
  integration tests for simultaneous open in both identifier orders and
  both arrival orders, Established default, restart replacement with route
  flush, distinct identifiers, slot release, and Bad BGP Identifier.
  Existing fan-out test now gives its two peers distinct identifiers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-19 15:05:13 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 a137160b48 feature: lab topology, student handout; drop peer routes on session end 
Phase 5 of the teaching demo plus two fixes found while exercising the
dialer on the status page.

- Adj-RIB-In: routes learned from a peer are removed when its session ends
  (RFC 4271 §8.2.2), emitting a Withdrawn event per prefix before the
  session Down event. Previously stale routes from dead sessions stayed in
  the received table.
- statusz: poll the peers table every 5 s so dial state (Connecting /
  Backoff and its retry countdown) stays current; those changes carry no
  event.
- docs/lab: docker-compose topology with ferrix (AS65000) and two FRR
  routers. r2 accepts nothing directly from ferrix so student prefixes
  only reach it via r1, then come back with AS path 65002 65001 65000. r1
  applies community-driven policy: NO_EXPORT stops forwarding, 65001:100
  prepends, BLACKHOLE drops. Dockerfile for the daemon.
- docs/lab/HANDOUT.md: student exercises walking through announcement,
  reading the bytes, prepending, NO_EXPORT, action communities,
  blackholing and withdrawal.

The container files have not been executed (no runtime available here);
docs/lab/README.md lists what is most likely to need adjusting.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 23:26:10 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e2de44ba78 feature: active dialer for configured peers 
Phase 4 of the teaching demo: the daemon now brings up its own sessions.

- New dialer task per configured peer: connect with a timeout, run the
  same FSM as the passive side, back off exponentially (from
  connect_retry_seconds, capped at two minutes) and retry forever. Stays
  idle while a session with the same remote IP and ASN already exists.
- The peer's OPEN must carry the configured ASN, otherwise the session is
  refused with a Bad Peer AS notification. Inbound connections from a
  configured peer's address get the same check plus the configured name
  and hold time.
- SessionRegistry tracks configured peers and their dial state
  (Idle / Connecting / Backoff with last error and retry time).
- /api/peers merges configured peers with live sessions so the page shows
  peers that are down, why, and when the next attempt is; unconfigured
  inbound sessions are listed after them.
- fake_peer gains --listen so it can stand in for a router the daemon
  dials.
- Integration tests: dial and establish, reconnect after the peer closes,
  wrong ASN refused, inbound from a configured address checked and named.

Not implemented: RFC 4271 §6.8 collision detection; a real router on the
other end resolves the rare race by closing one connection.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 23:22:08 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 fb48fc0a85 feature: statusz page and fake_peer example 
Phase 3 of the teaching demo.

- crates/web serves an embedded, dependency-free status page at /statusz
  (/ redirects there). Sections: router header with live indicator, peers,
  announce form (owner remembered locally, well-known community picker,
  prepend / MED / large communities), local routes with per-session UPDATE
  bytes segmented into header / lengths / attributes / NLRI, received
  routes with rows highlighted when a locally announced prefix has come
  back, and a live event log. Tables refresh on every SSE event. Light
  and dark themes.
- fake_peer example in crates/integration-tests: connects, announces,
  prints received UPDATEs and with --echo re-announces everything back
  with its ASN prepended (dropping NO_EXPORT), so one daemon plus one
  process demonstrates the full loop without lab routers.
- README for the binary documents the endpoints and the demo setup.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 23:17:27 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c3f7713965 feature: HTTP API, event stream and Prometheus metrics (bgp-web) 
Phase 2 of the teaching demo: everything the status page will need is now
reachable over HTTP.

- New crate crates/web (axum). JSON endpoints for status, peers, local
  routes (announce / withdraw), received routes, the exact UPDATE bytes a
  local route produces per session, and well-known community names.
  Communities are accepted as ASN:VALUE or by name (NO_EXPORT, BLACKHOLE,
  ...); large communities as A:B:C. Policy errors map to 403/409/422.
- GET /api/events is a server-sent event stream merging session up/down,
  local route and Adj-RIB-In changes, with lag notifications.
- GET /metricz serves Prometheus metrics; the server crate now counts
  messages sent/received by type and sessions established.
- The runnable main moves from crates/server to bin/bgp so it can depend
  on both server and web without a cycle. `cargo run -- config.textpb`.
- Config gains http_endpoint; unset disables the API.
- Integration tests drive the API against a live server with a real BGP
  peer, including verifying the wire view matches the bytes the peer got.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 23:10:16 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 73e9fbf0ad fix: sample config used bracketed IPv6 address which the endpoint parser rejects 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 22:57:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ad07233196 feature: locally originated routes, session registry, outbound UPDATEs 
Phase 1 of the BGP teaching demo: the daemon can now originate routes and
advertise them to every established peer.

- rib: add LocalRib actor holding locally originated routes with owner,
  origin, AS-path prepend, MED, standard and large communities. Enforces a
  LocalRibPolicy (prefix allowlist, per-owner limit, owner match on
  replace/withdraw) and emits LocalRibEvents.
- server: add SessionRegistry with per-session outbound queues; sessions
  register on Established and deregister via a drop guard. Emits
  SessionEvent::{Up,Down}.
- server: advertiser task fans LocalRib changes out to all sessions and
  replays the table to new sessions. UPDATE builders handle IPv4 (legacy
  NLRI) and IPv6 (MP_REACH/MP_UNREACH).
- fsm: merge the timer and timer-less established loops into one select!
  with an outbound branch; reset the hold timer on any received message
  per RFC 4271 §8.2.2 (previously only on KEEPALIVE).
- config: new local_route_policy proto message.
- packet: fix Communitites typo, expose CommunitiesPathAttribute inner vec.
- primitives: IpPrefix::contains.
- integration tests: announce/withdraw/replay/fan-out/policy coverage.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-16 22:56:29 +02:00
137 changed files with 21070 additions and 2923 deletions
Expand all files Collapse all files

No files matched your search

.dockerignore
+2
View File
@@ -0,0 +1,2 @@
target
.git
.github/workflows/ci.yml
+50
View File
@@ -0,0 +1,50 @@
# Runs on every push and pull request. Gitea Actions reads this file too
# (it falls back to .github/workflows), so one definition serves both.
name: ci
on:
push:
pull_request:
env:
CARGO_TERM_COLOR: always
# Warnings fail the build here, and only here: locally they stay warnings
# so that work in progress compiles.
RUSTFLAGS: -D warnings
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- name: Format
run: cargo fmt --all -- --check
- name: Lint
run: cargo clippy --workspace --all-targets -- -D warnings
- name: Test
run: cargo test --workspace
# A short run of each fuzz target: enough to notice a parser change that
# breaks on input the seeds are close to. Long runs are done by hand; see
# fuzz/README.md.
fuzz:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
- name: Install cargo-fuzz
run: cargo install cargo-fuzz --locked
- name: Fuzz
working-directory: fuzz
# cargo-fuzz sets its own flags; the workspace's -D warnings would
# replace them.
env:
RUSTFLAGS: ""
run: |
for target in message message_body prefix_text config; do
mkdir -p "corpus/$target"
cargo fuzz run "$target" "corpus/$target" "seeds/$target" -- -max_total_time=60
done
.gitignore
+3
View File
@@ -1 +1,4 @@
/target
__pycache__/
.DS_Store
.claude/
Cargo.lock
Generated
+1023 -87
View File
File diff suppressed because it is too large. Load diff
Cargo.toml
+23 -2
View File
@@ -1,6 +1,8 @@
[workspace]
default-members = ["bin/bgp"]
members = ["bin/*", "crates/*"]
# Fuzz targets need a nightly toolchain; see fuzz/README.md.
exclude = ["fuzz"]
resolver = "3"
[workspace.package]
@@ -9,17 +11,29 @@ description = "A Border Gateway Protocol implementation"
edition = "2024"
homepage = "https://rayhaan.ch"
license = "Apache-2.0"
name = "bgp"
repository = "https://git.rayhaan.net/rayhaan/bgp"
repository = "https://git.rayhaan.net/rayhaan/ferrix"
version = "0.0.2"
# Every crate opts in with `[lints] workspace = true`. CI turns warnings into
# errors, so a lint added here is enforced.
[workspace.lints.rust]
unsafe_code = "forbid"
[workspace.lints.clippy]
all = { level = "warn", priority = -1 }
dbg_macro = "warn"
todo = "warn"
unimplemented = "warn"
[workspace.dependencies]
# --- Our crates ---
bgp-packet = { path = "crates/packet" }
bgp-primitives = { path = "crates/primitives" }
bgp-rib = { path = "crates/rib" }
bgp-server = { path = "crates/server" }
bgp-web = { path = "crates/web" }
bgp-integration-tests = { path = "crates/integration-tests" }
bgp-bench = { path = "crates/bench" }
# --- General ---
bitfield = "0.19.0"
@@ -27,6 +41,7 @@ bytes = "1.10.1"
eyre = "0.6.12"
nom = "8.0.0"
serde = { version = "1.0.219", features = ["derive"] }
socket2 = "0.6"
serde_json = "1.0.140"
serde_repr = "0.1.20"
thiserror = "2.0.12"
@@ -35,7 +50,13 @@ strum = { version = "0.27.1", features = ["derive"] }
tracing = "0.1.44"
tracing-subscriber = { version = "0.3.22", features = ["env-filter"] }
prometheus = "0.14.0"
proptest = "1.7"
protobuf = { package = "protobuf", version = "3.7.2" }
protobuf-codegen = { package = "protobuf-codegen", version = "3.7.2" }
clap = { version = "4.5.53", features = ["derive"] }
foldhash = "0.2"
futures = "0.3"
hashbrown = "0.17"
axum = "0.8.9"
tokio-stream = { version = "0.1.19", features = ["sync"] }
reqwest = { version = "0.13.5", default-features = false, features = ["json"] }
Dockerfile
+11
View File
@@ -0,0 +1,11 @@
# Builds the daemon into a small image. Mount a configuration file at
# /etc/ferrix/server.textpb, or pass another path as the first argument.
FROM rust:1.92-bookworm AS build
WORKDIR /src
COPY . .
RUN cargo build --release -p bgp
FROM debian:bookworm-slim
COPY --from=build /src/target/release/bgp /usr/local/bin/bgp
ENTRYPOINT ["/usr/local/bin/bgp"]
CMD ["/etc/ferrix/server.textpb", "--level", "info"]
LICENSE
+176
View File
@@ -0,0 +1,176 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
README.md
+35 -5
View File
@@ -1,8 +1,8 @@
# BGP
The `bgp` crate is an implementation of the Border Gateway Protocol, which is used to exchange routing and control plane information between Autonomous Systms.
The `bgp` crate is an implementation of the Border Gateway Protocol, which is used to exchange routing and control plane information between Autonomous Systems.
The aim of this project is to provide a modern, high performance, secure and programatic and customizable software package that can be useful to network operators, researchers, and people interested in Internet.
The aim of this project is to provide a modern, high performance, secure, programmatic and customizable software package that can be useful to network operators, researchers, and people interested in Internet.
## Current status.
@@ -20,8 +20,38 @@ We aim for the 1.0 release to be functional and stable to operate peering sessio
- ✅ Capabilities from RFC4271
- ✅ Update message
- ☐ Notification message
- ☐ Implement scaffolding required to start and configure a server
- ☐ Implement connection logic to establish sessions with peers
- ✅ Implement scaffolding required to start and configure a server
- ✅ Implement connection logic to establish sessions with peers (passive and active)
- ✅ Routes originated from the configuration, with communities, announced to all peers
- ✅ Read-only HTTP API, status page and Prometheus metrics
- ☐ Implement an efficient data structure for storing prefixes (e.g. Poptrie)
- ☐ Filtering for peer RIB in / out
- ☐ Main / auxillary RIBs for exchanging routes between peers
- ☐ Main / auxiliary RIBs for exchanging routes between peers
## Development
CI runs these three on every push, and fails on any warning:
```
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
```
Lints are configured once, under `[workspace.lints]` in the root `Cargo.toml`.
`unsafe` code is forbidden throughout.
`cargo test` includes property tests: generated messages must survive being
written and read back, and no input, however damaged, may make a parser
panic. They take about a second; for a longer hunt:
```
PROPTEST_CASES=1000000 cargo test --release -p bgp-packet --test properties
```
Coverage-guided fuzz targets for the same parsers are in `fuzz/`; see
`fuzz/README.md`. CI runs each of them for a minute.
## License
Apache License 2.0; see `LICENSE`.
bin/bgp/Cargo.toml
+17 -7
View File
@@ -1,11 +1,21 @@
[package]
description = "A Border Gateway Protocol implementation"
edition.workspace = true
name = "bgp"
description = "A Border Gateway Protocol implementation"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
name = "bgp"
version.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
eyre.workspace = true
tokio.workspace = true
bgp-server.workspace = true
bgp-web.workspace = true
clap.workspace = true
eyre.workspace = true
protobuf.workspace = true
tokio.workspace = true
tracing.workspace = true
[lints]
workspace = true
bin/bgp/README.md
+52 -4
View File
@@ -1,9 +1,57 @@
# BGP
This crate provides a binary which runs the Border Gateway Protocol (BGP).
This crate provides the `bgp` binary, which runs the Border Gateway Protocol
daemon together with its read-only HTTP status interface.
Please refer to the other implementation crates part of this repository where most of the actual implemntation lives.
```
cargo run -- path/to/config.textpb
```
Warning: This is still very much a work in progress and is not yet ready for use.
The configuration format is defined in `crates/server/proto/config.proto`;
`testdata/server.textpb` is a worked example. The daemon is configured
entirely through that file: peers, and the routes it originates
(`originate { prefix: ... }`). There is no way to change either at runtime.
The main ongoing work is happening currently in the BGP packet crate.
When `http_endpoint` is set the binary also serves:
| Path | Purpose |
|---|---|
| `GET /statusz` | Status page: configuration, peers and session state, originated and received routes, live event log |
| `GET /api/status` | Router ID, ASN, uptime and table sizes |
| `GET /api/config` | The configuration file as read at startup |
| `GET /api/peers` | Configured peers and established sessions |
| `GET /api/routes/local` | Originated routes |
| `GET /api/routes/received` | Adj-RIB-In contents |
| `GET /api/events` | Server-sent events for session and route changes |
| `GET /metricz` | Prometheus metrics |
Everything under it is read-only and unauthenticated. Bind it to loopback or
put it behind a reverse proxy.
## What it does today
It establishes sessions (both dialling configured peers and accepting
connections, with RFC 4271 collision detection), announces its originated
routes to every peer with itself as next hop, and records what peers send in
an Adj-RIB-In. It does not select best paths, re-advertise received routes or
program a forwarding table, and it has no import or export policy.
## Trying it without a router
`fake_peer` is a minimal BGP speaker that connects to the daemon, announces a
prefix, prints every UPDATE it receives and, with `--echo`, re-announces each
received prefix back with its own ASN prepended (dropping NO_EXPORT routes).
That makes originated prefixes show up in the received table as if they had
travelled through a neighbouring AS:
```
cargo run -- testdata/server.textpb
cargo run -p bgp-integration-tests --example fake_peer -- \
--server 127.0.0.1:179 --asn 65001 --announce 203.0.113.0/24 --echo
```
Please refer to the other implementation crates in this repository where most
of the actual implementation lives.
Warning: This is still very much a work in progress and is not yet ready for
production use.
bin/bgp/src/main.rs
+110 -3
View File
@@ -1,6 +1,113 @@
use eyre::Result;
use std::sync::Arc;
use bgp_server::config::Config;
use bgp_server::proto::config::BgpConfig;
use bgp_server::server::{BgpServer, close_sessions};
use bgp_server::tracing::TracingArgs;
use bgp_web::{AppState, RouterInfo};
use clap::Parser;
use eyre::{Context, Result};
use tracing::{error, info};
#[derive(Parser, Debug)]
struct ServerArgs {
/// Path to the textproto configuration file.
config: String,
#[command(flatten)]
tracing: TracingArgs,
}
#[tokio::main]
pub async fn main() -> Result<()> {
todo!("Implement the server");
async fn main() -> Result<()> {
let args = ServerArgs::parse();
args.tracing.install()?;
log_panics();
let config_string =
std::fs::read_to_string(&args.config).wrap_err("Failed to open config file")?;
let proto_config = protobuf::text_format::parse_from_str::<BgpConfig>(&config_string)
.wrap_err("Failed to parse config file")?;
let config = Config::from_proto(proto_config).wrap_err("Invalid configuration")?;
let server = BgpServer::bind(config).await?;
if let Some(http_endpoint) = server.config().http_endpoint {
// Bound here, so that an address that cannot be bound stops the
// daemon from starting, as it does for the BGP listeners.
let listener = tokio::net::TcpListener::bind(http_endpoint)
.await
.wrap_err_with(|| format!("Failed to bind HTTP listener on {http_endpoint}"))?;
let state = AppState {
info: Arc::new(
RouterInfo::new(
server.config().router_id,
server.config().local_asn,
env!("CARGO_PKG_VERSION"),
)
.with_config(&args.config, &config_string),
),
registry: server.registry.clone(),
local_rib: server.local_rib.clone(),
rib: Arc::clone(&server.rib),
};
tokio::spawn(async move {
if let Err(e) = bgp_web::serve_on(state, listener).await {
error!(err = %e, "HTTP server exited");
}
});
}
// Leave politely: tell every peer we are going (Cease / Administrative
// Shutdown) rather than letting them find a dead TCP connection, which
// makes them slower to take us back.
let registry = server.registry.clone();
tokio::select! {
result = server.run() => Ok(result?),
signal = shutdown_signal() => {
info!(signal, "Shutting down, closing BGP sessions");
close_sessions(&registry).await;
Ok(())
}
}
}
/// Sends panics to the log, with where they happened. A panic is a bug, and
/// the default report goes to stderr in a format of its own, apart from
/// everything else the daemon says. The task that panicked still unwinds;
/// the server deals with what it leaves behind.
fn log_panics() {
std::panic::set_hook(Box::new(|info| {
let location = info
.location()
.map(|l| format!("{}:{}", l.file(), l.line()))
.unwrap_or_else(|| "an unknown location".into());
let message = info
.payload()
.downcast_ref::<&str>()
.map(|s| s.to_string())
.or_else(|| info.payload().downcast_ref::<String>().cloned())
.unwrap_or_else(|| "no message".into());
// Captured only if RUST_BACKTRACE is set.
let backtrace = std::backtrace::Backtrace::capture();
error!(%location, %message, %backtrace, "Panic: this is a bug");
}));
}
/// Resolves with the name of the first termination signal received.
async fn shutdown_signal() -> &'static str {
#[cfg(unix)]
{
use tokio::signal::unix::{SignalKind, signal};
let mut term = signal(SignalKind::terminate()).expect("install SIGTERM handler");
tokio::select! {
_ = tokio::signal::ctrl_c() => "SIGINT",
_ = term.recv() => "SIGTERM",
}
}
#[cfg(not(unix))]
{
let _ = tokio::signal::ctrl_c().await;
"ctrl-c"
}
}
crates/bench/Cargo.toml
+42
View File
@@ -0,0 +1,42 @@
[package]
name = "bgp-bench"
description = "Benchmarks at the scale of the Internet routing table"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
publish = false
[dependencies]
bgp-packet.workspace = true
bgp-primitives.workspace = true
bytes.workspace = true
flate2 = "1"
stats_alloc = { version = "0.1.10", optional = true }
[dev-dependencies]
bgp-integration-tests.workspace = true
bgp-rib.workspace = true
bgp-server.workspace = true
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
[features]
# Counts every allocation, to report how much memory a table holds. The
# counters are shared by all threads and slow them down, so timings are
# taken without this and memory is measured in a run of its own.
count-memory = ["dep:stats_alloc"]
[[bench]]
name = "full_table"
harness = false
[[bench]]
name = "real_table"
harness = false
[lints]
workspace = true
crates/bench/benches/full_table.rs
+215
View File
@@ -0,0 +1,215 @@
//! What it costs to take in the Internet routing table, from the bytes on
//! the wire to a chosen best route.
//!
//! Run with `cargo bench -p bgp-bench --bench full_table`. Set
//! `BENCH_PEERS` (default 4) and `BENCH_SCALE_DOWN` (default 1; 10 makes the
//! table ten times smaller) to change the size. The table is synthetic (see
//! the crate documentation); `real_table` runs on a real one.
//!
//! This is a stopwatch around whole operations at full size, not a
//! statistical micro-benchmark: the questions are "how long until a full
//! table is in" and "how much memory does it take", and their answers are
//! seconds and gigabytes.
use std::hint::black_box;
use std::net::Ipv4Addr;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::memory::{Region, describe};
use bgp_bench::{Prefixes, Shape, announcements, encoded, wire_context};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{Message, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
use bytes::BytesMut;
fn env(name: &str, default: usize) -> usize {
std::env::var(name)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
}
fn main() {
let peers = env("BENCH_PEERS", 4) as u32;
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
let threads = std::thread::available_parallelism().map_or(1, |n| n.get());
println!(
"== Table: {} IPv4 + {} IPv6 prefixes, from each of {peers} peers",
shape.ipv4_prefixes, shape.ipv6_prefixes
);
println!(
"(IpPrefix is {} bytes, PathAttribute {})",
size_of::<IpPrefix>(),
size_of::<PathAttribute>()
);
let prefixes = Prefixes::generate(shape);
let wire: Vec<Vec<BytesMut>> = (1..=peers)
.map(|peer| encoded(&announcements(&prefixes, shape, peer)))
.collect();
let updates: usize = wire.iter().map(Vec::len).sum();
let routes = shape.prefixes() * peers as usize;
let wire_bytes: usize = wire.iter().flatten().map(BytesMut::len).sum();
println!(
"{updates} UPDATE messages, {:.1} prefixes each, {:.1} MiB on the wire, {routes} routes",
routes as f64 / updates as f64,
mib(wire_bytes),
);
println!("\n== Reading the messages (one thread)");
let ctx = wire_context();
let started = Instant::now();
for message in wire.iter().flatten() {
black_box(Message::decode(&ctx, message).expect("what we wrote is well formed"));
}
let elapsed = started.elapsed();
row(
"decode UPDATEs",
elapsed,
format!(
"{} routes, {} messages, {:.0} MiB/s",
per_second(routes, elapsed),
per_second(updates, elapsed),
mib(wire_bytes) / elapsed.as_secs_f64()
),
);
println!("\n== From bytes to best routes, without the network");
for threads in [1, threads] {
into_table(&wire, routes, threads);
}
println!("\n== End to end: {peers} peers send the table over TCP at once");
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
let server = TestServer::start(65000).await.expect("a server");
let mut senders = Vec::new();
for (index, messages) in wire.iter().enumerate() {
let peer = index as u32 + 1;
let mut session = TestPeer::connect(server.addr).await.expect("a connection");
session
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 90)
.await
.expect("a session");
let stream: Vec<u8> = messages.iter().flat_map(|m| m.iter().copied()).collect();
senders.push((session, stream));
}
let started = Instant::now();
let sending: Vec<_> = senders
.into_iter()
.map(|(mut session, stream)| {
tokio::spawn(async move {
session.send_raw(&stream).await.expect("the server reads");
// Held open until the table is in.
session
})
})
.collect();
while server.rib.len() < routes {
tokio::time::sleep(Duration::from_millis(2)).await;
}
let elapsed = started.elapsed();
row(
"sessions, state machine, table",
elapsed,
format!(
"{} routes; {} prefixes with a best route",
per_second(routes, elapsed),
server.rib.prefix_count()
),
);
for session in sending {
drop(session.await);
}
});
}
/// What a session does with each message, without the session: decode it,
/// find its attribute set, put its routes in the table, choose the best.
/// `threads` threads each take whole peers.
fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) {
let region = Region::begin();
let table = RibTable::new(threads * 16);
let peers: Vec<PeerIndex> = (0..wire.len())
.map(|peer| {
table
.add_peer(PeerInfo {
asn: 64512 + peer as u32 + 1,
router_id: Ipv4Addr::from(peer as u32 + 1),
address: Ipv4Addr::new(192, 0, 2, peer as u8 + 1).into(),
ibgp: false,
})
.expect("room for the peer")
})
.collect();
let ctx = wire_context();
let next = AtomicUsize::new(0);
let started = Instant::now();
std::thread::scope(|scope| {
for _ in 0..threads {
scope.spawn(|| {
loop {
let at = next.fetch_add(1, Ordering::Relaxed);
let Some(messages) = wire.get(at) else {
break;
};
for message in messages {
let decoded =
Message::decode_with_report(&ctx, message).expect("well formed");
let Message::Update(update) = decoded.message else {
continue;
};
let wire = decoded.attributes_wire.as_deref().expect("kept");
if !update.announced_routes.is_empty() {
let attrs = table.attrs().intern_received(wire, None).expect("valid");
table.announce(peers[at], &attrs, &update.announced_routes);
}
for attribute in &update.path_attributes {
if let PathAttribute::MpReachNlri(mp) = attribute {
let attrs = table
.attrs()
.intern_received(wire, Some(&mp.next_hop))
.expect("valid");
table.announce(peers[at], &attrs, &mp.prefixes);
}
}
}
}
});
}
});
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
row(
&format!("decode, intern, insert, choose: {threads} thread(s)"),
elapsed,
format!(
"{} routes; {}",
per_second(routes, elapsed),
describe(region.held(), routes, "route")
),
);
}
crates/bench/benches/real_table.rs
+431
View File
@@ -0,0 +1,431 @@
//! The same questions as `full_table`, asked of a real table dump: every
//! route that the peers of a RIPE RIS route collector had announced to it.
//!
//! Needs a dump in `crates/bench/data/` (see the README there) or at the
//! path in `BENCH_MRT`. Without one it says so and does nothing, so that
//! `cargo bench` works on a fresh checkout. `BENCH_PEERS` limits the run to
//! the peers with the most routes (default: all of them).
use std::collections::HashSet;
use std::net::Ipv4Addr;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::memory::Region;
use bgp_bench::mrt::{Dump, parse_attributes};
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{Event, PeerIndex, PeerInfo, RibTable};
/// Bytes allocated and not freed since `region` began; zero when memory
/// is not being counted (see `bgp_bench::memory`).
fn retained(region: &Region) -> i64 {
region.held().unwrap_or(0)
}
fn mib(bytes: i64) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
format!("{:.2} M/s", count as f64 / elapsed.as_secs_f64() / 1e6)
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
}
fn dump_path() -> Option<PathBuf> {
if let Ok(path) = std::env::var("BENCH_MRT") {
return Some(path.into());
}
let data = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("data");
let mut dumps: Vec<PathBuf> = std::fs::read_dir(data)
.ok()?
.filter_map(|entry| Some(entry.ok()?.path()))
.filter(|path| path.extension().is_some_and(|e| e == "gz"))
.collect();
dumps.sort();
dumps.pop()
}
type Attributes = (Arc<[PathAttribute]>, Option<NlriNextHop>);
fn main() {
if Region::begin().held().is_none() {
println!(
"Memory is not counted in this run: sizes print as 0. For sizes (and slower\ntimings) add --features count-memory.\n"
);
}
let Some(path) = dump_path() else {
println!("No table dump found; see crates/bench/data/README.md. Nothing to do.");
return;
};
println!("== Reading {}", path.display());
let started = Instant::now();
let dump = Dump::read(&path).expect("a readable dump");
println!(
"{} peers, {} routes, {} distinct attribute sets, read in {:.0} s",
dump.peers.len(),
dump.route_count(),
dump.attribute_sets.len(),
started.elapsed().as_secs_f64()
);
let limit = std::env::var("BENCH_PEERS")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(dump.peers.len());
let peers = dump.largest_peers(limit);
let peers: Vec<usize> = peers
.into_iter()
.filter(|&peer| !dump.routes[peer].is_empty())
.collect();
describe(&dump, &peers);
// Parsed once, outside the timings: reading messages is measured by
// `full_table`, and here the attributes arrive already parsed.
let region = Region::begin();
let parsed: Vec<Option<Attributes>> = dump
.attribute_sets
.iter()
.map(|wire| parse_attributes(wire).map(|(a, next_hop)| (Arc::from(a), next_hop)))
.collect();
let parsed_bytes = retained(&region);
println!(
"\nattribute sets, parsed (Arc<[PathAttribute]>) {:.0} MiB = {:.0} bytes a set",
mib(parsed_bytes),
parsed_bytes as f64 / parsed.len() as f64
);
let unparsed = parsed.iter().filter(|a| a.is_none()).count();
if unparsed > 0 {
println!("{unparsed} attribute sets could not be parsed; their routes are left out");
}
let threads = std::thread::available_parallelism().map_or(1, |n| n.get());
sharded_table(&dump, &peers, &parsed, 1);
sharded_table(&dump, &peers, &parsed, threads);
followed_table(&dump, &peers, &parsed, threads);
}
/// What the table looks like, as far as it decides what a RIB costs.
fn describe(dump: &Dump, peers: &[usize]) {
let routes: usize = peers.iter().map(|&p| dump.routes[p].len()).sum();
let full = peers
.iter()
.filter(|&&p| dump.routes[p].len() > 800_000)
.count();
let prefixes: HashSet<IpPrefix> = peers
.iter()
.flat_map(|&p| dump.routes[p].iter().map(|(prefix, _)| *prefix))
.collect();
let mut sets_used = HashSet::new();
let mut sets_per_peer = 0usize;
for &peer in peers {
let own: HashSet<u32> = dump.routes[peer].iter().map(|(_, set)| *set).collect();
sets_per_peer += own.len();
sets_used.extend(own);
}
let attribute_bytes: usize = sets_used
.iter()
.map(|&set| dump.attribute_sets[set as usize].len())
.sum();
println!("\n== The table ({} peers used)", peers.len());
println!("routes {routes:>12}");
println!("peers with more than 800,000 routes {full:>12}");
println!(
"distinct prefixes {:>12}",
prefixes.len()
);
println!(
"routes per prefix {:>12.1}",
routes as f64 / prefixes.len() as f64
);
println!(
"distinct attribute sets {:>12}",
sets_used.len()
);
println!(
" if each peer's were kept apart {sets_per_peer:>12} ({:.2}x: sets shared between peers)",
sets_per_peer as f64 / sets_used.len() as f64
);
println!(
"routes per attribute set {:>12.1}",
routes as f64 / sets_used.len() as f64
);
println!(
"attribute set on the wire, mean {:>12.0} bytes",
attribute_bytes as f64 / sets_used.len() as f64
);
}
/// The sharded table, filled by `threads` threads. Each takes whole peers,
/// as sessions would, and for every UPDATE interns the attributes and then
/// announces the prefixes: everything between a decoded message and a
/// chosen best route.
fn sharded_table(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>], threads: usize) {
println!("\n== The sharded table, {threads} thread(s)");
let updates = grouped(dump, peers, parsed);
let routes: usize = updates
.iter()
.flatten()
.map(|(_, prefixes)| prefixes.len())
.sum();
let region = Region::begin();
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let started = Instant::now();
let best_changes = load(&table, dump, &updates, &indices, threads);
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
let held = retained(&region);
row(
"intern attributes, insert, choose best",
elapsed,
format!(
"{} routes; {:.0} MiB = {:.0} bytes a route, attribute sets included",
per_second(routes, elapsed),
mib(held),
held as f64 / routes as f64
),
);
println!(
" {} prefixes, {} attribute sets, best route changed {} times",
table.prefix_count(),
table.attrs().len(),
best_changes
);
let started = Instant::now();
let mut best = 0usize;
table.for_each_best(|prefix, _, _| {
best += usize::from(std::hint::black_box(prefix).length() > 0)
});
row(
"read every best route",
started.elapsed(),
format!("{best} of them, in place, table not stopped"),
);
// Reading has to look at each route for the timing to mean anything.
let started = Instant::now();
let mut all = 0usize;
let mut hops = 0u64;
table.for_each_route(|route| {
all += 1;
hops += u64::from(route.attrs.key().as_path_len);
});
row(
"read every route and its attributes",
started.elapsed(),
format!(
"{all} of them, mean AS path {:.1}",
hops as f64 / all as f64
),
);
let started = Instant::now();
let outcome = table.remove_peer(indices[0]);
let elapsed = started.elapsed();
row(
"remove one peer (session down)",
elapsed,
format!(
"{} routes; best route changed for {} prefixes",
per_second(outcome.removed, elapsed),
outcome.best_changed
),
);
let started = Instant::now();
let swept = table.attrs().sweep();
row(
"drop attribute sets no longer used",
started.elapsed(),
format!("{swept} dropped"),
);
}
/// A peer's routes grouped as UPDATEs: the attribute set and the prefixes
/// announced with it.
type Updates = Vec<Vec<(u32, Vec<IpPrefix>)>>;
fn grouped(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>]) -> Updates {
peers
.iter()
.map(|&peer| {
let mut routes = dump.routes[peer].clone();
routes.sort_by_key(|(_, set)| *set);
routes
.chunk_by(|a, b| a.1 == b.1)
.filter(|group| parsed[group[0].1 as usize].is_some())
.map(|group| {
(
group[0].1,
group.iter().map(|(prefix, _)| *prefix).collect(),
)
})
.collect()
})
.collect()
}
fn add_peers(table: &RibTable, dump: &Dump, peers: &[usize]) -> Vec<PeerIndex> {
peers
.iter()
.map(|&peer| {
table
.add_peer(PeerInfo {
asn: dump.peers[peer].asn,
router_id: Ipv4Addr::from(peer as u32 + 1),
address: dump.peers[peer].address,
ibgp: false,
})
.expect("room for the peer")
})
.collect()
}
/// Announces everything in `updates` on `threads` threads, each taking
/// whole peers. Returns how many times a best route changed.
fn load(
table: &RibTable,
dump: &Dump,
updates: &Updates,
indices: &[PeerIndex],
threads: usize,
) -> usize {
let next = AtomicUsize::new(0);
let best_changes = AtomicUsize::new(0);
std::thread::scope(|scope| {
for _ in 0..threads {
scope.spawn(|| {
let mut changes = 0;
loop {
let at = next.fetch_add(1, Ordering::Relaxed);
let Some(updates) = updates.get(at) else {
break;
};
for (set, prefixes) in updates {
// From the bytes, as a dump has them and as a
// session could hand them over without parsing.
let Some(attrs) = table
.attrs()
.intern_wire(&dump.attribute_sets[*set as usize])
else {
continue;
};
changes += table.announce(indices[at], &attrs, prefixes).best_changed;
}
}
best_changes.fetch_add(changes, Ordering::Relaxed);
});
}
});
best_changes.load(Ordering::Relaxed)
}
/// The same load with someone following the best routes through the
/// journal: once reading as fast as it can while the table fills, once not
/// reading at all until the table is full.
fn followed_table(dump: &Dump, peers: &[usize], parsed: &[Option<Attributes>], threads: usize) {
println!("\n== The sharded table with a follower, {threads} thread(s)");
let updates = grouped(dump, peers, parsed);
let routes: usize = updates
.iter()
.flatten()
.map(|(_, prefixes)| prefixes.len())
.sum();
let count = |counts: &mut (usize, usize, usize), event: Event<'_>| match event {
Event::Best { attrs, .. } => {
// Look at the route, as a follower would.
counts.0 += usize::from(attrs.key().as_path_len > 0);
}
Event::Withdrawn { .. } => counts.1 += 1,
Event::Reset | Event::Synced => counts.2 += 1,
// Not given to a follower of the best routes.
Event::Routes { .. } => {}
};
// Reading while the table fills.
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let done = AtomicBool::new(false);
let started = Instant::now();
let (best_changes, counts, longest, after_load) = std::thread::scope(|scope| {
let follower = scope.spawn(|| {
let mut reader = table.reader();
let mut counts = (0, 0, 0);
let mut longest = 0;
while !done.load(Ordering::Acquire) {
reader.poll(4096, |event| count(&mut counts, event));
longest = longest.max(table.journal_len());
}
let finishing = Instant::now();
while !reader
.poll(4096, |event| count(&mut counts, event))
.caught_up
{}
(counts, longest, finishing.elapsed())
});
let best_changes = load(&table, dump, &updates, &indices, threads);
done.store(true, Ordering::Release);
let (counts, longest, after_load) = follower.join().expect("the follower runs");
(best_changes, counts, longest, after_load)
});
let elapsed = started.elapsed();
assert_eq!(table.len(), routes);
row(
"load, follower reading throughout",
elapsed,
format!(
"{} routes; caught up {:.3} s after the last route",
per_second(routes, elapsed),
after_load.as_secs_f64()
),
);
println!(
" best route changed {best_changes} times; the follower was given {} events for it; \n journal at most {longest} entries for {} prefixes",
counts.0 + counts.1,
table.prefix_count()
);
drop(table);
// Not reading: the journal has to stay small on its own account.
let table = Arc::new(RibTable::new(threads * 16));
let indices = add_peers(&table, dump, peers);
let mut reader = table.reader();
let mut counts = (0, 0, 0);
reader.poll(usize::MAX, |event| count(&mut counts, event));
let started = Instant::now();
load(&table, dump, &updates, &indices, threads);
let elapsed = started.elapsed();
row(
"load, follower not reading",
elapsed,
format!(
"{} routes; journal holds {} entries for {} prefixes",
per_second(routes, elapsed),
table.journal_len(),
table.prefix_count()
),
);
let started = Instant::now();
let mut counts = (0, 0, 0);
while !reader
.poll(4096, |event| count(&mut counts, event))
.caught_up
{}
let elapsed = started.elapsed();
row(
"the follower then catches up",
elapsed,
format!(
"{} events, {} of them a reset",
counts.0 + counts.1,
counts.2
),
);
}
crates/bench/data/.gitignore
+4
View File
@@ -0,0 +1,4 @@
# Table dumps downloaded for benchmarks; see README.md.
*
!.gitignore
!README.md
crates/bench/data/README.md
+18
View File
@@ -0,0 +1,18 @@
# Table dumps for the benchmarks
`cargo bench -p bgp-bench --bench real_table` runs on a real routing table:
a dump of every route that the peers of a route collector had announced to
it. The dumps are large and are not checked in. Fetch one into this
directory:
```bash
curl -O https://data.ris.ripe.net/rrc03/2026.09/bview.20260915.0000.gz
```
That is RIPE RIS collector `rrc03`, which peers at AMS-IX: about 340 MB,
3.4 GB uncompressed. Any `bview` file from <https://data.ris.ripe.net/> or
a RouteViews RIB works; the most recent `.gz` here is used, or the one named
by `BENCH_MRT`.
The benchmark needs memory in proportion to the dump. `BENCH_PEERS=8`
limits it to the eight peers with the most routes.
crates/bench/src/lib.rs
+285
View File
@@ -0,0 +1,285 @@
//! A synthetic copy of the Internet routing table, as the UPDATE messages a
//! peer would send to announce it.
//!
//! It is made up, not captured, so that the benchmarks need no data files
//! and give the same input every run. It is shaped like the real thing where
//! that matters for cost:
//!
//! - about a million IPv4 prefixes, most of them /24s, and about a quarter
//! of a million IPv6 prefixes, most of them /48s;
//! - prefixes that share their attributes are announced together, a handful
//! to an UPDATE, as routers do;
//! - AS paths a few hops long, and communities on most routes;
//! - several peers announce the same prefixes, each with attributes of its
//! own.
pub mod memory;
pub mod mrt;
use std::collections::HashSet;
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LargeCommunitiesPathAttribute,
LargeCommunity, Message, MpReachNlriPathAttribute, MultiExitDiscPathAttribute,
NextHopPathAttribute, NlriNextHop, OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::ParserContext;
use bytes::BytesMut;
/// How big a table to make.
#[derive(Debug, Clone, Copy)]
pub struct Shape {
pub ipv4_prefixes: usize,
pub ipv6_prefixes: usize,
/// How many prefixes share one set of attributes, on average.
pub prefixes_per_attribute_set: usize,
}
impl Shape {
/// Roughly the default-free zone in 2026.
pub const INTERNET: Self = Self {
ipv4_prefixes: 1_000_000,
ipv6_prefixes: 230_000,
prefixes_per_attribute_set: 6,
};
pub fn prefixes(&self) -> usize {
self.ipv4_prefixes + self.ipv6_prefixes
}
/// The same shape, `factor` times smaller.
pub fn scaled_down(&self, factor: usize) -> Self {
Self {
ipv4_prefixes: self.ipv4_prefixes / factor,
ipv6_prefixes: self.ipv6_prefixes / factor,
..*self
}
}
}
/// A small deterministic generator (SplitMix64); no need for a dependency.
pub struct Rng(u64);
impl Rng {
pub fn new(seed: u64) -> Self {
Self(seed)
}
pub fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
z ^ (z >> 31)
}
/// A number below `bound`.
pub fn below(&mut self, bound: u64) -> u64 {
self.next_u64() % bound
}
/// An entry of `weighted`, picked with probability proportional to its
/// weight.
fn pick<T: Copy>(&mut self, weighted: &[(T, u64)]) -> T {
let total: u64 = weighted.iter().map(|(_, w)| w).sum();
let mut at = self.below(total);
for (value, weight) in weighted {
if at < *weight {
return *value;
}
at -= weight;
}
weighted[weighted.len() - 1].0
}
}
/// The prefixes of the table. Every peer announces these same prefixes.
pub struct Prefixes {
pub ipv4: Vec<IpPrefix>,
pub ipv6: Vec<IpPrefix>,
}
impl Prefixes {
pub fn generate(shape: Shape) -> Self {
let mut rng = Rng::new(0x1234_5678);
// Shares of prefix lengths, close to what the table looks like.
let v4_lengths: [(u8, u64); 9] = [
(24, 60),
(23, 10),
(22, 11),
(21, 5),
(20, 4),
(19, 3),
(18, 2),
(16, 4),
(15, 1),
];
let v6_lengths: [(u8, u64); 7] = [
(48, 48),
(32, 14),
(40, 10),
(44, 10),
(36, 6),
(29, 6),
(47, 6),
];
let mut seen = HashSet::new();
let mut ipv4 = Vec::with_capacity(shape.ipv4_prefixes);
while ipv4.len() < shape.ipv4_prefixes {
let length = rng.pick(&v4_lengths);
// Unicast space, 1.0.0.0 to 223.255.255.255.
let address = 0x0100_0000 + (rng.next_u64() as u32) % 0xdf00_0000;
let prefix = IpPrefix::new(IpVersion::Ipv4, address.to_be_bytes(), length)
.expect("four bytes cover any IPv4 prefix");
if seen.insert(prefix) {
ipv4.push(prefix);
}
}
let mut ipv6 = Vec::with_capacity(shape.ipv6_prefixes);
while ipv6.len() < shape.ipv6_prefixes {
let length = rng.pick(&v6_lengths);
// Global unicast, 2000::/3.
let address = (0x2000_u128 << 112) | (u128::from(rng.next_u64()) << 61 >> 3);
let prefix = IpPrefix::new(IpVersion::Ipv6, address.to_be_bytes(), length)
.expect("sixteen bytes cover any IPv6 prefix");
if seen.insert(prefix) {
ipv6.push(prefix);
}
}
Self { ipv4, ipv6 }
}
}
/// The UPDATE messages with which the peer numbered `peer` announces the
/// whole table. The peer's AS is `64512 + peer`.
pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<UpdateMessage> {
let mut rng = Rng::new(0x9999 + u64::from(peer));
let peer_asn = 64512 + peer;
let attributes = |rng: &mut Rng| {
let hops = 2 + rng.below(4) as usize;
let mut path = vec![peer_asn];
path.extend((0..hops).map(|_| 1 + rng.below(400_000) as u32));
let mut attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
PathAttribute::AsPath(AsPathAttribute {
segments: vec![AsPathSegment {
ordered: true,
path,
}],
}),
];
let mut later = Vec::new();
if rng.below(4) == 0 {
later.push(PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(
rng.below(1000) as u32,
)));
}
let communities = rng.below(7) as usize;
if communities > 0 {
later.push(PathAttribute::Communities(CommunitiesPathAttribute(
(0..communities)
.map(|_| (rng.below(65000) as u16, rng.below(1000) as u16))
.collect(),
)));
}
let large = if rng.below(5) == 0 {
Some(PathAttribute::LargeCommunities(
LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: peer_asn,
data_1: rng.below(100) as u32,
data_2: rng.below(100) as u32,
}],
},
))
} else {
None
};
(std::mem::take(&mut attributes), later, large)
};
let mut updates = Vec::new();
// Prefixes that share attributes travel together. A set's prefixes are a
// run of the (randomly ordered) list, a few long.
let mut grouped = |list: &[IpPrefix], rng: &mut Rng, ipv6: bool| {
let mut rest = list;
while !rest.is_empty() {
let size = (1 + rng.below(2 * shape.prefixes_per_attribute_set as u64 - 1) as usize)
.min(rest.len());
let (group, after) = rest.split_at(size);
rest = after;
let (mut path_attributes, later, large) = attributes(rng);
let update = if ipv6 {
// In order of type code: MED and COMMUNITIES come before
// MP_REACH_NLRI.
path_attributes.extend(later);
path_attributes.push(PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(
0x2001,
0xdb8,
0,
0,
0,
0,
0,
peer as u16,
)),
prefixes: Nlri::plain(group.iter().copied()),
}));
path_attributes.extend(large);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: vec![],
}
} else {
path_attributes.push(PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(
192, 0, 2, peer as u8,
))));
path_attributes.extend(later);
path_attributes.extend(large);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: Nlri::plain(group.iter().copied()),
}
};
updates.push(update);
}
};
grouped(&prefixes.ipv4, &mut rng, false);
grouped(&prefixes.ipv6, &mut rng, true);
updates
}
/// `updates` as they are on the wire, on a session with four-octet AS
/// numbers.
pub fn encoded(updates: &[UpdateMessage]) -> Vec<BytesMut> {
let ctx = wire_context();
updates
.iter()
.map(|update| {
let mut wire = BytesMut::new();
Message::Update(update.clone())
.to_wire(&ctx, &mut wire)
.expect("a few prefixes fit one message");
wire
})
.collect()
}
/// The context of the sessions the benchmarks pretend to have: an external
/// peer with four-octet AS numbers.
pub fn wire_context() -> ParserContext {
ParserContext {
four_octet_asn: true,
ibgp: false,
add_path: Vec::new(),
}
}
crates/bench/src/memory.rs
+60
View File
@@ -0,0 +1,60 @@
//! Counting the memory a benchmark holds.
//!
//! With the `count-memory` feature every allocation is counted, and
//! [`Region::held`] says how many bytes were allocated and not freed since
//! the region began. Without it nothing is counted and the answer is
//! `None`: the counters are shared by all threads, and counting slows down
//! exactly the code whose speed is being measured.
#[cfg(feature = "count-memory")]
mod counting {
use std::alloc::System;
use stats_alloc::{INSTRUMENTED_SYSTEM, StatsAlloc};
#[global_allocator]
static GLOBAL: &StatsAlloc<System> = &INSTRUMENTED_SYSTEM;
pub struct Region(stats_alloc::Region<'static, System>);
impl Region {
pub fn begin() -> Self {
Self(stats_alloc::Region::new(GLOBAL))
}
pub fn held(&self) -> Option<i64> {
let stats = self.0.change();
Some(stats.bytes_allocated as i64 - stats.bytes_deallocated as i64)
}
}
}
#[cfg(not(feature = "count-memory"))]
mod counting {
pub struct Region;
impl Region {
pub fn begin() -> Self {
Self
}
pub fn held(&self) -> Option<i64> {
None
}
}
}
pub use counting::Region;
/// "`bytes` MiB = n bytes a `unit`", or a note that memory is not being
/// counted.
pub fn describe(held: Option<i64>, count: usize, unit: &str) -> String {
match held {
Some(bytes) => format!(
"{:.0} MiB = {:.0} bytes a {unit}",
bytes as f64 / (1024.0 * 1024.0),
bytes as f64 / count as f64
),
None => "memory not counted (run with --features count-memory)".into(),
}
}
crates/bench/src/mrt.rs
+237
View File
@@ -0,0 +1,237 @@
//! Reads a table dump in MRT format (RFC 6396, TABLE_DUMP_V2), as published
//! by RIPE RIS and RouteViews: the routes that every peer of a route
//! collector had announced to it at one moment.
//!
//! Only what the benchmarks need is read: the IPv4 and IPv6 unicast tables.
use std::collections::HashMap;
use std::fs::File;
use std::io::{self, BufReader, Read};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::path::Path;
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use flate2::read::MultiGzDecoder;
use crate::wire_context;
const TABLE_DUMP_V2: u16 = 13;
const PEER_INDEX_TABLE: u16 = 1;
const RIB_IPV4_UNICAST: u16 = 2;
const RIB_IPV6_UNICAST: u16 = 4;
const MP_REACH_NLRI: u8 = 14;
/// A peer of the collector.
#[derive(Debug, Clone)]
pub struct DumpPeer {
pub address: IpAddr,
pub asn: u32,
}
/// The contents of a table dump.
pub struct Dump {
/// When the dump was taken, in seconds since the Unix epoch.
pub timestamp: u32,
pub peers: Vec<DumpPeer>,
/// Every distinct list of path attributes in the dump, as it is on the
/// wire. Routes refer to these by index.
pub attribute_sets: Vec<Box<[u8]>>,
/// For each peer (by index into `peers`), its routes: a prefix and the
/// index of its attributes.
pub routes: Vec<Vec<(IpPrefix, u32)>>,
}
impl Dump {
/// Reads the dump at `path`, gzip-compressed as it is published.
pub fn read(path: &Path) -> io::Result<Self> {
let mut input = BufReader::with_capacity(
1 << 20,
MultiGzDecoder::new(BufReader::new(File::open(path)?)),
);
let mut dump = Dump {
timestamp: 0,
peers: Vec::new(),
attribute_sets: Vec::new(),
routes: Vec::new(),
};
let mut interned: HashMap<Box<[u8]>, u32> = HashMap::new();
let mut body = Vec::new();
loop {
let mut header = [0u8; 12];
match input.read_exact(&mut header) {
Ok(()) => {}
Err(e) if e.kind() == io::ErrorKind::UnexpectedEof => break,
Err(e) => return Err(e),
}
let timestamp = u32::from_be_bytes([header[0], header[1], header[2], header[3]]);
let kind = u16::from_be_bytes([header[4], header[5]]);
let subtype = u16::from_be_bytes([header[6], header[7]]);
let length = u32::from_be_bytes([header[8], header[9], header[10], header[11]]);
body.resize(length as usize, 0);
input.read_exact(&mut body)?;
if kind != TABLE_DUMP_V2 {
continue;
}
let mut record = Cursor(&body);
let malformed = || io::Error::new(io::ErrorKind::InvalidData, "truncated MRT record");
match subtype {
PEER_INDEX_TABLE => {
dump.timestamp = timestamp;
dump.peers = peer_index_table(&mut record).ok_or_else(malformed)?;
dump.routes = vec![Vec::new(); dump.peers.len()];
}
RIB_IPV4_UNICAST | RIB_IPV6_UNICAST => {
let version = if subtype == RIB_IPV4_UNICAST {
IpVersion::Ipv4
} else {
IpVersion::Ipv6
};
rib_entries(&mut record, version, &mut dump, &mut interned)
.ok_or_else(malformed)?;
}
_ => {}
}
}
Ok(dump)
}
pub fn route_count(&self) -> usize {
self.routes.iter().map(Vec::len).sum()
}
/// The peers with the most routes, most first: the ones that send a
/// full table.
pub fn largest_peers(&self, count: usize) -> Vec<usize> {
let mut peers: Vec<usize> = (0..self.peers.len()).collect();
peers.sort_by_key(|&peer| std::cmp::Reverse(self.routes[peer].len()));
peers.truncate(count);
peers
}
}
struct Cursor<'a>(&'a [u8]);
impl<'a> Cursor<'a> {
fn take(&mut self, n: usize) -> Option<&'a [u8]> {
if self.0.len() < n {
return None;
}
let (taken, rest) = self.0.split_at(n);
self.0 = rest;
Some(taken)
}
fn u8(&mut self) -> Option<u8> {
self.take(1).map(|b| b[0])
}
fn u16(&mut self) -> Option<u16> {
self.take(2).map(|b| u16::from_be_bytes([b[0], b[1]]))
}
fn u32(&mut self) -> Option<u32> {
self.take(4)
.map(|b| u32::from_be_bytes([b[0], b[1], b[2], b[3]]))
}
}
fn peer_index_table(record: &mut Cursor<'_>) -> Option<Vec<DumpPeer>> {
record.u32()?; // collector's BGP identifier
let name = record.u16()?;
record.take(name.into())?;
let count = record.u16()?;
let mut peers = Vec::with_capacity(count.into());
for _ in 0..count {
let kind = record.u8()?;
record.u32()?; // peer's BGP identifier
let address = if kind & 1 == 1 {
let bytes: [u8; 16] = record.take(16)?.try_into().ok()?;
IpAddr::from(Ipv6Addr::from(bytes))
} else {
let bytes: [u8; 4] = record.take(4)?.try_into().ok()?;
IpAddr::from(Ipv4Addr::from(bytes))
};
let asn = if kind & 2 == 2 {
record.u32()?
} else {
record.u16()?.into()
};
peers.push(DumpPeer { address, asn });
}
Some(peers)
}
fn rib_entries(
record: &mut Cursor<'_>,
version: IpVersion,
dump: &mut Dump,
interned: &mut HashMap<Box<[u8]>, u32>,
) -> Option<()> {
record.u32()?; // sequence number
let length = record.u8()?;
let bytes = record.take(usize::from(length.div_ceil(8)))?;
// A prefix the collector should not have had is not ours to fix.
let prefix = IpPrefix::new(version, bytes, length).ok();
let count = record.u16()?;
for _ in 0..count {
let peer = usize::from(record.u16()?);
record.u32()?; // when the route was received
let attributes_len = record.u16()?;
let attributes = record.take(attributes_len.into())?;
let (Some(prefix), Some(routes)) = (prefix, dump.routes.get_mut(peer)) else {
continue;
};
let id = match interned.get(attributes) {
Some(id) => *id,
None => {
let id = u32::try_from(dump.attribute_sets.len()).ok()?;
dump.attribute_sets.push(attributes.into());
interned.insert(attributes.into(), id);
id
}
};
routes.push((prefix, id));
}
Some(())
}
/// The path attributes of a route in a dump, parsed.
///
/// In a dump the AS numbers are always four octets wide, and
/// MP_REACH_NLRI holds only the next hop (RFC 6396 §4.3.4): the prefix is in
/// the record. The next hop is returned beside the other attributes, as the
/// RIB keeps it. `None` if the attributes cannot be parsed.
pub fn parse_attributes(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
let ctx = wire_context();
let mut attributes = Vec::new();
let mut next_hop = None;
while !wire.is_empty() {
if wire.get(1) == Some(&MP_REACH_NLRI) {
let extended = wire[0] & 0x10 != 0;
let (length, header) = if extended {
(
usize::from(u16::from_be_bytes([*wire.get(2)?, *wire.get(3)?])),
4,
)
} else {
(usize::from(*wire.get(2)?), 3)
};
let value = wire.get(header..header + length)?;
let address = |bytes: &[u8]| Some(Ipv6Addr::from(<[u8; 16]>::try_from(bytes).ok()?));
next_hop = match value {
[4, a, b, c, d] => Some(NlriNextHop::Ipv4(Ipv4Addr::new(*a, *b, *c, *d))),
[16, rest @ ..] => Some(NlriNextHop::Ipv6(address(rest)?)),
[32, rest @ ..] if rest.len() == 32 => Some(NlriNextHop::Ipv6WithLl {
global: address(&rest[..16])?,
link_local: address(&rest[16..])?,
}),
_ => return None,
};
wire = &wire[header + length..];
continue;
}
let (rest, attribute) = PathAttribute::from_wire(&ctx, wire).ok()?;
attributes.push(attribute);
wire = rest;
}
Some((attributes, next_hop))
}
crates/integration-tests/Cargo.toml
+12
View File
@@ -1,6 +1,11 @@
[package]
name = "bgp-integration-tests"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
publish = false
[dependencies]
@@ -8,6 +13,13 @@ bgp-packet.workspace = true
bgp-primitives.workspace = true
bgp-rib.workspace = true
bgp-server.workspace = true
bgp-web.workspace = true
bytes.workspace = true
clap.workspace = true
eyre.workspace = true
reqwest.workspace = true
serde_json.workspace = true
tokio.workspace = true
[lints]
workspace = true
crates/integration-tests/examples/fake_peer.rs
+184
View File
@@ -0,0 +1,184 @@
//! A minimal stand-in for a neighbouring router: connects to the daemon,
//! completes the handshake, optionally announces some prefixes, and prints
//! every UPDATE it receives. Useful for trying things out when no real
//! router is around.
//!
//! cargo run -p bgp-integration-tests --example fake_peer -- \
//! --server 127.0.0.1:1790 --asn 65001 --announce 203.0.113.0/24
//!
//! With `--echo`, every prefix received is re-announced back to the server
//! with this peer's ASN prepended, which lets a single daemon show its own
//! routes "coming back" in the received table.
//!
//! With `--listen ADDR` it instead waits for the server to dial it (one
//! connection at a time), which exercises the server's active side.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use clap::Parser;
use eyre::Result;
use tokio::net::TcpListener;
#[derive(Parser, Debug)]
struct Args {
/// Address of the daemon's BGP listener to connect to.
#[arg(long, default_value = "127.0.0.1:1790", conflicts_with = "listen")]
server: String,
/// Instead of connecting, listen here and accept connections from the
/// server, one at a time, forever.
#[arg(long)]
listen: Option<String>,
/// ASN this peer speaks as.
#[arg(long, default_value_t = 65001)]
asn: u32,
/// Router ID to send in OPEN.
#[arg(long, default_value = "10.0.0.2")]
router_id: Ipv4Addr,
/// Hold time to negotiate.
#[arg(long, default_value_t = 30)]
hold_time: u16,
/// Prefixes to announce after the session is established.
#[arg(long)]
announce: Vec<String>,
/// Re-announce every received prefix back to the server, with our ASN
/// prepended and NO_EXPORT-tagged routes dropped.
#[arg(long)]
echo: bool,
}
const NO_EXPORT: (u16, u16) = (0xFFFF, 0xFF01);
#[tokio::main]
async fn main() -> Result<()> {
let args = Args::parse();
match &args.listen {
Some(listen) => {
let listener = TcpListener::bind(listen).await?;
println!("listening on {}", listener.local_addr()?);
loop {
let (stream, from) = listener.accept().await?;
println!("accepted connection from {from}");
if let Err(e) = run(&args, TestPeer::from_stream(stream), from).await {
println!("session ended: {e}");
}
}
}
None => {
let addr = args.server.parse()?;
run(&args, TestPeer::connect(addr).await?, addr).await
}
}
}
async fn run(args: &Args, mut peer: TestPeer, addr: std::net::SocketAddr) -> Result<()> {
let open = peer
.handshake(args.asn, args.router_id, args.hold_time)
.await?;
println!(
"established with {} (AS{} router-id {})",
addr, open.asn, open.identifier
);
for p in &args.announce {
let prefix = IpPrefix::try_from(p.as_str())?;
let update = announce(prefix, vec![args.asn], vec![(args.asn as u16, 1)]);
peer.send(&Message::Update(update)).await?;
println!("announced {prefix}");
}
let keepalive = Duration::from_secs((args.hold_time / 3).max(1) as u64);
let mut ticker = tokio::time::interval(keepalive);
loop {
tokio::select! {
_ = ticker.tick() => peer.send(&Message::KeepAlive).await?,
msg = peer.recv() => match msg? {
Message::KeepAlive => {}
Message::Update(u) => {
for w in &u.withdrawn_routes {
println!("WITHDRAW {w}");
}
for a in &u.announced_routes {
println!("UPDATE {a} attrs={:?}", u.path_attributes);
}
if args.echo {
echo_back(&mut peer, &u, args.asn).await?;
}
}
Message::Notification(n) => {
println!("NOTIFICATION {}; session over", n.error);
return Ok(());
}
other => println!("unexpected {other:?}"),
},
}
}
}
fn announce(prefix: IpPrefix, as_path: Vec<u32>, communities: Vec<(u16, u16)>) -> UpdateMessage {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 2))),
];
if !communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
communities,
)));
}
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: vec![prefix.into()],
}
}
async fn echo_back(peer: &mut TestPeer, u: &UpdateMessage, asn: u32) -> Result<()> {
let mut as_path = vec![asn];
let mut communities = vec![];
for attr in &u.path_attributes {
match attr {
PathAttribute::AsPath(p) => {
as_path.extend(p.segments.iter().flat_map(|s| s.path.iter().copied()))
}
PathAttribute::Communities(c) => communities = c.0.clone(),
_ => {}
}
}
if communities.contains(&NO_EXPORT) {
println!(" not echoing: NO_EXPORT");
return Ok(());
}
for nlri in &u.announced_routes {
let prefix = nlri.prefix;
peer.send(&Message::Update(announce(
prefix,
as_path.clone(),
communities.clone(),
)))
.await?;
println!(" echoed {prefix} back with path {as_path:?}");
}
if !u.withdrawn_routes.is_empty() {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: u.withdrawn_routes.clone(),
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
println!(
" echoed withdraw of {:?}",
u.withdrawn_routes
.iter()
.map(|p| p.to_string())
.collect::<Vec<_>>()
);
}
Ok(())
}
crates/integration-tests/src/test_peer.rs
+105 -19
View File
@@ -1,8 +1,8 @@
use std::net::{Ipv4Addr, SocketAddr};
use bgp_packet::message::{BGP_HEADER_LEN, Message};
use bgp_packet::message::{BGP_HEADER_LEN, Message, UpdateMessage};
use bgp_packet::open::{Capability, OpenMessage, OpenOption};
use bgp_primitives::constants::{AddressFamilyId, SubsequentAfi};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::ParserContext;
use bytes::BytesMut;
use eyre::{Result, bail, eyre};
@@ -14,18 +14,41 @@ use tokio::net::TcpStream;
pub struct TestPeer {
stream: TcpStream,
pub ctx: ParserContext,
/// The address families for which this peer sends path identifiers
/// with its prefixes (ADD-PATH, RFC 7911). What it expects to be sent
/// is `ctx.add_path`. A test sets both to match the capabilities it
/// exchanged.
pub send_add_path: Vec<AfiSafi>,
/// Whether the OPEN we sent offered four-octet AS numbers. They are in
/// use only if the server's did too.
offered_four_octet_asn: bool,
}
impl TestPeer {
pub async fn connect(addr: SocketAddr) -> Result<Self> {
let stream = TcpStream::connect(addr).await?;
Ok(Self { stream, ctx: ParserContext::default() })
Ok(Self::from_stream(stream))
}
/// Wraps an already-connected stream, e.g. one accepted from a server
/// that dialled us.
pub fn from_stream(stream: TcpStream) -> Self {
Self {
stream,
ctx: ParserContext::default(),
send_add_path: Vec::new(),
offered_four_octet_asn: true,
}
}
/// Send a BGP message.
pub async fn send(&mut self, msg: &Message) -> Result<()> {
let send_ctx = ParserContext {
add_path: self.send_add_path.clone(),
..self.ctx.clone()
};
let mut buf = BytesMut::new();
msg.to_wire(&self.ctx, &mut buf)?;
msg.to_wire(&send_ctx, &mut buf)?;
self.stream.write_all(&buf).await?;
Ok(())
}
@@ -42,11 +65,23 @@ impl TestPeer {
let mut buf = vec![0u8; total_len];
buf[..BGP_HEADER_LEN as usize].copy_from_slice(&header);
self.stream.read_exact(&mut buf[BGP_HEADER_LEN as usize..]).await?;
self.stream
.read_exact(&mut buf[BGP_HEADER_LEN as usize..])
.await?;
let (_, msg) = Message::from_wire(&self.ctx, &buf)
.map_err(|e| eyre!("Failed to parse BGP message: {e:?}"))?;
Ok(msg)
Message::decode(&self.ctx, &buf).map_err(|e| eyre!("Failed to parse BGP message: {e}"))
}
/// Read messages until an UPDATE arrives, skipping KEEPALIVEs. Any other
/// message type is an error.
pub async fn recv_update(&mut self) -> Result<UpdateMessage> {
loop {
match self.recv().await? {
Message::Update(u) => return Ok(u),
Message::KeepAlive => {}
other => bail!("Expected UPDATE, got {:?}", other),
}
}
}
/// Receive and unwrap as `OpenMessage`, failing if a different type arrives.
@@ -58,34 +93,67 @@ impl TestPeer {
matches!(opt, OpenOption::Capabilities(caps)
if caps.iter().any(|c| matches!(c, Capability::FourByteAsn { .. })))
});
self.ctx.four_octet_asn = Some(four_byte);
self.ctx.four_octet_asn = four_byte && self.offered_four_octet_asn;
Ok(o)
}
other => bail!("Expected OPEN, got {:?}", other),
}
}
/// Send an OPEN message with the given parameters and standard capabilities.
/// Send an OPEN message with the given parameters and the capabilities
/// of a typical dual-stack speaker: four-octet AS numbers, IPv4 and IPv6
/// unicast, route refresh.
pub async fn send_open(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
) -> Result<()> {
let asn_field = if local_asn > u16::MAX as u32 { 23456 } else { local_asn as u16 };
let capabilities = vec![
Capability::FourByteAsn { asn: local_asn },
Capability::MultiProtocol {
afi: Afi::IPV4,
safi: Safi::UNICAST,
},
Capability::MultiProtocol {
afi: Afi::IPV6,
safi: Safi::UNICAST,
},
Capability::RouteRefresh,
];
self.send_open_with(local_asn, router_id, hold_time, capabilities)
.await
}
/// Send an OPEN message carrying exactly `capabilities`.
pub async fn send_open_with(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
capabilities: Vec<Capability>,
) -> Result<()> {
self.offered_four_octet_asn = capabilities
.iter()
.any(|c| matches!(c, Capability::FourByteAsn { .. }));
// If the server's OPEN has been read already, what was settled
// then still depends on what is offered now.
self.ctx.four_octet_asn &= self.offered_four_octet_asn;
let asn_field = if local_asn > u16::MAX as u32 {
23456
} else {
local_asn as u16
};
let open = Message::Open(OpenMessage {
version: 4,
asn: asn_field,
hold_time,
identifier: router_id,
options: vec![OpenOption::Capabilities(vec![
Capability::FourByteAsn { asn: local_asn },
Capability::MultiProtocol {
afi: AddressFamilyId::Ipv4,
safi: SubsequentAfi::Unicast,
},
Capability::RouteRefresh,
])],
options: if capabilities.is_empty() {
vec![]
} else {
vec![OpenOption::Capabilities(capabilities)]
},
});
self.send(&open).await
}
@@ -99,6 +167,24 @@ impl TestPeer {
hold_time: u16,
) -> Result<OpenMessage> {
self.send_open(local_asn, router_id, hold_time).await?;
self.finish_handshake().await
}
/// As [`TestPeer::handshake`], with an OPEN carrying exactly
/// `capabilities`.
pub async fn handshake_with(
&mut self,
local_asn: u32,
router_id: Ipv4Addr,
hold_time: u16,
capabilities: Vec<Capability>,
) -> Result<OpenMessage> {
self.send_open_with(local_asn, router_id, hold_time, capabilities)
.await?;
self.finish_handshake().await
}
async fn finish_handshake(&mut self) -> Result<OpenMessage> {
let server_open = self.recv_open().await?;
// Expect server KEEPALIVE
match self.recv().await? {
crates/integration-tests/src/test_server.rs
+170 -12
View File
@@ -1,19 +1,32 @@
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use bgp_rib::AdjRibIn;
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::LocalRib;
use bgp_rib::table::{Event, PeerIndex, Reader, RibTable};
use bgp_server::config::Config;
use bgp_server::server::BgpServerBuilder;
use bgp_server::error::ServerError;
use bgp_server::server::BgpServer;
use bgp_server::session::SessionRegistry;
use bgp_web::{AppState, RouterInfo};
use eyre::Result;
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
/// A BGP server started on an ephemeral port for testing.
/// Aborts the server task when dropped.
pub struct TestServer {
handle: JoinHandle<Result<()>>,
handle: JoinHandle<Result<(), ServerError>>,
/// The single IPv4 loopback address the server is bound to.
pub addr: SocketAddr,
/// Handle to the server's Adj-RIB-In for test introspection.
pub adj_rib_in: AdjRibIn,
/// The server's routing table.
pub rib: Arc<RibTable>,
/// Handle to the server's locally originated routes.
pub local_rib: LocalRib,
/// Live view of the server's established sessions.
pub registry: SessionRegistry,
pub local_asn: u32,
}
impl TestServer {
@@ -24,28 +37,173 @@ impl TestServer {
/// Start a server with an explicit hold time (useful for timer tests).
pub async fn start_with_hold_time(local_asn: u32, hold_time_secs: u16) -> Result<Self> {
Self::start_inner(local_asn, hold_time_secs).await
Self::start_with(local_asn, |c| c.hold_time_secs = hold_time_secs).await
}
async fn start_inner(local_asn: u32, hold_time_secs: u16) -> Result<Self> {
let config = Config {
/// Start a server after letting `customize` edit the default test config
/// (e.g. to add configured peers).
pub async fn start_with(local_asn: u32, customize: impl FnOnce(&mut Config)) -> Result<Self> {
let mut config = Config {
router_id: Ipv4Addr::new(127, 0, 0, 1),
local_asn,
hold_time_secs,
hold_time_secs: 90,
listen_endpoints: vec![SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0)],
peers: vec![],
connect_retry: std::time::Duration::from_secs(5),
collision_detect_established_state: false,
originate: vec![],
http_endpoint: None,
};
customize(&mut config);
let bound = BgpServerBuilder::new(config).bind().await?;
let bound = BgpServer::bind(config).await?;
let addr = bound
.local_addrs()
.into_iter()
.next()
.expect("at least one listener");
let adj_rib_in = bound.adj_rib_in.clone();
let rib = Arc::clone(&bound.rib);
let local_rib = bound.local_rib.clone();
let registry = bound.registry.clone();
let handle = tokio::spawn(bound.run());
Ok(Self { handle, addr, adj_rib_in })
Ok(Self {
handle,
addr,
rib,
local_rib,
registry,
local_asn,
})
}
}
/// A route in the server's table, as a test wants to look at it.
#[derive(Debug, Clone)]
pub struct ReceivedRoute {
pub prefix: IpPrefix,
pub peer: PeerIndex,
/// The peer's identifier for the route; zero without ADD-PATH.
pub path_id: u32,
/// The route's attributes, without MP_REACH_NLRI.
pub attributes: Vec<PathAttribute>,
/// The next hop, if the route came in MP_REACH_NLRI.
pub next_hop: Option<NlriNextHop>,
/// Whether it is the best route for its prefix.
pub best: bool,
}
/// A change of a prefix's best route.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RouteChange {
/// The prefix has a best route, or another one than before.
Announced { prefix: IpPrefix, peer: PeerIndex },
/// The prefix has no route any more.
Withdrawn { prefix: IpPrefix },
}
/// The changes of best route in a server's table from the moment this was
/// made, read from the table's journal.
pub struct RouteChanges {
reader: Reader,
pending: std::collections::VecDeque<RouteChange>,
}
impl RouteChanges {
/// The next change, waiting for one if need be. Changes to one prefix
/// that happen before they are read arrive as one.
pub async fn recv(&mut self) -> Result<RouteChange> {
loop {
if let Some(change) = self.pending.pop_front() {
return Ok(change);
}
self.reader.changed().await;
self.read();
}
}
fn read(&mut self) {
let pending = &mut self.pending;
let mut read = |event: Event<'_>| match event {
Event::Best { prefix, peer, .. } => {
pending.push_back(RouteChange::Announced { prefix, peer })
}
Event::Withdrawn { prefix } => pending.push_back(RouteChange::Withdrawn { prefix }),
// Not given to a follower of the best routes.
Event::Routes { .. } => {}
Event::Reset | Event::Synced => {}
};
while !self.reader.poll(1024, &mut read).caught_up {}
}
}
impl TestServer {
/// Every route in the server's table.
pub fn received_routes(&self) -> Vec<ReceivedRoute> {
let mut routes = Vec::new();
self.rib.for_each_route(|route| {
let (attributes, next_hop) = route.attrs.attributes().expect("stored attributes parse");
routes.push(ReceivedRoute {
prefix: route.prefix,
peer: route.peer,
path_id: route.path_id,
attributes,
next_hop,
best: route.best,
});
});
routes
}
/// Follows the changes of best route from now on.
pub fn route_changes(&self) -> RouteChanges {
let mut changes = RouteChanges {
reader: self.rib.reader(),
pending: Default::default(),
};
// A new reader is first given the table as it is. That is not a
// change.
changes.read();
changes.pending.clear();
changes
}
/// Waits for the server to stop by itself and returns why. A healthy
/// server never does.
pub async fn stopped(&mut self) -> Result<(), ServerError> {
(&mut self.handle)
.await
.expect("the server task does not panic")
}
/// Starts the HTTP API on an ephemeral loopback port and returns its
/// base URL (e.g. `http://127.0.0.1:12345`). The API task is aborted
/// when the returned guard drops.
pub async fn serve_http(&self) -> Result<(String, HttpGuard)> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let base_url = format!("http://{}", listener.local_addr()?);
let state = AppState {
info: std::sync::Arc::new(
RouterInfo::new(Ipv4Addr::new(127, 0, 0, 1), self.local_asn, "test")
.with_config("test.textpb", format!("local_asn: {}\n", self.local_asn)),
),
registry: self.registry.clone(),
local_rib: self.local_rib.clone(),
rib: Arc::clone(&self.rib),
};
let handle = tokio::spawn(bgp_web::serve_on(state, listener));
Ok((base_url, HttpGuard { handle }))
}
}
/// Aborts the HTTP API task when dropped.
pub struct HttpGuard {
handle: JoinHandle<std::io::Result<()>>,
}
impl Drop for HttpGuard {
fn drop(&mut self) {
self.handle.abort();
}
}
crates/integration-tests/tests/add_path.rs
+288
View File
@@ -0,0 +1,288 @@
//! End-to-end tests of ADD-PATH (RFC 7911): a peer sending several routes
//! for one prefix, told apart by path identifiers.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_packet::open::{AddPathEntry, AddPathSendRecv, Capability, OpenMessage};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::{FamilySet, SUPPORTED};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A server on which connections from this machine are those of a
/// configured peer, with ADD-PATH offered to it as `add_path` says.
async fn server_offering(
add_path: Option<AddPathSendRecv>,
originate: &[&str],
) -> Result<TestServer> {
let originate: Vec<LocalRoute> = originate
.iter()
.map(|p| LocalRoute::new(prefix(p), LocalRouteAttrs::default()))
.collect();
TestServer::start_with(LOCAL_ASN, |c| {
c.peers = vec![PeerConfig {
name: "route-server-client".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path,
}];
c.originate = originate;
// Keep the dialer out of the way.
c.connect_retry = Duration::from_secs(3600);
})
.await
}
/// A session from a peer whose OPEN offers ADD-PATH as `offer` says, for
/// IPv4 unicast. Returns the peer and the server's OPEN.
async fn peer_offering(
server: &TestServer,
offer: Option<AddPathSendRecv>,
) -> Result<(TestPeer, OpenMessage)> {
let mut capabilities = vec![Capability::FourByteAsn { asn: PEER_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
if let Some(send_recv) = offer {
capabilities.push(Capability::AddPath {
entries: vec![AddPathEntry {
afi: AfiSafi::IPV4_UNICAST.afi,
safi: AfiSafi::IPV4_UNICAST.safi,
send_recv,
}],
});
}
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90, capabilities)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok((peer, open))
}
fn add_path_offered(open: &OpenMessage) -> Vec<(AfiSafi, AddPathSendRecv)> {
open.capabilities()
.filter_map(|capability| match capability {
Capability::AddPath { entries } => Some(entries),
_ => None,
})
.flatten()
.map(|entry| (entry.family(), entry.send_recv))
.collect()
}
fn announcement(nlri: Nlri, as_path: &[u32]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: vec![nlri],
})
}
async fn wait_for_routes(server: &TestServer, routes: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != routes {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(())
}
/// The routes the server holds, as (path identifier, AS path, best).
fn held(server: &TestServer) -> Vec<(u32, Vec<u32>, bool)> {
let mut held: Vec<(u32, Vec<u32>, bool)> = server
.received_routes()
.into_iter()
.map(|route| {
let path = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.unwrap_or_default();
(route.path_id, path, route.best)
})
.collect();
held.sort();
held
}
/// A peer that sends several routes for one prefix: each is kept, the best
/// of them is chosen, and each is withdrawn on its own.
#[tokio::test]
async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
let server = server_offering(Some(AddPathSendRecv::Receive), &[]).await?;
let (base, _http) = server.serve_http().await?;
let (mut peer, open) = peer_offering(&server, Some(AddPathSendRecv::Send)).await?;
// The server offered to receive, for every family it offers.
assert_eq!(
add_path_offered(&open),
SUPPORTED.map(|family| (family, AddPathSendRecv::Receive))
);
// In use for the one family the peer sends for, and in that direction.
let session = &server.registry.list()[0];
assert_eq!(
session.add_path_receive,
[AfiSafi::IPV4_UNICAST].into_iter().collect()
);
assert!(session.add_path_send.is_empty());
peer.send_add_path = vec![AfiSafi::IPV4_UNICAST];
let p = prefix("203.0.113.0/24");
let path = |path_id| Nlri { prefix: p, path_id };
peer.send(&announcement(path(1), &[PEER_ASN, 64999, 64998]))
.await?;
peer.send(&announcement(path(2), &[PEER_ASN, 64998]))
.await?;
wait_for_routes(&server, 2).await?;
assert_eq!(
held(&server),
[
(1, vec![PEER_ASN, 64999, 64998], false),
(2, vec![PEER_ASN, 64998], true),
]
);
assert_eq!(server.rib.prefix_count(), 1);
// The API tells them apart too.
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
let mut ids: Vec<u64> = routes
.as_array()
.unwrap()
.iter()
.map(|route| route["path_id"].as_u64().unwrap())
.collect();
ids.sort_unstable();
assert_eq!(ids, [1, 2]);
let peers: Value = reqwest::get(format!("{base}/api/peers"))
.await?
.json()
.await?;
assert_eq!(
peers[0]["add_path_receive"],
serde_json::json!(["IPv4 unicast"])
);
// Announcing a path again replaces that path only.
peer.send(&announcement(path(1), &[PEER_ASN])).await?;
tokio::time::timeout(TIMEOUT, async {
while held(&server)[0].1.len() != 1 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
assert_eq!(
held(&server),
[(1, vec![PEER_ASN], true), (2, vec![PEER_ASN, 64998], false),]
);
// And withdrawing one leaves the other.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![path(1)],
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
wait_for_routes(&server, 1).await?;
assert_eq!(held(&server), [(2, vec![PEER_ASN, 64998], true)]);
// When the session ends, all of its routes go.
peer.send(&announcement(path(3), &[PEER_ASN, 1, 2, 3]))
.await?;
wait_for_routes(&server, 2).await?;
drop(peer);
wait_for_routes(&server, 0).await?;
Ok(())
}
/// ADD-PATH changes how prefixes are written. It must be in use only when
/// both sides said so; a peer that offers it to a server that does not is
/// spoken to, and listened to, in plain BGP.
#[tokio::test]
async fn test_add_path_is_not_used_unless_both_sides_offer_it() -> Result<()> {
let server = server_offering(None, &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, Some(AddPathSendRecv::Both)).await?;
assert!(add_path_offered(&open).is_empty());
let session = &server.registry.list()[0];
assert!(session.add_path_receive.is_empty() && session.add_path_send.is_empty());
// Plain prefixes both ways.
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
peer.send(&announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
wait_for_routes(&server, 1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
// And the other way round: offered by the server, not by the peer.
let server = server_offering(Some(AddPathSendRecv::Both), &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, None).await?;
assert_eq!(add_path_offered(&open).len(), SUPPORTED.len());
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
Ok(())
}
/// Where the peer offered to receive and the server to send, the server's
/// prefixes carry a path identifier, and a peer that reads them so
/// understands them.
#[tokio::test]
async fn test_prefixes_are_sent_with_path_identifiers_when_agreed() -> Result<()> {
let server = server_offering(Some(AddPathSendRecv::Both), &["10.100.0.0/16"]).await?;
let (mut peer, _) = peer_offering(&server, Some(AddPathSendRecv::Receive)).await?;
let session = &server.registry.list()[0];
assert_eq!(
session.add_path_send,
[AfiSafi::IPV4_UNICAST].into_iter().collect()
);
assert!(session.add_path_receive.is_empty());
// Read as agreed: with identifiers.
peer.ctx.add_path = vec![AfiSafi::IPV4_UNICAST];
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
assert_eq!(update.announced_routes.len(), 1);
assert_eq!(update.announced_routes[0].prefix, prefix("10.100.0.0/16"));
// The peer did not offer to send, so the server reads its prefixes
// plain.
peer.send(&announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
wait_for_routes(&server, 1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
Ok(())
}
crates/integration-tests/tests/announce.rs
+500
View File
@@ -0,0 +1,500 @@
//! End-to-end tests for locally originated routes: announcing via the
//! `LocalRib` handle must produce UPDATEs on every established session.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
LocalPrefPathAttribute, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::session::SessionEvent;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(p: &str, attrs: LocalRouteAttrs) -> LocalRoute {
LocalRoute::new(prefix(p), attrs)
}
async fn established_peer(server: &TestServer) -> Result<TestPeer> {
established_peer_with_id(server, Ipv4Addr::new(10, 0, 0, 1)).await
}
/// Distinct routers need distinct BGP Identifiers, or the second connection
/// is (correctly) treated as a collision with the first.
async fn established_peer_with_id(server: &TestServer, router_id: Ipv4Addr) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, router_id, 90).await?;
Ok(peer)
}
async fn next_update(peer: &mut TestPeer) -> Result<UpdateMessage> {
tokio::time::timeout(TIMEOUT, peer.recv_update()).await?
}
/// Waits until the registry reports `n` established sessions.
async fn wait_for_sessions(server: &TestServer, n: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().len() != n {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
}
// ---------------------------------------------------------------------------
// Session registry
// ---------------------------------------------------------------------------
#[tokio::test]
async fn test_registry_tracks_session_lifecycle() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut events = server.registry.subscribe();
assert!(server.registry.list().is_empty());
let peer = established_peer(&server).await?;
let up = tokio::time::timeout(TIMEOUT, events.recv()).await??;
let SessionEvent::Up(handle) = up else {
panic!("expected Up, got {up:?}")
};
assert_eq!(handle.peer.asn, PEER_ASN);
assert_eq!(handle.local_addr, server.addr);
assert!(handle.four_octet_asn);
assert_eq!(server.registry.list().len(), 1);
drop(peer);
let down = tokio::time::timeout(TIMEOUT, events.recv()).await??;
assert!(
matches!(down, SessionEvent::Down(p) if p == handle.peer),
"got {down:?}"
);
assert!(server.registry.list().is_empty());
Ok(())
}
// ---------------------------------------------------------------------------
// Announce / withdraw
// ---------------------------------------------------------------------------
/// A route announced while the session is up is sent immediately, with the
/// attributes the announcer chose and our address as the next hop.
#[tokio::test]
async fn test_announce_after_session_up() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
origin: OriginPathAttribute::IGP,
prepend: 0,
med: None,
communities: vec![(65000, 100), (65535, 65281)],
large_communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 7,
data_2: 8,
}],
},
))
.await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(update.withdrawn_routes.is_empty());
assert_eq!(
update.path_attributes,
vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![LOCAL_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::LOCALHOST)),
PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65000, 100),
(65535, 65281)
])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 7,
data_2: 8
}],
}),
]
);
Ok(())
}
/// Routes announced before any session exists are replayed when a session
/// reaches Established.
#[tokio::test]
async fn test_routes_replayed_on_session_up() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
server
.local_rib
.announce(route("10.100.2.0/24", LocalRouteAttrs::default()))
.await?;
let mut peer = established_peer(&server).await?;
let mut seen = vec![next_update(&mut peer).await?, next_update(&mut peer).await?]
.into_iter()
.flat_map(|u| u.announced_routes)
.map(|p| p.to_string())
.collect::<Vec<_>>();
seen.sort();
assert_eq!(seen, vec!["10.100.1.0/24", "10.100.2.0/24"]);
Ok(())
}
/// An announcement is fanned out to every established session.
#[tokio::test]
async fn test_announce_reaches_all_sessions() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
wait_for_sessions(&server, 2).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
for peer in [&mut a, &mut b] {
let update = next_update(peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
}
Ok(())
}
#[tokio::test]
async fn test_withdraw_sends_withdrawn_route() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let _ = next_update(&mut peer).await?;
server.local_rib.withdraw(prefix("10.100.1.0/24")).await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.withdrawn_routes, vec![prefix("10.100.1.0/24")]);
assert!(update.announced_routes.is_empty());
assert!(update.path_attributes.is_empty());
Ok(())
}
/// Re-announcing with new attributes sends a fresh UPDATE (implicit replace).
#[tokio::test]
async fn test_reannounce_with_prepend_and_med() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let first = next_update(&mut peer).await?;
assert_eq!(
first.path_attributes[1],
AsPathAttribute::from_asns(vec![LOCAL_ASN])
);
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
prepend: 3,
med: Some(42),
..LocalRouteAttrs::default()
},
))
.await?;
let second = next_update(&mut peer).await?;
assert_eq!(second.announced_routes, vec![prefix("10.100.1.0/24")]);
assert_eq!(
second.path_attributes[1],
AsPathAttribute::from_asns(vec![LOCAL_ASN; 4])
);
assert!(
second
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::MultiExitDisc(m) if m.0 == 42))
);
Ok(())
}
/// The session stays healthy (keepalives keep flowing) while UPDATEs are sent.
#[tokio::test]
async fn test_keepalives_continue_alongside_updates() -> Result<()> {
let server = TestServer::start_with_hold_time(LOCAL_ASN, 3).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 3)
.await?;
wait_for_sessions(&server, 1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
let mut saw_update = false;
let mut saw_keepalive = false;
tokio::time::timeout(TIMEOUT, async {
while !(saw_update && saw_keepalive) {
match peer.recv().await? {
bgp_packet::message::Message::Update(_) => saw_update = true,
bgp_packet::message::Message::KeepAlive => saw_keepalive = true,
other => eyre::bail!("unexpected {other:?}"),
}
}
Ok::<_, eyre::Error>(())
})
.await??;
Ok(())
}
/// Routes learned from a peer disappear from the Adj-RIB-In when its
/// session ends, with a Withdrawn event for each.
#[tokio::test]
async fn test_received_routes_removed_when_session_closes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut rib_events = server.route_changes();
let mut session_events = server.registry.subscribe();
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
peer.send(&bgp_packet::message::Message::Update(
bgp_packet::message::UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 1))),
],
announced_routes: vec![
prefix("203.0.113.0/24").into(),
prefix("198.51.100.0/24").into(),
],
},
))
.await?;
for _ in 0..2 {
let ev = tokio::time::timeout(TIMEOUT, rib_events.recv()).await??;
assert!(matches!(ev, RouteChange::Announced { .. }), "{ev:?}");
}
assert_eq!(server.received_routes().len(), 2);
drop(peer);
let mut withdrawn = vec![];
for _ in 0..2 {
match tokio::time::timeout(TIMEOUT, rib_events.recv()).await?? {
RouteChange::Withdrawn { prefix, .. } => withdrawn.push(prefix.to_string()),
other => panic!("expected Withdrawn, got {other:?}"),
}
}
withdrawn.sort();
assert_eq!(withdrawn, vec!["198.51.100.0/24", "203.0.113.0/24"]);
assert!(server.received_routes().is_empty());
// Withdrawals precede the Down event.
let _up = tokio::time::timeout(TIMEOUT, session_events.recv()).await??;
let down = tokio::time::timeout(TIMEOUT, session_events.recv()).await??;
assert!(matches!(down, SessionEvent::Down(_)), "{down:?}");
Ok(())
}
// ---------------------------------------------------------------------------
// Route refresh and shutdown
// ---------------------------------------------------------------------------
/// ROUTE-REFRESH for IPv4 unicast exactly as a router sends it (RFC 2918):
/// 19-byte header plus AFI, reserved, SAFI.
#[rustfmt::skip]
const ROUTE_REFRESH_IPV4: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x17, 0x05,
0x00, 0x01, 0x00, 0x01,
];
/// A peer whose import policy changed asks for our routes again; it must get
/// every local route re-sent, and the session must stay up.
#[tokio::test]
async fn test_route_refresh_resends_local_routes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server
.local_rib
.announce(route("10.100.1.0/24", LocalRouteAttrs::default()))
.await?;
server
.local_rib
.announce(route("10.100.2.0/24", LocalRouteAttrs::default()))
.await?;
let _ = next_update(&mut peer).await?;
let _ = next_update(&mut peer).await?;
peer.send_raw(ROUTE_REFRESH_IPV4).await?;
let mut resent = vec![next_update(&mut peer).await?, next_update(&mut peer).await?]
.into_iter()
.flat_map(|u| u.announced_routes)
.map(|p| p.to_string())
.collect::<Vec<_>>();
resent.sort();
assert_eq!(resent, vec!["10.100.1.0/24", "10.100.2.0/24"]);
assert_eq!(
server.registry.list().len(),
1,
"session must survive a route refresh"
);
Ok(())
}
/// Shutting down tells each peer why (Cease / Administrative Shutdown) and
/// closes the sessions, instead of leaving dead TCP connections behind.
#[tokio::test]
async fn test_shutdown_sends_cease_to_every_peer() -> Result<()> {
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
wait_for_sessions(&server, 2).await?;
let left = server.registry.shutdown(Duration::from_secs(2)).await;
assert_eq!(left, 0, "sessions still open after shutdown");
assert!(server.registry.is_shutting_down());
for peer in [&mut a, &mut b] {
let notif = tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {}
other => eyre::bail!("unexpected {other:?}"),
}
}
})
.await??;
assert!(
matches!(notif.error, BgpError::Cease(CeaseSubcode::AdminShutdown)),
"{notif:?}"
);
assert!(peer.is_closed().await);
}
Ok(())
}
// ---------------------------------------------------------------------------
// Routes from the configuration
// ---------------------------------------------------------------------------
/// Routes listed under `originate` are in the table before any peer
/// connects, and each session is sent them when it comes up.
#[tokio::test]
async fn test_configured_routes_are_announced_to_every_new_session() -> Result<()> {
let attrs = LocalRouteAttrs {
med: Some(7),
communities: vec![(65000, 100)],
..Default::default()
};
let configured = vec![
route("10.100.1.0/24", attrs),
route("10.100.2.0/24", LocalRouteAttrs::default()),
];
let server = TestServer::start_with(LOCAL_ASN, |c| c.originate = configured).await?;
assert_eq!(server.local_rib.snapshot().await.unwrap().len(), 2);
for router_id in [Ipv4Addr::new(10, 0, 0, 1), Ipv4Addr::new(10, 0, 0, 2)] {
let mut peer = established_peer_with_id(&server, router_id).await?;
let mut seen = [next_update(&mut peer).await?, next_update(&mut peer).await?];
seen.sort_by_key(|u| u.announced_routes[0].to_string());
assert_eq!(seen[0].announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(
seen[0]
.path_attributes
.contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65000, 100)
])))
);
assert_eq!(seen[1].announced_routes, vec![prefix("10.100.2.0/24")]);
}
Ok(())
}
/// A peer in our own AS is sent a route we originate with an empty AS path
/// and a LOCAL_PREF, whatever prepending is configured: our AS number goes
/// on the path only as the route leaves the AS. It used to be sent as to
/// an external peer, which an internal one refuses.
#[tokio::test]
async fn test_a_peer_in_our_own_as_gets_an_empty_path_and_a_local_pref() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(LOCAL_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
// The peer reads as a router in the server's AS does: one outside it
// would drop the LOCAL_PREF.
peer.ctx.ibgp = true;
wait_for_sessions(&server, 1).await?;
assert!(server.registry.list()[0].ibgp);
server
.local_rib
.announce(route(
"10.100.1.0/24",
LocalRouteAttrs {
prepend: 2,
..LocalRouteAttrs::default()
},
))
.await?;
let update = next_update(&mut peer).await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(
update
.path_attributes
.contains(&PathAttribute::AsPath(AsPathAttribute { segments: vec![] })),
"{:?}",
update.path_attributes
);
assert!(
update
.path_attributes
.contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(100))),
"{:?}",
update.path_attributes
);
Ok(())
}
crates/integration-tests/tests/basic_connection.rs
+225 -54
View File
@@ -2,17 +2,41 @@ use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::RouteChange;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::errors::{BgpError, CeaseSubcode, FsmSubcode, MsgHeaderSubcode, UpdateMsgSubcode};
use bgp_packet::message::{
Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute, NlriNextHop,
NotificationMessage, PathAttribute, UpdateMessage,
AsPathAttribute, Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute,
NextHopPathAttribute, NlriNextHop, NotificationMessage, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::constants::{AddressFamilyId, SubsequentAfi};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::RibEvent;
use eyre::Result;
const TIMEOUT: Duration = Duration::from_secs(5);
/// The attributes every announcement of IPv4 routes must carry. An UPDATE
/// without them has its routes withdrawn instead (RFC 7606).
fn mandatory_attributes() -> Vec<PathAttribute> {
let mut attributes = with_origin_and_as_path(PathAttribute::NextHop(NextHopPathAttribute(
Ipv4Addr::new(192, 0, 2, 1),
)));
// In order of type code: ORIGIN, AS_PATH, NEXT_HOP.
attributes.rotate_left(1);
attributes
}
/// `first` followed by the attributes every announcement must carry.
/// Routes in MP_REACH_NLRI have their next hop there.
fn with_origin_and_as_path(first: PathAttribute) -> Vec<PathAttribute> {
vec![
first,
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65001]),
]
}
// ---------------------------------------------------------------------------
// Connectivity
// ---------------------------------------------------------------------------
@@ -74,6 +98,147 @@ async fn test_bad_marker_does_not_crash_server() -> Result<()> {
Ok(())
}
/// Reads until a NOTIFICATION arrives, skipping KEEPALIVEs.
async fn expect_notification(peer: &mut TestPeer) -> Result<NotificationMessage> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => eyre::bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
/// A header with the given length and type, followed by `body`.
fn framed(length: u16, msg_type: u8, body: &[u8]) -> Vec<u8> {
let mut buf = vec![0xFF; 16];
buf.extend_from_slice(&length.to_be_bytes());
buf.push(msg_type);
buf.extend_from_slice(body);
buf
}
/// What the peer sent decides the NOTIFICATION it gets back (RFC 4271 §6),
/// in every state, and the session ends.
#[tokio::test]
async fn test_malformed_messages_are_answered_with_the_matching_notification() -> Result<()> {
let server = TestServer::start(65000).await?;
let header = |subcode| BgpError::MsgHeader(subcode);
let mut bad_marker = framed(19, 4, &[]);
bad_marker[0] = 0;
let cases: Vec<(&str, Vec<u8>, BgpError, Vec<u8>)> = vec![
(
"length beyond the maximum",
framed(5000, 2, &[]),
header(MsgHeaderSubcode::BadMessageLength),
5000u16.to_be_bytes().to_vec(),
),
(
"length below a header",
framed(5, 4, &[]),
header(MsgHeaderSubcode::BadMessageLength),
vec![0, 5],
),
(
"unknown message type",
framed(19, 9, &[]),
header(MsgHeaderSubcode::BadMessageType),
vec![9],
),
(
"marker not all ones",
bad_marker,
header(MsgHeaderSubcode::ConnNotSynchronized),
vec![],
),
(
"UPDATE with attributes running past its end",
framed(23, 2, &[0, 0, 0, 40]),
BgpError::UpdateMsg(UpdateMsgSubcode::MalformedAttrs),
vec![],
),
];
for (i, (what, bytes, error, data)) in cases.into_iter().enumerate() {
// Established: a distinct router ID each time, so that a session
// still closing does not collide with the next one.
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 1, i as u8 + 1), 90)
.await?;
peer.send_raw(&bytes).await?;
let notification = expect_notification(&mut peer).await?;
assert_eq!(
(notification.error, notification.data),
(error.clone(), data.clone()),
"{what}, Established"
);
// OpenSent: the same bytes instead of an OPEN.
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send_raw(&bytes).await?;
let notification = expect_notification(&mut peer).await?;
assert_eq!(
(notification.error, notification.data),
(error, data),
"{what}, OpenSent"
);
}
// A well-formed message at the wrong time is an FSM error, not a
// message error.
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send(&Message::KeepAlive).await?;
let notification = expect_notification(&mut peer).await?;
assert_eq!(
notification.error,
BgpError::FsmError(FsmSubcode::UnexpectedOpenSent)
);
Ok(())
}
/// A message may arrive in pieces, with the server's own timers firing in
/// between. The server once threw away the piece it had already read when
/// that happened, so the message was silently lost (and the stream was only
/// back in step if the remainder happened to be thrown away as well).
#[tokio::test]
async fn test_message_split_across_a_keepalive_tick_is_still_one_message() -> Result<()> {
// Hold time 3 s: the server sends a KEEPALIVE every second.
let server = TestServer::start_with_hold_time(65000, 3).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 3).await?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
let mut update = bytes::BytesMut::new();
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
})
.to_wire(&peer.ctx, &mut update)?;
// The first half, a pause long enough for the server's keepalive timer
// to fire, then the rest.
peer.send_raw(&update[..10]).await?;
tokio::time::sleep(Duration::from_millis(1300)).await;
peer.send_raw(&update[10..]).await?;
tokio::time::timeout(TIMEOUT, async {
while !server.received_routes().iter().any(|e| e.prefix == prefix) {
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.map_err(|_| eyre::eyre!("the split UPDATE never reached the RIB"))?;
assert_eq!(server.registry.list().len(), 1, "session was dropped");
Ok(())
}
// ---------------------------------------------------------------------------
// OpenConfirm → Established
// ---------------------------------------------------------------------------
@@ -173,9 +338,9 @@ async fn test_server_accepts_update() -> Result<()> {
let update = Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
path_attributes: mandatory_attributes(),
// 10.0.0.0/8 — one byte of prefix for a /8.
announced_routes: vec![IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?],
announced_routes: vec![IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?.into()],
});
peer.send(&update).await?;
@@ -193,15 +358,16 @@ async fn test_server_accepts_ipv6_update() -> Result<()> {
.await?;
// 2001:db8::/32
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let update = Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
});
peer.send(&update).await?;
@@ -219,26 +385,27 @@ async fn test_server_accepts_ipv6_update() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv4_announce() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
announced_routes: vec![prefix.clone()],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
}))
.await?;
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Announced { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Announced { prefix: p, .. } if p == &prefix),
"Expected Announced for 10.0.0.0/8, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
snapshot.iter().any(|e| e.prefix == prefix),
"10.0.0.0/8 missing from RIB snapshot",
@@ -250,25 +417,26 @@ async fn test_rib_records_ipv4_announce() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv4_withdraw() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv4, vec![10], 8)?;
let prefix = IpPrefix::new(IpVersion::Ipv4, vec![10], 8)?;
// Announce first.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![],
announced_routes: vec![prefix.clone()],
path_attributes: mandatory_attributes(),
announced_routes: vec![prefix.into()],
}))
.await?;
tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
// Now withdraw.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![prefix.clone()],
withdrawn_routes: vec![prefix.into()],
path_attributes: vec![],
announced_routes: vec![],
}))
@@ -276,11 +444,11 @@ async fn test_rib_records_ipv4_withdraw() -> Result<()> {
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Withdrawn { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Withdrawn { prefix: p, .. } if p == &prefix),
"Expected Withdrawn for 10.0.0.0/8, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
!snapshot.iter().any(|e| e.prefix == prefix),
"10.0.0.0/8 should have been removed from RIB snapshot",
@@ -292,31 +460,33 @@ async fn test_rib_records_ipv4_withdraw() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv6_announce() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.clone()],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
}))
.await?;
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Announced { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Announced { prefix: p, .. } if p == &prefix),
"Expected Announced for 2001:db8::/32, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
snapshot.iter().any(|e| e.prefix == prefix),
"2001:db8::/32 missing from RIB snapshot",
@@ -328,22 +498,24 @@ async fn test_rib_records_ipv6_announce() -> Result<()> {
#[tokio::test]
async fn test_rib_records_ipv6_withdraw() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut events = server.adj_rib_in.subscribe();
let mut events = server.route_changes();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
let prefix = IpPrefix::new(AddressFamilyId::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
let prefix = IpPrefix::new(IpVersion::Ipv6, vec![0x20, 0x01, 0x0d, 0xb8], 32)?;
// Announce first.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.clone()],
})],
path_attributes: with_origin_and_as_path(PathAttribute::MpReachNlri(
MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
prefixes: vec![prefix.into()],
},
)),
announced_routes: vec![],
}))
.await?;
@@ -353,9 +525,8 @@ async fn test_rib_records_ipv6_withdraw() -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpUnreachNlri(MpUnreachNlriPathAttribute {
afi: AddressFamilyId::Ipv6,
safi: SubsequentAfi::Unicast,
prefixes: vec![prefix.clone()],
family: AfiSafi::IPV6_UNICAST,
prefixes: vec![prefix.into()],
})],
announced_routes: vec![],
}))
@@ -363,11 +534,11 @@ async fn test_rib_records_ipv6_withdraw() -> Result<()> {
let event = tokio::time::timeout(Duration::from_secs(1), events.recv()).await??;
assert!(
matches!(&event, RibEvent::Withdrawn { prefix: p, .. } if p == &prefix),
matches!(&event, RouteChange::Withdrawn { prefix: p, .. } if p == &prefix),
"Expected Withdrawn for 2001:db8::/32, got {event:?}",
);
let snapshot = server.adj_rib_in.snapshot().await;
let snapshot = server.received_routes();
assert!(
!snapshot.iter().any(|e| e.prefix == prefix),
"2001:db8::/32 should have been removed from RIB snapshot",
crates/integration-tests/tests/best_route.rs
+239
View File
@@ -0,0 +1,239 @@
//! End-to-end tests of route selection: routes for one prefix from several
//! peers, which of them the server chooses, and how that is shown.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A session from the peer numbered `number`, in AS 65000 + `number`.
async fn peer(server: &TestServer, number: u8) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(
LOCAL_ASN + u32::from(number),
Ipv4Addr::new(10, 0, 0, number),
90,
)
.await?;
Ok(peer)
}
fn announcement(prefixes: &[IpPrefix], as_path: &[u32]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: Nlri::plain(prefixes.iter().copied()),
})
}
fn withdrawal(prefixes: &[IpPrefix]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: Nlri::plain(prefixes.iter().copied()),
path_attributes: vec![],
announced_routes: vec![],
})
}
/// Waits until the table holds `routes` routes.
async fn wait_for_routes(server: &TestServer, routes: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != routes {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(())
}
/// The AS path of the route the server holds as best for `prefix`.
fn best_path(server: &TestServer, prefix: &IpPrefix) -> Option<Vec<u32>> {
let best = server
.received_routes()
.into_iter()
.find(|route| route.prefix == *prefix && route.best)?;
best.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
}
async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
}
/// Two peers offer the same prefix. The route with the shorter AS path is
/// chosen; when it goes, the other is.
#[tokio::test]
async fn test_the_best_route_is_chosen_and_replaced_when_it_goes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut changes = server.route_changes();
let p = prefix("203.0.113.0/24");
let mut far = peer(&server, 1).await?;
far.send(&announcement(&[p], &[65001, 64999, 64998]))
.await?;
wait_for_routes(&server, 1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
let mut near = peer(&server, 2).await?;
near.send(&announcement(&[p], &[65002, 64998])).await?;
wait_for_routes(&server, 2).await?;
assert_eq!(best_path(&server, &p), Some(vec![65002, 64998]));
// Both routes are kept and shown; one is marked.
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["received_routes"], 2);
assert_eq!(status["received_prefixes"], 1);
let routes = get_json(&client, &format!("{base}/api/routes/received")).await?;
let best: Vec<&Value> = routes
.as_array()
.unwrap()
.iter()
.filter(|route| route["best"] == true)
.collect();
assert_eq!(routes.as_array().unwrap().len(), 2);
assert_eq!(best.len(), 1);
assert_eq!(best[0]["peer_asn"], 65002);
// The session with the best route ends: the other route takes over.
drop(near);
wait_for_routes(&server, 1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
far.send(&withdrawal(&[p])).await?;
wait_for_routes(&server, 0).await?;
assert_eq!(best_path(&server, &p), None);
// A follower of the best routes is told of each change: the first
// route, the better one, back to the first, none.
let mut seen = Vec::new();
while seen.last() != Some(&RouteChange::Withdrawn { prefix: p }) {
seen.push(tokio::time::timeout(TIMEOUT, changes.recv()).await??);
}
assert!(
seen.iter()
.all(|change| matches!(change, RouteChange::Announced { prefix, .. } | RouteChange::Withdrawn { prefix } if *prefix == p)),
"{seen:?}"
);
Ok(())
}
/// The event stream reports changes of best route, and only those.
#[tokio::test]
async fn test_changes_of_best_route_are_on_the_event_stream() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let p = prefix("203.0.113.0/24");
// A route that is there before the client connects is not an event.
let mut first = peer(&server, 1).await?;
first
.send(&announcement(&[prefix("198.51.100.0/24")], &[65001]))
.await?;
wait_for_routes(&server, 1).await?;
let mut stream = reqwest::Client::new()
.get(format!("{base}/api/events"))
.send()
.await?
.error_for_status()?;
first.send(&announcement(&[p], &[65001, 64998])).await?;
wait_for_routes(&server, 2).await?;
first.send(&withdrawal(&[p])).await?;
wait_for_routes(&server, 1).await?;
let mut text = String::new();
let mut events: Vec<Value> = Vec::new();
tokio::time::timeout(TIMEOUT, async {
while !events.iter().any(|event| event["type"] == "best_withdrawn") {
let chunk = stream
.chunk()
.await?
.ok_or_else(|| eyre::eyre!("stream ended"))?;
text.push_str(std::str::from_utf8(&chunk)?);
events = text
.lines()
.filter_map(|line| line.strip_prefix("data:"))
.filter_map(|data| serde_json::from_str(data.trim()).ok())
.collect();
}
Ok::<_, eyre::Error>(())
})
.await??;
let about_routes: Vec<&Value> = events
.iter()
.filter(|event| {
event["type"]
.as_str()
.is_some_and(|t| t.starts_with("best_"))
})
.collect();
// The announcement and the withdrawal may have been read as one change
// or as two; either way nothing is said of the route that was there
// from the start, and the last word is that the prefix is gone.
assert!(about_routes.len() <= 2, "{about_routes:?}");
for event in &about_routes {
let named = event["prefix"]
.as_str()
.or(event["route"]["prefix"].as_str());
assert_eq!(named, Some("203.0.113.0/24"), "{event}");
}
assert_eq!(about_routes.last().unwrap()["type"], "best_withdrawn");
Ok(())
}
/// The table can be far too large to send whole: the API gives a part of
/// it, and says how much there is.
#[tokio::test]
async fn test_received_routes_are_returned_up_to_a_limit() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let prefixes: Vec<IpPrefix> = (0..40).map(|n| prefix(&format!("10.{n}.0.0/16"))).collect();
let mut peer = peer(&server, 1).await?;
peer.send(&announcement(&prefixes, &[65001])).await?;
wait_for_routes(&server, 40).await?;
let count = |routes: Value| routes.as_array().unwrap().len();
let url = format!("{base}/api/routes/received");
assert_eq!(count(get_json(&client, &url).await?), 40);
assert_eq!(
count(get_json(&client, &format!("{url}?limit=7")).await?),
7
);
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["received_routes"], 40);
Ok(())
}
crates/integration-tests/tests/collision.rs
+315
View File
@@ -0,0 +1,315 @@
//! Connection collision detection (RFC 4271 §6.8) against a live server.
//!
//! The test server's BGP Identifier is 127.0.0.1. A test peer picks an
//! identifier above or below that to steer which connection must survive.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, NotificationMessage, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::collision::Direction;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use eyre::Result;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
/// Above the server's 127.0.0.1.
const HIGH_ID: Ipv4Addr = Ipv4Addr::new(192, 0, 2, 1);
/// Below the server's 127.0.0.1.
const LOW_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
async fn server_dialing(listener: &TcpListener) -> Result<TestServer> {
let endpoint: SocketAddr = listener.local_addr()?;
TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
name: "r1".into(),
endpoint,
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
}];
})
.await
}
/// Exchanges OPENs only, leaving the server's side in OpenConfirm (it has
/// sent its KEEPALIVE and is waiting for ours).
async fn open_only(peer: &mut TestPeer, router_id: Ipv4Addr) -> Result<()> {
peer.send_open(PEER_ASN, router_id, 90).await?;
peer.recv_open().await?;
Ok(())
}
/// Finishes the handshake begun by [`open_only`].
async fn confirm(peer: &mut TestPeer) -> Result<()> {
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
Message::KeepAlive => {}
other => eyre::bail!("expected KEEPALIVE, got {other:?}"),
}
peer.send(&Message::KeepAlive).await
}
/// Reads until a NOTIFICATION arrives, skipping KEEPALIVEs.
async fn expect_notification(peer: &mut TestPeer) -> Result<NotificationMessage> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => eyre::bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
fn is_collision_cease(n: &NotificationMessage) -> bool {
matches!(
n.error,
BgpError::Cease(CeaseSubcode::ConnCollisionResolution)
)
}
async fn wait_for_sessions(server: &TestServer, n: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().len() != n {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
}
/// Peer has the higher identifier, so the connection the *peer* initiated
/// must survive: the server closes its own outbound connection.
#[tokio::test]
async fn test_simultaneous_open_peer_higher_keeps_inbound() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
// Server's outbound connection reaches OpenConfirm.
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
open_only(&mut dialled, HIGH_ID).await?;
// The peer now also dials the server.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, HIGH_ID).await?;
let notif = expect_notification(&mut dialled).await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(dialled.is_closed().await);
confirm(&mut inbound).await?;
wait_for_sessions(&server, 1).await?;
let session = server.registry.list().remove(0);
assert_eq!(session.direction, Direction::Inbound);
assert_eq!(session.name.as_deref(), Some("r1"));
// The survivor is fully functional.
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, inbound.recv_update()).await??;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
// The dialer notices the surviving session and does not open another.
let extra = tokio::time::timeout(Duration::from_millis(700), listener.accept()).await;
assert!(
extra.is_err(),
"dialer opened a second connection despite an established session"
);
Ok(())
}
/// Peer has the lower identifier, so the connection the *server* initiated
/// must survive: the peer's inbound connection is refused.
#[tokio::test]
async fn test_simultaneous_open_peer_lower_keeps_outbound() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
open_only(&mut dialled, LOW_ID).await?;
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
let notif = expect_notification(&mut inbound).await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(inbound.is_closed().await);
confirm(&mut dialled).await?;
wait_for_sessions(&server, 1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
/// The order in which the two OPENs are processed must not change the
/// outcome: here the inbound connection reaches OpenConfirm first.
#[tokio::test]
async fn test_outcome_is_independent_of_arrival_order() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
// Inbound first this time; peer is lower so outbound must still win.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
open_only(&mut dialled, LOW_ID).await?;
let notif = expect_notification(&mut inbound).await?;
assert!(is_collision_cease(&notif), "{notif:?}");
confirm(&mut dialled).await?;
wait_for_sessions(&server, 1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
/// RFC default: a collision with an Established session closes the new
/// connection, whatever the identifiers say.
#[tokio::test]
async fn test_established_session_wins_by_default() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
let mut second = TestPeer::connect(server.addr).await?;
open_only(&mut second, HIGH_ID).await?;
let notif = expect_notification(&mut second).await?;
assert!(is_collision_cease(&notif), "{notif:?}");
// The original is untouched.
assert_eq!(server.registry.list().len(), 1);
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, first.recv_update()).await??;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
/// With collision_detect_established_state, a peer that restarts and dials
/// again replaces the stale session at once, and the routes learned over the
/// old session are flushed.
#[tokio::test]
async fn test_restarted_peer_replaces_stale_session_when_enabled() -> Result<()> {
let server =
TestServer::start_with(LOCAL_ASN, |c| c.collision_detect_established_state = true).await?;
let mut stale = TestPeer::connect(server.addr).await?;
stale.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
stale
.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 9))),
],
announced_routes: vec![IpPrefix::try_from("203.0.113.0/24")?.into()],
}))
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.received_routes().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let stale_addr = server.registry.list()[0].peer.addr;
// "Restart": a fresh connection from the same router.
let mut fresh = TestPeer::connect(server.addr).await?;
fresh.handshake(PEER_ASN, HIGH_ID, 90).await?;
let notif = expect_notification(&mut stale).await?;
assert!(is_collision_cease(&notif), "{notif:?}");
tokio::time::timeout(TIMEOUT, async {
loop {
let sessions = server.registry.list();
if sessions.len() == 1 && sessions[0].peer.addr != stale_addr {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
assert!(
server.received_routes().is_empty(),
"stale routes were not flushed"
);
Ok(())
}
/// Different routers (different identifiers) never collide.
#[tokio::test]
async fn test_distinct_identifiers_do_not_collide() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = TestPeer::connect(server.addr).await?;
a.handshake(PEER_ASN, HIGH_ID, 90).await?;
let mut b = TestPeer::connect(server.addr).await?;
b.handshake(PEER_ASN, LOW_ID, 90).await?;
wait_for_sessions(&server, 2).await?;
Ok(())
}
/// A failed connection frees its slot: the same router can come back.
#[tokio::test]
async fn test_identifier_is_released_when_connection_ends() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
drop(first);
wait_for_sessions(&server, 0).await?;
let mut second = TestPeer::connect(server.addr).await?;
second.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
Ok(())
}
/// A peer claiming our own identifier is refused outright.
#[tokio::test]
async fn test_peer_with_our_identifier_is_refused() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
open_only(&mut peer, Ipv4Addr::new(127, 0, 0, 1)).await?;
let notif = expect_notification(&mut peer).await?;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadBgpId)),
"{notif:?}"
);
Ok(())
}
crates/integration-tests/tests/dialer.rs
+319
View File
@@ -0,0 +1,319 @@
//! The active side: the server dials its configured peers and keeps
//! retrying.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::Message;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::session::DialState;
use eyre::Result;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
/// Starts a server configured to dial `listener`'s address as peer "r1".
async fn server_dialing(listener: &TcpListener, remote_asn: u32) -> Result<TestServer> {
let endpoint: SocketAddr = listener.local_addr()?;
TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
name: "r1".into(),
endpoint,
remote_asn,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
}];
})
.await
}
async fn accept(listener: &TcpListener) -> Result<TestPeer> {
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
Ok(TestPeer::from_stream(stream))
}
#[tokio::test]
async fn test_server_dials_configured_peer_and_establishes() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let mut peer = accept(&listener).await?;
// Both sides send OPEN on connect; the test peer's handshake order works
// whichever side opened TCP.
let open = peer
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
assert_eq!(open.asn, LOCAL_ASN as u16);
let mut events = server.registry.subscribe();
// Either the Up event is still queued or the session is already listed.
let listed = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(s) = server.registry.list().first().cloned() {
return s;
}
let _ = events.recv().await;
}
})
.await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
assert_eq!(listed.peer.asn, PEER_ASN);
assert_eq!(listed.peer.addr, listener.local_addr()?);
// Configured view reflects it; the dialer is not in backoff.
let (cfg, state) = server
.registry
.configured_peers()
.into_iter()
.next()
.unwrap();
assert_eq!(cfg.name, "r1");
assert!(
matches!(state, DialState::Connecting | DialState::Idle),
"{state:?}"
);
// Local routes flow over the dialled session too.
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
#[tokio::test]
async fn test_server_reconnects_after_peer_closes() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let mut first = accept(&listener).await?;
first
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
drop(first);
// Session goes away, then the dialer comes back.
tokio::time::timeout(TIMEOUT, async {
while !server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let (_, state) = server
.registry
.configured_peers()
.into_iter()
.next()
.unwrap();
assert!(matches!(state, DialState::Backoff { .. }), "{state:?}");
let mut second = accept(&listener).await?;
second
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
}
#[tokio::test]
async fn test_wrong_asn_is_refused_with_bad_peer_as() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let _server = server_dialing(&listener, PEER_ASN).await?;
let mut peer = accept(&listener).await?;
peer.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let _ = peer.recv_open().await?;
let notif = tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {}
other => eyre::bail!("unexpected {other:?}"),
}
}
})
.await??;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{notif:?}"
);
// The dialer retries rather than giving up.
let _again = accept(&listener).await?;
Ok(())
}
/// An inbound connection from a configured peer's address is also checked
/// against the configured ASN and gets the configured name.
#[tokio::test]
async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
// Configure a peer at the loopback address on a port nobody listens on,
// so the dialer never succeeds; then connect inbound from loopback.
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_secs(60);
c.peers = vec![PeerConfig {
name: "r1".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
}];
})
.await?;
// Wrong ASN inbound → refused.
let mut bad = TestPeer::connect(server.addr).await?;
bad.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90).await?;
let _ = bad.recv_open().await?;
let msg = tokio::time::timeout(TIMEOUT, bad.recv()).await??;
assert!(
matches!(&msg, Message::Notification(n) if matches!(n.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs))),
"{msg:?}"
);
// Right ASN inbound → established, named.
let mut good = TestPeer::connect(server.addr).await?;
good.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let listed = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(s) = server.registry.list().first().cloned() {
return s;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
Ok(())
}
/// With `local_address` set the connection must come from that address (the
/// peer may identify us by it), and an address the host does not have is
/// reported as the reason for backing off rather than silently ignored.
#[tokio::test]
async fn test_local_address_is_used_as_the_source() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint: SocketAddr = listener.local_addr()?;
let _server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
name: "r1".into(),
endpoint,
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: Some(Ipv4Addr::LOCALHOST.into()),
families: FamilySet::supported(),
add_path: None,
}];
})
.await?;
let (_stream, from) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
assert_eq!(from.ip(), std::net::IpAddr::from(Ipv4Addr::LOCALHOST));
// 192.0.2.1 is documentation space; no host has it.
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
name: "r1".into(),
endpoint,
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: Some(Ipv4Addr::new(192, 0, 2, 1).into()),
families: FamilySet::supported(),
add_path: None,
}];
})
.await?;
let reason = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some((_, DialState::Backoff { last_error, .. })) =
server.registry.configured_peers().into_iter().next()
{
return last_error;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await?;
assert!(reason.contains("local_address 192.0.2.1"), "{reason}");
Ok(())
}
/// A peer that takes the connection and then refuses the session, here
/// because it is not the AS we are configured to expect, will refuse it
/// again. It is dialled ever more slowly, like a peer that cannot be
/// reached: connecting used to count as success and reset the wait, so
/// such a peer was dialled every few seconds for ever.
#[tokio::test]
async fn test_a_peer_that_refuses_the_session_is_dialled_ever_more_slowly() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let mut waits = Vec::new();
for _ in 0..3 {
let mut peer = accept(&listener).await?;
peer.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let _ = peer.recv_open().await?;
let refusal = tokio::time::timeout(TIMEOUT, peer.recv()).await??;
assert!(
matches!(&refusal, Message::Notification(n) if n.error == BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{refusal:?}"
);
drop(peer);
// How long the dialer says it will wait before trying again.
let wait = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some((_, DialState::Backoff { until, last_error })) =
server.registry.configured_peers().into_iter().next()
{
assert!(last_error.contains("AS65099"), "{last_error}");
return until
.duration_since(std::time::SystemTime::now())
.unwrap_or_default();
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
waits.push(wait);
}
// 200 ms, then 400, then 800, less what passed before each was read.
assert!(
waits[2] > Duration::from_millis(500),
"the waits were {waits:?}"
);
Ok(())
}
crates/integration-tests/tests/failure.rs
+87
View File
@@ -0,0 +1,87 @@
//! What the server does when a part of it that it cannot do without fails.
//!
//! The local RIB's actor stops only by panicking. Before, nothing noticed:
//! sessions stayed up, nothing was announced any more, and the status page
//! showed an empty table. Now the server tells its peers it is going and
//! stops with an error, for whatever supervises it to restart it.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
use bgp_rib::RibStopped;
use bgp_server::error::ServerError;
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
async fn established(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(peer)
}
/// Reads until the server's NOTIFICATION, skipping KEEPALIVEs.
async fn farewell(peer: &mut TestPeer) -> Result<BgpError> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::KeepAlive => {}
Message::Notification(n) => return Ok(n.error),
other => bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
#[tokio::test]
async fn test_losing_the_local_rib_stops_the_server_and_tells_the_peers() -> Result<()> {
let mut server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
server.local_rib.stop_for_testing();
let outcome = tokio::time::timeout(TIMEOUT, server.stopped()).await?;
assert!(
matches!(outcome, Err(ServerError::RibStopped(RibStopped { rib })) if rib == "local RIB"),
"{outcome:?}"
);
assert_eq!(
farewell(&mut peer).await?,
BgpError::Cease(CeaseSubcode::AdminShutdown)
);
// The status interface says the table cannot be read, where it used to
// show an empty one.
let client = reqwest::Client::new();
for path in ["/api/status", "/api/routes/local", "/metricz"] {
let response = client.get(format!("{base}{path}")).send().await?;
assert_eq!(response.status(), 503, "{path}");
assert_eq!(
response.text().await?,
"the local RIB has stopped",
"{path}"
);
}
// What does not need that table still answers.
let response = client
.get(format!("{base}/api/routes/received"))
.send()
.await?;
assert_eq!(response.status(), 200);
Ok(())
}
crates/integration-tests/tests/http_api.rs
+303
View File
@@ -0,0 +1,303 @@
//! Drives the HTTP API against a real server with a real BGP peer attached.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, LargeCommunity, Message, NextHopPathAttribute,
OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
use serde_json::{Value, json};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
async fn established_peer(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(peer)
}
async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
}
#[tokio::test]
async fn test_status_and_peers() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let config = get_json(&client, &format!("{base}/api/config")).await?;
assert_eq!(config["path"], "test.textpb");
assert_eq!(config["text"], format!("local_asn: {LOCAL_ASN}\n"));
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["local_asn"], LOCAL_ASN);
assert_eq!(status["router_id"], "127.0.0.1");
assert_eq!(status["sessions"], 0);
assert!(status["started_at_unix"].as_u64().unwrap() > 1_700_000_000);
assert!(!status["hostname"].as_str().unwrap().is_empty());
let _peer = established_peer(&server).await?;
let status = get_json(&client, &format!("{base}/api/status")).await?;
assert_eq!(status["sessions"], 1);
let peers = get_json(&client, &format!("{base}/api/peers")).await?;
assert_eq!(peers.as_array().unwrap().len(), 1);
assert_eq!(peers[0]["peer_asn"], PEER_ASN);
assert_eq!(peers[0]["local_addr"], server.addr.to_string());
assert_eq!(peers[0]["four_octet_asn"], true);
assert_eq!(
peers[0]["address_families"],
serde_json::json!(["IPv4 unicast", "IPv6 unicast"])
);
Ok(())
}
/// Originated routes are listed with their attributes as text.
#[tokio::test]
async fn test_originated_routes_are_listed() -> Result<()> {
let attrs = LocalRouteAttrs {
prepend: 1,
med: Some(10),
communities: vec![(65000, 100), (65535, 65281)],
large_communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2,
}],
..Default::default()
};
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.originate = vec![LocalRoute::new(prefix("10.100.1.0/24"), attrs)];
})
.await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let local = get_json(&client, &format!("{base}/api/routes/local")).await?;
assert_eq!(local.as_array().unwrap().len(), 1);
assert_eq!(local[0]["prefix"], "10.100.1.0/24");
assert_eq!(local[0]["origin"], "IGP");
assert_eq!(local[0]["prepend"], 1);
assert_eq!(local[0]["med"], 10);
assert_eq!(local[0]["communities"], json!(["65000:100", "65535:65281"]));
assert_eq!(local[0]["large_communities"], json!(["65000:1:2"]));
assert_eq!(
get_json(&client, &format!("{base}/api/status")).await?["local_routes"],
1
);
Ok(())
}
/// The API only reports. Nothing reachable over HTTP changes what the
/// router announces.
#[tokio::test]
async fn test_api_is_read_only() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let body = json!({ "prefix": "10.100.1.0/24" });
for path in [
"/api/routes/local",
"/api/routes/received",
"/api/peers",
"/api/status",
] {
let post = client
.post(format!("{base}{path}"))
.json(&body)
.send()
.await?;
assert_eq!(post.status(), 405, "POST {path}");
let delete = client
.delete(format!("{base}{path}"))
.json(&body)
.send()
.await?;
assert_eq!(delete.status(), 405, "DELETE {path}");
}
assert!(server.local_rib.snapshot().await.unwrap().is_empty());
Ok(())
}
/// Routes the peer sends us show up in the received table with a summary.
#[tokio::test]
async fn test_received_routes_summary() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut peer = established_peer(&server).await?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN, 65002]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 1))),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65001, 7)])),
],
announced_routes: vec![prefix("203.0.113.0/24").into()],
}))
.await?;
let received = tokio::time::timeout(TIMEOUT, async {
loop {
let v = get_json(&client, &format!("{base}/api/routes/received")).await?;
if !v.as_array().unwrap().is_empty() {
return Ok::<_, eyre::Error>(v);
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await??;
assert_eq!(received[0]["prefix"], "203.0.113.0/24");
assert_eq!(received[0]["peer_asn"], PEER_ASN);
assert_eq!(received[0]["summary"]["as_path"], json!([PEER_ASN, 65002]));
assert_eq!(received[0]["summary"]["next_hop"], "10.0.0.1");
assert_eq!(received[0]["summary"]["communities"], json!(["65001:7"]));
Ok(())
}
/// The SSE stream reports session and route changes as they happen.
#[tokio::test]
async fn test_event_stream() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut resp = client
.get(format!("{base}/api/events"))
.send()
.await?
.error_for_status()?;
assert!(
resp.headers()["content-type"]
.to_str()?
.starts_with("text/event-stream")
);
let _peer = established_peer(&server).await?;
server
.local_rib
.announce(LocalRoute::new(prefix("10.100.1.0/24"), Default::default()))
.await?;
// Collect SSE frames until we have seen both events.
let mut buf = String::new();
let mut seen_up = false;
let mut seen_announce = false;
tokio::time::timeout(TIMEOUT, async {
while !(seen_up && seen_announce) {
let chunk = resp
.chunk()
.await?
.ok_or_else(|| eyre::eyre!("stream ended"))?;
buf.push_str(std::str::from_utf8(&chunk)?);
for line in buf.lines().filter_map(|l| l.strip_prefix("data:")) {
let ev: Value = serde_json::from_str(line.trim())?;
match ev["type"].as_str() {
Some("session_up") if ev["peer"]["peer_asn"] == PEER_ASN => seen_up = true,
Some("local_announced") if ev["route"]["prefix"] == "10.100.1.0/24" => {
seen_announce = true
}
_ => {}
}
}
}
Ok::<_, eyre::Error>(())
})
.await??;
Ok(())
}
#[tokio::test]
async fn test_statusz_page_is_served() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let resp = client.get(format!("{base}/statusz")).send().await?;
assert_eq!(resp.status(), 200);
assert!(
resp.headers()["content-type"]
.to_str()?
.starts_with("text/html")
);
let body = resp.text().await?;
assert!(body.contains("<title>Status for ferrix_bgp</title>"));
assert!(body.contains("/api/events"));
// The root redirects to the page.
let no_redirect = reqwest::Client::builder()
.redirect(reqwest::redirect::Policy::none())
.build()?;
let resp = no_redirect.get(format!("{base}/")).send().await?;
assert_eq!(resp.status(), 307);
assert_eq!(resp.headers()["location"], "/statusz");
Ok(())
}
#[tokio::test]
async fn test_metricz() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let _peer = established_peer(&server).await?;
let text = client
.get(format!("{base}/metricz"))
.send()
.await?
.error_for_status()?
.text()
.await?;
// Metrics are process-global, so other tests in this binary contribute
// too; assert on presence and a lower bound rather than exact values.
let metric = |name: &str| -> Option<i64> {
text.lines()
.find(|l| l.starts_with(name) && l[name.len()..].starts_with(' '))
.and_then(|l| l.rsplit(' ').next()?.parse().ok())
};
assert!(
metric("bgp_sessions_established").unwrap_or(0) >= 1,
"{text}"
);
assert!(
metric("bgp_messages_sent_total{type=\"open\"}").unwrap_or(0) >= 1,
"{text}"
);
assert!(
metric("bgp_messages_received_total{type=\"keepalive\"}").unwrap_or(0) >= 1,
"{text}"
);
assert!(metric("bgp_local_routes").is_some(), "{text}");
assert!(metric("bgp_rib_routes").is_some(), "{text}");
Ok(())
}
crates/integration-tests/tests/interop.rs
+148
View File
@@ -0,0 +1,148 @@
//! Interoperability with what real routers put on the wire.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{Message, PathAttribute};
use bgp_packet::open::{Capability, OpenOption};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
const TIMEOUT: Duration = Duration::from_secs(5);
/// The OPEN a BIRD 2 router sends for an IPv6-only session from AS210036:
/// one optional parameter carrying MP(IPv6 unicast), route refresh,
/// extended message, graceful restart (aware only), 4-octet AS, enhanced
/// route refresh and long-lived graceful restart (aware only).
#[rustfmt::skip]
const BIRD_OPEN: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x37, 0x01, // length 55, type OPEN
0x04, // version
0x5b, 0xa0, // AS_TRANS
0x00, 0xf0, // hold time 240
0x0a, 0xff, 0x00, 0x01, // router id 10.255.0.1
0x1a, // optional parameters length 26
0x02, 0x18, // capabilities, 24 bytes
0x01, 0x04, 0x00, 0x02, 0x00, 0x01, // MP: IPv6 unicast
0x02, 0x00, // route refresh
0x06, 0x00, // extended message
0x40, 0x02, 0x00, 0x00, // graceful restart, no address families
0x41, 0x04, 0x00, 0x03, 0x34, 0x74, // 4-octet AS 210036
0x46, 0x00, // enhanced route refresh
0x47, 0x00, // long-lived graceful restart, none
];
/// IPv6 End-of-RIB: an UPDATE whose only attribute is an empty
/// MP_UNREACH_NLRI for IPv6 unicast (RFC 4724 §2).
#[rustfmt::skip]
const IPV6_END_OF_RIB: &[u8] = &[
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0x00, 0x1e, 0x02, // length 30, type UPDATE
0x00, 0x00, // no withdrawn routes
0x00, 0x07, // 7 bytes of path attributes
0x90, 0x0f, 0x00, 0x03, // optional + extended length, MP_UNREACH_NLRI, len 3
0x00, 0x02, 0x01, // IPv6 unicast, no prefixes
];
#[tokio::test]
async fn test_server_advertises_ipv4_and_ipv6_unicast() -> Result<()> {
let server = TestServer::start(65000).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer.recv_open().await?;
let caps: Vec<&Capability> = open
.options
.iter()
.flat_map(|o| match o {
OpenOption::Capabilities(c) => c.iter(),
})
.collect();
for afi in [Afi::IPV4, Afi::IPV6] {
assert!(
caps.iter().any(|c| matches!(c, Capability::MultiProtocol { afi: a, safi: Safi::UNICAST } if *a == afi)),
"missing MP capability for {afi:?}: {caps:?}"
);
}
Ok(())
}
#[tokio::test]
async fn test_bird_open_and_end_of_rib_are_accepted() -> Result<()> {
let server = TestServer::start(4200000001).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_raw(BIRD_OPEN).await?;
peer.recv_open().await?;
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
Message::KeepAlive => {}
other => eyre::bail!("server did not accept BIRD's OPEN: {other:?}"),
}
peer.send(&Message::KeepAlive).await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let session = server.registry.list().remove(0);
assert_eq!(
session.peer.asn, 210036,
"4-octet ASN must come from the capability, not AS_TRANS"
);
assert!(session.four_octet_asn);
// End-of-RIB must neither break the session nor create a route.
peer.send_raw(IPV6_END_OF_RIB).await?;
peer.send(&Message::KeepAlive).await?;
tokio::time::sleep(Duration::from_millis(200)).await;
assert_eq!(
server.registry.list().len(),
1,
"session dropped after End-of-RIB"
);
assert!(server.received_routes().is_empty());
Ok(())
}
/// Over an IPv4 test transport an IPv6 route cannot be sent (no IPv6 next
/// hop), so this only checks the builder the advertiser uses: the UPDATE for
/// an IPv6 prefix carries MP_REACH_NLRI with an IPv6 next hop and no NEXT_HOP.
#[tokio::test]
async fn test_ipv6_prefix_update_shape() -> Result<()> {
let route = LocalRoute::new(
IpPrefix::try_from("2001:db8:1::/48")?,
LocalRouteAttrs::default(),
);
let nh: std::net::Ipv6Addr = "fd00:179::2".parse()?;
let update = bgp_server::advertise::build_announce(
&route,
&bgp_server::advertise::Towards {
local_asn: 4200000001,
next_hop: nh.into(),
four_octet_asn: true,
ibgp: false,
},
)
.expect("ipv6 over ipv6");
assert!(update.announced_routes.is_empty());
assert!(
!update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::NextHop(_)))
);
assert!(
update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::MpReachNlri(m)
if m.family == AfiSafi::IPV6_UNICAST && m.prefixes[0].to_string() == "2001:db8:1::/48"))
);
let _ = Ipv4Addr::LOCALHOST;
Ok(())
}
crates/integration-tests/tests/negotiation.rs
+321
View File
@@ -0,0 +1,321 @@
//! End-to-end tests for capability negotiation: what the two OPENs agree on
//! decides which routes are sent and accepted, and how they are encoded.
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NextHopPathAttribute, NlriNextHop,
OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_packet::open::Capability;
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::{FamilySet, SUPPORTED};
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn only(family: AfiSafi) -> FamilySet {
[family].into_iter().collect()
}
fn four_octet() -> Capability {
Capability::FourByteAsn { asn: PEER_ASN }
}
async fn wait_for_session(server: &TestServer) -> Result<bgp_server::session::SessionHandle> {
Ok(tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(session) = server.registry.list().into_iter().next() {
return session;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?)
}
/// An UPDATE announcing one IPv4 route in the body and one IPv6 route in
/// MP_REACH_NLRI.
fn dual_stack_update() -> UpdateMessage {
UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1)),
prefixes: vec![prefix("2001:db8:1::/48").into()],
}),
],
announced_routes: vec![prefix("192.0.2.0/24").into()],
}
}
/// The session carries the families both OPENs offered, and the server's
/// OPEN offers every family by default.
#[tokio::test]
async fn test_families_are_the_ones_both_sides_offered() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
for family in SUPPORTED {
assert!(
open.capabilities()
.any(|c| *c == Capability::multiprotocol(family)),
"the server's OPEN offers {family}"
);
}
let session = wait_for_session(&server).await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
Ok(())
}
/// A route of a family the peer did not offer is not accepted from it. The
/// rest of the UPDATE is, and the session stays up.
#[tokio::test]
async fn test_routes_of_a_family_not_negotiated_are_not_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
wait_for_session(&server).await?;
peer.send(&Message::Update(dual_stack_update())).await?;
let routes = tokio::time::timeout(TIMEOUT, async {
loop {
let routes = server.received_routes();
if !routes.is_empty() {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
// One UPDATE is applied as a whole, so the IPv6 route is not on its way.
assert_eq!(routes.len(), 1);
assert_eq!(routes[0].prefix, prefix("192.0.2.0/24"));
assert_eq!(server.registry.list().len(), 1, "the session survives");
Ok(())
}
/// A route of a family the peer did not offer is not sent to it.
#[tokio::test]
async fn test_routes_of_a_family_not_negotiated_are_not_sent() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.originate = vec![LocalRoute::new(
prefix("10.100.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
// This peer wants IPv6 only; the server has an IPv4 route.
let mut v6_only = TestPeer::connect(server.addr).await?;
v6_only
.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
],
)
.await?;
let session = wait_for_session(&server).await?;
assert_eq!(session.families, only(AfiSafi::IPV6_UNICAST));
// A second peer that takes IPv4 is sent the route. The advertiser
// handles sessions in the order they came up, so by the time this one
// has its route the first has been dealt with.
let mut dual = TestPeer::connect(server.addr).await?;
dual.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let update = tokio::time::timeout(TIMEOUT, dual.recv_update()).await??;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
let nothing = tokio::time::timeout(Duration::from_millis(200), v6_only.recv_update()).await;
assert!(nothing.is_err(), "got {nothing:?}");
Ok(())
}
/// A configured peer is offered only the families configured for it, and a
/// peer that shares none of them is refused with the capability we needed.
#[tokio::test]
async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// Inbound connections from this address inherit the peer's settings.
c.peers = vec![PeerConfig {
name: "v6-only".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: only(AfiSafi::IPV6_UNICAST),
add_path: None,
}];
// Keep the dialer out of the way.
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_open_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
let open = peer.recv_open().await?;
let offered: Vec<_> = open
.capabilities()
.filter(|c| matches!(c, Capability::MultiProtocol { .. }))
.collect();
assert_eq!(offered, [&Capability::multiprotocol(AfiSafi::IPV6_UNICAST)]);
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
Message::Notification(n) => {
assert_eq!(
n.error,
BgpError::OpenMsg(OpenMsgSubcode::UnsupportedCapability)
);
// The Multiprotocol capability for IPv6 unicast.
assert_eq!(n.data, [1, 4, 0, 2, 0, 1]);
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
assert!(server.registry.list().is_empty());
Ok(())
}
/// Real routers offer families we do not support. That must not stop
/// the families we do share from working.
#[tokio::test]
async fn test_a_peer_offering_an_unsupported_family_is_still_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(
PEER_ASN,
PEER_ID,
90,
vec![
four_octet(),
Capability::multiprotocol(AfiSafi::new(Afi::L2VPN, Safi::EVPN)),
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
],
)
.await?;
let session = wait_for_session(&server).await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
assert!(
session.four_octet_asn,
"capabilities after it are still read"
);
Ok(())
}
/// A peer that offers no capabilities at all is a plain RFC 4271 speaker:
/// IPv4 unicast, two-octet AS numbers. Our four-octet AS number reaches it
/// as AS_TRANS in AS_PATH and in full in AS4_PATH (RFC 6793 §4.2.2).
#[tokio::test]
async fn test_a_peer_without_capabilities_gets_two_octet_as_paths() -> Result<()> {
const WIDE_ASN: u32 = 4_200_000_001;
let server = TestServer::start_with(WIDE_ASN, |c| {
c.originate = vec![LocalRoute::new(
prefix("10.100.0.0/16"),
LocalRouteAttrs::default(),
)];
})
.await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, vec![]).await?;
let session = wait_for_session(&server).await?;
assert!(!session.four_octet_asn);
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
// The test peer now decodes AS_PATH two octets wide, as such a speaker
// would; an UPDATE encoded four octets wide would not parse.
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
assert!(
update
.path_attributes
.contains(&AsPathAttribute::from_asns(vec![23456])),
"{update:?}"
);
let as4_path = update.path_attributes.iter().find_map(|a| match a {
PathAttribute::As4Path(path) => Some(path.segments[0].path.clone()),
_ => None,
});
assert_eq!(as4_path, Some(vec![WIDE_ASN]));
// And its own two-octet paths are read correctly.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN, 64512]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: vec![prefix("192.0.2.0/24").into()],
}))
.await?;
let routes = tokio::time::timeout(TIMEOUT, async {
loop {
let routes = server.received_routes();
if !routes.is_empty() {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
assert!(
routes[0]
.attributes
.contains(&AsPathAttribute::from_asns(vec![PEER_ASN, 64512]))
);
Ok(())
}
crates/integration-tests/tests/update_errors.rs
+228
View File
@@ -0,0 +1,228 @@
//! End-to-end tests for errors in UPDATE messages (RFC 7606): most of them
//! cost the routes in the UPDATE, not the session.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ReceivedRoute, TestServer};
use bgp_packet::errors::{BgpError, UpdateMsgSubcode};
use bgp_packet::message::{LocalPrefPathAttribute, Message, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::metrics::UPDATE_ERRORS;
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
const WELL_KNOWN: u8 = 0x40;
const OPTIONAL_TRANSITIVE: u8 = 0xc0;
fn attr(flags: u8, type_code: u8, value: &[u8]) -> Vec<u8> {
let mut wire = vec![flags, type_code, value.len() as u8];
wire.extend_from_slice(value);
wire
}
fn origin() -> Vec<u8> {
attr(WELL_KNOWN, 1, &[0])
}
/// AS_SEQUENCE of the peer's AS, four octets wide.
fn as_path() -> Vec<u8> {
attr(WELL_KNOWN, 2, &[2, 1, 0, 0, 0xfd, 0xe9])
}
fn next_hop() -> Vec<u8> {
attr(WELL_KNOWN, 3, &[192, 0, 2, 1])
}
/// A whole UPDATE message announcing `nlri` with `attributes`, as bytes, so
/// that it can be wrong in ways the encoder would not write.
fn update(attributes: &[Vec<u8>], nlri: &[u8]) -> Vec<u8> {
let attributes = attributes.concat();
let mut body = vec![0, 0];
body.extend_from_slice(&(attributes.len() as u16).to_be_bytes());
body.extend_from_slice(&attributes);
body.extend_from_slice(nlri);
let mut wire = vec![0xff; 16];
wire.extend_from_slice(&((19 + body.len()) as u16).to_be_bytes());
wire.push(2);
wire.extend_from_slice(&body);
wire
}
/// 10.1.0.0/16 and 10.2.0.0/16.
const ROUTE_A: [u8; 3] = [16, 10, 1];
const ROUTE_B: [u8; 3] = [16, 10, 2];
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
async fn established(server: &TestServer, asn: u32) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(asn, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
Ok(peer)
}
/// Waits until the server's Adj-RIB-In holds exactly `prefixes`.
async fn wait_for_routes(server: &TestServer, prefixes: &[&str]) -> Result<Vec<ReceivedRoute>> {
let mut expected: Vec<IpPrefix> = prefixes.iter().map(|p| prefix(p)).collect();
expected.sort_by_key(|p| p.to_string());
Ok(tokio::time::timeout(TIMEOUT, async {
loop {
let mut routes = server.received_routes();
routes.sort_by_key(|r| r.prefix.to_string());
if routes.iter().map(|r| &r.prefix).eq(expected.iter()) {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?)
}
/// The situation RFC 7606 exists for: a route is in the table, and then an
/// UPDATE for it arrives with a broken attribute. The route goes; the
/// session, and the other routes learned over it, stay.
#[tokio::test]
async fn test_a_malformed_attribute_withdraws_the_route_and_keeps_the_session() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
let handled_before = UPDATE_ERRORS
.with_label_values(&["treat-as-withdraw"])
.get();
let good = [origin(), as_path(), next_hop()];
peer.send_raw(&update(&good, &[ROUTE_A, ROUTE_B].concat()))
.await?;
wait_for_routes(&server, &["10.1.0.0/16", "10.2.0.0/16"]).await?;
// ORIGIN with a value that is not defined.
let bad = [attr(WELL_KNOWN, 1, &[7]), as_path(), next_hop()];
peer.send_raw(&update(&bad, &ROUTE_A)).await?;
wait_for_routes(&server, &["10.2.0.0/16"]).await?;
// The session goes on: the route can be announced again.
peer.send_raw(&update(&good, &ROUTE_A)).await?;
wait_for_routes(&server, &["10.1.0.0/16", "10.2.0.0/16"]).await?;
assert_eq!(server.registry.list().len(), 1);
assert!(
UPDATE_ERRORS
.with_label_values(&["treat-as-withdraw"])
.get()
> handled_before
);
Ok(())
}
/// An announcement without a mandatory attribute is not a route we can use.
#[tokio::test]
async fn test_a_route_without_as_path_is_not_accepted() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
peer.send_raw(&update(&[origin(), next_hop()], &ROUTE_A))
.await?;
// A well-formed one after it shows that the first was dealt with.
peer.send_raw(&update(&[origin(), as_path(), next_hop()], &ROUTE_B))
.await?;
wait_for_routes(&server, &["10.2.0.0/16"]).await?;
assert_eq!(server.registry.list().len(), 1);
Ok(())
}
/// A broken attribute that plays no part in choosing a route is dropped,
/// and the route is kept.
#[tokio::test]
async fn test_a_malformed_aggregator_is_dropped_and_the_route_kept() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
let attributes = [
origin(),
as_path(),
next_hop(),
// AGGREGATOR is eight octets on this session.
attr(OPTIONAL_TRANSITIVE, 7, &[0, 0, 0xfd, 0xe9, 192]),
];
peer.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert_eq!(routes[0].attributes.len(), 3);
assert!(
!routes[0]
.attributes
.iter()
.any(|a| matches!(a, PathAttribute::Aggregator(_)))
);
Ok(())
}
/// LOCAL_PREF is a matter between routers of one AS. From an external peer
/// it is dropped; from an internal one it is kept.
#[tokio::test]
async fn test_local_pref_is_kept_only_from_an_internal_peer() -> Result<()> {
let local_pref = attr(WELL_KNOWN, 5, &[0, 0, 0, 200]);
let has_local_pref = |routes: &[ReceivedRoute]| {
routes[0]
.attributes
.contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(200)))
};
let server = TestServer::start(LOCAL_ASN).await?;
let mut external = established(&server, PEER_ASN).await?;
let attributes = [origin(), as_path(), next_hop(), local_pref.clone()];
external.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert!(!has_local_pref(&routes));
let server = TestServer::start(LOCAL_ASN).await?;
let mut internal = established(&server, LOCAL_ASN).await?;
// An internal peer has not added its AS to the path.
let attributes = [origin(), attr(WELL_KNOWN, 2, &[]), next_hop(), local_pref];
internal.send_raw(&update(&attributes, &ROUTE_A)).await?;
let routes = wait_for_routes(&server, &["10.1.0.0/16"]).await?;
assert!(has_local_pref(&routes));
Ok(())
}
/// When the prefixes themselves cannot be read there is nothing to
/// withdraw, and the session is reset as before.
#[tokio::test]
async fn test_an_unreadable_prefix_still_resets_the_session() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established(&server, PEER_ASN).await?;
// An IPv4 prefix of length 33.
peer.send_raw(&update(
&[origin(), as_path(), next_hop()],
&[33, 10, 1, 2, 3, 4],
))
.await?;
loop {
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
Message::KeepAlive => {}
Message::Notification(n) => {
assert_eq!(
n.error,
BgpError::UpdateMsg(UpdateMsgSubcode::InvalidNetworkField)
);
break;
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
}
tokio::time::timeout(TIMEOUT, async {
while !server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
}
crates/packet/Cargo.toml
+9 -3
View File
@@ -1,7 +1,7 @@
[package]
name = "bgp-packet"
description.workspace = true
description = "Parsing and serialisation of BGP messages"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
@@ -11,10 +11,16 @@ version.workspace = true
[dependencies]
bitfield.workspace = true
bytes.workspace = true
eyre.workspace = true
nom.workspace = true
serde.workspace = true
serde_repr.workspace = true
thiserror.workspace = true
strum.workspace = true
bgp-primitives.workspace = true
[dev-dependencies]
eyre.workspace = true
proptest.workspace = true
[lints]
workspace = true
crates/packet/src/community.rs
+98
View File
@@ -0,0 +1,98 @@
//! Parsing and formatting of community values as text, the way they are
//! written in configuration and shown in status output.
//!
//! Standard communities are written `ASN:VALUE` (e.g. `65000:100`) or as one
//! of the well-known names from RFC 1997 / RFC 7999 / RFC 8326. Large
//! communities are written `GLOBAL:LOCAL1:LOCAL2` (RFC 8092).
use thiserror::Error;
use crate::message::LargeCommunity;
/// Text that is not a community.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum CommunityError {
#[error("community {0:?} must be ASN:VALUE with 16-bit numbers, or a well-known name")]
Standard(String),
#[error("large community {0:?} must be GLOBAL:LOCAL1:LOCAL2 with 32-bit numbers")]
Large(String),
}
/// Well-known standard communities, by name.
pub const WELL_KNOWN: &[(&str, (u16, u16))] = &[
("GRACEFUL_SHUTDOWN", (0xFFFF, 0x0000)),
("BLACKHOLE", (0xFFFF, 0x029A)),
("NO_EXPORT", (0xFFFF, 0xFF01)),
("NO_ADVERTISE", (0xFFFF, 0xFF02)),
("NO_EXPORT_SUBCONFED", (0xFFFF, 0xFF03)),
("NOPEER", (0xFFFF, 0xFF04)),
];
pub fn parse_standard(s: &str) -> Result<(u16, u16), CommunityError> {
let s = s.trim();
let invalid = || CommunityError::Standard(s.to_string());
if let Some((_, value)) = WELL_KNOWN
.iter()
.find(|(name, _)| name.eq_ignore_ascii_case(s))
{
return Ok(*value);
}
let (high, low) = s.split_once(':').ok_or_else(invalid)?;
let parse = |part: &str| part.trim().parse::<u16>().map_err(|_| invalid());
Ok((parse(high)?, parse(low)?))
}
pub fn format_standard((high, low): (u16, u16)) -> String {
format!("{high}:{low}")
}
pub fn parse_large(s: &str) -> Result<LargeCommunity, CommunityError> {
let s = s.trim();
let invalid = || CommunityError::Large(s.to_string());
let parts: Vec<&str> = s.split(':').collect();
if parts.len() != 3 {
return Err(invalid());
}
let parse = |part: &str| part.trim().parse::<u32>().map_err(|_| invalid());
Ok(LargeCommunity {
global_admin: parse(parts[0])?,
data_1: parse(parts[1])?,
data_2: parse(parts[2])?,
})
}
pub fn format_large(c: &LargeCommunity) -> String {
format!("{}:{}:{}", c.global_admin, c.data_1, c.data_2)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn standard_roundtrip_and_names() {
assert_eq!(parse_standard("65000:100").unwrap(), (65000, 100));
assert_eq!(parse_standard(" no_export ").unwrap(), (65535, 65281));
assert_eq!(parse_standard("BLACKHOLE").unwrap(), (65535, 666));
assert_eq!(format_standard((65000, 100)), "65000:100");
assert!(parse_standard("65000").is_err());
assert!(parse_standard("70000:1").is_err());
assert!(parse_standard("bogus").is_err());
}
#[test]
fn large_roundtrip() {
let c = parse_large("65000:1:2").unwrap();
assert_eq!(
c,
LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2
}
);
assert_eq!(format_large(&c), "65000:1:2");
assert!(parse_large("65000:1").is_err());
assert!(parse_large("a:b:c").is_err());
}
}
crates/packet/src/errors.rs
+68 -95
View File
@@ -1,68 +1,79 @@
use serde::{Deserialize, Serialize};
use serde::Serialize;
use strum::FromRepr;
use thiserror::Error;
#[derive(Debug, Error)]
#[error("Unknown BGP error code {code} subcode {subcode}")]
pub struct UnknownErrorCode {
pub code: u8,
pub subcode: u8,
/// A field holds a number that is not one of the values defined for it.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)]
#[error("invalid {what} value {value}")]
pub struct InvalidValue {
pub what: &'static str,
pub value: u32,
}
#[derive(Debug, Clone, PartialEq, Eq, Error, Serialize, Deserialize)]
#[repr(u8)]
/// A message that could not be decoded, together with the NOTIFICATION that
/// reports the problem to the peer (RFC 4271 §6).
#[derive(Debug, Clone, PartialEq, Eq, Error)]
#[error("{detail}")]
pub struct DecodeError {
/// The error code and subcode to send.
pub notification: BgpError,
/// The Data field of that NOTIFICATION, where the RFC defines one.
pub data: Vec<u8>,
/// What was wrong, for the log.
pub detail: String,
}
/// The error a NOTIFICATION reports: its code and subcode (RFC 4271 §4.5).
#[derive(Debug, Clone, PartialEq, Eq, Error, Serialize)]
pub enum BgpError {
#[error("Message header error")]
MsgHeader(MsgHeaderSubcode) = 1,
#[error("Message header error: {0}")]
MsgHeader(MsgHeaderSubcode),
#[error("Open message error")]
OpenMsg(OpenMsgSubcode) = 2,
#[error("Open message error: {0}")]
OpenMsg(OpenMsgSubcode),
#[error("Update message error")]
UpdateMsg(UpdateMsgSubcode) = 3,
#[error("Update message error: {0}")]
UpdateMsg(UpdateMsgSubcode),
#[error("Hold timer expired")]
HoldTimer = 4,
HoldTimer,
#[error("Finite state machine error")]
FsmError(FsmSubcode) = 5,
#[error("Finite state machine error: {0}")]
FsmError(FsmSubcode),
#[error("Cease")]
Cease(CeaseSubcode) = 6,
#[error("Cease: {0}")]
Cease(CeaseSubcode),
#[error("Route refresh message error")]
RouteRefresh(RouteRefreshSubcode) = 7,
#[error("Route refresh message error: {0}")]
RouteRefresh(RouteRefreshSubcode),
#[error("Send hold timer expired")]
SendHoldTimer = 8,
SendHoldTimer,
/// A code or subcode this implementation has no name for. A peer may
/// send any: new ones are assigned, and zero ("unspecific") is always
/// allowed. The session ends the same way, and the numbers are kept
/// for the log.
#[error("error code {code} subcode {subcode}")]
Other { code: u8, subcode: u8 },
}
impl BgpError {
pub fn from_code_subcode(code: u8, subcode: u8) -> Result<Self, UnknownErrorCode> {
let err = || UnknownErrorCode { code, subcode };
match code {
1 => Ok(Self::MsgHeader(
MsgHeaderSubcode::try_from(subcode).map_err(|_| err())?,
)),
2 => Ok(Self::OpenMsg(
OpenMsgSubcode::try_from(subcode).map_err(|_| err())?,
)),
3 => Ok(Self::UpdateMsg(
UpdateMsgSubcode::try_from(subcode).map_err(|_| err())?,
)),
4 => Ok(Self::HoldTimer),
5 => Ok(Self::FsmError(
FsmSubcode::try_from(subcode).map_err(|_| err())?,
)),
6 => Ok(Self::Cease(
CeaseSubcode::try_from(subcode).map_err(|_| err())?,
)),
7 => Ok(Self::RouteRefresh(
RouteRefreshSubcode::try_from(subcode).map_err(|_| err())?,
)),
8 => Ok(Self::SendHoldTimer),
_ => Err(err()),
}
/// The error for a code and subcode off the wire. Every pair has one,
/// so that a peer's NOTIFICATION can always be read.
pub fn from_code_subcode(code: u8, subcode: u8) -> Self {
let known = match code {
1 => MsgHeaderSubcode::from_repr(subcode).map(Self::MsgHeader),
2 => OpenMsgSubcode::from_repr(subcode).map(Self::OpenMsg),
3 => UpdateMsgSubcode::from_repr(subcode).map(Self::UpdateMsg),
4 => (subcode == 0).then_some(Self::HoldTimer),
5 => FsmSubcode::from_repr(subcode).map(Self::FsmError),
6 => CeaseSubcode::from_repr(subcode).map(Self::Cease),
7 => RouteRefreshSubcode::from_repr(subcode).map(Self::RouteRefresh),
8 => (subcode == 0).then_some(Self::SendHoldTimer),
_ => None,
};
known.unwrap_or(Self::Other { code, subcode })
}
pub fn code_subcode(&self) -> (u8, u8) {
@@ -75,11 +86,12 @@ impl BgpError {
Self::Cease(s) => (6, *s as u8),
Self::RouteRefresh(s) => (7, *s as u8),
Self::SendHoldTimer => (8, 0),
Self::Other { code, subcode } => (*code, *subcode),
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum MsgHeaderSubcode {
#[error("Connection not synchronized")]
@@ -92,16 +104,12 @@ pub enum MsgHeaderSubcode {
BadMessageType = 3,
}
impl TryFrom<u8> for MsgHeaderSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum OpenMsgSubcode {
#[error("Unspecific")]
Unspecific = 0,
#[error("Unsupported BGP version number")]
UnsupportedVersion = 1,
@@ -124,14 +132,7 @@ pub enum OpenMsgSubcode {
RoleMismatch = 8,
}
impl TryFrom<u8> for OpenMsgSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum UpdateMsgSubcode {
#[error("Malformed attribute list")]
@@ -165,14 +166,7 @@ pub enum UpdateMsgSubcode {
MalformedAsPath = 11,
}
impl TryFrom<u8> for UpdateMsgSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum FsmSubcode {
#[error("Received an unexpected message in OpenSent state")]
@@ -185,14 +179,7 @@ pub enum FsmSubcode {
UnexpectedEstablished = 3,
}
impl TryFrom<u8> for FsmSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum CeaseSubcode {
#[error("Maximum number of prefixes reached")]
@@ -226,23 +213,9 @@ pub enum CeaseSubcode {
BfdDown = 10,
}
impl TryFrom<u8> for CeaseSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, Deserialize, FromRepr)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error, Serialize, FromRepr)]
#[repr(u8)]
pub enum RouteRefreshSubcode {
#[error("Invalid message length")]
InvalidLength = 1,
}
impl TryFrom<u8> for RouteRefreshSubcode {
type Error = ();
fn try_from(v: u8) -> Result<Self, ()> {
Self::from_repr(v).ok_or(())
}
}
crates/packet/src/lib.rs
+2
View File
@@ -1,3 +1,5 @@
pub mod community;
pub mod errors;
pub mod message;
pub mod open;
pub mod update_decode;
crates/packet/src/message.rs
+1321 -1209
View File
File diff suppressed because it is too large. Load diff
crates/packet/src/open.rs
+888 -583
View File
File diff suppressed because it is too large. Load diff
crates/packet/src/update_decode.rs
+1008
View File
File diff suppressed because it is too large. Load diff
crates/packet/tests/properties.rs
+674
View File
@@ -0,0 +1,674 @@
//! Property tests for the wire codec.
//!
//! Two things are checked for every kind of message:
//!
//! - **Round trip.** A message that can be written can be read back, comes
//! back equal.
//! - **Hostile input.** Whatever bytes arrive, decoding returns a message or
//! an error: it never panics. What it does accept can be shown, written
//! again, and read back as the same message.
//!
//! The second is what a peer can do to us, so it is run on three kinds of
//! input: plain noise, noise behind a valid header (so it reaches the body
//! parsers), and valid messages with a few bytes damaged (so it reaches deep
//! into them).
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::errors::BgpError;
use bgp_packet::message::{
AggregatorPathAttribute, AsPathAttribute, AsPathSegment, AtomicAggregatePathAttribute,
BGP_HEADER_LEN, BGP_MAX_MSG_LEN, CommunitiesPathAttribute, ExtendedCommunitiesPathAttribute,
ExtendedCommunity, LargeCommunitiesPathAttribute, LargeCommunity, LocalPrefPathAttribute,
Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute, MultiExitDiscPathAttribute,
NextHopPathAttribute, NlriNextHop, NotificationMessage, OriginPathAttribute, PathAttribute,
PathAttributeFlags, RouteRefreshMessage, UpdateMessage,
};
use bgp_packet::open::{
AddPathEntry, AddPathSendRecv, BgpRole, BgpSecHeader, Capability, ExtendedNextHop,
GracefulRestartEntry, LongLivedGracefulRestart, MultiLabelEntry, OpenMessage, OpenOption,
OrfFamily, OrfSendRecv, OrfType, OutboundRouteFilterEntry,
};
use bgp_primitives::constants::{Afi, AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::{EncodeError, ParserContext};
use bytes::BytesMut;
use proptest::collection::vec;
use proptest::prelude::*;
/// What a session settled that the codec depends on.
#[derive(Debug, Clone, Copy)]
struct Session {
four_octet_asn: bool,
ibgp: bool,
/// Whether prefixes carry path identifiers (RFC 7911), for IPv4 and
/// for IPv6 unicast.
add_path: [bool; 2],
}
impl Session {
fn ctx(self) -> ParserContext {
ParserContext {
four_octet_asn: self.four_octet_asn,
ibgp: self.ibgp,
add_path: [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST]
.into_iter()
.zip(self.add_path)
.filter_map(|(family, in_use)| in_use.then_some(family))
.collect(),
}
}
fn has_add_path(self, version: IpVersion, safi: Safi) -> bool {
safi == Safi::UNICAST
&& match version {
IpVersion::Ipv4 => self.add_path[0],
IpVersion::Ipv6 => self.add_path[1],
}
}
}
fn session() -> impl Strategy<Value = Session> {
any::<(bool, bool, [bool; 2])>().prop_map(|(four_octet_asn, ibgp, add_path)| Session {
four_octet_asn,
ibgp,
add_path,
})
}
// ---------------------------------------------------------------------------
// Generators
// ---------------------------------------------------------------------------
fn prefix(version: IpVersion) -> impl Strategy<Value = IpPrefix> {
let max_len = match version {
IpVersion::Ipv4 => 32u8,
IpVersion::Ipv6 => 128,
};
(0..=max_len, any::<[u8; 16]>()).prop_map(move |(length, bytes)| {
IpPrefix::new(version, bytes, length).expect("16 bytes are enough for any prefix")
})
}
/// Prefixes as an UPDATE names them. Without ADD-PATH for the family there
/// is no identifier on the wire, and it reads back as zero.
fn nlri(session: Session, version: IpVersion, safi: Safi) -> impl Strategy<Value = Vec<Nlri>> {
let with_path_id = session.has_add_path(version, safi);
vec(
(prefix(version), any::<u32>()).prop_map(move |(prefix, path_id)| Nlri {
prefix,
path_id: if with_path_id { path_id } else { 0 },
}),
0..5,
)
}
fn afi() -> impl Strategy<Value = Afi> {
any::<u16>().prop_map(Afi)
}
fn safi() -> impl Strategy<Value = Safi> {
any::<u8>().prop_map(Safi)
}
/// AS numbers that fit the session: any on a four-octet one, two-octet ones
/// otherwise (wider ones are replaced by AS_TRANS on the wire, by design).
fn asn(four_octet_asn: bool) -> BoxedStrategy<u32> {
if four_octet_asn {
any::<u32>().boxed()
} else {
any::<u16>().prop_map(u32::from).boxed()
}
}
fn as_path(asn: BoxedStrategy<u32>) -> impl Strategy<Value = AsPathAttribute> {
// A segment has at least one AS number.
vec((any::<bool>(), vec(asn, 1..6)), 0..4).prop_map(|segments| AsPathAttribute {
segments: segments
.into_iter()
.map(|(ordered, path)| AsPathSegment { ordered, path })
.collect(),
})
}
fn extended_community() -> impl Strategy<Value = ExtendedCommunity> {
any::<[u8; 8]>().prop_map(ExtendedCommunity)
}
fn mp_reach(session: Session) -> impl Strategy<Value = MpReachNlriPathAttribute> {
let ip_safi = || prop_oneof![Just(Safi::UNICAST), Just(Safi::MULTICAST)];
let v4 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
any::<[u8; 4]>(),
nlri(session, IpVersion::Ipv4, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV4, safi),
next_hop: NlriNextHop::Ipv4(Ipv4Addr::from(next_hop)),
prefixes,
});
let v6_next_hop = || {
prop_oneof![
any::<[u8; 16]>().prop_map(|a| NlriNextHop::Ipv6(Ipv6Addr::from(a))),
(any::<[u8; 16]>(), any::<[u8; 16]>()).prop_map(|(global, link_local)| {
NlriNextHop::Ipv6WithLl {
global: Ipv6Addr::from(global),
link_local: Ipv6Addr::from(link_local),
}
}),
]
};
let v6 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
v6_next_hop(),
nlri(session, IpVersion::Ipv6, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV6, safi),
next_hop,
prefixes,
});
prop_oneof![v4, v6]
}
fn mp_unreach(session: Session) -> impl Strategy<Value = MpUnreachNlriPathAttribute> {
let of = move |version: IpVersion| {
nlri(session, version, Safi::UNICAST).prop_map(move |prefixes| MpUnreachNlriPathAttribute {
family: AfiSafi::new(version.into(), Safi::UNICAST),
prefixes,
})
};
prop_oneof![of(IpVersion::Ipv4), of(IpVersion::Ipv6)]
}
/// Type codes whose definition the decoder knows; an unknown attribute has
/// none of these.
const DEFINED_ATTRIBUTES: [u8; 16] = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 32];
/// Unknown attributes, each with a type code of its own: a repeated
/// attribute is dropped by the decoder.
fn unknown_attributes() -> impl Strategy<Value = Vec<PathAttribute>> {
proptest::collection::btree_map(
any::<u8>().prop_filter("a type code without a definition", |code| {
!DEFINED_ATTRIBUTES.contains(code)
}),
(any::<u8>(), vec(any::<u8>(), 0..300)),
0..3,
)
.prop_map(|attributes| {
attributes
.into_iter()
.map(|(type_code, (flags, payload))| {
// Optional (bit 7): an unknown well-known attribute is an
// error. Bit 4 selects the two-octet length, which a payload
// this long needs.
let mut flags = flags | 0x80;
if payload.len() > 255 {
flags |= 0x10;
}
PathAttribute::UnknownPathAttribute {
flags: PathAttributeFlags(flags),
type_code,
payload,
}
})
.collect()
})
}
/// The attributes of a well-formed UPDATE on `session`: the mandatory ones,
/// and any of the others at most once.
fn path_attributes(session: Session) -> impl Strategy<Value = Vec<PathAttribute>> {
let mandatory = (
prop_oneof![
Just(OriginPathAttribute::IGP),
Just(OriginPathAttribute::EGP),
Just(OriginPathAttribute::INCOMPLETE)
],
as_path(asn(session.four_octet_asn)),
any::<[u8; 4]>(),
any::<u32>(),
)
.prop_map(move |(origin, as_path, next_hop, local_pref)| {
let mut attributes = vec![
PathAttribute::Origin(origin),
PathAttribute::AsPath(as_path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::from(next_hop))),
];
// Mandatory between internal peers, dropped from external ones.
if session.ibgp {
attributes.push(PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)));
}
attributes
});
let optional = (
proptest::option::of(
any::<u32>().prop_map(|v| PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(v))),
),
proptest::option::of(Just(PathAttribute::AtomicAggregate(
AtomicAggregatePathAttribute {},
))),
proptest::option::of((asn(session.four_octet_asn), any::<[u8; 4]>()).prop_map(
|(asn, ip)| {
PathAttribute::Aggregator(AggregatorPathAttribute {
asn,
ip: Ipv4Addr::from(ip),
})
},
)),
proptest::option::of(
vec(any::<(u16, u16)>(), 1..8)
.prop_map(|c| PathAttribute::Communities(CommunitiesPathAttribute(c))),
),
proptest::option::of(mp_reach(session).prop_map(PathAttribute::MpReachNlri)),
proptest::option::of(mp_unreach(session).prop_map(PathAttribute::MpUnreachNlri)),
proptest::option::of(
vec(extended_community(), 1..6).prop_map(|extended_communities| {
PathAttribute::ExtendedCommunities(ExtendedCommunitiesPathAttribute {
extended_communities,
})
}),
),
// AS4_PATH is four octets wide, and only a two-octet speaker sends
// it.
proptest::option::of(as_path(any::<u32>().boxed()).prop_map(PathAttribute::As4Path))
.prop_map(move |as4_path| as4_path.filter(|_| !session.four_octet_asn)),
proptest::option::of(vec(any::<(u32, u32, u32)>(), 1..6).prop_map(|communities| {
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: communities
.into_iter()
.map(|(global_admin, data_1, data_2)| LargeCommunity {
global_admin,
data_1,
data_2,
})
.collect(),
})
})),
unknown_attributes(),
);
(mandatory, optional).prop_map(|(mut attributes, optional)| {
let (med, atomic, aggregator, communities, reach, unreach, extended, as4, large, unknown) =
optional;
attributes.extend(
[
med,
atomic,
aggregator,
communities,
reach,
unreach,
extended,
as4,
large,
]
.into_iter()
.flatten(),
);
attributes.extend(unknown);
attributes
})
}
fn update(session: Session) -> impl Strategy<Value = UpdateMessage> {
(
nlri(session, IpVersion::Ipv4, Safi::UNICAST),
path_attributes(session),
nlri(session, IpVersion::Ipv4, Safi::UNICAST),
)
.prop_map(
|(withdrawn_routes, path_attributes, announced_routes)| UpdateMessage {
withdrawn_routes,
path_attributes,
announced_routes,
},
)
}
/// Capability codes that have their own variant.
const KNOWN_CAPABILITIES: [u8; 13] = [1, 2, 3, 5, 6, 7, 8, 9, 64, 65, 69, 70, 71];
fn capability() -> impl Strategy<Value = Capability> {
let orf = (
any::<u8>().prop_map(OrfType),
prop_oneof![
Just(OrfSendRecv::Receive),
Just(OrfSendRecv::Send),
Just(OrfSendRecv::Both)
],
)
.prop_map(|(r#type, send_recv)| OutboundRouteFilterEntry { r#type, send_recv });
let add_path = prop_oneof![
Just(AddPathSendRecv::Receive),
Just(AddPathSendRecv::Send),
Just(AddPathSendRecv::Both)
];
prop_oneof![
(afi(), safi()).prop_map(|(afi, safi)| Capability::MultiProtocol { afi, safi }),
Just(Capability::RouteRefresh),
vec((afi(), safi(), vec(orf, 0..4)), 0..3).prop_map(|families| {
Capability::OutboundRouteFilter(
families
.into_iter()
.map(|(afi, safi, orfs)| OrfFamily { afi, safi, orfs })
.collect(),
)
}),
vec((afi(), safi(), afi()), 1..4).prop_map(|entries| Capability::ExtendedNextHop {
entries: entries
.into_iter()
.map(|(afi, safi, nh_afi)| ExtendedNextHop { afi, safi, nh_afi })
.collect(),
}),
Just(Capability::ExtendedMessage),
(any::<u8>(), afi()).prop_map(|(header, afi)| Capability::BgpSec {
header: BgpSecHeader(header),
afi,
}),
vec((afi(), safi(), any::<u8>()), 1..4).prop_map(|entries| {
Capability::MultiLabelCompat {
entries: entries
.into_iter()
.map(|(afi, safi, count)| MultiLabelEntry { afi, safi, count })
.collect(),
}
}),
any::<u8>().prop_map(|role| Capability::BgpRole {
role: BgpRole::from(role),
}),
(
any::<bool>(),
0u16..4096,
vec((afi(), safi(), any::<bool>()), 0..4)
)
.prop_map(|(restart_state, restart_time, entries)| {
Capability::GracefulRestart {
restart_state,
restart_time,
entries: entries
.into_iter()
.map(|(afi, safi, preserve_forwarding)| GracefulRestartEntry {
afi,
safi,
preserve_forwarding,
})
.collect(),
}
}),
any::<u32>().prop_map(|asn| Capability::FourByteAsn { asn }),
vec((afi(), safi(), add_path), 1..4).prop_map(|entries| Capability::AddPath {
entries: entries
.into_iter()
.map(|(afi, safi, send_recv)| AddPathEntry {
afi,
safi,
send_recv,
})
.collect(),
}),
Just(Capability::EnhancedRouteRefresh),
vec((afi(), safi(), any::<u8>(), 0u32..(1 << 24)), 0..4).prop_map(|entries| {
Capability::LongLivedGracefulRestart(
entries
.into_iter()
.map(|(afi, safi, flags, stale_time)| LongLivedGracefulRestart {
afi,
safi,
flags,
stale_time,
})
.collect(),
)
}),
(
any::<u8>().prop_filter("a code without a variant", |code| {
!KNOWN_CAPABILITIES.contains(code)
}),
vec(any::<u8>(), 0..20)
)
.prop_map(|(type_code, value)| Capability::UnknownCapability { type_code, value }),
]
}
fn open() -> impl Strategy<Value = OpenMessage> {
(
any::<u16>(),
any::<u16>(),
any::<[u8; 4]>(),
// Enough to go past 255 octets of options now and then, which
// switches to the extended length of RFC 9072.
vec(vec(capability(), 0..10), 0..8),
)
.prop_map(|(asn, hold_time, identifier, options)| OpenMessage {
version: 4,
asn,
hold_time,
identifier: Ipv4Addr::from(identifier),
options: options.into_iter().map(OpenOption::Capabilities).collect(),
})
}
fn notification() -> impl Strategy<Value = NotificationMessage> {
// Any code and subcode: a peer may send ones we have no name for.
(any::<u8>(), any::<u8>(), vec(any::<u8>(), 0..40)).prop_map(|(code, subcode, data)| {
NotificationMessage {
error: BgpError::from_code_subcode(code, subcode),
data,
}
})
}
fn message(session: Session) -> impl Strategy<Value = Message> {
prop_oneof![
open().prop_map(Message::Open),
update(session).prop_map(Message::Update),
notification().prop_map(Message::Notification),
Just(Message::KeepAlive),
(afi(), any::<u8>(), safi()).prop_map(|(afi, subtype, safi)| {
Message::RouteRefresh(RouteRefreshMessage { afi, subtype, safi })
}),
]
}
/// A session, and a well-formed message that fits it.
fn session_and_message() -> impl Strategy<Value = (Session, Message)> {
session().prop_flat_map(|session| (Just(session), message(session)))
}
// ---------------------------------------------------------------------------
// Properties
// ---------------------------------------------------------------------------
/// Decodes `wire` and checks what must hold for anything the decoder
/// accepts: it can be shown and written, and unless the decoder had to
/// repair it, it reads back as itself.
fn check_decoding(ctx: &ParserContext, wire: &[u8]) -> Result<(), TestCaseError> {
let Ok(decoded) = Message::decode_with_report(ctx, wire) else {
return Ok(());
};
let (msg, repairs) = (decoded.message, decoded.update_errors);
// Display of what a peer sent must not be able to panic: it is logged.
let _ = format!("{msg:?} {repairs:?}");
if let Message::Update(update) = &msg {
for prefix in update
.withdrawn_routes
.iter()
.chain(&update.announced_routes)
{
let _ = prefix.to_string();
}
for attribute in &update.path_attributes {
match attribute {
PathAttribute::MpReachNlri(mp) => {
mp.prefixes.iter().for_each(|p| drop(p.to_string()))
}
PathAttribute::MpUnreachNlri(mp) => {
mp.prefixes.iter().for_each(|p| drop(p.to_string()))
}
_ => {}
}
}
}
let mut rewritten = BytesMut::new();
match msg.to_wire(ctx, &mut rewritten) {
Ok(()) => {}
// Withdrawing what a full-size UPDATE announced can need more room
// than one message has. The result is acted on, never sent.
Err(EncodeError::TooLong { .. }) if !repairs.is_empty() => return Ok(()),
Err(e) => {
return Err(TestCaseError::fail(format!(
"accepted but cannot be written: {e}"
)));
}
}
if repairs.is_empty() {
let again = Message::decode_with_report(ctx, &rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e}")))?;
prop_assert!(again.update_errors.is_empty(), "{:?}", again.update_errors);
prop_assert_eq!(again.message, msg);
}
Ok(())
}
/// Writes a generated message. `None` when it is too big for one BGP
/// message, or some part of it for its length field, which the generators
/// can produce and the encoder must refuse.
fn written(ctx: &ParserContext, msg: &Message) -> Option<BytesMut> {
let mut wire = BytesMut::new();
match msg.to_wire(ctx, &mut wire) {
Ok(()) => Some(wire),
Err(EncodeError::TooLong { .. }) => None,
}
}
fn framed(msg_type: u8, body: &[u8]) -> Vec<u8> {
let mut wire = vec![0xff; 16];
let length = u16::try_from(usize::from(BGP_HEADER_LEN) + body.len()).unwrap();
wire.extend_from_slice(&length.to_be_bytes());
wire.push(msg_type);
wire.extend_from_slice(body);
wire
}
/// Ways to damage a message.
#[derive(Debug, Clone)]
enum Damage {
Set { at: prop::sample::Index, to: u8 },
Remove { at: prop::sample::Index },
Insert { at: prop::sample::Index, byte: u8 },
Truncate { to: prop::sample::Index },
}
fn damage() -> impl Strategy<Value = Damage> {
prop_oneof![
(any::<prop::sample::Index>(), any::<u8>()).prop_map(|(at, to)| Damage::Set { at, to }),
any::<prop::sample::Index>().prop_map(|at| Damage::Remove { at }),
(any::<prop::sample::Index>(), any::<u8>())
.prop_map(|(at, byte)| Damage::Insert { at, byte }),
any::<prop::sample::Index>().prop_map(|to| Damage::Truncate { to }),
]
}
/// Applies `damage` to the body of `wire` and makes the header's length
/// right again, so that the damage is the body parsers' problem.
fn damaged(wire: &[u8], damage: &[Damage]) -> Vec<u8> {
let header = usize::from(BGP_HEADER_LEN);
let msg_type = wire[header - 1];
let mut body = wire[header..].to_vec();
for d in damage {
match d {
Damage::Set { at, to } if !body.is_empty() => {
let i = at.index(body.len());
body[i] = *to;
}
Damage::Remove { at } if !body.is_empty() => {
body.remove(at.index(body.len()));
}
Damage::Insert { at, byte } => body.insert(at.index(body.len() + 1), *byte),
Damage::Truncate { to } if !body.is_empty() => body.truncate(to.index(body.len())),
_ => {}
}
}
body.truncate(usize::from(BGP_MAX_MSG_LEN - BGP_HEADER_LEN));
framed(msg_type, &body)
}
/// 2000 cases a property, which takes about a second. For a longer hunt
/// set `PROPTEST_CASES`, e.g. `PROPTEST_CASES=1000000 cargo test --release
/// -p bgp-packet --test properties`.
fn config() -> ProptestConfig {
let cases = std::env::var("PROPTEST_CASES")
.ok()
.and_then(|cases| cases.parse().ok())
.unwrap_or(2000);
ProptestConfig {
cases,
// Failures are not saved to a file: one that is found becomes a
// named test next to the code it broke.
failure_persistence: None,
..ProptestConfig::default()
}
}
proptest! {
#![proptest_config(config())]
#[test]
fn a_message_that_can_be_written_reads_back_equal((session, msg) in session_and_message()) {
let ctx = session.ctx();
let Some(wire) = written(&ctx, &msg) else {
return Ok(());
};
let decoded = Message::decode_with_report(&ctx, &wire)
.map_err(|e| TestCaseError::fail(format!("{e}")))?;
// What we write is well formed: nothing for the decoder to repair.
prop_assert!(decoded.update_errors.is_empty(), "{:?}", decoded.update_errors);
prop_assert_eq!(&decoded.message, &msg);
}
#[test]
fn noise_is_never_more_than_an_error(session in session(), bytes in vec(any::<u8>(), 0..200)) {
check_decoding(&session.ctx(), &bytes)?;
}
#[test]
fn noise_behind_a_valid_header_is_never_more_than_an_error(
session in session(),
msg_type in 0u8..7,
body in vec(any::<u8>(), 0..300),
) {
check_decoding(&session.ctx(), &framed(msg_type, &body))?;
}
#[test]
fn a_damaged_message_is_never_more_than_an_error(
(session, msg) in session_and_message(),
damage in vec(damage(), 1..4),
) {
let ctx = session.ctx();
let Some(wire) = written(&ctx, &msg) else {
return Ok(());
};
check_decoding(&ctx, &damaged(&wire, &damage))?;
}
/// The parsers below the message level are public too.
#[test]
fn noise_is_never_more_than_an_error_for_the_parts(
session in session(),
bytes in vec(any::<u8>(), 0..120),
) {
let ctx = session.ctx();
let _ = OpenMessage::from_wire(&bytes);
let _ = Capability::from_wire(&bytes);
let _ = UpdateMessage::decode(&ctx, &bytes);
let _ = PathAttribute::from_wire(&ctx, &bytes);
}
}
crates/primitives/Cargo.toml
+7 -2
View File
@@ -1,7 +1,7 @@
[package]
name = "bgp-primitives"
description = "Primitive types for working with routing information"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
@@ -11,9 +11,14 @@ version.workspace = true
[dependencies]
bitfield.workspace = true
bytes.workspace = true
eyre.workspace = true
nom.workspace = true
serde.workspace = true
serde_repr.workspace = true
thiserror.workspace = true
strum.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/primitives/src/constants.rs
+150 -56
View File
@@ -1,80 +1,174 @@
use std::fmt::Display;
use eyre::bail;
use serde_repr::{Deserialize_repr, Serialize_repr};
use serde::Serialize;
use serde_repr::Serialize_repr;
use thiserror::Error;
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Serialize_repr, Deserialize_repr)]
/// An address family number this implementation has no name for.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)]
#[error("unknown address family identifier {0}")]
pub struct UnknownAfi(pub u16);
/// The IP version of a prefix. For the address family field of a capability
/// or an MP_REACH_NLRI, which can name families that are not IP, see [`Afi`].
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Serialize_repr)]
#[repr(u16)]
pub enum AddressFamilyId {
pub enum IpVersion {
Ipv4 = 1,
Ipv6 = 2,
}
impl TryFrom<u16> for AddressFamilyId {
type Error = eyre::Error;
fn try_from(value: u16) -> Result<Self, Self::Error> {
Ok(match value {
x if x == Self::Ipv4 as u16 => Self::Ipv4,
x if x == Self::Ipv6 as u16 => Self::Ipv6,
other => bail!("Unknown AddressFamily: {}", other),
})
}
}
impl Display for AddressFamilyId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
impl IpVersion {
/// How many bits an address of this version has.
pub fn bits(self) -> u8 {
match self {
AddressFamilyId::Ipv4 => write!(f, "Ipv4"),
AddressFamilyId::Ipv6 => write!(f, "Ipv6"),
Self::Ipv4 => 32,
Self::Ipv6 => 128,
}
}
}
/// Represents a Subsequent Address Family Identifier.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, Serialize_repr, Deserialize_repr)]
#[repr(u8)]
pub enum SubsequentAfi {
Unicast = 1,
Multicast = 2,
NlriWithMpls = 4,
MplsLabelledVpn = 128,
MulticastMplsVpn = 129,
impl TryFrom<u16> for IpVersion {
type Error = UnknownAfi;
fn try_from(value: u16) -> Result<Self, Self::Error> {
match value {
x if x == Self::Ipv4 as u16 => Ok(Self::Ipv4),
x if x == Self::Ipv6 as u16 => Ok(Self::Ipv6),
other => Err(UnknownAfi(other)),
}
}
}
impl TryFrom<u8> for SubsequentAfi {
type Error = eyre::Error;
impl Display for IpVersion {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
IpVersion::Ipv4 => write!(f, "Ipv4"),
IpVersion::Ipv6 => write!(f, "Ipv6"),
}
}
}
fn try_from(value: u8) -> Result<Self, Self::Error> {
Ok(match value {
x if x == Self::Unicast as u8 => Self::Unicast,
x if x == Self::Multicast as u8 => Self::Multicast,
x if x == Self::NlriWithMpls as u8 => Self::NlriWithMpls,
x if x == Self::MplsLabelledVpn as u8 => Self::MplsLabelledVpn,
x if x == Self::MulticastMplsVpn as u8 => Self::MulticastMplsVpn,
other => bail!("Unknown SubsequentAfi: {}", other),
/// An Address Family Identifier as it appears on the wire (RFC 4760).
///
/// Any number is a valid value: a peer may name a family this implementation
/// has never heard of, and that must be readable so it can be ignored. Which
/// families are *supported* is a separate question, answered where routes
/// are stored.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
#[serde(transparent)]
pub struct Afi(pub u16);
impl Afi {
pub const IPV4: Self = Self(1);
pub const IPV6: Self = Self(2);
pub const L2VPN: Self = Self(25);
pub const BGP_LS: Self = Self(16388);
}
impl From<IpVersion> for Afi {
fn from(version: IpVersion) -> Self {
Self(version as u16)
}
}
impl TryFrom<Afi> for IpVersion {
type Error = UnknownAfi;
fn try_from(afi: Afi) -> Result<Self, Self::Error> {
Self::try_from(afi.0)
}
}
/// A Subsequent Address Family Identifier as it appears on the wire. Like
/// [`Afi`], any number is a valid value.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
#[serde(transparent)]
pub struct Safi(pub u8);
impl Safi {
pub const UNICAST: Self = Self(1);
pub const MULTICAST: Self = Self(2);
pub const MPLS_LABEL: Self = Self(4);
pub const EVPN: Self = Self(70);
pub const BGP_LS: Self = Self(71);
pub const MPLS_VPN: Self = Self(128);
pub const MULTICAST_MPLS_VPN: Self = Self(129);
pub const FLOWSPEC: Self = Self(133);
}
/// An address family: the pair that says what kind of routes a capability
/// or an MP_REACH_NLRI is about.
#[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Serialize)]
pub struct AfiSafi {
pub afi: Afi,
pub safi: Safi,
}
impl AfiSafi {
pub const IPV4_UNICAST: Self = Self::new(Afi::IPV4, Safi::UNICAST);
pub const IPV6_UNICAST: Self = Self::new(Afi::IPV6, Safi::UNICAST);
pub const fn new(afi: Afi, safi: Safi) -> Self {
Self { afi, safi }
}
/// The unicast family of an IP version.
pub fn unicast(version: IpVersion) -> Self {
Self::new(version.into(), Safi::UNICAST)
}
/// The usual name of the family, if it has one here.
fn name(self) -> Option<&'static str> {
Some(match (self.afi, self.safi) {
(Afi::IPV4, Safi::UNICAST) => "IPv4 unicast",
(Afi::IPV6, Safi::UNICAST) => "IPv6 unicast",
(Afi::IPV4, Safi::MULTICAST) => "IPv4 multicast",
(Afi::IPV6, Safi::MULTICAST) => "IPv6 multicast",
(Afi::IPV4, Safi::MPLS_LABEL) => "IPv4 labeled unicast",
(Afi::IPV6, Safi::MPLS_LABEL) => "IPv6 labeled unicast",
(Afi::IPV4, Safi::MPLS_VPN) => "VPNv4 unicast",
(Afi::IPV6, Safi::MPLS_VPN) => "VPNv6 unicast",
(Afi::IPV4, Safi::FLOWSPEC) => "IPv4 flowspec",
(Afi::IPV6, Safi::FLOWSPEC) => "IPv6 flowspec",
(Afi::L2VPN, Safi::EVPN) => "L2VPN EVPN",
(Afi::BGP_LS, Safi::BGP_LS) => "BGP-LS",
_ => return None,
})
}
}
/// The implementation try_from for u16 just reuses the u8 implementation since
/// we don't have any cases where the SAFI is longer than u8::MAX yet.
impl TryFrom<u16> for SubsequentAfi {
type Error = eyre::Error;
fn try_from(value: u16) -> Result<Self, Self::Error> {
TryFrom::<u8>::try_from(value as u8)
}
}
impl Display for SubsequentAfi {
impl Display for AfiSafi {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
SubsequentAfi::Unicast => write!(f, "Unicast"),
SubsequentAfi::Multicast => write!(f, "Multicast"),
SubsequentAfi::NlriWithMpls => write!(f, "NlriWithMpls"),
SubsequentAfi::MplsLabelledVpn => write!(f, "MplsLabelledVpn"),
SubsequentAfi::MulticastMplsVpn => write!(f, "MulticastMplsVpn"),
match self.name() {
Some(name) => f.write_str(name),
None => write!(f, "AFI {} SAFI {}", self.afi.0, self.safi.0),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn any_family_is_a_value_and_the_common_ones_have_names() {
assert_eq!(AfiSafi::IPV6_UNICAST.to_string(), "IPv6 unicast");
assert_eq!(
AfiSafi::new(Afi::L2VPN, Safi::EVPN).to_string(),
"L2VPN EVPN"
);
assert_eq!(
AfiSafi::new(Afi(999), Safi(7)).to_string(),
"AFI 999 SAFI 7"
);
assert_eq!(AfiSafi::unicast(IpVersion::Ipv4), AfiSafi::IPV4_UNICAST);
}
#[test]
fn only_ipv4_and_ipv6_are_ip_versions() {
assert_eq!(IpVersion::try_from(Afi::IPV6), Ok(IpVersion::Ipv6));
assert_eq!(IpVersion::try_from(Afi::L2VPN), Err(UnknownAfi(25)));
assert_eq!(IpVersion::try_from(3u16), Err(UnknownAfi(3)));
}
}
crates/primitives/src/ip_prefix.rs
+274 -111
View File
@@ -2,130 +2,187 @@ use std::fmt::Display;
use std::net::{Ipv4Addr, Ipv6Addr};
use std::str::FromStr;
use bytes::{Buf, BufMut, BytesMut};
use eyre::{Result, bail};
use serde::{Deserialize, Serialize};
use serde::Serialize;
use thiserror::Error;
use crate::constants::AddressFamilyId;
use crate::parser::{ParserContext, ToWireError};
use crate::constants::IpVersion;
/// IpPrefix represents some IP address prefix, for a specific AddressFamilyId.
#[derive(Debug, PartialEq, Eq, Clone, Hash, Serialize, Deserialize)]
/// Why something is not a valid prefix.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum PrefixError {
#[error("prefix {0:?} is not in address/length form")]
Format(String),
#[error("prefix {0:?} has an address that is neither IPv4 nor IPv6")]
Address(String),
#[error("prefix length {length} is longer than the {max} bits of an address")]
LengthTooLong { length: u8, max: u8 },
#[error("prefix length {length} needs {needed} address bytes, got {got}")]
TooFewBytes {
length: u8,
needed: usize,
got: usize,
},
}
/// An IPv4 or IPv6 prefix.
///
/// It is a small value with no allocation behind it, cheap to copy, compare
/// and hash: a routing table holds millions of them and looks them up by
/// value. A prefix has one representation: the bits past its length are
/// always zero, so two prefixes are equal exactly when they are the same
/// prefix.
#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash, Serialize)]
pub struct IpPrefix {
pub address_family: AddressFamilyId,
pub prefix: Vec<u8>,
pub length: u8,
address_family: IpVersion,
length: u8,
/// The address, left-aligned, zero past `length` bits.
address: [u8; 16],
}
impl IpPrefix {
pub fn new(address_family: AddressFamilyId, prefix: Vec<u8>, length: u8) -> Result<Self> {
// Ensure that the prefix we are given contains the right number of bytes corresponding to the prefix length.
if prefix.len() < ((length + 7) / 8).into() {
bail!(
"Mismatched prefix {:?} for given prefix length: {}",
prefix,
length
);
/// Makes a prefix from the leading bytes of its address. `prefix` must
/// hold at least the bytes that `length` bits cover; anything past
/// those bits is ignored (RFC 4271 §4.3: such bits are irrelevant).
pub fn new(
address_family: IpVersion,
prefix: impl AsRef<[u8]>,
length: u8,
) -> Result<Self, PrefixError> {
let prefix = prefix.as_ref();
let max = address_family.bits();
if length > max {
return Err(PrefixError::LengthTooLong { length, max });
}
let needed = usize::from(length.div_ceil(8));
if prefix.len() < needed {
return Err(PrefixError::TooFewBytes {
length,
needed,
got: prefix.len(),
});
}
let mut address = [0u8; 16];
address[..needed].copy_from_slice(&prefix[..needed]);
if let (Some(last), rem @ 1..) = (address[..needed].last_mut(), length % 8) {
*last &= u8::MAX << (8 - rem);
}
Ok(Self {
address_family,
prefix,
length,
address,
})
}
pub fn to_wire(&self, _: &ParserContext, out: &mut BytesMut) -> Result<(), ToWireError> {
// Verify that there is enough space to write the IpPrefix.
if out.remaining() < (self.prefix.len() + 1) {
Err(ToWireError::OutBufferOverflow)?;
pub fn address_family(&self) -> IpVersion {
self.address_family
}
/// The prefix length in bits.
pub fn length(&self) -> u8 {
self.length
}
/// The bytes the prefix length covers, as they go on the wire: none for
/// a default route, three for a /24.
pub fn bytes(&self) -> &[u8] {
&self.address[..usize::from(self.length.div_ceil(8))]
}
/// Returns true if `other` is equal to or more specific than `self`, i.e.
/// `self` covers `other`. Prefixes of different address families never
/// contain each other.
pub fn contains(&self, other: &IpPrefix) -> bool {
if self.address_family != other.address_family || other.length < self.length {
return false;
}
let full_bytes = usize::from(self.length / 8);
if self.address[..full_bytes] != other.address[..full_bytes] {
return false;
}
let rem_bits = self.length % 8;
if rem_bits == 0 {
return true;
}
let mask = u8::MAX << (8 - rem_bits);
(self.address[full_bytes] & mask) == (other.address[full_bytes] & mask)
}
}
// Write length and prefix.
out.put_u8(self.length);
out.put(self.prefix.as_slice());
/// A prefix as an UPDATE names it: the prefix and, on a session where
/// ADD-PATH is in use for the address family (RFC 7911), the identifier
/// that tells apart the several routes a peer may announce for it.
///
/// Whether the identifier is on the wire is a matter of what the session
/// negotiated, not of its value. Where it is not, it is zero here.
#[derive(Debug, PartialEq, Eq, Clone, Copy, Hash, Serialize)]
pub struct Nlri {
pub prefix: IpPrefix,
pub path_id: u32,
}
Ok(())
impl Nlri {
/// `prefixes` as they are named on a session without ADD-PATH.
pub fn plain(prefixes: impl IntoIterator<Item = IpPrefix>) -> Vec<Self> {
prefixes.into_iter().map(Self::from).collect()
}
/// The bytes this takes on the wire, with or without its identifier.
pub fn wire_len(&self, with_path_id: bool) -> usize {
1 + self.prefix.bytes().len() + if with_path_id { 4 } else { 0 }
}
}
impl From<IpPrefix> for Nlri {
fn from(prefix: IpPrefix) -> Self {
Self { prefix, path_id: 0 }
}
}
/// So that a list of these compares with a list of plain prefixes.
impl PartialEq<IpPrefix> for Nlri {
fn eq(&self, prefix: &IpPrefix) -> bool {
self.path_id == 0 && self.prefix == *prefix
}
}
impl Display for Nlri {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self.path_id {
0 => write!(f, "{}", self.prefix),
id => write!(f, "{} (path {id})", self.prefix),
}
}
}
impl TryFrom<&str> for IpPrefix {
type Error = eyre::Error;
type Error = PrefixError;
fn try_from(value: &str) -> Result<Self, Self::Error> {
let parts: Vec<&str> = value.split("/").collect();
let format = || PrefixError::Format(value.to_string());
let address = || PrefixError::Address(value.to_string());
let (addr, length) = value.split_once('/').ok_or_else(format)?;
let length: u8 = length.parse().map_err(|_| format())?;
if parts.len() != 2 {
bail!(
"Expected IpPrefix in format prefix/length but got: {}",
value
);
}
let length: u8 = u8::from_str_radix(parts[1], 10).map_err(eyre::Error::from)?;
let mut octets;
let afi: AddressFamilyId;
if parts[0].contains(":") {
afi = AddressFamilyId::Ipv6;
let addr: Ipv6Addr = Ipv6Addr::from_str(parts[0]).map_err(eyre::Error::from)?;
octets = addr.octets().to_vec();
} else if parts[0].contains(".") {
afi = AddressFamilyId::Ipv4;
let addr: Ipv4Addr = Ipv4Addr::from_str(parts[0]).map_err(eyre::Error::from)?;
octets = addr.octets().to_vec();
let (afi, octets) = if addr.contains(':') {
let addr = Ipv6Addr::from_str(addr).map_err(|_| address())?;
(IpVersion::Ipv6, addr.octets().to_vec())
} else {
bail!("Could not figure out address type")
}
// Truncate the octets we have to the right number of bytes to match the prefix length..
if length % 8 == 0 {
// We can cleanly truncate the number of bytes since we are at a byte boundary.
octets.truncate((length / 8).into());
} else {
// We need to keep length % 8 bits of the last byte.
let num_bytes = (length / 8) + 1;
let mask = u8::MAX << (8 - (length % 8));
octets.truncate(num_bytes.into());
// Fix up the last byte.
let last_pos = octets.len() - 1;
octets[last_pos] &= mask;
}
Ok(IpPrefix {
address_family: afi,
prefix: octets,
length,
})
let addr = Ipv4Addr::from_str(addr).map_err(|_| address())?;
(IpVersion::Ipv4, addr.octets().to_vec())
};
IpPrefix::new(afi, octets, length)
}
}
impl Display for IpPrefix {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self.address_family {
AddressFamilyId::Ipv4 => {
// Pad the prefix with 0 bytes if it's less than 4 bytes long.
let bytes = &mut self.prefix.clone();
if bytes.len() < 4 {
bytes.extend(std::iter::repeat(0).take(4 - bytes.len()));
}
let four_bytes: [u8; 4] = bytes
.as_slice()
.try_into()
.map_err(|_| std::fmt::Error {})?;
let ipv4_addr = Ipv4Addr::from(four_bytes);
write!(f, "{}/{}", ipv4_addr, self.length)
IpVersion::Ipv4 => {
let [a, b, c, d, ..] = self.address;
write!(f, "{}/{}", Ipv4Addr::new(a, b, c, d), self.length)
}
AddressFamilyId::Ipv6 => {
let bytes = &mut self.prefix.clone();
if bytes.len() < 16 {
bytes.extend(std::iter::repeat(0).take(16 - bytes.len()));
}
let sixteen_bytes: [u8; 16] = bytes
.as_slice()
.try_into()
.map_err(|_| std::fmt::Error {})?;
let ipv6_addr = Ipv6Addr::from(sixteen_bytes);
write!(f, "{}/{}", ipv6_addr, self.length)
IpVersion::Ipv6 => {
write!(f, "{}/{}", Ipv6Addr::from(self.address), self.length)
}
}
}
@@ -133,9 +190,11 @@ impl Display for IpPrefix {
#[cfg(test)]
mod tests {
use crate::constants::AddressFamilyId;
use crate::constants::IpVersion;
use super::IpPrefix;
use proptest::prelude::*;
use super::{IpPrefix, PrefixError};
macro_rules! verify_roundtrip {
($name:ident, $prefix_str:expr, $afi:expr, $bytes:expr, $length:expr) => {
@@ -143,9 +202,9 @@ mod tests {
fn $name() {
let ip_prefix = IpPrefix::try_from($prefix_str).unwrap();
assert_eq!(ip_prefix.address_family, $afi);
assert_eq!(ip_prefix.prefix, $bytes);
assert_eq!(ip_prefix.length, $length);
assert_eq!(ip_prefix.address_family(), $afi);
assert_eq!(ip_prefix.bytes(), $bytes);
assert_eq!(ip_prefix.length(), $length);
let to_str: &str = &ip_prefix.to_string();
assert_eq!(IpPrefix::try_from(to_str).unwrap(), ip_prefix);
@@ -156,7 +215,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_24,
"10.1.2.0/24",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![10, 1, 2],
24
);
@@ -165,7 +224,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_19,
"10.245.123.0/19",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![10, 245, 96],
19
);
@@ -174,7 +233,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_3,
"192.168.1.0/3",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![192],
3
);
@@ -183,7 +242,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv4_0,
"0.0.0.0/0",
AddressFamilyId::Ipv4,
IpVersion::Ipv4,
vec![],
0
);
@@ -191,7 +250,7 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv6_48,
"2001:db8:cafe::/48",
AddressFamilyId::Ipv6,
IpVersion::Ipv6,
vec![32, 1, 13, 184, 202, 254],
48
);
@@ -200,16 +259,120 @@ mod tests {
verify_roundtrip!(
verify_roundtrip_ipv6_32,
"2001:db8:cafe::/32",
AddressFamilyId::Ipv6,
IpVersion::Ipv6,
vec![32, 1, 13, 184],
32
);
verify_roundtrip!(
verify_roundtrip_ipv6_0,
"::/0",
AddressFamilyId::Ipv6,
vec![],
0
);
#[test]
fn text_that_is_not_a_prefix_is_rejected_with_the_reason() {
for (text, expected) in [
("10.0.0.0", PrefixError::Format("10.0.0.0".into())),
("10.0.0.0/x", PrefixError::Format("10.0.0.0/x".into())),
("10.0.0/8", PrefixError::Address("10.0.0/8".into())),
("nonsense/8", PrefixError::Address("nonsense/8".into())),
(
"10.0.0.0/33",
PrefixError::LengthTooLong {
length: 33,
max: 32,
},
),
(
"2001:db8::/129",
PrefixError::LengthTooLong {
length: 129,
max: 128,
},
),
] {
assert_eq!(IpPrefix::try_from(text), Err(expected), "{text}");
}
assert!(IpPrefix::try_from("0.0.0.0/0").is_ok());
assert!(IpPrefix::try_from("2001:db8::1/128").is_ok());
assert_eq!(
IpPrefix::new(IpVersion::Ipv4, vec![10], 16),
Err(PrefixError::TooFewBytes {
length: 16,
needed: 2,
got: 1
})
);
}
#[test]
fn a_prefix_has_one_representation() {
// Extra bytes and stray bits past the length are dropped.
let noisy = IpPrefix::new(IpVersion::Ipv4, vec![10, 0xff, 3, 4], 12).unwrap();
assert_eq!(noisy, IpPrefix::try_from("10.240.0.0/12").unwrap());
assert_eq!(noisy.bytes(), [10, 0xf0]);
assert_eq!(
IpPrefix::new(IpVersion::Ipv4, vec![0xff; 13], 99),
Err(PrefixError::LengthTooLong {
length: 99,
max: 32
})
);
}
#[test]
fn a_prefix_is_a_small_value() {
assert!(size_of::<IpPrefix>() <= 20);
let prefix = IpPrefix::try_from("10.1.2.0/24").unwrap();
let copy = prefix;
assert_eq!(copy, prefix);
assert_eq!(prefix.bytes(), [10, 1, 2]);
assert_eq!(IpPrefix::try_from("0.0.0.0/0").unwrap().bytes(), [0u8; 0]);
}
proptest! {
#[test]
fn text_round_trips(version: bool, bytes: [u8; 16], length in 0u8..=128) {
let version = if version { IpVersion::Ipv6 } else { IpVersion::Ipv4 };
let length = length.min(version.bits());
let prefix = IpPrefix::new(version, bytes, length).unwrap();
let text = prefix.to_string();
prop_assert_eq!(IpPrefix::try_from(text.as_str()), Ok(prefix));
prop_assert!(prefix.contains(&prefix));
}
#[test]
fn text_that_is_noise_is_an_error_or_a_prefix_that_round_trips(text in "\\PC{0,40}") {
if let Ok(prefix) = IpPrefix::try_from(text.as_str()) {
let shown = prefix.to_string();
prop_assert_eq!(IpPrefix::try_from(shown.as_str()), Ok(prefix));
}
}
/// `contains` agrees with comparing the leading bits by hand.
#[test]
fn contains_is_a_comparison_of_leading_bits(
a: [u8; 4], a_len in 0u8..=32, b: [u8; 4], b_len in 0u8..=32,
) {
let outer = IpPrefix::new(IpVersion::Ipv4, a, a_len).unwrap();
let inner = IpPrefix::new(IpVersion::Ipv4, b, b_len).unwrap();
let mask = if a_len == 0 { 0 } else { u32::MAX << (32 - a_len) };
let expected = b_len >= a_len
&& (u32::from_be_bytes(a) & mask) == (u32::from_be_bytes(b) & mask);
prop_assert_eq!(outer.contains(&inner), expected);
}
}
#[test]
fn contains_covers_more_specifics() {
let p = |s: &str| IpPrefix::try_from(s).unwrap();
assert!(p("10.100.0.0/16").contains(&p("10.100.7.0/24")));
assert!(p("10.100.0.0/16").contains(&p("10.100.0.0/16")));
assert!(!p("10.100.0.0/16").contains(&p("10.101.0.0/24")));
assert!(!p("10.100.7.0/24").contains(&p("10.100.0.0/16")));
assert!(p("0.0.0.0/0").contains(&p("192.0.2.0/24")));
// Non-byte-aligned boundary.
assert!(p("10.100.0.0/19").contains(&p("10.100.31.0/24")));
assert!(!p("10.100.0.0/19").contains(&p("10.100.32.0/24")));
// Cross-family never matches.
assert!(!p("::/0").contains(&p("10.0.0.0/8")));
assert!(p("2001:db8::/32").contains(&p("2001:db8:cafe::/48")));
}
verify_roundtrip!(verify_roundtrip_ipv6_0, "::/0", IpVersion::Ipv6, vec![], 0);
}
crates/primitives/src/parser.rs
+33 -25
View File
@@ -1,35 +1,43 @@
use std::fmt::Display;
use nom::error::{ErrorKind, ParseError};
use crate::constants::AddressFamilyId;
use crate::constants::AfiSafi;
use thiserror::Error;
#[derive(Debug, Default)]
/// What reading or writing a message has to know about the session it
/// belongs to. The default is a session before its OPENs are exchanged;
/// only an UPDATE depends on anything here, and there are none until then.
#[derive(Debug, Clone, Default)]
pub struct ParserContext {
/// Whether thi parser is being run with a peer that is RFC6793 compliant.
pub four_octet_asn: Option<bool>,
/// Which address family should be parsed by default with this parser.
pub address_family: Option<AddressFamilyId>,
/// Whether AS numbers are four octets wide on this session (RFC 6793).
pub four_octet_asn: bool,
/// Whether the peer is in our own AS. Some attributes mean something
/// only between internal peers and are dropped when an external one
/// sends them (RFC 7606 §7).
pub ibgp: bool,
/// The address families whose prefixes carry a path identifier in the
/// direction this context is for (RFC 7911). What a session sends and
/// what it receives are negotiated separately, so a session has one
/// context for reading and one for writing.
pub add_path: Vec<AfiSafi>,
}
#[derive(Debug)]
pub enum ToWireError {
/// There was not enough space in the output buffer to serialize the data into.
OutBufferOverflow,
/// Another error.
Other(eyre::Error),
}
impl Display for ToWireError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ToWireError::OutBufferOverflow => write!(f, "OutBufferOverflow"),
ToWireError::Other(report) => report.fmt(f),
}
impl ParserContext {
pub fn has_add_path(&self, family: AfiSafi) -> bool {
self.add_path.contains(&family)
}
}
impl std::error::Error for ToWireError {}
/// Why a value could not be written to the wire.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum EncodeError {
/// A length does not fit the field that has to carry it.
#[error("{what} is too long to encode: {len} exceeds {max}")]
TooLong {
what: &'static str,
len: usize,
max: usize,
},
}
// Custom error type for the parser.
#[derive(Debug)]
@@ -37,8 +45,8 @@ pub enum BgpParserError<I> {
/// ProtocolError encapsulates protocol level semantics problems that require
/// action from the caller of the parser.
ProtocolError(ProtocolErrorValues),
CustomText(&'static str),
Eyre(eyre::ErrReport),
/// The input does not follow the format; the text says how.
Malformed(String),
Nom(I, ErrorKind),
}
crates/primitives/src/poptrie/mod.rs
+14 -5
View File
@@ -1,3 +1,10 @@
//! A Poptrie (Asai and Ohara, SIGCOMM 2015): a compressed multiway trie for
//! longest-prefix matching.
//!
//! Work in progress. Nothing in the daemon uses it yet, and the construction
//! side has no public entry point, hence the lint expectation below.
#![expect(dead_code, reason = "not yet wired into a RIB")]
use std::collections::BTreeMap;
#[derive(Debug)]
@@ -75,7 +82,8 @@ impl<T: Clone> Poptrie<T> {
result
}
fn new(&mut self, rt: BTreeMap<(u32, u8), T>) {
/// Fills the trie from a routing table keyed by (address, prefix length).
fn build(&mut self, rt: BTreeMap<(u32, u8), T>) {
const DEPTH: u8 = 6;
let mut forest = vec![(0, rt)];
@@ -168,9 +176,10 @@ mod tests {
leaves: vec![],
};
poptrie.new(map);
println!("Success");
dbg!(poptrie);
poptrie.build(map);
assert!(
!poptrie.inners.is_empty(),
"building from a non-empty table yields nodes"
);
}
}
crates/rib/Cargo.toml
+16
View File
@@ -1,9 +1,25 @@
[package]
name = "bgp-rib"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
bgp-packet.workspace = true
bgp-primitives.workspace = true
bytes.workspace = true
foldhash.workspace = true
hashbrown.workspace = true
serde.workspace = true
thiserror.workspace = true
tokio = { workspace = true, features = ["full"] }
tracing.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/rib/proptest-regressions/table/journal_tests.txt
+7
View File
@@ -0,0 +1,7 @@
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 7107cf5580fe7bb95e138f1fead386d89273532f83aa1d11808aa9586aa9da63 # shrinks to ops = [Withdraw { peer: 0, prefix: 1 }, Withdraw { peer: 0, prefix: 1 }, Poll { reader: 2, budget: 2 }, Withdraw { peer: 0, prefix: 10 }, Poll { reader: 2, budget: 4 }, Announce { peer: 2, prefix: 7, route: Route { local_pref: None, path: [1, 2], origin: 0, med: Some(0) } }, Announce { peer: 3, prefix: 6, route: Route { local_pref: None, path: [1], origin: 0, med: None } }, Announce { peer: 3, prefix: 6, route: Route { local_pref: None, path: [1, 1], origin: 0, med: None } }, Withdraw { peer: 1, prefix: 5 }, Withdraw { peer: 0, prefix: 11 }, Withdraw { peer: 3, prefix: 6 }, Announce { peer: 0, prefix: 3, route: Route { local_pref: None, path: [1, 2], origin: 1, med: None } }, Poll { reader: 2, budget: 3 }, Withdraw { peer: 0, prefix: 8 }, Flap { peer: 3 }, Withdraw { peer: 1, prefix: 1 }, Announce { peer: 2, prefix: 5, route: Route { local_pref: None, path: [1], origin: 1, med: Some(0) } }, Announce { peer: 0, prefix: 1, route: Route { local_pref: None, path: [1, 1], origin: 1, med: None } }, Withdraw { peer: 0, prefix: 1 }, Withdraw { peer: 2, prefix: 8 }, Poll { reader: 1, budget: 2 }, Flap { peer: 1 }, Poll { reader: 2, budget: 2 }, Announce { peer: 0, prefix: 9, route: Route { local_pref: None, path: [1], origin: 0, med: Some(1) } }, Announce { peer: 2, prefix: 2, route: Route { local_pref: Some(100), path: [2], origin: 0, med: Some(1) } }, Poll { reader: 1, budget: 3 }, Withdraw { peer: 3, prefix: 1 }, Announce { peer: 1, prefix: 4, route: Route { local_pref: None, path: [2], origin: 1, med: None } }, Announce { peer: 3, prefix: 7, route: Route { local_pref: None, path: [2], origin: 0, med: None } }, Announce { peer: 1, prefix: 11, route: Route { local_pref: None, path: [1], origin: 1, med: Some(1) } }, Withdraw { peer: 0, prefix: 6 }, Withdraw { peer: 0, prefix: 4 }, Announce { peer: 3, prefix: 10, route: Route { local_pref: Some(100), path: [2, 2], origin: 1, med: None } }, Announce { peer: 1, prefix: 2, route: Route { local_pref: None, path: [2], origin: 0, med: Some(2) } }, Announce { peer: 1, prefix: 3, route: Route { local_pref: None, path: [1], origin: 1, med: None } }, Withdraw { peer: 0, prefix: 2 }, Announce { peer: 1, prefix: 4, route: Route { local_pref: None, path: [1, 1], origin: 0, med: None } }, Poll { reader: 1, budget: 2 }, Poll { reader: 1, budget: 4 }, Poll { reader: 2, budget: 1 }, Poll { reader: 1, budget: 3 }, Poll { reader: 1, budget: 5 }, Poll { reader: 1, budget: 1 }, Withdraw { peer: 3, prefix: 5 }, Withdraw { peer: 3, prefix: 9 }, Withdraw { peer: 2, prefix: 10 }], shards = 3
crates/rib/proptest-regressions/table/tests.txt
+7
View File
@@ -0,0 +1,7 @@
# Seeds for failure cases proptest has generated in the past. It is
# automatically read and these particular cases re-run before any
# novel cases are generated.
#
# It is recommended to check this file in to source control so that
# everyone who runs the test benefits from these saved cases.
cc 6bcfc317c976928c2fbb29e943f385127b91770a76f18e7061b80b31f4276955 # shrinks to ops = [Announce { peer: 1, prefix: 1, route: Route { local_pref: None, path: [2], origin: 0, med: None } }, Announce { peer: 2, prefix: 1, route: Route { local_pref: None, path: [1], origin: 0, med: Some(2) } }, Announce { peer: 3, prefix: 1, route: Route { local_pref: None, path: [1], origin: 0, med: None } }, Announce { peer: 3, prefix: 1, route: Route { local_pref: None, path: [1], origin: 1, med: None } }]
crates/rib/src/adj_rib_in.rs
-96
View File
@@ -1,96 +0,0 @@
use std::collections::HashMap;
use bgp_primitives::ip_prefix::IpPrefix;
use tokio::sync::{broadcast, mpsc, oneshot};
use tracing::warn;
use crate::entry::{PeerId, RibEntry, SnapshotEntry};
use crate::event::RibEvent;
const EVENT_CHANNEL_CAPACITY: usize = 1024;
const CMD_CHANNEL_CAPACITY: usize = 256;
enum RibCommand {
Announce { peer: PeerId, prefix: IpPrefix, entry: RibEntry },
Withdraw { peer: PeerId, prefix: IpPrefix },
Snapshot { reply: oneshot::Sender<Vec<SnapshotEntry>> },
}
/// Handle to the Adj-RIB-In actor. Cheap to clone — all clones share the same
/// backing task. The actor is stopped when all handles are dropped.
#[derive(Clone)]
pub struct AdjRibIn {
tx: mpsc::Sender<RibCommand>,
events: broadcast::Sender<RibEvent>,
}
impl AdjRibIn {
/// Spawns the backing actor and returns a handle.
pub fn spawn() -> Self {
let (tx, rx) = mpsc::channel(CMD_CHANNEL_CAPACITY);
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let events_clone = events.clone();
tokio::spawn(run_actor(rx, events_clone));
Self { tx, events }
}
pub async fn announce(&self, peer: PeerId, prefix: IpPrefix, entry: RibEntry) {
let _ = self.tx.send(RibCommand::Announce { peer, prefix, entry }).await;
}
pub async fn withdraw(&self, peer: PeerId, prefix: IpPrefix) {
let _ = self.tx.send(RibCommand::Withdraw { peer, prefix }).await;
}
/// Returns a point-in-time snapshot of all routes in the RIB.
pub async fn snapshot(&self) -> Vec<SnapshotEntry> {
let (reply_tx, reply_rx) = oneshot::channel();
let _ = self.tx.send(RibCommand::Snapshot { reply: reply_tx }).await;
reply_rx.await.unwrap_or_default()
}
/// Subscribe to a stream of RIB mutation events.
///
/// The receiver will see every [`RibEvent`] from the point of subscription.
/// If the consumer falls behind by more than [`EVENT_CHANNEL_CAPACITY`]
/// events it will receive a [`broadcast::error::RecvError::Lagged`] error.
pub fn subscribe(&self) -> broadcast::Receiver<RibEvent> {
self.events.subscribe()
}
}
async fn run_actor(mut rx: mpsc::Receiver<RibCommand>, events: broadcast::Sender<RibEvent>) {
let mut table: HashMap<(PeerId, IpPrefix), RibEntry> = HashMap::new();
while let Some(cmd) = rx.recv().await {
match cmd {
RibCommand::Announce { peer, prefix, entry } => {
table.insert((peer, prefix.clone()), entry.clone());
if let Err(e) = events.send(RibEvent::Announced { peer, prefix, entry }) {
// No active subscribers — not an error.
let _ = e;
}
}
RibCommand::Withdraw { peer, prefix } => {
if table.remove(&(peer, prefix.clone())).is_some() {
if let Err(e) = events.send(RibEvent::Withdrawn { peer, prefix }) {
let _ = e;
}
}
}
RibCommand::Snapshot { reply } => {
let snap = table
.iter()
.map(|((peer, prefix), entry)| SnapshotEntry {
peer: *peer,
prefix: prefix.clone(),
entry: entry.clone(),
})
.collect();
if reply.send(snap).is_err() {
warn!("Snapshot reply channel closed before send");
}
}
}
}
}
crates/rib/src/entry.rs
-39
View File
@@ -1,39 +0,0 @@
use std::net::SocketAddr;
use std::time::SystemTime;
use bgp_packet::message::PathAttribute;
use bgp_primitives::ip_prefix::IpPrefix;
/// Identifies a BGP peer by its TCP address and negotiated ASN.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PeerId {
pub addr: SocketAddr,
pub asn: u32,
}
impl std::fmt::Display for PeerId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}(AS{})", self.addr, self.asn)
}
}
/// A single route stored in the Adj-RIB-In.
#[derive(Debug, Clone)]
pub struct RibEntry {
pub path_attributes: Vec<PathAttribute>,
pub received_at: SystemTime,
}
impl RibEntry {
pub fn new(path_attributes: Vec<PathAttribute>) -> Self {
Self { path_attributes, received_at: SystemTime::now() }
}
}
/// A flattened row returned by [`crate::AdjRibIn::snapshot`].
#[derive(Debug, Clone)]
pub struct SnapshotEntry {
pub peer: PeerId,
pub prefix: IpPrefix,
pub entry: RibEntry,
}
crates/rib/src/error.rs
+14
View File
@@ -0,0 +1,14 @@
/// A RIB's actor is no longer running, so the table can be neither read nor
/// changed.
///
/// The actor runs for as long as a handle to it exists; it stops early only
/// by panicking. That is a bug, and one the server cannot work around: it
/// could no longer say what it originates. Callers pass this up
/// until the process exits and is restarted, rather than carry on with a
/// table that looks empty.
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("the {rib} has stopped")]
pub struct RibStopped {
/// Which table.
pub rib: &'static str,
}
crates/rib/src/event.rs
-9
View File
@@ -1,9 +0,0 @@
use bgp_primitives::ip_prefix::IpPrefix;
use crate::entry::{PeerId, RibEntry};
#[derive(Debug, Clone)]
pub enum RibEvent {
Announced { peer: PeerId, prefix: IpPrefix, entry: RibEntry },
Withdrawn { peer: PeerId, prefix: IpPrefix },
}
crates/rib/src/lib.rs
+8 -6
View File
@@ -1,7 +1,9 @@
mod adj_rib_in;
mod entry;
mod event;
//! The router's tables: the routes it has learned and chosen
//! ([`table::RibTable`]) and the routes it originates ([`LocalRib`]).
pub use adj_rib_in::AdjRibIn;
pub use entry::{PeerId, RibEntry, SnapshotEntry};
pub use event::RibEvent;
mod error;
mod local_rib;
pub mod table;
pub use error::RibStopped;
pub use local_rib::{LocalRib, LocalRibEvent, LocalRoute, LocalRouteAttrs};
crates/rib/src/local_rib.rs
+262
View File
@@ -0,0 +1,262 @@
//! Locally originated routes (the "Loc-RIB" of routes this router announces
//! on its own behalf, as opposed to routes learned from peers).
//!
//! The actor stores one route per prefix together with its attributes. It
//! does not know about peers or sessions: the server subscribes to
//! [`LocalRibEvent`]s and turns them into UPDATEs.
use std::collections::HashMap;
use std::time::SystemTime;
use bgp_packet::message::{LargeCommunity, OriginPathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use serde::Serialize;
use tokio::sync::{broadcast, mpsc, oneshot};
use tokio::task::AbortHandle;
use crate::error::RibStopped;
const EVENT_CHANNEL_CAPACITY: usize = 1024;
const CMD_CHANNEL_CAPACITY: usize = 256;
/// Attributes of a locally originated route.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct LocalRouteAttrs {
pub origin: OriginPathAttribute,
/// Number of *additional* copies of the local ASN to prepend to the
/// AS_PATH. `0` yields the plain `[local_asn]` path.
pub prepend: u8,
pub med: Option<u32>,
/// RFC 1997 standard communities as `(high, low)` 16-bit halves.
pub communities: Vec<(u16, u16)>,
/// RFC 8092 large communities.
pub large_communities: Vec<LargeCommunity>,
}
impl Default for LocalRouteAttrs {
fn default() -> Self {
Self {
origin: OriginPathAttribute::IGP,
prepend: 0,
med: None,
communities: vec![],
large_communities: vec![],
}
}
}
/// A route originated by this router.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct LocalRoute {
pub prefix: IpPrefix,
pub attrs: LocalRouteAttrs,
pub announced_at: SystemTime,
}
impl LocalRoute {
pub fn new(prefix: IpPrefix, attrs: LocalRouteAttrs) -> Self {
Self {
prefix,
attrs,
announced_at: SystemTime::now(),
}
}
}
const STOPPED: RibStopped = RibStopped { rib: "local RIB" };
#[derive(Debug, Clone)]
pub enum LocalRibEvent {
/// A route was added or its attributes replaced.
Announced(LocalRoute),
Withdrawn {
prefix: IpPrefix,
},
}
enum Command {
Announce {
route: LocalRoute,
reply: oneshot::Sender<()>,
},
Withdraw {
prefix: IpPrefix,
reply: oneshot::Sender<Option<LocalRoute>>,
},
Snapshot {
reply: oneshot::Sender<Vec<LocalRoute>>,
},
Len {
reply: oneshot::Sender<usize>,
},
}
/// Handle to the local RIB actor. Cheap to clone; all clones share one table.
///
/// Every operation fails with [`RibStopped`] if the actor is gone; see
/// there for what that means.
#[derive(Clone)]
pub struct LocalRib {
tx: mpsc::Sender<Command>,
events: broadcast::Sender<LocalRibEvent>,
actor: AbortHandle,
}
impl LocalRib {
pub fn spawn() -> Self {
let (tx, rx) = mpsc::channel(CMD_CHANNEL_CAPACITY);
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let actor = tokio::spawn(run_actor(rx, events.clone())).abort_handle();
Self { tx, events, actor }
}
/// Adds a route, or replaces the existing route for the same prefix.
pub async fn announce(&self, route: LocalRoute) -> Result<(), RibStopped> {
self.ask(|reply| Command::Announce { route, reply }).await
}
/// Removes a route. Returns the removed route, or `None` if the prefix
/// was not present.
pub async fn withdraw(&self, prefix: IpPrefix) -> Result<Option<LocalRoute>, RibStopped> {
self.ask(|reply| Command::Withdraw { prefix, reply }).await
}
/// Point-in-time copy of every route, in no particular order.
pub async fn snapshot(&self) -> Result<Vec<LocalRoute>, RibStopped> {
self.ask(|reply| Command::Snapshot { reply }).await
}
/// How many routes the RIB holds, without copying them.
pub async fn len(&self) -> Result<usize, RibStopped> {
self.ask(|reply| Command::Len { reply }).await
}
/// Whether the RIB holds no routes.
pub async fn is_empty(&self) -> Result<bool, RibStopped> {
Ok(self.len().await? == 0)
}
/// Sends a command and waits for its answer.
async fn ask<T>(
&self,
command: impl FnOnce(oneshot::Sender<T>) -> Command,
) -> Result<T, RibStopped> {
let (reply, answer) = oneshot::channel();
self.tx.send(command(reply)).await.map_err(|_| STOPPED)?;
answer.await.map_err(|_| STOPPED)
}
/// Resolves when the actor has stopped. It never does in a healthy
/// process; whoever owns the RIB waits on this to notice a failure even
/// when nothing happens to be using the table.
pub async fn stopped(&self) {
self.tx.closed().await
}
/// Stops the actor, as a panic in it would. For tests of how the rest
/// of the server copes.
#[doc(hidden)]
pub fn stop_for_testing(&self) {
self.actor.abort();
}
/// Subscribe to mutations. A subscriber that falls too far behind is
/// told so with [`broadcast::error::RecvError::Lagged`].
pub fn subscribe(&self) -> broadcast::Receiver<LocalRibEvent> {
self.events.subscribe()
}
}
async fn run_actor(mut rx: mpsc::Receiver<Command>, events: broadcast::Sender<LocalRibEvent>) {
let mut table: HashMap<IpPrefix, LocalRoute> = HashMap::new();
while let Some(cmd) = rx.recv().await {
match cmd {
Command::Announce { route, reply } => {
table.insert(route.prefix, route.clone());
let _ = events.send(LocalRibEvent::Announced(route));
let _ = reply.send(());
}
Command::Withdraw { prefix, reply } => {
let removed = table.remove(&prefix);
if removed.is_some() {
let _ = events.send(LocalRibEvent::Withdrawn { prefix });
}
let _ = reply.send(removed);
}
Command::Snapshot { reply } => {
// A failed reply means the asker stopped waiting.
let _ = reply.send(table.values().cloned().collect());
}
Command::Len { reply } => {
let _ = reply.send(table.len());
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(p: &str) -> LocalRoute {
LocalRoute::new(prefix(p), LocalRouteAttrs::default())
}
#[tokio::test]
async fn announce_and_withdraw_emit_events() {
let rib = LocalRib::spawn();
let mut events = rib.subscribe();
rib.announce(route("10.0.0.0/24")).await.unwrap();
assert!(
matches!(events.recv().await.unwrap(), LocalRibEvent::Announced(r) if r.prefix == prefix("10.0.0.0/24"))
);
assert_eq!(rib.len().await, Ok(1));
let removed = rib.withdraw(prefix("10.0.0.0/24")).await.unwrap();
assert_eq!(removed.unwrap().prefix, prefix("10.0.0.0/24"));
assert!(
matches!(events.recv().await.unwrap(), LocalRibEvent::Withdrawn { prefix: p } if p == prefix("10.0.0.0/24"))
);
assert_eq!(rib.is_empty().await, Ok(true));
// Withdrawing an absent prefix is a no-op, not an error.
assert_eq!(rib.withdraw(prefix("10.0.0.0/24")).await.unwrap(), None);
}
#[tokio::test]
async fn announcing_a_prefix_again_replaces_its_attributes() {
let rib = LocalRib::spawn();
rib.announce(route("10.0.0.0/24")).await.unwrap();
let attrs = LocalRouteAttrs {
med: Some(50),
..Default::default()
};
rib.announce(LocalRoute::new(prefix("10.0.0.0/24"), attrs.clone()))
.await
.unwrap();
let table = rib.snapshot().await.unwrap();
assert_eq!(table.len(), 1);
assert_eq!(table[0].attrs, attrs);
}
#[tokio::test]
async fn a_stopped_actor_is_an_error_and_not_an_empty_table() {
let rib = LocalRib::spawn();
rib.announce(route("10.0.0.0/24")).await.unwrap();
rib.stop_for_testing();
rib.stopped().await;
assert_eq!(rib.announce(route("10.0.1.0/24")).await, Err(STOPPED));
assert_eq!(rib.withdraw(prefix("10.0.0.0/24")).await, Err(STOPPED));
assert_eq!(rib.snapshot().await, Err(STOPPED));
assert_eq!(rib.len().await, Err(STOPPED));
}
}
crates/rib/src/table/attrs.rs
+449
View File
@@ -0,0 +1,449 @@
//! Attribute sets, stored once.
//!
//! A full table has millions of routes and far fewer distinct lists of path
//! attributes: a peer announces prefixes that share their attributes
//! together, about six to an UPDATE on the Internet today. A route in the
//! table is therefore a pointer to an [`AttrSet`], and each distinct set
//! exists once, in the [`AttrStore`].
//!
//! A set is kept as its bytes on the wire, which is a fifth the size of the
//! parsed form. What choosing between routes needs from it is worked out
//! once, when the set is stored, and kept beside the bytes.
use std::hash::BuildHasher;
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use bgp_packet::message::{NlriNextHop, PathAttribute, attr_type, split_attribute};
use bgp_primitives::parser::{EncodeError, ParserContext};
use bytes::{BufMut, BytesMut};
use foldhash::fast::RandomState;
use hashbrown::HashTable;
const OPTIONAL: u8 = 0x80;
/// What route selection reads from a set of attributes (RFC 4271 §9.1.2.2).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SelectionKey {
/// LOCAL_PREF, if the set has one.
pub local_pref: Option<u32>,
/// MULTI_EXIT_DISC, if the set has one.
pub med: Option<u32>,
/// The AS that the route was learned from: the first AS of the path, or
/// zero if the path is empty or begins with a set. MEDs are comparable
/// only between routes with the same one.
pub neighbor_as: u32,
/// The length of the AS path as selection counts it: an AS_SET is one
/// hop however many AS numbers it holds.
pub as_path_len: u16,
/// ORIGIN: 0 for IGP, 1 for EGP, 2 for INCOMPLETE.
pub origin: u8,
}
/// Reads the selection key straight off attributes in the form an
/// [`AttrSet`] keeps, without building anything. `None` if the bytes are
/// not a well-formed list of attributes.
fn scan(mut wire: &[u8]) -> Option<SelectionKey> {
let mut key = SelectionKey {
local_pref: None,
med: None,
neighbor_as: 0,
as_path_len: 0,
// A route without ORIGIN does not get this far; if one did, it
// would be the least preferred.
origin: 2,
};
let number = |value: &[u8]| Some(u32::from_be_bytes(<[u8; 4]>::try_from(value).ok()?));
while !wire.is_empty() {
let (attribute, rest) = split_attribute(wire).ok()?;
wire = rest;
let value = attribute.value;
match attribute.type_code {
attr_type::ORIGIN => key.origin = *value.first()?,
attr_type::AS_PATH => (key.as_path_len, key.neighbor_as) = scan_as_path(value)?,
attr_type::MULTI_EXIT_DISC => key.med = Some(number(value)?),
attr_type::LOCAL_PREF => key.local_pref = Some(number(value)?),
_ => {}
}
}
Some(key)
}
/// The length of an AS_PATH as selection counts it, and the neighbouring
/// AS, from the attribute's value with four-octet AS numbers.
fn scan_as_path(mut segments: &[u8]) -> Option<(u16, u32)> {
const AS_SEQUENCE: u8 = 2;
let mut hops = 0usize;
let mut neighbor_as = 0;
let mut first = true;
while let [kind, count, rest @ ..] = segments {
let (asns, rest) = rest.split_at_checked(usize::from(*count) * 4)?;
let ordered = *kind == AS_SEQUENCE;
hops += if ordered { usize::from(*count) } else { 1 };
if first && ordered {
neighbor_as = asns
.first_chunk::<4>()
.map_or(0, |asn| u32::from_be_bytes(*asn));
}
first = false;
segments = rest;
}
if !segments.is_empty() {
return None;
}
Some((u16::try_from(hops).unwrap_or(u16::MAX), neighbor_as))
}
/// One distinct list of path attributes, with the next hop that goes with
/// it. Made by an [`AttrStore`]; two routes have the same attributes
/// exactly when they point to the same `AttrSet`.
#[derive(Debug)]
pub struct AttrSet {
hash: u64,
key: SelectionKey,
/// The attributes as on the wire, with four-octet AS numbers. A next
/// hop that arrived in MP_REACH_NLRI is kept as an MP_REACH_NLRI that
/// holds nothing else, the form RFC 6396 §4.3.4 uses in table dumps.
wire: Box<[u8]>,
}
impl AttrSet {
pub fn key(&self) -> &SelectionKey {
&self.key
}
/// The attributes as they are on the wire.
pub fn wire(&self) -> &[u8] {
&self.wire
}
/// The attributes, parsed, and the next hop if it came in
/// MP_REACH_NLRI.
///
/// `None` if they do not parse. Sets made from parsed attributes or
/// from an UPDATE the decoder accepted always do; one made from bytes
/// of another source ([`AttrStore::intern_wire`]) was checked only as
/// far as selection reads it.
pub fn attributes(&self) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
parse(&self.wire)
}
}
/// How the attributes of a set are written: with four-octet AS numbers.
fn four_octet() -> ParserContext {
ParserContext {
four_octet_asn: true,
ibgp: true,
add_path: Vec::new(),
}
}
/// Writes `attributes` and `next_hop` in the form an [`AttrSet`] keeps.
fn encode(
attributes: &[PathAttribute],
next_hop: Option<&NlriNextHop>,
out: &mut BytesMut,
) -> Result<(), EncodeError> {
let ctx = four_octet();
for attribute in attributes {
// These name prefixes, which are not a property of the route.
if matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
) {
continue;
}
attribute.to_wire(&ctx, out)?;
}
if let Some(next_hop) = next_hop {
encode_next_hop(next_hop, out);
}
Ok(())
}
/// Writes a next hop as an MP_REACH_NLRI that holds nothing else.
fn encode_next_hop(next_hop: &NlriNextHop, out: &mut BytesMut) {
out.put_u8(OPTIONAL);
out.put_u8(attr_type::MP_REACH_NLRI);
out.put_u8(1 + next_hop.wire_len());
out.put_u8(next_hop.wire_len());
next_hop.to_wire(out);
}
/// Parses path attributes in the form [`AttrSet`] keeps and table dumps
/// use (RFC 6396 §4.3.4): four-octet AS numbers, and an MP_REACH_NLRI that
/// holds only a next hop. `None` if they are malformed.
fn parse(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<NlriNextHop>)> {
let ctx = four_octet();
let mut attributes = Vec::new();
let mut next_hop = None;
while !wire.is_empty() {
let (raw, rest) = split_attribute(wire).ok()?;
if raw.type_code == attr_type::MP_REACH_NLRI {
let (length, address) = raw.value.split_first()?;
if usize::from(*length) != address.len() {
return None;
}
next_hop = Some(NlriNextHop::from_bytes(address)?);
} else {
let (_, attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()?;
attributes.push(attribute);
}
wire = rest;
}
Some((attributes, next_hop))
}
/// How many locks the store is split over. Sessions intern concurrently,
/// once per UPDATE; this many locks keep them out of each other's way.
const STRIPES: usize = 64;
/// The one place attribute sets are kept. Safe to share between threads.
pub struct AttrStore {
stripes: Box<[Mutex<HashTable<Arc<AttrSet>>>]>,
hasher: RandomState,
}
impl Default for AttrStore {
fn default() -> Self {
Self::new()
}
}
impl AttrStore {
pub fn new() -> Self {
Self {
stripes: (0..STRIPES).map(|_| Mutex::default()).collect(),
hasher: RandomState::default(),
}
}
/// The stored set equal to `attributes` and `next_hop`, added if it was
/// not there. `next_hop` is the one from MP_REACH_NLRI; routes whose
/// next hop is the NEXT_HOP attribute pass `None`.
///
/// Fails only if the attributes cannot be encoded, which attributes that
/// were decoded from a message always can.
pub fn intern(
&self,
attributes: &[PathAttribute],
next_hop: Option<&NlriNextHop>,
) -> Result<Arc<AttrSet>, EncodeError> {
let mut wire = BytesMut::with_capacity(256);
encode(attributes, next_hop, &mut wire)?;
Ok(self
.intern_wire(&wire)
.expect("attributes written here are well formed"))
}
/// As [`AttrStore::intern`], for attributes that are already in the
/// form a set keeps: on the wire, with four-octet AS numbers, no
/// MP_UNREACH_NLRI, and an MP_REACH_NLRI (if any) that holds only the
/// next hop. That is how a table dump has them (RFC 6396 §4.3.4).
///
/// Nothing is parsed: finding a set that is already stored costs a hash
/// of the bytes and a comparison. `None` if the bytes are not a
/// well-formed list of attributes.
pub fn intern_wire(&self, wire: &[u8]) -> Option<Arc<AttrSet>> {
let hash = self.hasher.hash_one(wire);
let mut stripe = self.stripe(hash);
if let Some(existing) = stripe.find(hash, |set| *set.wire == *wire) {
return Some(Arc::clone(existing));
}
let set = Arc::new(AttrSet {
hash,
key: scan(wire)?,
wire: wire.into(),
});
stripe.insert_unique(hash, Arc::clone(&set), |set| set.hash);
Some(set)
}
/// The set for attributes as a session received them: `wire` is what
/// the decoder kept of the UPDATE's attributes
/// (`DecodedUpdate::attributes_wire`), and `next_hop` the next hop of
/// the routes if they came in MP_REACH_NLRI.
pub fn intern_received(
&self,
wire: &[u8],
next_hop: Option<&NlriNextHop>,
) -> Option<Arc<AttrSet>> {
let Some(next_hop) = next_hop else {
return self.intern_wire(wire);
};
let mut with_next_hop = BytesMut::with_capacity(wire.len() + 36);
with_next_hop.put_slice(wire);
encode_next_hop(next_hop, &mut with_next_hop);
self.intern_wire(&with_next_hop)
}
/// How many distinct sets are stored.
pub fn len(&self) -> usize {
(0..STRIPES).map(|i| self.stripe_at(i).len()).sum()
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// Drops the sets that no route refers to any more, and returns how
/// many that was. Sets are not dropped as their last route goes, since
/// that would mean a lock here for every route removed; this is called
/// now and then instead.
pub fn sweep(&self) -> usize {
let mut dropped = 0;
for i in 0..STRIPES {
let mut stripe = self.stripe_at(i);
let before = stripe.len();
// The store's own reference is the only one left. Nobody can
// be taking another: that is done under this lock.
stripe.retain(|set| Arc::strong_count(set) > 1);
dropped += before - stripe.len();
}
dropped
}
fn stripe(&self, hash: u64) -> MutexGuard<'_, HashTable<Arc<AttrSet>>> {
self.stripe_at(super::spread(hash, STRIPES))
}
fn stripe_at(&self, index: usize) -> MutexGuard<'_, HashTable<Arc<AttrSet>>> {
// A panic while holding the lock cannot leave the table half
// changed in a way that matters: see `IgnorePoison` in the server.
self.stripes[index]
.lock()
.unwrap_or_else(PoisonError::into_inner)
}
}
#[cfg(test)]
mod tests {
use std::net::Ipv4Addr;
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LocalPrefPathAttribute,
MultiExitDiscPathAttribute, NextHopPathAttribute, OriginPathAttribute,
};
use super::*;
fn attributes(path: Vec<u32>) -> Vec<PathAttribute> {
vec![
PathAttribute::Origin(OriginPathAttribute::EGP),
AsPathAttribute::from_asns(path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65000, 1)])),
]
}
#[test]
fn equal_attributes_are_one_set() {
let store = AttrStore::new();
let a = store.intern(&attributes(vec![65001, 65002]), None).unwrap();
let b = store.intern(&attributes(vec![65001, 65002]), None).unwrap();
let c = store.intern(&attributes(vec![65001, 65003]), None).unwrap();
assert!(Arc::ptr_eq(&a, &b));
assert!(!Arc::ptr_eq(&a, &c));
assert_eq!(store.len(), 2);
// The next hop is part of what a set is.
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let d = store
.intern(&attributes(vec![65001, 65002]), Some(&next_hop))
.unwrap();
assert!(!Arc::ptr_eq(&a, &d));
assert_eq!(store.len(), 3);
}
#[test]
fn a_set_gives_back_what_went_in() {
let store = AttrStore::new();
let next_hop = NlriNextHop::Ipv6WithLl {
global: "2001:db8::1".parse().unwrap(),
link_local: "fe80::1".parse().unwrap(),
};
let original = attributes(vec![65001, 4_200_000_001]);
let set = store.intern(&original, Some(&next_hop)).unwrap();
assert_eq!(set.attributes(), Some((original, Some(next_hop))));
}
#[test]
fn the_selection_key_is_read_off_the_attributes() {
let store = AttrStore::new();
let mut attrs = vec![
PathAttribute::Origin(OriginPathAttribute::INCOMPLETE),
PathAttribute::AsPath(AsPathAttribute {
segments: vec![
AsPathSegment {
ordered: true,
path: vec![65001, 65002],
},
// A set counts as one hop.
AsPathSegment {
ordered: false,
path: vec![65003, 65004, 65005],
},
],
}),
PathAttribute::LocalPref(LocalPrefPathAttribute(200)),
];
let key = *store.intern(&attrs, None).unwrap().key();
assert_eq!(
key,
SelectionKey {
local_pref: Some(200),
med: None,
neighbor_as: 65001,
as_path_len: 3,
origin: 2,
}
);
attrs.push(PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(7)));
assert_eq!(store.intern(&attrs, None).unwrap().key().med, Some(7));
}
#[test]
fn attributes_on_the_wire_are_interned_without_parsing_them() {
let store = AttrStore::new();
let next_hop = NlriNextHop::Ipv6("2001:db8::1".parse().unwrap());
let mut attrs = attributes(vec![65001, 65002, 65003]);
attrs.push(PathAttribute::LocalPref(LocalPrefPathAttribute(300)));
let parsed = store.intern(&attrs, Some(&next_hop)).unwrap();
// The same set by its bytes is the same set, key and all.
let by_wire = store.intern_wire(parsed.wire()).unwrap();
assert!(Arc::ptr_eq(&parsed, &by_wire));
// And one that arrives as bytes first gets the same key as it would
// have had from the parsed form.
let other = AttrStore::new();
let fresh = other.intern_wire(parsed.wire()).unwrap();
assert_eq!(fresh.key(), parsed.key());
assert_eq!(fresh.key().as_path_len, 3);
assert_eq!(fresh.key().neighbor_as, 65001);
assert_eq!(fresh.attributes(), Some((attrs, Some(next_hop))));
// Bytes that are not attributes are refused.
assert!(other.intern_wire(&[0x40, 0x01, 0x05, 0x00]).is_none());
assert!(
other
.intern_wire(&[0x40, 0x02, 0x03, 0x02, 0x01, 0x00])
.is_none()
);
}
#[test]
fn sweeping_drops_only_sets_nobody_holds() {
let store = AttrStore::new();
let kept = store.intern(&attributes(vec![65001]), None).unwrap();
drop(store.intern(&attributes(vec![65002]), None).unwrap());
assert_eq!(store.len(), 2);
assert_eq!(store.sweep(), 1);
assert_eq!(store.len(), 1);
// The one that is left is still the one that is found.
let again = store.intern(&attributes(vec![65001]), None).unwrap();
assert!(Arc::ptr_eq(&kept, &again));
}
}
crates/rib/src/table/journal.rs
+755
View File
@@ -0,0 +1,755 @@
//! The journal: how the rest of the router follows the best routes.
//!
//! Whatever acts on the table's choices (the peers that are sent the best
//! routes, the forwarding table, the status page) has to learn when a best
//! route changes, at its own pace, without the table waiting for it. Each
//! shard keeps a journal for that, and each follower reads it through a
//! [`Reader`].
//!
//! # What the journal holds
//!
//! It records *that* a prefix's best route changed, not what it changed to.
//! A reader that comes to an entry looks the prefix up and is given the best
//! route as it is at that moment. A follower does not want the history; it
//! wants to end up with what the table has now.
//!
//! This is what keeps the journal small. If a prefix changes again before a
//! reader has got to its entry, the older entry no longer says anything: the
//! newer one will deliver the current route. Each row remembers the
//! sequence number of its latest change, an entry with any other number is
//! dead, and dead entries are thrown out when the journal is tidied. So a
//! prefix that flaps a thousand times costs a reader one event, and the
//! journal never has more live entries than the table has prefixes.
//!
//! # Readers that fall behind
//!
//! Entries are dropped once every reader has passed them. A reader that
//! stops reading would hold them all back, so there is a limit: when the
//! journal is too long even after tidying, its oldest half is dropped
//! regardless. A reader that was behind that point is told so the next time
//! it reads ([`Event::Reset`]) and is given the whole table again. Reading
//! too slowly costs the slow reader a resynchronisation; it costs the
//! table, and the other readers, nothing.
//!
//! A new reader starts the same way: `Reset`, every best route, `Synced`,
//! and changes from then on.
//!
//! # All routes, or only the best
//!
//! There are two journals in each shard. One records changes of best route,
//! for followers that want one route for each prefix: the forwarding table,
//! and peers that are sent the best route. The other records every change
//! to any route, for followers that want them all: peers that are sent
//! several routes for a prefix (ADD-PATH, RFC 7911), or a choice made for
//! them alone. A [`Reader`] follows one or the other. The second journal
//! costs nothing while nobody follows it.
//!
//! # Order
//!
//! For any one prefix, a reader is given its states in the order they came
//! about, and always, in the end, the latest. Between prefixes there is no
//! order worth relying on.
use std::collections::VecDeque;
use std::hash::BuildHasher;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use bgp_primitives::ip_prefix::IpPrefix;
use foldhash::fast::RandomState;
use hashbrown::{HashMap, HashTable};
use super::{AttrSet, PeerIndex, RibTable, RouteRef, Row, Shard, read, write};
/// A cursor that holds nothing back: a reader slot that is free, or a
/// reader that has not read this shard yet.
const NO_CURSOR: u64 = u64::MAX;
/// When a journal is tidied, and when it is too long.
#[derive(Debug, Clone, Copy)]
pub(super) struct Limits {
/// Below this many entries a journal is not worth tidying.
pub(super) tidy_at: usize,
/// Room allowed beyond the number of prefixes before the journal is
/// thought too long.
pub(super) slack: usize,
}
impl Default for Limits {
fn default() -> Self {
Self {
tidy_at: 1024,
slack: 64,
}
}
}
struct Entry {
seq: u64,
prefix: IpPrefix,
}
/// One journal of one shard.
struct Journal {
kind: Kind,
/// The sequence number of the latest change. Counts every change this
/// journal is for, whether or not anyone was reading.
seq: u64,
/// Entries up to and including this number were dropped without regard
/// to the readers. A reader whose cursor is below it missed something.
floor: u64,
/// In order of sequence number. Not every number is here: dead entries
/// are removed, and nothing is recorded while there are no readers.
entries: VecDeque<Entry>,
/// Prefixes that have no route any more and whose disappearance is
/// still to be read, with the sequence number of the entry that says
/// so. A row keeps this number itself; a prefix without a row cannot.
gone: HashMap<IpPrefix, u64, RandomState>,
/// Where each reader has read to, by reader slot. Written by the
/// readers themselves while they hold the shard for reading.
cursors: Vec<AtomicU64>,
/// How many reader slots are in use.
readers: usize,
limits: Limits,
}
impl Journal {
fn new(kind: Kind, limits: Limits) -> Self {
Self {
kind,
limits,
seq: 0,
floor: 0,
entries: VecDeque::new(),
gone: HashMap::default(),
cursors: Vec::new(),
readers: 0,
}
}
/// Notes a change to `prefix`, which has a row. Returns the sequence
/// number for the row to remember.
fn changed(&mut self, prefix: IpPrefix) -> u64 {
self.seq += 1;
if self.readers > 0 {
self.entries.push_back(Entry {
seq: self.seq,
prefix,
});
}
self.seq
}
/// As [`Journal::changed`], for a prefix whose row was just made:
/// if it had gone and is back, the note that it had gone is void.
fn appeared(&mut self, prefix: IpPrefix) -> u64 {
if !self.gone.is_empty() {
self.gone.remove(&prefix);
}
self.changed(prefix)
}
/// Notes that `prefix` lost its last route and has no row any more.
fn disappeared(&mut self, prefix: IpPrefix) {
self.seq += 1;
if self.readers > 0 {
self.entries.push_back(Entry {
seq: self.seq,
prefix,
});
self.gone.insert(prefix, self.seq);
}
}
/// Forgets that `entry`'s prefix had gone, if `entry` is what said so.
fn drop_entry(gone: &mut HashMap<IpPrefix, u64, RandomState>, entry: &Entry) {
if gone.get(&entry.prefix) == Some(&entry.seq) {
gone.remove(&entry.prefix);
}
}
/// What `entry` says, if it still says something: it is the latest
/// change of its prefix. The row, or `None` for a prefix that has gone.
fn live<'a>(
&self,
entry: &Entry,
rows: &'a HashTable<Row>,
hasher: &RandomState,
) -> Option<Option<&'a Row>> {
let hash = hasher.hash_one(entry.prefix);
match rows.find(hash, |row| row.prefix == entry.prefix) {
Some(row) => (row.seqs.of(self.kind) == entry.seq).then_some(Some(row)),
None => (self.gone.get(&entry.prefix) == Some(&entry.seq)).then_some(None),
}
}
/// Keeps the journal from growing. Called after every change to the
/// shard; does nothing until there is something worth doing.
fn tidy(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
if self.entries.len() <= self.limits.tidy_at {
return;
}
// What every reader has passed can go.
let passed = self
.cursors
.iter()
.map(|cursor| cursor.load(Ordering::Acquire))
.min()
.unwrap_or(NO_CURSOR);
while let Some(front) = self.entries.front() {
if front.seq > passed {
break;
}
Self::drop_entry(&mut self.gone, front);
self.entries.pop_front();
}
// Entries for prefixes that have a row are at most as many as the
// rows once the dead ones are out. Entries for prefixes that have
// gone are not counted in: they are what a reader that has stopped
// reading makes pile up.
let limit = 2 * (rows.len() + self.limits.slack);
if self.entries.len() <= limit {
return;
}
// Dead entries: their prefix has changed again since. This looks
// every entry up, but only once the journal has doubled, so the
// cost per change stays small.
let mut entries = std::mem::take(&mut self.entries);
entries.retain(|entry| self.live(entry, rows, hasher).is_some());
self.entries = entries;
if self.entries.len() <= limit {
return;
}
// Still too long: a reader is not reading while prefixes come and
// go, or more prefixes went at once than are left. The older half
// goes, and a reader that had not read it loses its place. Half,
// so that readers that are only a little behind keep theirs, and
// so that getting a long journal down to size takes few passes.
for entry in self.entries.drain(..self.entries.len() / 2) {
Self::drop_entry(&mut self.gone, &entry);
self.floor = entry.seq;
}
}
}
/// Which of a shard's two journals.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum Kind {
/// Changes of best route.
Best,
/// Changes of any route.
All,
}
/// The sequence numbers a row remembers, one for each journal: of the
/// latest change of its best route, and of any of its routes.
#[derive(Debug, Clone, Copy)]
pub(super) struct Seqs {
best: u64,
all: u64,
}
impl Seqs {
fn of(self, kind: Kind) -> u64 {
match kind {
Kind::Best => self.best,
Kind::All => self.all,
}
}
}
/// The two journals of a shard, written together: every change to a row is
/// a change of one of its routes, and some are a change of its best route.
pub(super) struct Journals {
best: Journal,
all: Journal,
}
impl Journals {
pub(super) fn new(limits: Limits) -> Self {
Self {
best: Journal::new(Kind::Best, limits),
all: Journal::new(Kind::All, limits),
}
}
fn of(&self, kind: Kind) -> &Journal {
match kind {
Kind::Best => &self.best,
Kind::All => &self.all,
}
}
fn of_mut(&mut self, kind: Kind) -> &mut Journal {
match kind {
Kind::Best => &mut self.best,
Kind::All => &mut self.all,
}
}
/// Notes that `prefix` has its first route, and so a row. Returns the
/// sequence numbers for the new row.
pub(super) fn appeared(&mut self, prefix: IpPrefix) -> Seqs {
Seqs {
best: self.best.appeared(prefix),
all: self.all.appeared(prefix),
}
}
/// Notes that a route of `row` changed, and whether its best route did.
pub(super) fn changed(&mut self, row: &mut Row, best_changed: bool) {
if best_changed {
row.seqs.best = self.best.changed(row.prefix);
}
row.seqs.all = self.all.changed(row.prefix);
}
/// Notes that `prefix` lost its last route and has no row any more.
pub(super) fn disappeared(&mut self, prefix: IpPrefix) {
self.best.disappeared(prefix);
self.all.disappeared(prefix);
}
/// Keeps the journals from growing; see [`Journal::tidy`].
pub(super) fn tidy(&mut self, rows: &HashTable<Row>, hasher: &RandomState) {
self.best.tidy(rows, hasher);
self.all.tidy(rows, hasher);
}
/// Where the journals are now, to ask [`Journals::has_news_since`]
/// about later.
pub(super) fn mark(&self) -> Mark {
Mark(self.best.seq, self.all.seq)
}
/// Whether anything was written since `mark` that some reader is there
/// to read.
pub(super) fn has_news_since(&self, mark: Mark) -> bool {
(self.best.readers > 0 && self.best.seq != mark.0)
|| (self.all.readers > 0 && self.all.seq != mark.1)
}
fn len(&self) -> usize {
self.best.entries.len() + self.all.entries.len()
}
}
/// A point in a shard's journals; see [`Journals::mark`].
#[derive(Clone, Copy)]
pub(super) struct Mark(u64, u64);
/// The routes of one prefix, as a follower of all routes is given them.
#[derive(Clone, Copy)]
pub struct Routes<'a>(&'a Row);
impl<'a> Routes<'a> {
pub fn iter(&self) -> impl Iterator<Item = RouteRef<'a>> + use<'a> {
self.0.routes()
}
/// How many there are: at least one.
pub fn len(&self) -> usize {
self.0.paths.len()
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
}
impl std::fmt::Debug for Routes<'_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_list().entries(self.iter()).finish()
}
}
/// What a [`Reader`] is given.
#[derive(Debug, Clone, Copy)]
pub enum Event<'a> {
/// Forget what you know. The whole table follows, as [`Event::Best`]
/// or [`Event::Routes`] for every prefix, and then [`Event::Synced`].
/// The first thing a new
/// reader is given, and what a reader that fell too far behind is
/// given in place of what it missed.
Reset,
/// This is now the best route for `prefix`. Given to a follower of the
/// best routes.
Best {
prefix: IpPrefix,
peer: PeerIndex,
/// The peer's identifier for the route: zero unless the peer
/// sends several routes for a prefix (ADD-PATH, RFC 7911).
path_id: u32,
attrs: &'a Arc<AttrSet>,
},
/// These are now all the routes for `prefix`: whatever was known of its
/// routes before is replaced. Given to a follower of all routes.
Routes {
prefix: IpPrefix,
routes: Routes<'a>,
},
/// `prefix` has no route any more.
Withdrawn { prefix: IpPrefix },
/// The table that began with [`Event::Reset`] is complete: a prefix
/// not heard of since the reset has no route.
Synced,
}
/// What a call to [`Reader::poll`] got done.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Progress {
/// How many events were given out.
pub events: usize,
/// Whether the reader has now been given everything there was when the
/// call began. If not, call again.
pub caught_up: bool,
}
enum Phase {
/// Giving out the whole table; `next` is the shard to do next, and
/// `announced` whether [`Event::Reset`] has been given.
Resync { next: usize, announced: bool },
/// Following the journals; `next` is the shard to read next.
Follow { next: usize },
}
impl Phase {
/// The beginning of a resynchronisation.
const RESYNC: Self = Self::Resync {
next: 0,
announced: false,
};
}
/// Follows the routes of a table: the best of each prefix, or all of them.
/// Made by [`RibTable::reader`] and its siblings.
///
/// A reader holds its place in every shard's journal. Dropping it gives
/// the place up, so that the journals need not keep entries for it.
pub struct Reader {
table: Arc<RibTable>,
/// Which journal it follows.
kind: Kind,
slot: usize,
/// Where this reader has read to in each shard; the copy in the shard
/// is for the shard's benefit.
cursors: Vec<u64>,
phase: Phase,
/// The table's version when the reader was last found caught up.
seen: u64,
caught_up: bool,
}
impl RibTable {
/// A new follower of this table's best routes. It is given the whole
/// table first and every change after.
pub fn reader(self: &Arc<Self>) -> Reader {
self.register(Kind::Best, Phase::RESYNC)
}
/// A new follower that is given only the changes from now on, for
/// those who want the news and not the table. If it falls too far
/// behind it is reset like any other, and is then given the table.
pub fn reader_from_now(self: &Arc<Self>) -> Reader {
let mut reader = self.register(Kind::Best, Phase::Follow { next: 0 });
for (index, shard) in self.shards.iter().enumerate() {
let shard = read(shard);
reader.set_cursor(index, &shard, shard.journals.best.seq);
}
reader
}
/// A new follower of every route in the table, not only the best: it
/// is told whenever any route of a prefix changes, with all the
/// prefix's routes ([`Event::Routes`]). The whole table first, like
/// [`RibTable::reader`].
pub fn reader_of_all_routes(self: &Arc<Self>) -> Reader {
self.register(Kind::All, Phase::RESYNC)
}
fn register(self: &Arc<Self>, kind: Kind, phase: Phase) -> Reader {
let mut slots = self
.reader_slots
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
let slot = match slots.iter().position(|used| !used) {
Some(free) => free,
None => {
slots.push(false);
slots.len() - 1
}
};
slots[slot] = true;
for shard in &self.shards {
let mut shard = write(shard);
let journal = shard.journals.of_mut(kind);
while journal.cursors.len() <= slot {
journal.cursors.push(AtomicU64::new(NO_CURSOR));
}
journal.cursors[slot].store(NO_CURSOR, Ordering::Release);
journal.readers += 1;
}
self.readers.fetch_add(1, Ordering::AcqRel);
Reader {
table: Arc::clone(self),
kind,
slot,
cursors: vec![NO_CURSOR; self.shards.len()],
phase,
seen: 0,
caught_up: false,
}
}
/// How many entries the journals hold, over all shards. For tests and
/// for watching: it stays bounded whatever the readers do.
pub fn journal_len(&self) -> usize {
self.shards
.iter()
.map(|shard| read(shard).journals.len())
.sum()
}
/// Tells readers that are waiting that there is something to read.
pub(super) fn wake_readers(&self) {
if self.readers.load(Ordering::Acquire) > 0 {
self.version.fetch_add(1, Ordering::AcqRel);
self.wake.notify_waiters();
}
}
}
impl Drop for Reader {
fn drop(&mut self) {
for shard in &self.table.shards {
let mut shard = write(shard);
let journal = shard.journals.of_mut(self.kind);
journal.cursors[self.slot].store(NO_CURSOR, Ordering::Release);
journal.readers -= 1;
if journal.readers == 0 {
// Nobody left to read it.
journal.entries.clear();
journal.gone.clear();
}
}
self.table.readers.fetch_sub(1, Ordering::AcqRel);
let mut slots = self
.table
.reader_slots
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
slots[self.slot] = false;
}
}
enum Drained {
/// Entries were dropped that this reader had not read.
Lagged,
/// `events` were given out; `more` if the budget ran out first.
Read { events: usize, more: bool },
}
impl Reader {
/// Gives `visit` what has changed since the last call, in up to about
/// `budget` events (a shard's part of a resynchronisation is not cut
/// short, so it can be more).
///
/// `visit` is called with a shard of the table held for reading. It
/// should be quick, and it must not call into the table, even to read:
/// taking the shard's lock again can wait for ever on a writer that is
/// waiting for this hold to end.
pub fn poll(&mut self, budget: usize, mut visit: impl FnMut(Event<'_>)) -> Progress {
// A call that may give nothing out would never get anywhere.
let budget = budget.max(1);
let shards = self.table.shards.len();
let mut events = 0;
loop {
match self.phase {
Phase::Resync { next, announced } => {
if !announced {
self.forget_place();
visit(Event::Reset);
events += 1;
}
if next == shards {
visit(Event::Synced);
events += 1;
self.phase = Phase::Follow { next: 0 };
continue;
}
events += self.scan(next, &mut visit);
self.phase = Phase::Resync {
next: next + 1,
announced: true,
};
if events >= budget {
return self.progress(events, false);
}
}
Phase::Follow { next } => {
// Read before the pass: a change made during it shows as
// a newer version, and `changed` then returns at once.
let version = if next == 0 {
self.table.version.load(Ordering::Acquire)
} else {
self.seen
};
self.seen = version;
let mut shard = next;
let mut lagged = false;
while shard < shards {
if events >= budget {
self.phase = Phase::Follow { next: shard };
return self.progress(events, false);
}
match self.drain(shard, budget - events, &mut visit) {
Drained::Lagged => {
lagged = true;
break;
}
Drained::Read { events: read, more } => {
events += read;
if more {
self.phase = Phase::Follow { next: shard };
return self.progress(events, false);
}
}
}
shard += 1;
}
if lagged {
self.phase = Phase::RESYNC;
continue;
}
self.phase = Phase::Follow { next: 0 };
if next != 0 {
// This pass took up where an earlier call stopped.
// The shards before that point have not been looked
// at in this call, so go round again from the first.
continue;
}
return self.progress(events, true);
}
}
}
}
fn progress(&mut self, events: usize, caught_up: bool) -> Progress {
self.caught_up = caught_up;
Progress { events, caught_up }
}
/// Waits until there may be something to read. Returns at once if the
/// last [`Reader::poll`] did not catch up.
pub async fn changed(&mut self) {
loop {
// Registered before the check, so that a change between the
// check and the wait is not missed.
let woken = self.table.wake.notified();
tokio::pin!(woken);
woken.as_mut().enable();
if !self.caught_up || self.table.version.load(Ordering::Acquire) != self.seen {
return;
}
woken.await;
}
}
/// What this reader is told about a row.
fn event_for<'a>(&self, row: &'a Row) -> Event<'a> {
match self.kind {
Kind::Best => {
let best = row.route(row.best());
Event::Best {
prefix: row.prefix,
peer: best.peer,
path_id: best.path_id,
attrs: best.attrs,
}
}
Kind::All => Event::Routes {
prefix: row.prefix,
routes: Routes(row),
},
}
}
/// Gives up this reader's place in every journal, before a
/// resynchronisation takes a new one.
fn forget_place(&mut self) {
for (index, shard) in self.table.shards.iter().enumerate() {
read(shard).journals.of(self.kind).cursors[self.slot]
.store(NO_CURSOR, Ordering::Release);
self.cursors[index] = NO_CURSOR;
}
}
fn set_cursor(&mut self, index: usize, shard: &Shard, cursor: u64) {
self.cursors[index] = cursor;
shard.journals.of(self.kind).cursors[self.slot].store(cursor, Ordering::Release);
}
/// Gives out every prefix in a shard and takes a place at the end of
/// its journal, in one hold of the shard: what changes afterwards is in
/// the journal after that place.
fn scan(&mut self, index: usize, visit: &mut impl FnMut(Event<'_>)) -> usize {
let table = Arc::clone(&self.table);
let shard = read(&table.shards[index]);
for row in &shard.rows {
visit(self.event_for(row));
}
self.set_cursor(index, &shard, shard.journals.of(self.kind).seq);
shard.rows.len()
}
/// Reads a shard's journal from this reader's place, for at most `max`
/// events.
fn drain(&mut self, index: usize, max: usize, visit: &mut impl FnMut(Event<'_>)) -> Drained {
let table = Arc::clone(&self.table);
let shard = read(&table.shards[index]);
let journal = shard.journals.of(self.kind);
let mut cursor = self.cursors[index];
if cursor < journal.floor {
return Drained::Lagged;
}
if cursor == journal.seq {
return Drained::Read {
events: 0,
more: false,
};
}
let start = journal.entries.partition_point(|entry| entry.seq <= cursor);
let mut events = 0;
let mut more = false;
for entry in journal.entries.range(start..) {
if events == max {
more = true;
break;
}
cursor = entry.seq;
let event = match journal.live(entry, &shard.rows, &shard.hasher) {
Some(Some(row)) => self.event_for(row),
Some(None) => Event::Withdrawn {
prefix: entry.prefix,
},
// The prefix has changed again; a later entry has it.
None => continue,
};
visit(event);
events += 1;
}
if !more {
// Past the last entry there is nothing up to the latest change.
cursor = journal.seq;
}
self.set_cursor(index, &shard, cursor);
Drained::Read { events, more }
}
}
#[cfg(test)]
#[path = "journal_tests.rs"]
mod tests;
crates/rib/src/table/journal_tests.rs
+713
View File
@@ -0,0 +1,713 @@
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use proptest::prelude::*;
use super::super::tests::{Fixture, Route, external, internal, prefix, route};
use super::super::{AttrSet, PeerIndex, PeerInfo, RibTable};
use super::{Event, Limits, Reader};
/// What a follower would keep: its own copy of the best routes, brought
/// up to date from the events it is given.
#[derive(Default)]
struct Mirror {
best: HashMap<IpPrefix, (PeerIndex, Arc<AttrSet>)>,
/// What was held before a reset and has not been heard of since.
stale: HashMap<IpPrefix, (PeerIndex, Arc<AttrSet>)>,
/// For a follower of all routes: every route of every prefix, as
/// (peer, path identifier, attributes, whether best).
routes: HashMap<IpPrefix, Vec<Held>>,
resets: usize,
synced: usize,
best_events: usize,
routes_events: usize,
withdrawn_events: usize,
}
/// A route as a mirror holds it. The attributes are held, so that the
/// address used to compare them stays theirs.
type Held = (PeerIndex, u32, Arc<AttrSet>, bool);
fn same_routes(a: &[Held], b: &[Held]) -> bool {
a.len() == b.len()
&& a.iter()
.zip(b)
.all(|(a, b)| (a.0, a.1, a.3) == (b.0, b.1, b.3) && Arc::ptr_eq(&a.2, &b.2))
}
fn sorted(mut routes: Vec<Held>) -> Vec<Held> {
routes.sort_by_key(|(peer, path_id, _, _)| (*peer, *path_id));
routes
}
impl Mirror {
fn apply(&mut self, event: Event<'_>) {
match event {
Event::Reset => {
self.resets += 1;
self.stale = std::mem::take(&mut self.best);
self.routes.clear();
}
Event::Routes { prefix, routes } => {
self.routes_events += 1;
assert!(!routes.is_empty(), "a prefix with no routes is withdrawn");
self.routes.insert(
prefix,
sorted(
routes
.iter()
.map(|r| (r.peer, r.path_id, Arc::clone(r.attrs), r.best))
.collect(),
),
);
}
Event::Best {
prefix,
peer,
attrs,
..
} => {
self.best_events += 1;
self.stale.remove(&prefix);
self.best.insert(prefix, (peer, Arc::clone(attrs)));
}
Event::Withdrawn { prefix } => {
self.withdrawn_events += 1;
self.best.remove(&prefix);
self.routes.remove(&prefix);
}
Event::Synced => {
self.synced += 1;
self.stale.clear();
}
}
}
/// Reads until there is nothing more, in steps of `budget`.
fn catch_up(&mut self, reader: &mut Reader, budget: usize) {
while !reader.poll(budget, |event| self.apply(event)).caught_up {}
}
#[track_caller]
fn assert_matches(&self, table: &RibTable) {
let mut expected = HashMap::new();
table.for_each_best(|prefix, peer, attrs| {
expected.insert(*prefix, (peer, Arc::clone(attrs)));
});
assert_eq!(
self.best.len(),
expected.len(),
"the mirror has {} prefixes, the table {}",
self.best.len(),
expected.len()
);
for (prefix, (peer, attrs)) in &expected {
let Some((mirrored_peer, mirrored_attrs)) = self.best.get(prefix) else {
panic!("{prefix} is in the table and not in the mirror");
};
assert_eq!(mirrored_peer, peer, "{prefix}: from the wrong peer");
assert!(
Arc::ptr_eq(mirrored_attrs, attrs),
"{prefix}: wrong attributes"
);
}
}
}
impl Mirror {
/// For the mirror of a follower of all routes: it holds every route in
/// the table, and knows which is best.
#[track_caller]
fn assert_has_all_routes_of(&self, table: &RibTable) {
let mut expected: HashMap<IpPrefix, Vec<Held>> = HashMap::new();
table.for_each_route(|r| {
expected.entry(r.prefix).or_default().push((
r.peer,
r.path_id,
Arc::clone(r.attrs),
r.best,
));
});
assert_eq!(self.routes.len(), expected.len(), "prefixes");
for (prefix, routes) in expected {
let mirrored = self
.routes
.get(&prefix)
.unwrap_or_else(|| panic!("{prefix} is in the table and not in the mirror"));
assert!(
same_routes(mirrored, &sorted(routes)),
"{prefix}: the mirror's routes are not the table's"
);
}
}
}
fn nth_prefix(n: u32) -> IpPrefix {
IpPrefix::new(IpVersion::Ipv4, (n << 8).to_be_bytes(), 24).unwrap()
}
#[test]
fn a_new_reader_is_given_the_table_and_then_the_changes() {
let fixture = Fixture::new(&[external(1), external(2)]);
let (a, b) = (prefix("10.0.0.0/8"), prefix("2001:db8::/32"));
fixture.announce(0, &a, &Route::via(&[1, 2]));
fixture.announce(0, &b, &Route::via(&[1, 2]));
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
let mut events = Vec::new();
let progress = reader.poll(100, |event| {
events.push(
format!("{event:?}")
.split([' ', '{'])
.next()
.unwrap()
.to_string(),
);
mirror.apply(event);
});
assert!(progress.caught_up);
assert_eq!(events, ["Reset", "Best", "Best", "Synced"]);
mirror.assert_matches(&fixture.table);
// Nothing has happened: nothing to read.
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
// A change that does not change the best route is not an event.
fixture.announce(1, &a, &Route::via(&[3, 4, 5]));
assert_eq!(reader.poll(100, |_| panic!("no event expected")).events, 0);
fixture.announce(1, &a, &Route::via(&[3]));
fixture.withdraw(0, &b);
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (3, 1));
assert_eq!((mirror.resets, mirror.synced), (1, 1));
mirror.assert_matches(&fixture.table);
}
#[test]
fn a_reader_from_now_is_given_the_changes_and_not_the_table() {
let fixture = Fixture::new(&[external(1)]);
let (old, new) = (prefix("10.0.0.0/8"), prefix("10.1.0.0/16"));
fixture.announce(0, &old, &Route::via(&[1]));
let mut reader = fixture.table.reader_from_now();
assert_eq!(
reader
.poll(100, |event| panic!("unexpected {event:?}"))
.events,
0
);
fixture.announce(0, &new, &Route::via(&[1]));
fixture.withdraw(0, &old);
let (mut best, mut withdrawn) = (Vec::new(), Vec::new());
reader.poll(100, |event| match event {
Event::Best { prefix, .. } => best.push(prefix),
Event::Withdrawn { prefix } => withdrawn.push(prefix),
other => panic!("unexpected {other:?}"),
});
assert_eq!((best, withdrawn), (vec![new], vec![old]));
}
/// A change to a route that is not the best is nothing to a follower of
/// the best routes, and is what a follower of all routes is there for.
#[test]
fn a_follower_of_all_routes_is_told_of_every_change() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
fixture.announce(0, &p, &Route::via(&[1]));
let mut best = fixture.table.reader();
let mut all = fixture.table.reader_of_all_routes();
let (mut best_mirror, mut all_mirror) = (Mirror::default(), Mirror::default());
best_mirror.catch_up(&mut best, 100);
all_mirror.catch_up(&mut all, 100);
assert_eq!((best_mirror.best_events, all_mirror.routes_events), (1, 1));
// A worse route from another peer, then two more from that peer under
// other path identifiers, one of them changed, one withdrawn.
fixture.announce(1, &p, &Route::via(&[2, 3, 4]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 3, 5]));
fixture.announce_path(1, &p, 8, &Route::via(&[2, 3, 6]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 9, 5]));
fixture.withdraw_path(1, &p, 8);
assert_eq!(
best.poll(100, |event| panic!("unexpected {event:?}"))
.events,
0
);
all_mirror.catch_up(&mut all, 100);
// All of it between two reads is one event, with the routes as they
// are now.
assert_eq!(all_mirror.routes_events, 2);
assert_eq!(all_mirror.routes[&p].len(), 3);
all_mirror.assert_has_all_routes_of(&fixture.table);
// The last route going is a withdrawal for both.
fixture.table.remove_peer(fixture.peers[1]);
fixture.withdraw(0, &p);
best_mirror.catch_up(&mut best, 100);
all_mirror.catch_up(&mut all, 100);
assert_eq!(
(best_mirror.withdrawn_events, all_mirror.withdrawn_events),
(1, 1)
);
all_mirror.assert_has_all_routes_of(&fixture.table);
// Nobody following all routes: that journal keeps nothing. A change
// that is not of the best route is then recorded nowhere.
drop(all);
fixture.announce(0, &p, &Route::via(&[1]));
fixture.announce(1, &p, &Route::via(&[2, 3]));
let before = fixture.table.journal_len();
fixture.announce(1, &p, &Route::via(&[2, 4]));
fixture.announce_path(1, &p, 7, &Route::via(&[2, 5]));
assert_eq!(fixture.table.journal_len(), before);
}
/// A follower wants what the table has now, not how it got there.
#[test]
fn many_changes_to_a_prefix_between_reads_are_one_event() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
for round in 0..50 {
fixture.announce(0, &p, &Route::via(&[1, 2, 3]));
fixture.announce(1, &p, &Route::via(&[4, 5]));
fixture.withdraw(1, &p);
if round % 2 == 0 {
fixture.withdraw(0, &p);
}
}
// It ended with peer 0's route in place.
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (1, 0));
mirror.assert_matches(&fixture.table);
// And a prefix that came and went again is, at most, a withdrawal.
let q = prefix("10.1.0.0/16");
for _ in 0..50 {
fixture.announce(0, &q, &Route::via(&[1]));
fixture.withdraw(0, &q);
}
mirror.catch_up(&mut reader, 100);
assert_eq!((mirror.best_events, mirror.withdrawn_events), (1, 1));
mirror.assert_matches(&fixture.table);
}
#[test]
fn reading_in_small_steps_gives_the_same_as_reading_at_once() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..500 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1, n % 7 + 1]));
}
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 7);
mirror.assert_matches(&fixture.table);
for n in 0..500 {
if n % 3 == 0 {
fixture.withdraw(0, &nth_prefix(n));
} else {
fixture.announce(0, &nth_prefix(n), &Route::via(&[2, n % 5 + 1]));
}
}
let mut steps = 0;
while !reader.poll(7, |event| mirror.apply(event)).caught_up {
steps += 1;
}
assert!(steps > 50, "the budget was kept to: {steps} steps");
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.resets, 1);
}
#[test]
fn with_no_reader_nothing_is_kept() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..5000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
fixture.withdraw(0, &nth_prefix(n));
}
assert_eq!(fixture.table.journal_len(), 0);
// And what a reader was holding back goes when the reader does.
let reader = fixture.table.reader();
for n in 0..100 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
assert_eq!(fixture.table.journal_len(), 100);
drop(reader);
assert_eq!(fixture.table.journal_len(), 0);
}
/// A reader that stops reading must not make the table keep a growing
/// journal for it. It is the one that pays: it is given the table again.
#[test]
fn a_reader_that_stops_reading_is_reset_and_holds_nothing_up() {
// One shard, so that the journal's length is that shard's.
let fixture = Fixture::on(RibTable::new(1), &[external(1)]);
fixture.announce(0, &prefix("192.0.2.0/24"), &Route::via(&[1]));
let mut stuck = fixture.table.reader();
let mut stuck_mirror = Mirror::default();
stuck_mirror.catch_up(&mut stuck, 100);
let mut diligent = fixture.table.reader();
let mut diligent_mirror = Mirror::default();
// Prefixes come and go, each a different one, so no entry is made
// dead by a later one for the same prefix.
let limits = Limits::default();
let mut longest = 0;
for n in 0..20_000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
fixture.withdraw(0, &nth_prefix(n));
longest = longest.max(fixture.table.journal_len());
if n % 100 == 0 {
diligent_mirror.catch_up(&mut diligent, 1000);
}
}
assert!(
longest <= limits.tidy_at.max(2 * (1 + limits.slack)) + 2,
"the journal reached {longest} entries"
);
// The one that kept up never noticed.
diligent_mirror.catch_up(&mut diligent, 1000);
assert_eq!(
diligent_mirror.resets, 1,
"only the reset every reader starts with"
);
diligent_mirror.assert_matches(&fixture.table);
// The one that did not is told, and ends up right all the same.
stuck_mirror.catch_up(&mut stuck, 1000);
assert_eq!(stuck_mirror.resets, 2);
assert_eq!(stuck_mirror.synced, 2);
stuck_mirror.assert_matches(&fixture.table);
}
/// When a peer that was the only source of most prefixes goes, more
/// prefixes disappear at once than are left. The journal still comes down
/// to its bound, and a reader ends up with what the table has.
#[test]
fn many_prefixes_going_at_once_leaves_the_journal_bounded() {
let fixture = Fixture::on(RibTable::new(1), &[external(1), external(2)]);
for n in 0..5000 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
fixture.announce(1, &nth_prefix(0), &Route::via(&[2, 3]));
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 1000);
fixture.table.remove_peer(fixture.peers[0]);
// Some later change, as there always is, lets the journal be tidied.
for _ in 0..8 {
fixture.announce(1, &nth_prefix(1), &Route::via(&[2]));
fixture.withdraw(1, &nth_prefix(1));
}
let limits = Limits::default();
assert!(
fixture.table.journal_len() <= limits.tidy_at.max(2 * (1 + limits.slack)) + 2,
"the journal holds {} entries",
fixture.table.journal_len()
);
mirror.catch_up(&mut reader, 1000);
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.best.len(), 1);
}
/// Readers are woken to read. An announcement that changes nothing, or
/// nothing a reader follows, writes nothing for them and must not wake
/// them: each would look through every shard to find that out.
#[test]
fn readers_are_not_woken_when_nothing_was_written_for_them() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let version = || fixture.table.version.load(Ordering::Acquire);
// Nobody reading: nobody to wake.
fixture.announce(0, &p, &Route::via(&[1]));
assert_eq!(version(), 0);
let _best = fixture.table.reader();
let before = version();
// The same route again.
fixture.announce(0, &p, &Route::via(&[1]));
assert_eq!(version(), before);
// A worse route from another peer: the best route stands.
fixture.announce(1, &p, &Route::via(&[2, 3, 4]));
assert_eq!(version(), before);
// A better one.
fixture.announce(1, &p, &Route::via(&[]));
assert_eq!(version(), before + 1);
// A follower of all routes is told of any change to a route.
let _all = fixture.table.reader_of_all_routes();
let before = version();
fixture.announce(0, &p, &Route::via(&[1, 1, 1]));
assert_eq!(version(), before + 1);
fixture.announce(0, &p, &Route::via(&[1, 1, 1]));
assert_eq!(version(), before + 1);
}
/// A budget of nothing would let a reader ask for ever without getting
/// anywhere; it is taken as the smallest that does.
#[test]
fn a_budget_of_nothing_still_makes_progress() {
let fixture = Fixture::new(&[external(1)]);
for n in 0..20 {
fixture.announce(0, &nth_prefix(n), &Route::via(&[1]));
}
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 0);
for n in 0..20 {
fixture.withdraw(0, &nth_prefix(n));
}
mirror.catch_up(&mut reader, 0);
mirror.assert_matches(&fixture.table);
assert!(mirror.best.is_empty());
}
#[tokio::test]
async fn a_waiting_reader_is_woken_by_a_change_of_best_route() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
// Not caught up yet: no waiting.
reader.changed().await;
mirror.catch_up(&mut reader, 100);
let waiting = tokio::time::timeout(std::time::Duration::from_millis(50), reader.changed());
assert!(waiting.await.is_err(), "nothing changed, so it waits");
let table = Arc::clone(&fixture.table);
let attrs = table
.attrs()
.intern(&Route::via(&[1]).attributes(), None)
.unwrap();
let peer = fixture.peers[0];
let writer = tokio::spawn(async move {
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
table.announce(peer, &attrs, &[p]);
});
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("woken by the announcement");
writer.await.unwrap();
mirror.catch_up(&mut reader, 100);
mirror.assert_matches(&fixture.table);
// A change made before the wait began is not missed either.
fixture.withdraw(0, &p);
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("the change was already there");
}
/// Writers on several threads, a reader on another, reading while they
/// write. When the writers are done and the reader has caught up, it has
/// what the table has.
#[test]
fn a_reader_keeps_up_with_writers_on_other_threads() {
const WRITERS: usize = 4;
const PREFIXES: u32 = 300;
let infos: Vec<PeerInfo> = (0..WRITERS).map(|i| external(i as u8 + 1)).collect();
// Small journals, so that tidying and falling behind happen too.
let fixture = Fixture::on(
RibTable::with_journal_limits(
8,
Limits {
tidy_at: 16,
slack: 4,
},
),
&infos,
);
let table = &fixture.table;
let done = AtomicBool::new(false);
let mut mirror = Mirror::default();
std::thread::scope(|scope| {
let writers: Vec<_> = (0..WRITERS)
.map(|writer| {
let peer = fixture.peers[writer];
scope.spawn(move || {
// A simple generator; each writer its own sequence.
let mut state = 0x9e37_79b9_7f4a_7c15u64.wrapping_mul(writer as u64 + 1);
let mut next = || {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
state
};
for _ in 0..30_000 {
let p = nth_prefix((next() % u64::from(PREFIXES)) as u32);
if next() % 3 == 0 {
table.withdraw(peer, &[p]);
} else {
let route = Route {
med: Some((next() % 3) as u32),
..Route::via(
&[(next() % 3) as u32 + 1, 9][..(next() % 2 + 1) as usize],
)
};
let attrs = table.attrs().intern(&route.attributes(), None).unwrap();
table.announce(peer, &attrs, &[p]);
}
}
})
})
.collect();
let mirror = &mut mirror;
let done = &done;
let reading = scope.spawn(move || {
let mut reader = table.reader();
while !done.load(Ordering::Acquire) {
reader.poll(50, |event| mirror.apply(event));
}
mirror.catch_up(&mut reader, 50);
});
for writer in writers {
writer.join().unwrap();
}
done.store(true, Ordering::Release);
reading.join().unwrap();
});
mirror.assert_matches(table);
assert!(mirror.best_events > 0);
}
// ---------------------------------------------------------------------------
// Readers against the table, under any interleaving
// ---------------------------------------------------------------------------
#[derive(Debug, Clone)]
enum Op {
Announce {
peer: usize,
prefix: usize,
path_id: u32,
route: Route,
},
Withdraw {
peer: usize,
prefix: usize,
path_id: u32,
},
Flap {
peer: usize,
},
Poll {
reader: usize,
budget: usize,
},
/// The reader is dropped and a new one takes its place.
Replace {
reader: usize,
},
}
const PEERS: usize = 4;
const PREFIXES: usize = 12;
// The first three follow the best routes, the others all routes.
const READERS: usize = 5;
const OF_BEST: usize = 3;
fn op() -> impl Strategy<Value = Op> {
prop_oneof![
8 => (0..PEERS, 0..PREFIXES, 0u32..3, route()).prop_map(
|(peer, prefix, path_id, route)| Op::Announce { peer, prefix, path_id, route }
),
5 => (0..PEERS, 0..PREFIXES, 0u32..3)
.prop_map(|(peer, prefix, path_id)| Op::Withdraw { peer, prefix, path_id }),
1 => (0..PEERS).prop_map(|peer| Op::Flap { peer }),
// Reader 0 is polled rarely: it is the one that falls behind.
4 => (1..READERS, 1usize..6).prop_map(|(reader, budget)| Op::Poll { reader, budget }),
1 => (0..READERS, 1usize..6).prop_map(|(reader, budget)| Op::Poll { reader, budget }),
1 => (1..READERS).prop_map(|reader| Op::Replace { reader }),
]
}
proptest! {
#![proptest_config(ProptestConfig::with_cases(1500))]
/// However changes and reads are interleaved, and however small the
/// journals: a reader that has caught up holds exactly the table's best
/// routes, and every reader gets there in the end.
#[test]
fn a_reader_that_has_caught_up_has_what_the_table_has(
ops in proptest::collection::vec(op(), 1..150),
shards in 1usize..4,
) {
let infos: Vec<PeerInfo> = (0..PEERS)
.map(|i| if i == 0 { internal(9) } else { external(i as u8) })
.collect();
// Journals that are tidied, and found too long, almost at once.
let limits = Limits { tidy_at: 3, slack: 1 };
let mut fixture = Fixture::on(RibTable::with_journal_limits(shards, limits), &infos);
let prefixes: Vec<IpPrefix> = (0..PREFIXES as u32).map(nth_prefix).collect();
let new_reader = |table: &Arc<RibTable>, index: usize| {
if index < OF_BEST { table.reader() } else { table.reader_of_all_routes() }
};
let check = |index: usize, mirror: &Mirror, table: &RibTable| {
if index < OF_BEST {
mirror.assert_matches(table);
} else {
mirror.assert_has_all_routes_of(table);
}
};
let mut readers: Vec<(Reader, Mirror)> = (0..READERS)
.map(|index| (new_reader(&fixture.table, index), Mirror::default()))
.collect();
for op in ops {
match op {
Op::Announce { peer, prefix, path_id, route } => {
fixture.announce_path(peer, &prefixes[prefix], path_id, &route);
}
Op::Withdraw { peer, prefix, path_id } => {
fixture.withdraw_path(peer, &prefixes[prefix], path_id);
}
Op::Flap { peer } => {
fixture.table.remove_peer(fixture.peers[peer]);
fixture.peers[peer] = fixture.table.add_peer(infos[peer]).unwrap();
}
Op::Poll { reader: index, budget } => {
let (reader, mirror) = &mut readers[index];
let progress = reader.poll(budget, |event| mirror.apply(event));
if progress.caught_up {
check(index, mirror, &fixture.table);
}
}
Op::Replace { reader } => {
readers[reader] = (new_reader(&fixture.table, reader), Mirror::default());
}
}
// Never much longer than the table, whatever the readers do.
// Two journals to a shard.
let bound = 2 * shards * (2 * (PREFIXES + limits.slack) + limits.tidy_at + PREFIXES);
prop_assert!(fixture.table.journal_len() <= bound);
}
for (index, (reader, mirror)) in readers.iter_mut().enumerate() {
mirror.catch_up(reader, 5);
check(index, mirror, &fixture.table);
prop_assert_eq!(mirror.resets, mirror.synced);
}
}
}
crates/rib/src/table/mod.rs
+728
View File
@@ -0,0 +1,728 @@
//! The routing table: every route every peer has announced, and for each
//! prefix the one chosen as best.
//!
//! This is the Adj-RIBs-In and the Loc-RIB of RFC 4271 §3.2 in one
//! structure, organised by prefix: a prefix's row holds the routes for it
//! from all peers, side by side, and which of them is best. Choosing again
//! after a change looks at one row and nothing else.
//!
//! It is built for a table of hundreds of full feeds, hundreds of millions
//! of routes, taken in on many cores at once:
//!
//! - **A route is small**: sixteen bytes. A pointer to its attribute set,
//! which is shared with every other route that has the same attributes
//! ([`attrs`]), and one word that holds the peer it is from and the first
//! steps of route selection as a single number to compare.
//! - **The table is in shards**, by hash of the prefix, each behind its own
//! lock. A session applies an UPDATE itself, on its own thread, taking
//! the locks of the few shards its prefixes fall in. There is no queue in
//! front of the table and no thread that owns it; sessions that touch
//! different shards do not wait for each other, and one that has to wait
//! is slowed down, which is the back-pressure that is wanted.
//! - **Readers do not stop the table.** Reading takes a shard's lock for
//! reading, one shard at a time, and looks at the rows in place.
//! - **Followers go at their own pace.** What acts on the best routes reads
//! the changes from a [`journal`], and a slow reader holds nothing up.
pub mod attrs;
mod journal;
mod select;
#[cfg(test)]
mod tests;
use std::hash::BuildHasher;
use std::net::{IpAddr, Ipv4Addr};
use std::sync::atomic::{AtomicU64, AtomicUsize};
use std::sync::{Arc, Mutex, PoisonError, RwLock, RwLockReadGuard, RwLockWriteGuard};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use foldhash::fast::RandomState;
use hashbrown::HashTable;
use hashbrown::hash_table::Entry;
use tokio::sync::Notify;
pub use attrs::{AttrSet, AttrStore, SelectionKey};
pub use journal::{Event, Progress, Reader, Routes};
/// What route selection needs to know about a peer.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PeerInfo {
pub asn: u32,
/// The peer's BGP Identifier.
pub router_id: Ipv4Addr,
pub address: IpAddr,
/// Whether the peer is in our own AS.
pub ibgp: bool,
}
/// A peer's number in a table, given out by [`RibTable::add_peer`]. Small,
/// because every route carries one.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct PeerIndex(u16);
impl PeerIndex {
pub fn as_usize(self) -> usize {
self.0.into()
}
}
/// The LOCAL_PREF of a route that does not carry one: every route from an
/// external peer, until import policy says otherwise.
pub const DEFAULT_LOCAL_PREF: u32 = 100;
/// What a change to the table did.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct Outcome {
/// Routes that were not in the table before.
pub added: usize,
/// Routes that took the place of one from the same peer.
pub replaced: usize,
/// Routes that were taken out.
pub removed: usize,
/// Prefixes whose best route is now another one, or none.
pub best_changed: usize,
}
impl std::ops::AddAssign for Outcome {
fn add_assign(&mut self, other: Self) {
self.added += other.added;
self.replaced += other.replaced;
self.removed += other.removed;
self.best_changed += other.best_changed;
}
}
/// One route of a prefix: sixteen bytes.
struct Path {
/// Two things in one word. The high 48 bits are the first steps of
/// route selection as one number, the larger the better; see
/// [`select::preference`]. The low 16 are the peer the route is from.
word: u64,
attrs: Arc<AttrSet>,
}
impl Path {
fn new(preference: u64, peer: PeerIndex, attrs: &Arc<AttrSet>) -> Self {
Self {
word: (preference << 16) | u64::from(peer.0),
attrs: Arc::clone(attrs),
}
}
fn preference(&self) -> u64 {
self.word >> 16
}
fn peer(&self) -> PeerIndex {
PeerIndex(self.word as u16)
}
}
/// Everything about one prefix.
struct Row {
prefix: IpPrefix,
/// Index of the best route in `paths`. A row always has at least one
/// route, and so always a best one; and the best one always has the
/// highest [`Path::preference`] in the row, which is what lets most
/// changes be settled without looking at the other routes
/// ([`select`]).
best: u32,
/// The sequence numbers of the latest change of this row's best route
/// and of any of its routes, by which the journals tell their entries
/// for it that still say something from those that do not.
seqs: journal::Seqs,
/// The routes, in one run of memory: finding a peer's route and
/// choosing the best are scans of it. One from each peer that has
/// announced the prefix, or several from a peer that sends more than one
/// (ADD-PATH, RFC 7911).
paths: Vec<Path>,
/// The path identifier of each route in `paths`, for telling apart the
/// routes of a peer that sends several. Empty while every route has
/// the identifier zero, which is all routes of peers that send one
/// each: they pay nothing for this.
path_ids: Vec<u32>,
}
impl Row {
fn new(prefix: IpPrefix, path: Path, path_id: u32, seqs: journal::Seqs) -> Self {
Self {
prefix,
best: 0,
seqs,
paths: vec![path],
path_ids: if path_id == 0 { vec![] } else { vec![path_id] },
}
}
fn path_id(&self, index: usize) -> u32 {
self.path_ids.get(index).copied().unwrap_or(0)
}
/// Where the route from `peer` with the identifier `path_id` is.
fn position(&self, peer: PeerIndex, path_id: u32) -> Option<usize> {
self.paths
.iter()
.enumerate()
.position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id)
}
/// Where a route from `peer` is, whatever its identifier.
fn position_of_peer(&self, peer: PeerIndex) -> Option<usize> {
self.paths.iter().position(|path| path.peer() == peer)
}
fn push(&mut self, path: Path, path_id: u32) {
if path_id != 0 && self.path_ids.is_empty() {
self.path_ids = vec![0; self.paths.len()];
}
push_tight(&mut self.paths, path);
if !self.path_ids.is_empty() {
push_tight(&mut self.path_ids, path_id);
}
}
/// Takes out the route at `at`; the last route moves into its place.
fn swap_remove(&mut self, at: usize) -> Path {
if !self.path_ids.is_empty() {
self.path_ids.swap_remove(at);
}
self.paths.swap_remove(at)
}
fn best(&self) -> usize {
self.best as usize
}
fn set_best(&mut self, index: usize) {
// An index into `paths`, which cannot have that many: each is
// sixteen bytes.
self.best = index as u32;
}
fn route(&self, index: usize) -> RouteRef<'_> {
let path = &self.paths[index];
RouteRef {
prefix: self.prefix,
peer: path.peer(),
path_id: self.path_id(index),
attrs: &path.attrs,
best: index == self.best(),
}
}
fn routes(&self) -> impl Iterator<Item = RouteRef<'_>> {
(0..self.paths.len()).map(|index| self.route(index))
}
}
/// A route in the table, looked at in place.
#[derive(Debug, Clone, Copy)]
pub struct RouteRef<'a> {
pub prefix: IpPrefix,
/// The peer the route is from.
pub peer: PeerIndex,
/// The peer's identifier for the route: zero unless the peer sends
/// several routes for a prefix (ADD-PATH, RFC 7911).
pub path_id: u32,
pub attrs: &'a Arc<AttrSet>,
/// Whether it is the route chosen for its prefix.
pub best: bool,
}
impl RouteRef<'_> {
/// The route, to keep.
pub fn to_owned(&self) -> Route {
Route {
prefix: self.prefix,
peer: self.peer,
path_id: self.path_id,
attrs: Arc::clone(self.attrs),
best: self.best,
}
}
}
/// A route taken out of the table to keep; see [`RouteRef`].
#[derive(Debug, Clone)]
pub struct Route {
pub prefix: IpPrefix,
pub peer: PeerIndex,
pub path_id: u32,
pub attrs: Arc<AttrSet>,
pub best: bool,
}
struct Shard {
rows: HashTable<Row>,
routes: usize,
/// A copy of the peer list, so that route selection reads it under the
/// lock it already holds.
peers: Vec<Option<PeerInfo>>,
hasher: RandomState,
/// Reused between selections.
scratch: select::Scratch,
/// Which prefixes have changed, for those who follow.
journals: journal::Journals,
}
/// The routing table. Shared between threads by reference; every method
/// takes `&self`.
pub struct RibTable {
shards: Box<[RwLock<Shard>]>,
hasher: RandomState,
attrs: AttrStore,
/// Which peer numbers are in use. Changing the peer list also takes
/// every shard's lock in turn; this serialises that.
peers: Mutex<Vec<Option<PeerInfo>>>,
/// Which reader slots are in use; see [`journal`].
reader_slots: Mutex<Vec<bool>>,
/// How many readers there are, to skip waking them when there are none.
readers: AtomicUsize,
/// Goes up whenever something is written to a journal that has
/// readers.
version: AtomicU64,
wake: Notify,
}
/// No more peers can be added: every [`PeerIndex`] is in use.
#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]
#[error("the table has no room for another peer")]
pub struct TooManyPeers;
impl RibTable {
/// A table in `shards` shards. More shards let more sessions write at
/// once; a few times the number of cores is plenty.
pub fn new(shards: usize) -> Self {
Self::with_journal_limits(shards, journal::Limits::default())
}
/// A table whose journals are tidied, and found too long, at the given
/// sizes. Tests use small ones to get there quickly.
fn with_journal_limits(shards: usize, limits: journal::Limits) -> Self {
let hasher = RandomState::default();
Self {
shards: (0..shards.max(1))
.map(|_| {
RwLock::new(Shard {
rows: HashTable::new(),
routes: 0,
peers: Vec::new(),
hasher: hasher.clone(),
scratch: select::Scratch::default(),
journals: journal::Journals::new(limits),
})
})
.collect(),
hasher,
attrs: AttrStore::new(),
peers: Mutex::default(),
reader_slots: Mutex::default(),
readers: AtomicUsize::new(0),
version: AtomicU64::new(0),
wake: Notify::new(),
}
}
/// Where attribute sets for this table's routes are made.
pub fn attrs(&self) -> &AttrStore {
&self.attrs
}
/// Makes room for a peer's routes and returns the number to announce
/// them under.
///
/// The number is the peer's until [`RibTable::remove_peer`] is called
/// with it, and may then be given to another. Whoever was given it must
/// not use it after that: the table cannot tell a late announcement
/// from one of the number's next owner.
pub fn add_peer(&self, info: PeerInfo) -> Result<PeerIndex, TooManyPeers> {
let mut peers = self.peers.lock().unwrap_or_else(PoisonError::into_inner);
let index = match peers.iter().position(Option::is_none) {
Some(free) => free,
None => {
// A peer number shares a word with other things; see `Path`.
if peers.len() > usize::from(u16::MAX) {
return Err(TooManyPeers);
}
peers.push(None);
peers.len() - 1
}
};
peers[index] = Some(info);
for shard in &self.shards {
let mut shard = write(shard);
if shard.peers.len() <= index {
shard.peers.resize(index + 1, None);
}
shard.peers[index] = Some(info);
}
Ok(PeerIndex(index as u16))
}
/// Takes out every route from `peer` and gives its number back for
/// reuse (RFC 4271 §8.2.2: the routes of a session that ended are
/// deleted).
///
/// This looks at every prefix in the table. The list of peers is not
/// held while it does, so other peers can come and go meanwhile.
pub fn remove_peer(&self, peer: PeerIndex) -> Outcome {
let mut outcome = Outcome::default();
let mut news = false;
for shard in &self.shards {
let mut shard = write(shard);
let before = shard.journals.mark();
outcome += shard.remove_peer(peer);
// From here on the shard takes no more routes from the peer.
if let Some(slot) = shard.peers.get_mut(peer.as_usize()) {
*slot = None;
}
shard.tidy_journals();
news |= shard.journals.has_news_since(before);
}
// Only now can the number be given out again: no shard knows it.
let mut peers = self.peers.lock().unwrap_or_else(PoisonError::into_inner);
if let Some(slot) = peers.get_mut(peer.as_usize()) {
*slot = None;
}
drop(peers);
if news {
self.wake_readers();
}
outcome
}
/// Announces `routes` from `peer`, all with the attributes `attrs`:
/// what one UPDATE does. A route is known by its prefix and path
/// identifier together ([`Nlri`]; a plain [`IpPrefix`] is the route
/// with identifier zero, which is every route of a peer that sends one
/// for each prefix). Announcing a route the peer had announced before
/// replaces it.
pub fn announce<N>(&self, peer: PeerIndex, attrs: &Arc<AttrSet>, routes: &[N]) -> Outcome
where
N: Copy + Into<Nlri>,
{
let mut outcome = Outcome::default();
self.by_shard(routes, |shard, placed| {
let Some(info) = shard.peers.get(peer.as_usize()).copied().flatten() else {
// The peer was removed while this was on its way.
return;
};
let preference = select::preference(attrs.key(), &info);
for (hash, nlri) in placed {
outcome += shard.announce(*hash, *nlri, peer, preference, attrs);
}
});
outcome
}
/// Withdraws `routes` of `peer`. Withdrawing a route that is not there
/// does nothing.
pub fn withdraw<N>(&self, peer: PeerIndex, routes: &[N]) -> Outcome
where
N: Copy + Into<Nlri>,
{
let mut outcome = Outcome::default();
self.by_shard(routes, |shard, placed| {
for (hash, nlri) in placed {
outcome += shard.withdraw(*hash, nlri, peer);
}
});
outcome
}
fn shard_of(&self, hash: u64) -> usize {
spread(hash, self.shards.len())
}
/// Calls `apply` once for each shard that some of `routes` fall in,
/// with that shard locked and those routes. Wakes the readers if a
/// journal one of them follows was written to.
fn by_shard<N>(&self, routes: &[N], mut apply: impl FnMut(&mut Shard, &[(u64, Nlri)]))
where
N: Copy + Into<Nlri>,
{
let Some(first) = routes.first() else {
return;
};
let hashed = |route: &N| {
let nlri: Nlri = (*route).into();
(self.hasher.hash_one(nlri.prefix), nlri)
};
// An UPDATE has a handful of prefixes; they are sorted on the stack.
const INLINE: usize = 32;
let mut inline = [hashed(first); INLINE];
let mut spilled = Vec::new();
let placed: &mut [(u64, Nlri)] = if routes.len() <= INLINE {
for (slot, route) in inline.iter_mut().zip(routes) {
*slot = hashed(route);
}
&mut inline[..routes.len()]
} else {
spilled.extend(routes.iter().map(hashed));
&mut spilled
};
placed.sort_unstable_by_key(|(hash, _)| self.shard_of(*hash));
let mut news = false;
for run in placed.chunk_by(|a, b| self.shard_of(a.0) == self.shard_of(b.0)) {
let mut shard = write(&self.shards[self.shard_of(run[0].0)]);
let before = shard.journals.mark();
apply(&mut shard, run);
shard.tidy_journals();
news |= shard.journals.has_news_since(before);
}
if news {
self.wake_readers();
}
}
/// How many routes the table holds, from all peers.
pub fn len(&self) -> usize {
self.shards.iter().map(|shard| read(shard).routes).sum()
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// How many prefixes have at least one route.
pub fn prefix_count(&self) -> usize {
self.shards.iter().map(|shard| read(shard).rows.len()).sum()
}
/// The best route for `prefix`: who it is from, and its attributes.
pub fn best(&self, prefix: &IpPrefix) -> Option<(PeerIndex, Arc<AttrSet>)> {
self.with_row(prefix, |row| {
let best = &row.paths[row.best()];
(best.peer(), Arc::clone(&best.attrs))
})
}
/// Every route for `prefix`.
pub fn routes(&self, prefix: &IpPrefix) -> Vec<Route> {
self.with_row(prefix, |row| row.routes().map(|r| r.to_owned()).collect())
.unwrap_or_default()
}
fn with_row<T>(&self, prefix: &IpPrefix, read_row: impl FnOnce(&Row) -> T) -> Option<T> {
let hash = self.hasher.hash_one(prefix);
let shard = read(&self.shards[self.shard_of(hash)]);
shard
.rows
.find(hash, |row| row.prefix == *prefix)
.map(read_row)
}
/// Calls `visit` with the best route of every prefix, in no particular
/// order. The table is not stopped: one shard at a time is held for
/// reading, so the routes seen are not all of one moment.
///
/// `visit` is called with a shard held, and must not call into the
/// table itself: taking the shard's lock again can wait for ever on a
/// writer that is waiting for this hold to end.
pub fn for_each_best(&self, mut visit: impl FnMut(&IpPrefix, PeerIndex, &Arc<AttrSet>)) {
for shard in &self.shards {
for row in &read(shard).rows {
let best = &row.paths[row.best()];
visit(&row.prefix, best.peer(), &best.attrs);
}
}
}
/// Up to `limit` routes: every route of as many prefixes as fit, in no
/// particular order. For showing some of a table that is too large to
/// show all of.
pub fn some_routes(&self, limit: usize) -> Vec<Route> {
let mut routes = Vec::new();
for shard in &self.shards {
for row in &read(shard).rows {
for route in row.routes() {
if routes.len() == limit {
return routes;
}
routes.push(route.to_owned());
}
}
}
routes
}
/// Calls `visit` with every route in the table. As
/// [`RibTable::for_each_best`]: the table is not stopped, and `visit`
/// must not call into it.
pub fn for_each_route(&self, mut visit: impl FnMut(RouteRef<'_>)) {
for shard in &self.shards {
for row in &read(shard).rows {
row.routes().for_each(&mut visit);
}
}
}
}
fn read(shard: &RwLock<Shard>) -> RwLockReadGuard<'_, Shard> {
// A panic under the lock cannot leave a row half changed in a way that
// is unsafe to read; the poison is ignored, as for the server's tables.
shard.read().unwrap_or_else(PoisonError::into_inner)
}
fn write(shard: &RwLock<Shard>) -> RwLockWriteGuard<'_, Shard> {
shard.write().unwrap_or_else(PoisonError::into_inner)
}
/// Picks one of `count` buckets by a hash, for spreading keys over shards
/// or lock stripes that each hold a hash table keyed by the same hash.
///
/// That table takes a key's slot from the low bits of the hash and, from
/// the top seven, the tag it compares before it looks at the key itself
/// (hashbrown). The bucket is taken from bits in between: taken from
/// either end, the keys of one bucket would all share those bits, and the
/// table inside would lose them.
fn spread(hash: u64, count: usize) -> usize {
let middle = (hash << 7) >> 32;
((middle * count as u64) >> 32) as usize
}
/// Adds to a vector without the doubling that `push` does: a row of a few
/// hundred routes should not hold room for as many again. Grows by a
/// quarter, which keeps adding cheap on average.
fn push_tight<T>(vec: &mut Vec<T>, value: T) {
if vec.len() == vec.capacity() {
vec.reserve_exact((vec.len() / 4).max(1));
}
vec.push(value);
}
impl Shard {
fn tidy_journals(&mut self) {
self.journals.tidy(&self.rows, &self.hasher);
}
fn announce(
&mut self,
hash: u64,
nlri: Nlri,
peer: PeerIndex,
preference: u64,
attrs: &Arc<AttrSet>,
) -> Outcome {
let Nlri { prefix, path_id } = nlri;
let hasher = &self.hasher;
let entry = self.rows.entry(
hash,
|row| row.prefix == prefix,
|row| hasher.hash_one(row.prefix),
);
let path = Path::new(preference, peer, attrs);
let row = match entry {
Entry::Vacant(vacant) => {
let seqs = self.journals.appeared(prefix);
vacant.insert(Row::new(prefix, path, path_id, seqs));
self.routes += 1;
return Outcome {
added: 1,
best_changed: 1,
..Outcome::default()
};
}
Entry::Occupied(occupied) => occupied.into_mut(),
};
let (outcome, best_changed) = match row.position(peer, path_id) {
None => {
row.push(path, path_id);
self.routes += 1;
let outcome = Outcome {
added: 1,
..Outcome::default()
};
(
outcome,
select::after_add(row, &self.peers, &mut self.scratch),
)
}
Some(at) => {
let before = std::mem::replace(&mut row.paths[at], path);
let outcome = Outcome {
replaced: 1,
..Outcome::default()
};
if Arc::ptr_eq(&before.attrs, attrs) && before.preference() == preference {
// The same route again: nothing changed at all.
return outcome;
}
let best_changed = select::after_replace(
row,
at,
before.preference(),
&self.peers,
&mut self.scratch,
);
(outcome, best_changed)
}
};
self.journals.changed(row, best_changed);
Outcome {
best_changed: best_changed.into(),
..outcome
}
}
fn withdraw(&mut self, hash: u64, nlri: &Nlri, peer: PeerIndex) -> Outcome {
let prefix = nlri.prefix;
let Ok(mut entry) = self.rows.find_entry(hash, |row| row.prefix == prefix) else {
return Outcome::default();
};
let row = entry.get_mut();
let Some(at) = row.position(peer, nlri.path_id) else {
return Outcome::default();
};
self.routes -= 1;
if row.paths.len() == 1 {
entry.remove();
self.journals.disappeared(prefix);
return Outcome {
removed: 1,
best_changed: 1,
..Outcome::default()
};
}
let best_changed = select::after_remove(row, at, &self.peers, &mut self.scratch);
self.journals.changed(row, best_changed);
Outcome {
removed: 1,
best_changed: best_changed.into(),
..Outcome::default()
}
}
fn remove_peer(&mut self, peer: PeerIndex) -> Outcome {
let mut outcome = Outcome::default();
let peers = &self.peers;
let scratch = &mut self.scratch;
let journals = &mut self.journals;
self.rows.retain(|row| {
let mut removed = 0;
let mut best_changed = false;
// A peer that sends several routes for a prefix has several
// here.
while let Some(at) = row.position_of_peer(peer) {
removed += 1;
if row.paths.len() == 1 {
outcome.removed += removed;
outcome.best_changed += 1;
journals.disappeared(row.prefix);
return false;
}
best_changed |= select::after_remove(row, at, peers, scratch);
}
if removed > 0 {
outcome.removed += removed;
outcome.best_changed += usize::from(best_changed);
journals.changed(row, best_changed);
}
true
});
self.routes -= outcome.removed;
outcome
}
}
crates/rib/src/table/select.rs
+249
View File
@@ -0,0 +1,249 @@
//! Choosing the best route for a prefix (RFC 4271 §9.1.2.2).
//!
//! The steps, in order. A route wins at the first step where it differs:
//!
//! 1. highest LOCAL_PREF;
//! 2. shortest AS path;
//! 3. lowest ORIGIN;
//! 4. lowest MULTI_EXIT_DISC, among routes from the same neighbouring AS;
//! 5. a route from an external peer over one from an internal peer;
//! 6. the peer with the lowest BGP Identifier;
//! 7. the peer with the lowest address.
//!
//! (The RFC's step between 5 and 6, the interior cost to the next hop, has
//! nothing to compare until there is an IGP to ask.)
//!
//! Steps 1 to 3 are put into one number per route when it is stored,
//! [`preference`]. Most of the time that number alone settles it: a new
//! route with a larger one than the best is the new best, with a smaller
//! one changes nothing, and neither needs a look at any other route. Only a
//! tie goes on to [`choose`], which looks at the whole row, and that begins
//! with a pass over the numbers to find the routes still in the running.
//!
//! [`after_add`], [`after_replace`] and [`after_remove`] are what the table
//! calls when a row has changed. Each says whether the best route is now
//! another, and takes the short way where there is one. They rely on, and
//! keep, one fact about a row: its best route has the highest preference
//! of all its routes.
use super::attrs::SelectionKey;
use super::{DEFAULT_LOCAL_PREF, PeerInfo, Row};
/// Steps 1 to 3 as one number of 48 bits: the larger, the better.
///
/// LOCAL_PREF fills the high 32 bits. Below it come the AS path length in
/// 12 bits and the ORIGIN in 2, each turned upside down so that shorter and
/// lower compare as larger. Twelve bits are enough for any path: a BGP
/// message of 4096 octets has room for about a thousand AS numbers.
pub(super) fn preference(key: &SelectionKey, peer: &PeerInfo) -> u64 {
// LOCAL_PREF is a matter between routers of one AS. A route from an
// external peer has the default until import policy gives it another.
let local_pref = if peer.ibgp {
key.local_pref.unwrap_or(DEFAULT_LOCAL_PREF)
} else {
DEFAULT_LOCAL_PREF
};
const LONGEST: u16 = (1 << 12) - 1;
(u64::from(local_pref) << 16)
| (u64::from(LONGEST - key.as_path_len.min(LONGEST)) << 4)
| (u64::from(3 - key.origin.min(3)) << 2)
}
/// Buffers that [`choose`] reuses.
#[derive(Default)]
pub(super) struct Scratch {
/// Routes still in the running: (neighbouring AS, MED, index in row).
candidates: Vec<(u32, u32, u32)>,
}
/// The best route changed after a route was added at the end of `row`?
pub(super) fn after_add(row: &mut Row, peers: &[Option<PeerInfo>], scratch: &mut Scratch) -> bool {
let added = row.paths.len() - 1;
let preference = row.paths[added].preference();
let best = row.paths[row.best()].preference();
// Better than the best in the first steps of selection is better than
// everything; worse than it changes nothing. A tie has to be settled by
// the later steps.
if preference > best {
row.set_best(added);
true
} else if preference < best {
false
} else if let Some(wins) = settles_tie(row, added, peers) {
if wins {
row.set_best(added);
}
wins
} else {
choose(row, peers, scratch)
}
}
/// The best route changed after the route at `at` was replaced by another
/// from the same peer? `before` is the preference of the route it replaced.
/// The two are known to differ.
pub(super) fn after_replace(
row: &mut Row,
at: usize,
before: u64,
peers: &[Option<PeerInfo>],
scratch: &mut Scratch,
) -> bool {
let preference = row.paths[at].preference();
if at == row.best() {
// The best route itself changed. If it got better in the first
// steps it is still the best, but it is a different route now.
if preference <= before {
choose(row, peers, scratch);
}
return true;
}
let best = row.paths[row.best()].preference();
if preference > best {
row.set_best(at);
true
} else if preference < best && before < best {
// It was not in the running and still is not.
false
} else {
choose(row, peers, scratch)
}
}
/// Takes the route at `at` out of `row`, which has others, and returns
/// whether the best route changed.
pub(super) fn after_remove(
row: &mut Row,
at: usize,
peers: &[Option<PeerInfo>],
scratch: &mut Scratch,
) -> bool {
let best = row.best();
let best_preference = row.paths[best].preference();
let last = row.paths.len() - 1;
let removed = row.swap_remove(at);
if at == best {
choose(row, peers, scratch);
return true;
}
// The last route moved into the gap.
if best == last {
row.set_best(at);
}
// A route that tied with the best in the first steps may have been
// what kept another from being best, by its MED.
removed.preference() == best_preference && choose(row, peers, scratch)
}
/// Chooses the best route of `row` from all its routes, records it in the
/// row, and returns whether it is another route than before.
///
/// The row must have at least one route.
fn choose(row: &mut Row, peers: &[Option<PeerInfo>], scratch: &mut Scratch) -> bool {
let before = row.best;
let top = row
.paths
.iter()
.map(|path| path.preference())
.max()
.unwrap_or(0);
let candidates = &mut scratch.candidates;
candidates.clear();
for (index, path) in row.paths.iter().enumerate() {
if path.preference() == top {
let key = path.attrs.key();
// A route without a MED is taken to have the lowest (RFC 4271
// §9.1.2.2 c).
candidates.push((key.neighbor_as, key.med.unwrap_or(0), index as u32));
}
}
// Step 4. MEDs are compared only between routes from the same
// neighbouring AS, so "lower MED" is not an order over all the routes,
// and comparing them pair by pair can go round in circles. Instead:
// within each neighbouring AS only the routes with its lowest MED stay
// in, and the steps after are an order over what is left.
candidates.sort_unstable();
let mut best: Option<(TieBreak, u32)> = None;
let mut group = None;
for &(neighbor_as, med, index) in candidates.iter() {
// Sorted by AS and then MED: the first of each AS has its lowest.
let lowest = match group {
Some((group_as, lowest)) if group_as == neighbor_as => lowest,
_ => {
group = Some((neighbor_as, med));
med
}
};
if med != lowest {
continue;
}
let tie_break = TieBreak::of(row, index as usize, peers);
if best.as_ref().is_none_or(|(least, _)| tie_break < *least) {
best = Some((tie_break, index));
}
}
row.best = best.map_or(0, |(_, index)| index);
row.best != before
}
/// For a route just added at `added` that ties with the best in steps 1 to
/// 3: whether it is the new best, if that can be said from the two routes
/// alone. `None` if the whole row has to be looked at.
///
/// It can be said when MED cannot knock either of them out. The new route
/// has the lowest MED there is, so nothing knocks it out. It knocks out
/// only routes of its own neighbouring AS with a higher MED, and if the
/// best is not one of those, the best is still the best of all the others.
/// So the better of the two is the best of all. This is the usual case:
/// most routes have no MED.
fn settles_tie(row: &Row, added: usize, peers: &[Option<PeerInfo>]) -> Option<bool> {
let new = row.paths[added].attrs.key();
let best = row.paths[row.best()].attrs.key();
if new.med.unwrap_or(0) != 0 {
return None;
}
if best.neighbor_as == new.neighbor_as && best.med.unwrap_or(0) != 0 {
return None;
}
Some(TieBreak::of(row, added, peers) < TieBreak::of(row, row.best(), peers))
}
/// Steps 5 to 7, then the peer's number, and then the path identifier for
/// a peer that sent several routes: so that the choice never depends on the
/// order routes arrived in. The least wins.
#[derive(PartialEq, Eq, PartialOrd, Ord)]
struct TieBreak {
internal: bool,
router_id: u32,
address: std::net::IpAddr,
peer: u16,
path_id: u32,
}
impl TieBreak {
fn of(row: &Row, index: usize, peers: &[Option<PeerInfo>]) -> Self {
let peer = row.paths[index].peer();
let path_id = row.path_id(index);
match peers.get(peer.as_usize()).copied().flatten() {
Some(info) => Self {
internal: info.ibgp,
router_id: info.router_id.into(),
address: info.address,
peer: peer.0,
path_id,
},
// Not expected: a peer is forgotten only once its routes are
// gone. If it happened, anything known would be preferred.
None => Self {
internal: true,
router_id: u32::MAX,
address: std::net::Ipv6Addr::from(u128::MAX).into(),
peer: peer.0,
path_id,
},
}
}
}
crates/rib/src/table/tests.rs
+759
View File
@@ -0,0 +1,759 @@
use std::collections::{BTreeMap, HashMap};
use std::net::{IpAddr, Ipv4Addr};
use std::sync::Arc;
use bgp_packet::message::{
AsPathAttribute, LocalPrefPathAttribute, MultiExitDiscPathAttribute, NextHopPathAttribute,
OriginPathAttribute, PathAttribute,
};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use proptest::prelude::*;
use super::*;
pub(super) fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// What a test says about a route; everything selection looks at.
#[derive(Debug, Clone, PartialEq, Eq)]
pub(super) struct Route {
pub(super) local_pref: Option<u32>,
pub(super) path: Vec<u32>,
pub(super) origin: u8,
pub(super) med: Option<u32>,
}
impl Route {
pub(super) fn via(path: &[u32]) -> Self {
Self {
local_pref: None,
path: path.to_vec(),
origin: 0,
med: None,
}
}
pub(super) fn attributes(&self) -> Vec<PathAttribute> {
let mut attributes = vec![
PathAttribute::Origin(match self.origin {
0 => OriginPathAttribute::IGP,
1 => OriginPathAttribute::EGP,
_ => OriginPathAttribute::INCOMPLETE,
}),
AsPathAttribute::from_asns(self.path.clone()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
];
attributes.extend(
self.med
.map(|med| PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(med))),
);
attributes.extend(
self.local_pref
.map(|lp| PathAttribute::LocalPref(LocalPrefPathAttribute(lp))),
);
attributes
}
}
pub(super) fn external(number: u8) -> PeerInfo {
PeerInfo {
asn: 65000 + u32::from(number),
router_id: Ipv4Addr::new(10, 0, 0, number),
address: IpAddr::from(Ipv4Addr::new(192, 0, 2, number)),
ibgp: false,
}
}
pub(super) fn internal(number: u8) -> PeerInfo {
PeerInfo {
asn: 64999,
ibgp: true,
..external(number)
}
}
/// A table and the peers added to it, for tests that speak of peers by the
/// order they were added in.
pub(super) struct Fixture {
pub(super) table: Arc<RibTable>,
pub(super) peers: Vec<PeerIndex>,
}
impl Fixture {
pub(super) fn new(peers: &[PeerInfo]) -> Self {
// More than one shard, so that a test's prefixes are spread.
Self::on(RibTable::new(4), peers)
}
pub(super) fn on(table: RibTable, peers: &[PeerInfo]) -> Self {
let table = Arc::new(table);
let peers = peers.iter().map(|p| table.add_peer(*p).unwrap()).collect();
Self { table, peers }
}
pub(super) fn announce(&self, peer: usize, prefix: &IpPrefix, route: &Route) -> Outcome {
let attrs = self
.table
.attrs()
.intern(&route.attributes(), None)
.unwrap();
self.table.announce(self.peers[peer], &attrs, &[*prefix])
}
pub(super) fn withdraw(&self, peer: usize, prefix: &IpPrefix) -> Outcome {
self.table.withdraw(self.peers[peer], &[*prefix])
}
/// As `announce`, from a peer that sends several routes for a prefix
/// and calls this one `path_id`.
pub(super) fn announce_path(
&self,
peer: usize,
prefix: &IpPrefix,
path_id: u32,
route: &Route,
) -> Outcome {
let attrs = self
.table
.attrs()
.intern(&route.attributes(), None)
.unwrap();
let nlri = Nlri {
prefix: *prefix,
path_id,
};
self.table.announce(self.peers[peer], &attrs, &[nlri])
}
pub(super) fn withdraw_path(&self, peer: usize, prefix: &IpPrefix, path_id: u32) -> Outcome {
let nlri = Nlri {
prefix: *prefix,
path_id,
};
self.table.withdraw(self.peers[peer], &[nlri])
}
/// The best route for `prefix`: which peer (by order added) it is from
/// and that peer's identifier for it.
pub(super) fn best_path(&self, prefix: &IpPrefix) -> Option<(usize, u32)> {
let best = self.table.routes(prefix).into_iter().find(|r| r.best)?;
let peer = self.peers.iter().position(|p| *p == best.peer)?;
Some((peer, best.path_id))
}
/// The routes held for `prefix`, as (peer, path identifier), in order.
pub(super) fn held(&self, prefix: &IpPrefix) -> Vec<(usize, u32)> {
let mut held: Vec<(usize, u32)> = self
.table
.routes(prefix)
.iter()
.map(|r| {
let peer = self.peers.iter().position(|p| *p == r.peer).unwrap();
(peer, r.path_id)
})
.collect();
held.sort_unstable();
held
}
/// Which peer (by order added) the best route for `prefix` is from.
fn best(&self, prefix: &IpPrefix) -> Option<usize> {
let (peer, _) = self.table.best(prefix)?;
self.peers.iter().position(|p| *p == peer)
}
}
#[test]
fn each_step_of_selection_decides_when_the_ones_before_tie() {
let p = prefix("10.0.0.0/8");
// Peer 0 has the lower identifier, so it wins whenever nothing else
// decides; each case makes peer 1's route better in one step.
let cases: [(&str, [PeerInfo; 2], Route, Route); 6] = [
(
"higher LOCAL_PREF, despite a longer path",
[internal(1), internal(2)],
Route {
local_pref: Some(100),
..Route::via(&[1])
},
Route {
local_pref: Some(200),
..Route::via(&[1, 2, 3])
},
),
(
"shorter AS path",
[external(1), external(2)],
Route::via(&[1, 2, 3]),
Route::via(&[4, 5]),
),
(
"lower ORIGIN",
[external(1), external(2)],
Route {
origin: 2,
..Route::via(&[1, 2])
},
Route {
origin: 0,
..Route::via(&[3, 4])
},
),
(
"lower MED from the same neighbouring AS",
[external(1), external(2)],
Route {
med: Some(20),
..Route::via(&[7, 2])
},
Route {
med: Some(10),
..Route::via(&[7, 3])
},
),
(
"no MED counts as the lowest",
[external(1), external(2)],
Route {
med: Some(1),
..Route::via(&[7, 2])
},
Route::via(&[7, 3]),
),
(
"external over internal",
[internal(1), external(2)],
Route::via(&[7, 2]),
Route::via(&[8, 3]),
),
];
for (case, peers, first, second) in cases {
// In both orders of arrival: the choice must not depend on it.
for order in [[0, 1], [1, 0]] {
let fixture = Fixture::new(&peers);
for peer in order {
fixture.announce(peer, &p, [&first, &second][peer]);
}
assert_eq!(fixture.best(&p), Some(1), "{case}, arriving {order:?}");
}
}
// And when nothing decides: the lowest identifier.
let fixture = Fixture::new(&[external(2), external(1)]);
fixture.announce(0, &p, &Route::via(&[1, 2]));
fixture.announce(1, &p, &Route::via(&[3, 4]));
assert_eq!(fixture.best(&p), Some(1));
}
/// MEDs are compared only within a neighbouring AS. Taken pair by pair
/// that is not an order: A beats B, B beats C, C beats A. Whatever the
/// order of arrival, the answer must be the same.
#[test]
fn med_is_compared_only_within_a_neighbouring_as_and_never_goes_in_circles() {
let p = prefix("10.0.0.0/8");
let routes = [
// From AS 1, MED 10, and the highest identifier.
Route {
med: Some(10),
..Route::via(&[1, 9])
},
// From AS 2: its MED is not comparable with the others'.
Route {
med: Some(50),
..Route::via(&[2, 9])
},
// From AS 1, MED 20: loses to the first on MED.
Route {
med: Some(20),
..Route::via(&[1, 8])
},
];
// Identifiers: the third route's peer lowest, then the second's, then
// the first's. Without MED the third would win; it is knocked out by
// the first, and of the two left the second has the lower identifier.
let peers = [external(3), external(2), external(1)];
for order in [
[0, 1, 2],
[0, 2, 1],
[1, 0, 2],
[1, 2, 0],
[2, 0, 1],
[2, 1, 0],
] {
let fixture = Fixture::new(&peers);
for peer in order {
fixture.announce(peer, &p, &routes[peer]);
}
assert_eq!(fixture.best(&p), Some(1), "arriving {order:?}");
}
// Take away the route that knocked the third out, and the third wins.
let fixture = Fixture::new(&peers);
for (peer, route) in routes.iter().enumerate() {
fixture.announce(peer, &p, route);
}
let outcome = fixture.withdraw(0, &p);
assert_eq!(outcome.best_changed, 1);
assert_eq!(fixture.best(&p), Some(2));
}
/// A route that is not the best can still decide which is: by its MED it
/// keeps another route of its neighbouring AS out. When it gets worse or
/// goes, that other route is back in the running. The shortcuts must not
/// skip this.
#[test]
fn a_route_that_was_keeping_another_out_matters_when_it_changes() {
let p = prefix("10.0.0.0/8");
// Peer 0: from AS 1 with the lowest identifier, but MED 1.
// Peer 1: from AS 2. Best while peer 0's route is kept out.
// Peer 2: from AS 1 with MED 0, which keeps peer 0's route out.
let peers = [external(1), external(2), external(3)];
let held_out = Route {
med: Some(1),
..Route::via(&[1, 9])
};
let best = Route::via(&[2, 9]);
let holding_out = Route {
med: Some(0),
..Route::via(&[1, 8])
};
let set_up = || {
let fixture = Fixture::new(&peers);
fixture.announce(0, &p, &held_out);
fixture.announce(1, &p, &best);
fixture.announce(2, &p, &holding_out);
assert_eq!(fixture.best(&p), Some(1));
fixture
};
// It is replaced by a route with a longer path.
let fixture = set_up();
let outcome = fixture.announce(2, &p, &Route::via(&[1, 8, 7]));
assert_eq!(fixture.best(&p), Some(0));
assert_eq!(outcome.best_changed, 1);
// It is withdrawn.
let fixture = set_up();
let outcome = fixture.withdraw(2, &p);
assert_eq!(fixture.best(&p), Some(0));
assert_eq!(outcome.best_changed, 1);
// Its peer goes away.
let fixture = set_up();
let outcome = fixture.table.remove_peer(fixture.peers[2]);
assert_eq!(fixture.best(&p), Some(0));
assert_eq!(outcome.best_changed, 1);
// Its MED gets worse than the one it was keeping out.
let fixture = set_up();
fixture.announce(
2,
&p,
&Route {
med: Some(5),
..Route::via(&[1, 8])
},
);
assert_eq!(fixture.best(&p), Some(0));
}
#[test]
fn what_a_change_did_is_reported() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
let long = Route::via(&[1, 2, 3]);
let short = Route::via(&[4]);
let first = fixture.announce(0, &p, &long);
assert_eq!((first.added, first.best_changed), (1, 1));
let worse = fixture.announce(1, &p, &Route::via(&[1, 2, 3, 4]));
assert_eq!((worse.added, worse.best_changed), (1, 0));
// The same route again changes nothing.
let again = fixture.announce(0, &p, &long);
assert_eq!((again.replaced, again.best_changed), (1, 0));
// The best route replaced by a better one is still a change of best.
let better = fixture.announce(0, &p, &short);
assert_eq!((better.replaced, better.best_changed), (1, 1));
assert_eq!((fixture.table.len(), fixture.table.prefix_count()), (2, 1));
assert_eq!(fixture.withdraw(1, &p).best_changed, 0);
let last = fixture.withdraw(0, &p);
assert_eq!((last.removed, last.best_changed), (1, 1));
assert_eq!(fixture.withdraw(0, &p), Outcome::default(), "nothing there");
assert!(fixture.table.is_empty());
assert_eq!(fixture.best(&p), None);
}
#[test]
fn removing_a_peer_takes_its_routes_and_frees_its_number() {
let fixture = Fixture::new(&[external(1), external(2)]);
let prefixes: Vec<IpPrefix> = (0..200u32)
.map(|i| {
IpPrefix::new(
bgp_primitives::constants::IpVersion::Ipv4,
(i << 8).to_be_bytes(),
24,
)
.unwrap()
})
.collect();
let attrs = |route: &Route| {
fixture
.table
.attrs()
.intern(&route.attributes(), None)
.unwrap()
};
// Peer 0 is best everywhere; peer 1 has only the first hundred.
fixture
.table
.announce(fixture.peers[0], &attrs(&Route::via(&[1])), &prefixes);
fixture.table.announce(
fixture.peers[1],
&attrs(&Route::via(&[2, 3])),
&prefixes[..100],
);
assert_eq!(fixture.table.len(), 300);
let outcome = fixture.table.remove_peer(fixture.peers[0]);
assert_eq!(outcome.removed, 200);
assert_eq!(outcome.best_changed, 200);
assert_eq!(
(fixture.table.len(), fixture.table.prefix_count()),
(100, 100)
);
assert_eq!(fixture.best(&prefixes[0]), Some(1));
assert_eq!(fixture.best(&prefixes[150]), None);
// Its routes are gone, so its attribute set can be dropped.
assert_eq!(fixture.table.attrs().sweep(), 1);
// And its number is given to the next peer.
assert_eq!(fixture.table.add_peer(external(3)), Ok(fixture.peers[0]));
}
#[test]
fn readers_see_every_route_and_which_is_best() {
let fixture = Fixture::new(&[external(1), external(2)]);
let (a, b) = (prefix("10.0.0.0/8"), prefix("2001:db8::/32"));
fixture.announce(0, &a, &Route::via(&[1, 2]));
fixture.announce(1, &a, &Route::via(&[3]));
fixture.announce(0, &b, &Route::via(&[1, 2]));
let mut best = Vec::new();
fixture
.table
.for_each_best(|prefix, peer, _| best.push((*prefix, peer)));
best.sort_by_key(|(prefix, _)| prefix.to_string());
assert_eq!(best, [(a, fixture.peers[1]), (b, fixture.peers[0])]);
let mut all = 0;
let mut marked_best = 0;
fixture.table.for_each_route(|route| {
all += 1;
marked_best += usize::from(route.best);
});
assert_eq!((all, marked_best), (3, 2));
let routes = fixture.table.routes(&a);
assert_eq!(routes.len(), 2);
assert!(
routes
.iter()
.any(|route| route.peer == fixture.peers[1] && route.best)
);
// Routes with the same attributes share one set.
let shared = &fixture.table.routes(&b)[0].attrs;
assert!(routes.iter().any(|route| Arc::ptr_eq(&route.attrs, shared)));
}
/// With ADD-PATH a peer sends several routes for one prefix and tells them
/// apart by a path identifier. Each is a route of its own: kept, replaced
/// and withdrawn by prefix and identifier together.
#[test]
fn a_peer_can_have_several_routes_for_a_prefix() {
let fixture = Fixture::new(&[external(1), external(2)]);
let p = prefix("10.0.0.0/8");
fixture.announce_path(0, &p, 1, &Route::via(&[1, 2, 3]));
let second = fixture.announce_path(0, &p, 2, &Route::via(&[1, 4]));
assert_eq!(
(second.added, second.replaced, second.best_changed),
(1, 0, 1)
);
fixture.announce(1, &p, &Route::via(&[2, 5, 6, 7]));
assert_eq!(fixture.held(&p), [(0, 1), (0, 2), (1, 0)]);
assert_eq!(fixture.best_path(&p), Some((0, 2)), "the shorter path");
assert_eq!((fixture.table.len(), fixture.table.prefix_count()), (3, 1));
// Announcing a path again replaces that path and no other.
let replaced = fixture.announce_path(0, &p, 1, &Route::via(&[1]));
assert_eq!((replaced.added, replaced.replaced), (0, 1));
assert_eq!(fixture.held(&p), [(0, 1), (0, 2), (1, 0)]);
assert_eq!(fixture.best_path(&p), Some((0, 1)));
// Withdrawing one leaves the peer's other.
assert_eq!(fixture.withdraw_path(0, &p, 1).removed, 1);
assert_eq!(fixture.held(&p), [(0, 2), (1, 0)]);
assert_eq!(fixture.best_path(&p), Some((0, 2)));
// A path that is not there is not withdrawn: not the same prefix
// under another identifier, and not the peer's plain route.
assert_eq!(fixture.withdraw_path(0, &p, 9), Outcome::default());
assert_eq!(fixture.withdraw(0, &p), Outcome::default());
// When the session ends, all of the peer's routes go.
fixture.announce_path(0, &p, 3, &Route::via(&[1, 8, 9]));
let outcome = fixture.table.remove_peer(fixture.peers[0]);
assert_eq!((outcome.removed, outcome.best_changed), (2, 1));
assert_eq!(fixture.held(&p), [(1, 0)]);
}
/// Two routes from one peer that nothing else tells apart: the lower path
/// identifier is chosen, whichever arrived first.
#[test]
fn equal_routes_from_one_peer_are_chosen_by_path_identifier() {
let p = prefix("10.0.0.0/8");
for order in [[5, 3], [3, 5]] {
let fixture = Fixture::new(&[external(1)]);
for path_id in order {
fixture.announce_path(0, &p, path_id, &Route::via(&[1, path_id]));
}
assert_eq!(fixture.best_path(&p), Some((0, 3)), "arriving {order:?}");
}
}
// ---------------------------------------------------------------------------
// The table against a model
// ---------------------------------------------------------------------------
/// Route selection written the plain way: all routes of a prefix, step by
/// step, with no shortcuts. The table takes shortcuts and must agree.
/// Routes are known by peer and path identifier.
fn chosen_plainly(
routes: &BTreeMap<(usize, u32), Route>,
peers: &[PeerInfo],
) -> Option<(usize, u32)> {
let local_pref = |peer: usize, route: &Route| {
if peers[peer].ibgp {
route.local_pref.unwrap_or(DEFAULT_LOCAL_PREF)
} else {
DEFAULT_LOCAL_PREF
}
};
let mut left: Vec<((usize, u32), &Route)> =
routes.iter().map(|(k, route)| (*k, route)).collect();
let most = left.iter().map(|(k, r)| local_pref(k.0, r)).max()?;
left.retain(|(k, r)| local_pref(k.0, r) == most);
let shortest = left.iter().map(|(_, r)| r.path.len()).min()?;
left.retain(|(_, r)| r.path.len() == shortest);
let lowest = left.iter().map(|(_, r)| r.origin).min()?;
left.retain(|(_, r)| r.origin == lowest);
let neighbor = |route: &Route| route.path.first().copied().unwrap_or(0);
let med = |route: &Route| route.med.unwrap_or(0);
let mut lowest_med: HashMap<u32, u32> = HashMap::new();
for (_, route) in &left {
let entry = lowest_med.entry(neighbor(route)).or_insert(u32::MAX);
*entry = (*entry).min(med(route));
}
left.retain(|(_, r)| med(r) == lowest_med[&neighbor(r)]);
left.into_iter()
.map(|(key, _)| key)
.min_by_key(|&(peer, path_id)| {
(
peers[peer].ibgp,
peers[peer].router_id,
peers[peer].address,
peer,
path_id,
)
})
}
#[derive(Debug, Clone)]
enum Op {
Announce {
peer: usize,
prefix: usize,
path_id: u32,
route: Route,
},
Withdraw {
peer: usize,
prefix: usize,
path_id: u32,
},
/// The session goes down and comes straight back.
Flap { peer: usize },
}
const PEERS: usize = 6;
// Few prefixes, so that each gathers routes from most peers.
const PREFIXES: usize = 2;
// Most routes have the identifier zero, as without ADD-PATH; some peers
// send a second and third route.
const PATH_IDS: u32 = 3;
pub(super) fn route() -> impl Strategy<Value = Route> {
// Few values of everything, so that ties at every step are common:
// ties are where the shortcuts could be wrong.
(
proptest::option::of(100u32..102),
proptest::collection::vec(1u32..3, 1..3),
0u8..2,
proptest::option::of(0u32..3),
)
.prop_map(|(local_pref, path, origin, med)| Route {
local_pref,
path,
origin,
med,
})
}
fn path_id() -> impl Strategy<Value = u32> {
prop_oneof![3 => Just(0), 2 => 1..PATH_IDS]
}
fn op() -> impl Strategy<Value = Op> {
prop_oneof![
6 => (0..PEERS, 0..PREFIXES, path_id(), route()).prop_map(
|(peer, prefix, path_id, route)| Op::Announce { peer, prefix, path_id, route }
),
3 => (0..PEERS, 0..PREFIXES, path_id())
.prop_map(|(peer, prefix, path_id)| Op::Withdraw { peer, prefix, path_id }),
1 => (0..PEERS).prop_map(|peer| Op::Flap { peer }),
]
}
proptest! {
#![proptest_config(ProptestConfig::with_cases(2000))]
/// After any sequence of changes the table holds exactly the routes it
/// should, has chosen for every prefix what plain selection chooses,
/// and has reported a change of best route exactly when there was one.
#[test]
fn the_table_agrees_with_plain_selection(ops in proptest::collection::vec(op(), 1..80)) {
// Internal and external peers, and identifiers that are not in the
// order the peers are added in.
let infos: Vec<PeerInfo> = (0..PEERS)
.map(|i| {
let number = (PEERS - i) as u8;
if i % 3 == 0 { internal(number) } else { external(number) }
})
.collect();
let mut fixture = Fixture::new(&infos);
let prefixes: Vec<IpPrefix> = (0..PREFIXES)
.map(|i| prefix(&format!("10.{i}.0.0/16")))
.collect();
let mut model: Vec<BTreeMap<(usize, u32), Route>> = vec![BTreeMap::new(); PREFIXES];
for op in ops {
let before: Vec<Option<((usize, u32), Route)>> = model
.iter()
.map(|routes| chosen_plainly(routes, &infos).map(|key| (key, routes[&key].clone())))
.collect();
let outcome = match &op {
Op::Announce { peer, prefix, path_id, route } => {
model[*prefix].insert((*peer, *path_id), route.clone());
fixture.announce_path(*peer, &prefixes[*prefix], *path_id, route)
}
Op::Withdraw { peer, prefix, path_id } => {
model[*prefix].remove(&(*peer, *path_id));
fixture.withdraw_path(*peer, &prefixes[*prefix], *path_id)
}
Op::Flap { peer } => {
for routes in &mut model {
routes.retain(|(from, _), _| from != peer);
}
let outcome = fixture.table.remove_peer(fixture.peers[*peer]);
fixture.peers[*peer] = fixture.table.add_peer(infos[*peer]).unwrap();
outcome
}
};
let mut best_changes = 0;
for (i, routes) in model.iter().enumerate() {
let expected = chosen_plainly(routes, &infos);
prop_assert_eq!(fixture.best_path(&prefixes[i]), expected, "after {:?}", &op);
let now = expected.map(|key| (key, routes[&key].clone()));
best_changes += usize::from(now != before[i]);
prop_assert_eq!(
fixture.held(&prefixes[i]),
routes.keys().copied().collect::<Vec<_>>()
);
}
prop_assert_eq!(outcome.best_changed, best_changes, "after {:?}", &op);
prop_assert_eq!(
fixture.table.len(),
model.iter().map(BTreeMap::len).sum::<usize>()
);
}
}
}
/// Shards and lock stripes are picked by bits of the hash that the hash
/// table inside does not use, so that keys which share a shard still
/// differ where that table looks.
#[test]
fn keys_of_one_shard_differ_in_the_bits_their_table_uses() {
let hasher = RandomState::default();
let shards = 64;
let mut counts = vec![0usize; shards];
let mut tags = vec![std::collections::HashSet::new(); shards];
for n in 0..64_000u32 {
let hash = hasher.hash_one(n);
let shard = spread(hash, shards);
counts[shard] += 1;
// The seven bits hashbrown keeps as a tag.
tags[shard].insert(hash >> 57);
}
for shard in 0..shards {
assert!(
(700..1300).contains(&counts[shard]),
"shard {shard} has {} of 64000 keys",
counts[shard]
);
assert_eq!(tags[shard].len(), 128, "shard {shard}");
}
assert_eq!(spread(u64::MAX, 3), 2);
assert_eq!(spread(0, 3), 0);
}
/// A peer that sends several routes for a prefix can send more than fit a
/// sixteen-bit index. The best route was kept as one, and wrapped.
#[test]
fn a_row_can_hold_more_routes_than_a_sixteen_bit_index_counts() {
let store = AttrStore::new();
let worse = store
.intern(&Route::via(&[1, 2]).attributes(), None)
.unwrap();
let better = store.intern(&Route::via(&[1]).attributes(), None).unwrap();
let info = external(1);
let peers = [Some(info)];
let peer = PeerIndex(0);
let path =
|attrs: &Arc<AttrSet>| Path::new(select::preference(attrs.key(), &info), peer, attrs);
let seqs = journal::Journals::new(journal::Limits::default()).appeared(prefix("10.0.0.0/8"));
let mut row = Row::new(prefix("10.0.0.0/8"), path(&worse), 1, seqs);
let mut scratch = select::Scratch::default();
let count = usize::from(u16::MAX) + 10;
for path_id in 2..=count as u32 {
row.push(path(&worse), path_id);
}
// Equal routes from one peer: the lowest identifier is the best.
assert!(!select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), 0);
row.push(path(&better), count as u32 + 1);
assert!(select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), count);
assert_eq!(row.route(row.best()).path_id, count as u32 + 1);
// With the better one gone, the choice among all the rest is the
// lowest identifier again.
assert!(select::after_remove(&mut row, count, &peers, &mut scratch));
assert_eq!(row.route(row.best()).path_id, 1);
}
crates/server/Cargo.toml
+13 -1
View File
@@ -1,6 +1,11 @@
[package]
name = "bgp-server"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
bgp-packet.workspace = true
@@ -9,12 +14,19 @@ bgp-rib.workspace = true
bytes.workspace = true
clap.workspace = true
futures.workspace = true
eyre.workspace = true
prometheus.workspace = true
socket2.workspace = true
tokio.workspace = true
tracing-subscriber.workspace = true
tracing.workspace = true
protobuf.workspace = true
thiserror.workspace = true
[build-dependencies]
protobuf-codegen.workspace = true
[dev-dependencies]
proptest.workspace = true
[lints]
workspace = true
crates/server/build.rs
+4 -1
View File
@@ -2,8 +2,11 @@ use protobuf_codegen::Codegen;
fn main() {
Codegen::new()
// Parse the .proto in pure Rust so building needs no `protoc` binary
// on the machine (routers and minimal containers rarely have one).
.pure()
.input("proto/config.proto")
.includes(&["proto"])
.includes(["proto"])
.cargo_out_dir("protos")
.run_from_script();
}
crates/server/proto/config.proto
+79
View File
@@ -17,6 +17,49 @@ message BgpConfig {
// Peers configured on this server.
repeated Peer peers = 4;
// Routes this router originates and announces to every peer.
repeated OriginatedRoute originate = 5;
// Where to serve the HTTP API, status page and metrics. The port is
// required. Unset disables the HTTP server.
TcpEndpoint http_endpoint = 6;
// Initial delay between attempts to connect to a configured peer, in
// seconds. Doubles after each failure up to two minutes. Defaults to 5.
uint32 connect_retry_seconds = 7;
// Connection collisions (RFC 4271 §6.8) against a session that is already
// Established are, by default, resolved by closing the new connection.
// When true they are resolved like any other collision: the connection
// initiated by the higher BGP Identifier survives, and a second connection
// opened by the same side replaces the first. This lets a peer that
// restarted take over from a session we have not yet noticed is dead,
// instead of being refused until the hold timer expires.
bool collision_detect_established_state = 8;
}
// A route originated by this router. It is announced with an AS_PATH of the
// local AS and, on each session, that session's local address as next hop.
message OriginatedRoute {
// The prefix in CIDR notation, e.g. "192.0.2.0/24" or "2001:db8::/32".
string prefix = 1;
// ORIGIN attribute: "IGP" (the default), "EGP" or "INCOMPLETE".
string origin = 2;
// Extra copies of the local AS to prepend to the AS_PATH. At most 16.
uint32 as_path_prepend = 3;
// MULTI_EXIT_DISC. Left out of the announcement when unset.
optional uint32 med = 4;
// Standard communities (RFC 1997): "ASN:VALUE", or a well-known name such
// as "NO_EXPORT".
repeated string communities = 5;
// Large communities (RFC 8092): "GLOBAL:LOCAL1:LOCAL2".
repeated string large_communities = 6;
}
// An IP address and port pair.
@@ -41,4 +84,40 @@ message Peer {
// Hold time to advertise in the OPEN message, in seconds (RFC 4271 §4.2).
// Must be 0 or >= 3. Defaults to 90 when unset.
uint32 hold_time_seconds = 4;
// Source IP address for connections we open to this peer. It is also the
// next hop of the routes we send it. Required when the peer identifies us
// by address and the host has several, e.g. when both speakers run on the
// same machine. Must be the same address family as the endpoint. Unset
// lets the operating system choose.
string local_address = 5;
// The address families to exchange with this peer. Each is offered to the
// peer in our OPEN, and is in use on the session only if the peer offers
// it too (RFC 4760). Defaults to all of them when empty. A session with
// no family in common is refused.
repeated AddressFamily address_families = 6;
// ADD-PATH (RFC 7911): whether to take several routes for one prefix
// from this peer, to send it several, or both. It is in use, for each
// direction and address family, only if the peer agrees. Defaults to
// off: one route for each prefix, as in plain BGP.
AddPath add_path = 7;
}
// The directions ADD-PATH can be offered for.
enum AddPath {
ADD_PATH_OFF = 0;
// Accept several routes for a prefix from the peer.
ADD_PATH_RECEIVE = 1;
// Send the peer several routes for a prefix.
ADD_PATH_SEND = 2;
ADD_PATH_BOTH = 3;
}
// An address family routes can be exchanged for.
enum AddressFamily {
ADDRESS_FAMILY_UNSPECIFIED = 0;
IPV4_UNICAST = 1;
IPV6_UNICAST = 2;
}
crates/server/src/advertise.rs
+536
View File
@@ -0,0 +1,536 @@
//! Turns locally originated routes into UPDATE messages for every established
//! session.
//!
//! The advertiser task listens for [`LocalRibEvent`]s and [`SessionEvent`]s.
//! A route change is fanned out to all live sessions; a new session receives
//! the full local table. Sending the same route twice is harmless (BGP treats
//! a re-announcement as an implicit replace), so the two event streams need
//! no coordination beyond being handled by one task.
use std::net::IpAddr;
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LargeCommunitiesPathAttribute,
LocalPrefPathAttribute, Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute,
MultiExitDiscPathAttribute, NextHopPathAttribute, NlriNextHop, PathAttribute, UpdateMessage,
};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_rib::table::DEFAULT_LOCAL_PREF;
use bgp_rib::{LocalRib, LocalRibEvent, LocalRoute, RibStopped};
use tokio::sync::broadcast::Receiver;
use tokio::sync::broadcast::error::RecvError;
use tokio::task::JoinHandle;
use tracing::{debug, info, warn};
use crate::session::{SessionEvent, SessionHandle, SessionRegistry};
/// Starts the advertiser. It hears of every session that comes up after
/// this returns, so call it before anything that can bring one up.
pub fn spawn_advertiser(
local_rib: LocalRib,
registry: SessionRegistry,
local_asn: u32,
) -> JoinHandle<()> {
let advertiser = Advertiser {
rib_events: local_rib.subscribe(),
session_events: registry.subscribe(),
local_rib,
registry,
local_asn,
};
tokio::spawn(advertiser.run())
}
struct Advertiser {
local_rib: LocalRib,
registry: SessionRegistry,
local_asn: u32,
rib_events: Receiver<LocalRibEvent>,
session_events: Receiver<SessionEvent>,
}
impl Advertiser {
/// Runs until the local RIB stops. The server watches for this task
/// ending: without it nothing is announced.
async fn run(mut self) {
loop {
let handled = tokio::select! {
event = self.rib_events.recv() => self.on_route_event(event).await,
event = self.session_events.recv() => self.on_session_event(event).await,
};
if handled.is_err() {
return;
}
}
}
async fn on_route_event(
&self,
event: Result<LocalRibEvent, RecvError>,
) -> Result<(), RibStopped> {
match event {
Ok(LocalRibEvent::Announced(route)) => {
for session in self.registry.list() {
self.send_route(&session, &route).await;
}
Ok(())
}
Ok(LocalRibEvent::Withdrawn { prefix }) => {
for session in self.registry.list() {
send_withdraw(&session, &prefix).await;
}
Ok(())
}
Err(RecvError::Lagged(missed)) => {
warn!(
missed,
"Advertiser lagged on local RIB events, resyncing all sessions"
);
self.send_all_to(&self.registry.list()).await
}
Err(RecvError::Closed) => Err(RibStopped { rib: "local RIB" }),
}
}
async fn on_session_event(
&self,
event: Result<SessionEvent, RecvError>,
) -> Result<(), RibStopped> {
match event {
Ok(SessionEvent::Up(session)) => {
info!(peer = %session.peer, "Session up, sending local routes");
self.send_all_to(&[session]).await
}
Ok(SessionEvent::RefreshRequested(session)) => {
info!(peer = %session.peer, "Route refresh requested, re-sending local routes");
self.send_all_to(&[session]).await
}
Ok(SessionEvent::Down(_)) => Ok(()),
Err(RecvError::Lagged(missed)) => {
warn!(
missed,
"Advertiser lagged on session events, resyncing all sessions"
);
self.send_all_to(&self.registry.list()).await
}
// The registry outlives the server's tasks; nothing more will
// come up, and nothing more needs sending.
Err(RecvError::Closed) => Err(RibStopped { rib: "local RIB" }),
}
}
/// Sends every originated route to each of `sessions`.
async fn send_all_to(&self, sessions: &[SessionHandle]) -> Result<(), RibStopped> {
let routes = self.local_rib.snapshot().await?;
for session in sessions {
for route in &routes {
self.send_route(session, route).await;
}
}
Ok(())
}
async fn send_route(&self, session: &SessionHandle, route: &LocalRoute) {
if !session.families.carries(&route.prefix) {
debug!(
peer = %session.peer,
prefix = %route.prefix,
"Skipping route: its address family was not negotiated"
);
return;
}
let towards = Towards {
local_asn: self.local_asn,
next_hop: session.local_addr.ip(),
four_octet_asn: session.four_octet_asn,
ibgp: session.ibgp,
};
match build_announce(route, &towards) {
Some(update) => {
debug!(peer = %session.peer, prefix = %route.prefix, "Announcing local route");
if let Err(e) = session.send(Message::Update(update)).await {
warn!(peer = %session.peer, err = %e, "Failed to queue announce");
}
}
None => debug!(
peer = %session.peer,
prefix = %route.prefix,
"Skipping route: address family does not match session transport"
),
}
}
}
async fn send_withdraw(session: &SessionHandle, prefix: &IpPrefix) {
if !session.families.carries(prefix) {
// It was never announced on this session either.
return;
}
debug!(peer = %session.peer, %prefix, "Withdrawing local route");
if let Err(e) = session.send(Message::Update(build_withdraw(prefix))).await {
warn!(peer = %session.peer, err = %e, "Failed to queue withdraw");
}
}
/// What about a session decides how a route is announced on it.
#[derive(Debug, Clone, Copy)]
pub struct Towards {
pub local_asn: u32,
/// Our address on the session, which is the routes' next hop.
pub next_hop: IpAddr,
/// Whether the session negotiated four-octet AS numbers. If it did
/// not, and an AS number in the path needs four octets, the path is
/// also sent as AS4_PATH so that it survives the two-octet speaker
/// (RFC 6793 §4.2.2).
pub four_octet_asn: bool,
/// Whether the peer is in our own AS.
pub ibgp: bool,
}
/// Builds the UPDATE announcing `route` on a session.
///
/// Returns `None` when the prefix family does not match the next-hop family:
/// carrying IPv4 NLRI over an IPv6 next hop (RFC 8950) or vice versa is not
/// supported yet.
pub fn build_announce(route: &LocalRoute, towards: &Towards) -> Option<UpdateMessage> {
let attrs = &route.attrs;
// Our AS number goes on the path as a route leaves the AS, and not
// before (RFC 4271 §5.1.2): a route we originate has an empty path
// inside it. A peer in our AS that found our AS number on the path
// would take the route for one that had looped.
let as_path = AsPathAttribute {
segments: if towards.ibgp {
vec![]
} else {
vec![AsPathSegment {
ordered: true,
path: vec![towards.local_asn; 1 + usize::from(attrs.prepend)],
}]
},
};
let as4_path = if towards.four_octet_asn {
None
} else {
as_path.as4_path()
};
// Path attributes in ascending type-code order, as most implementations
// expect them.
let mut path_attributes = vec![
PathAttribute::Origin(attrs.origin),
PathAttribute::AsPath(as_path),
];
// IPv4 routes go in the body of the UPDATE with their next hop in
// NEXT_HOP; IPv6 routes go in MP_REACH_NLRI, which carries its own.
let mut announced_routes = vec![];
let mp_reach = match (route.prefix.address_family(), towards.next_hop) {
(IpVersion::Ipv4, IpAddr::V4(next_hop)) => {
path_attributes.push(PathAttribute::NextHop(NextHopPathAttribute(next_hop)));
announced_routes.push(route.prefix.into());
None
}
(IpVersion::Ipv6, IpAddr::V6(next_hop)) => {
Some(PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(next_hop),
prefixes: vec![route.prefix.into()],
}))
}
_ => return None,
};
if let Some(med) = attrs.med {
path_attributes.push(PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(
med,
)));
}
// LOCAL_PREF is sent to peers in our AS and to no others (RFC 4271
// §5.1.5).
if towards.ibgp {
path_attributes.push(PathAttribute::LocalPref(LocalPrefPathAttribute(
DEFAULT_LOCAL_PREF,
)));
}
if !attrs.communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
attrs.communities.clone(),
)));
}
path_attributes.extend(mp_reach);
path_attributes.extend(as4_path);
if !attrs.large_communities.is_empty() {
path_attributes.push(PathAttribute::LargeCommunities(
LargeCommunitiesPathAttribute {
communities: attrs.large_communities.clone(),
},
));
}
Some(UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes,
})
}
/// Builds the UPDATE withdrawing `prefix`.
pub fn build_withdraw(prefix: &IpPrefix) -> UpdateMessage {
match prefix.address_family() {
IpVersion::Ipv4 => UpdateMessage {
withdrawn_routes: vec![Nlri::from(*prefix)],
path_attributes: vec![],
announced_routes: vec![],
},
IpVersion::Ipv6 => UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::MpUnreachNlri(MpUnreachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
prefixes: vec![Nlri::from(*prefix)],
})],
announced_routes: vec![],
},
}
}
#[cfg(test)]
mod tests {
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::message::{LargeCommunity, OriginPathAttribute};
use bgp_primitives::parser::ParserContext;
use bgp_rib::LocalRouteAttrs;
use bytes::BytesMut;
use super::*;
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// An external peer.
fn towards(local_asn: u32, next_hop: IpAddr, four_octet_asn: bool) -> Towards {
Towards {
local_asn,
next_hop,
four_octet_asn,
ibgp: false,
}
}
#[test]
fn ipv4_announce_has_expected_attributes_and_roundtrips() {
let route = LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs {
origin: OriginPathAttribute::IGP,
prepend: 2,
med: Some(50),
communities: vec![(65000, 100), (65535, 65281)],
large_communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2,
}],
},
);
let update = build_announce(
&route,
&towards(65000, IpAddr::V4(Ipv4Addr::new(192, 0, 2, 1)), true),
)
.unwrap();
assert_eq!(update.announced_routes, vec![prefix("10.100.1.0/24")]);
assert!(update.withdrawn_routes.is_empty());
assert_eq!(
update.path_attributes,
vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65000, 65000, 65000]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65000, 100),
(65535, 65281)
])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![LargeCommunity {
global_admin: 65000,
data_1: 1,
data_2: 2
}],
}),
]
);
// Serialise and parse back to make sure the wire encoding is sound.
let ctx = ParserContext {
four_octet_asn: true,
ibgp: false,
add_path: Vec::new(),
};
let msg = Message::Update(update.clone());
let mut buf = BytesMut::new();
msg.to_wire(&ctx, &mut buf).unwrap();
let Message::Update(parsed) = Message::decode(&ctx, &buf).unwrap() else {
panic!("expected UPDATE")
};
assert_eq!(parsed, update);
}
#[test]
fn ipv6_announce_uses_mp_reach() {
let route = LocalRoute::new(prefix("2001:db8:1::/48"), LocalRouteAttrs::default());
let nh = Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1);
let update = build_announce(&route, &towards(65000, IpAddr::V6(nh), true)).unwrap();
assert!(update.announced_routes.is_empty());
assert!(
!update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::NextHop(_)))
);
assert!(update.path_attributes.iter().any(|a| matches!(
a,
PathAttribute::MpReachNlri(MpReachNlriPathAttribute { next_hop: NlriNextHop::Ipv6(h), prefixes, .. })
if *h == nh && *prefixes == vec![prefix("2001:db8:1::/48")]
)));
}
/// A two-octet session cannot carry a four-octet AS number in AS_PATH:
/// it goes out as AS_TRANS there, and in full in AS4_PATH.
#[test]
fn a_wide_as_number_reaches_a_two_octet_peer_in_as4_path() {
let route = LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs {
prepend: 1,
..LocalRouteAttrs::default()
},
);
let next_hop = IpAddr::V4(Ipv4Addr::new(192, 0, 2, 1));
let path = |update: &UpdateMessage, as4: bool| {
update.path_attributes.iter().find_map(|a| match a {
PathAttribute::As4Path(p) if as4 => Some(p.segments[0].path.clone()),
PathAttribute::AsPath(p) if !as4 => Some(p.segments[0].path.clone()),
_ => None,
})
};
let update = build_announce(&route, &towards(4_200_000_001, next_hop, false)).unwrap();
assert_eq!(path(&update, true), Some(vec![4_200_000_001; 2]));
// What the peer reads from the wire.
let ctx = ParserContext {
four_octet_asn: false,
ibgp: false,
add_path: Vec::new(),
};
let mut buf = BytesMut::new();
Message::Update(update).to_wire(&ctx, &mut buf).unwrap();
let Message::Update(seen) = Message::decode(&ctx, &buf).unwrap() else {
panic!("expected UPDATE")
};
assert_eq!(path(&seen, false), Some(vec![23456; 2]), "AS_TRANS");
assert_eq!(path(&seen, true), Some(vec![4_200_000_001; 2]));
// No AS4_PATH when it would say nothing new: the peer handles
// four-octet numbers, or ours fits in two.
let update = build_announce(&route, &towards(4_200_000_001, next_hop, true)).unwrap();
assert_eq!(path(&update, true), None);
let update = build_announce(&route, &towards(65000, next_hop, false)).unwrap();
assert_eq!(path(&update, true), None);
assert_eq!(path(&update, false), Some(vec![65000; 2]));
}
#[test]
fn family_mismatch_is_skipped() {
let v6 = LocalRoute::new(prefix("2001:db8::/32"), LocalRouteAttrs::default());
assert!(
build_announce(&v6, &towards(65000, IpAddr::V4(Ipv4Addr::LOCALHOST), true)).is_none()
);
let v4 = LocalRoute::new(prefix("10.0.0.0/8"), LocalRouteAttrs::default());
assert!(
build_announce(&v4, &towards(65000, IpAddr::V6(Ipv6Addr::LOCALHOST), true)).is_none()
);
}
#[test]
fn withdraw_by_family() {
let v4 = build_withdraw(&prefix("10.0.0.0/8"));
assert_eq!(v4.withdrawn_routes, vec![prefix("10.0.0.0/8")]);
assert!(v4.path_attributes.is_empty());
let v6 = build_withdraw(&prefix("2001:db8::/32"));
assert!(v6.withdrawn_routes.is_empty());
assert!(
matches!(&v6.path_attributes[..], [PathAttribute::MpUnreachNlri(m)] if m.prefixes == vec![prefix("2001:db8::/32")])
);
}
/// Inside our own AS a route we originate has an empty AS path and a
/// LOCAL_PREF. With our AS number on the path, a peer in our AS takes
/// the route for a loop; without LOCAL_PREF it treats the route as
/// withdrawn, as our own decoder does.
#[test]
fn a_route_announced_inside_our_as_has_an_empty_path_and_a_local_pref() {
let route = LocalRoute::new(
prefix("10.100.1.0/24"),
LocalRouteAttrs {
prepend: 3,
..LocalRouteAttrs::default()
},
);
let internal = Towards {
ibgp: true,
..towards(65000, IpAddr::V4(Ipv4Addr::new(192, 0, 2, 1)), true)
};
let update = build_announce(&route, &internal).unwrap();
assert_eq!(
update.path_attributes,
vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
PathAttribute::AsPath(AsPathAttribute { segments: vec![] }),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::LocalPref(LocalPrefPathAttribute(100)),
]
);
// An internal peer accepts it as it stands.
let ctx = ParserContext {
four_octet_asn: true,
ibgp: true,
add_path: Vec::new(),
};
let mut buf = BytesMut::new();
Message::Update(update.clone())
.to_wire(&ctx, &mut buf)
.unwrap();
let decoded = Message::decode_with_report(&ctx, &buf).unwrap();
assert!(
decoded.update_errors.is_empty(),
"{:?}",
decoded.update_errors
);
assert_eq!(decoded.message, Message::Update(update));
// To an external peer the same route has our AS on the path, as
// many times as configured, and no LOCAL_PREF.
let external = Towards {
ibgp: false,
..internal
};
let update = build_announce(&route, &external).unwrap();
assert_eq!(
update.path_attributes[1],
AsPathAttribute::from_asns(vec![65000; 4])
);
assert!(
!update
.path_attributes
.iter()
.any(|a| matches!(a, PathAttribute::LocalPref(_)))
);
}
}
crates/server/src/collision.rs
+304
View File
@@ -0,0 +1,304 @@
//! Connection collision detection (RFC 4271 §6.8).
//!
//! Two speakers that dial each other can end up with two TCP connections.
//! Exactly one must survive and both ends must pick the same one: the
//! connection *initiated by the speaker with the higher BGP Identifier*.
//! The other is closed with Cease / Connection Collision Resolution.
//!
//! This cannot be a check local to one connection: the rival connection is
//! usually still mid-handshake (so it is not in the session registry yet),
//! and sometimes it is the rival, running in another task, that must close.
//! Hence one small shared map, holding the single connection currently
//! preferred for each peer and a way to tell it to stop.
//!
//! A peer here is an address together with the BGP Identifier it announced.
//! The identifier alone would not do: a router commonly has one session
//! over IPv4 and another over IPv6 with the same neighbour, which are two
//! sessions and not a collision; and it would let any host that can reach
//! us displace a peer's connection by claiming the peer's identifier.
//!
//! A collision with an *Established* connection closes the new one, unless
//! `detect_established` is set (the RFC's optional
//! `CollisionDetectEstablishedState`), which lets a restarted peer replace a
//! session we have not yet noticed is dead. If both connections were opened
//! by the same side the newer wins: the initiator has abandoned the older.
use std::collections::HashMap;
use std::net::{IpAddr, Ipv4Addr};
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use tokio::sync::oneshot;
use crate::metrics::COLLISIONS;
use crate::sync::IgnorePoison;
/// Which side opened the TCP connection.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Direction {
/// The peer connected to us.
Inbound,
/// We connected to the peer.
Outbound,
}
impl Direction {
pub fn as_str(&self) -> &'static str {
match self {
Direction::Inbound => "inbound",
Direction::Outbound => "outbound",
}
}
}
struct Entry {
id: u64,
direction: Direction,
established: bool,
kill: Option<oneshot::Sender<()>>,
}
/// True if the new connection is the one to close.
fn new_loses(
local_id: Ipv4Addr,
remote_id: Ipv4Addr,
existing: &Entry,
new: Direction,
detect_established: bool,
) -> bool {
if existing.established && !detect_established {
return true;
}
if existing.direction == new {
return false;
}
// Identifiers compare as unsigned 32-bit integers.
let keep = if u32::from(local_id) > u32::from(remote_id) {
Direction::Outbound
} else {
Direction::Inbound
};
new != keep
}
/// Who a connection is with: the address at the other end, and the BGP
/// Identifier announced from it.
type Peer = (IpAddr, Ipv4Addr);
/// Shared by every connection of one server.
#[derive(Clone)]
pub struct CollisionTable {
entries: Arc<Mutex<HashMap<Peer, Entry>>>,
next_id: Arc<AtomicU64>,
detect_established: bool,
}
impl CollisionTable {
pub fn new(detect_established: bool) -> Self {
Self {
entries: Default::default(),
next_id: Default::default(),
detect_established,
}
}
/// Call when a connection receives the peer's OPEN. `None` means this
/// connection lost and must be closed with Cease / Connection Collision
/// Resolution. Otherwise any rival has been told to close and this
/// connection holds the slot until the guard drops.
pub fn admit(
&self,
local_id: Ipv4Addr,
remote_addr: IpAddr,
remote_id: Ipv4Addr,
direction: Direction,
) -> Option<ConnectionGuard> {
let peer = (remote_addr, remote_id);
let mut entries = self.entries.locked();
if let Some(existing) = entries.get_mut(&peer) {
if new_loses(
local_id,
remote_id,
existing,
direction,
self.detect_established,
) {
COLLISIONS.with_label_values(&["new"]).inc();
return None;
}
if let Some(kill) = existing.kill.take() {
COLLISIONS.with_label_values(&["existing"]).inc();
let _ = kill.send(());
}
}
let id = self.next_id.fetch_add(1, Ordering::Relaxed);
let (kill, killed) = oneshot::channel();
entries.insert(
peer,
Entry {
id,
direction,
established: false,
kill: Some(kill),
},
);
Some(ConnectionGuard {
table: self.clone(),
peer,
id,
killed,
})
}
}
/// Holds a connection's slot in the table; releases it on drop.
pub struct ConnectionGuard {
table: CollisionTable,
peer: Peer,
id: u64,
killed: oneshot::Receiver<()>,
}
impl ConnectionGuard {
pub fn set_established(&self) {
let mut entries = self.table.entries.locked();
if let Some(e) = entries.get_mut(&self.peer).filter(|e| e.id == self.id) {
e.established = true;
}
}
/// Completes when a later connection has won against this one; the
/// owner must then send Cease / Connection Collision Resolution and
/// close. Cancel-safe.
pub async fn killed(&mut self) {
// The sending half is dropped only when it has been used: the entry
// that holds it is removed by this guard alone, or replaced by a
// connection that signals first. Either way, the wait is over.
let _ = (&mut self.killed).await;
}
}
impl Drop for ConnectionGuard {
fn drop(&mut self) {
// A connection that lost has already been replaced in the map; only
// remove the slot if it is still ours.
let mut entries = self.table.entries.locked();
if entries.get(&self.peer).is_some_and(|e| e.id == self.id) {
entries.remove(&self.peer);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use Direction::*;
const LOW: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
const HIGH: Ipv4Addr = Ipv4Addr::new(192, 0, 2, 1);
fn loses(
local: Ipv4Addr,
remote: Ipv4Addr,
existing: Direction,
established: bool,
new: Direction,
detect: bool,
) -> bool {
let e = Entry {
id: 0,
direction: existing,
established,
kill: None,
};
new_loses(local, remote, &e, new, detect)
}
#[test]
fn higher_identifier_keeps_the_connection_it_initiated() {
// We are HIGH: our outbound connection survives.
assert!(loses(HIGH, LOW, Outbound, false, Inbound, false));
assert!(!loses(HIGH, LOW, Inbound, false, Outbound, false));
// We are LOW: the peer's connection (inbound to us) survives.
assert!(!loses(LOW, HIGH, Outbound, false, Inbound, false));
assert!(loses(LOW, HIGH, Inbound, false, Outbound, false));
}
#[test]
fn both_ends_agree() {
// A and B dial each other. Each end holds its own dial as the
// existing (Outbound) connection and sees the other's dial arrive as
// new (Inbound). Exactly one of the two dials may be closed, so the
// ends must give opposite answers to "close the other side's dial?".
for (a, b) in [(LOW, HIGH), (HIGH, LOW)] {
let a_closes_dial_from_b = loses(a, b, Outbound, false, Inbound, false);
let b_closes_dial_from_a = loses(b, a, Outbound, false, Inbound, false);
assert_ne!(
a_closes_dial_from_b, b_closes_dial_from_a,
"ends disagree for a={a} b={b}"
);
}
}
#[test]
fn established_wins_unless_enabled() {
for new in [Inbound, Outbound] {
for existing in [Inbound, Outbound] {
assert!(loses(LOW, HIGH, existing, true, new, false));
assert!(loses(HIGH, LOW, existing, true, new, false));
}
}
assert!(!loses(LOW, HIGH, Outbound, true, Inbound, true));
assert!(loses(HIGH, LOW, Outbound, true, Inbound, true));
}
#[test]
fn same_direction_newer_wins() {
assert!(!loses(LOW, HIGH, Inbound, false, Inbound, false));
assert!(!loses(HIGH, LOW, Inbound, false, Inbound, false));
assert!(loses(LOW, HIGH, Inbound, true, Inbound, false));
assert!(!loses(LOW, HIGH, Inbound, true, Inbound, true));
}
#[tokio::test]
async fn table_signals_loser_and_releases_slots() {
let table = CollisionTable::new(false);
let addr: IpAddr = Ipv4Addr::new(192, 0, 2, 1).into();
// We are LOW: outbound first, then the peer's inbound wins.
let mut out = table.admit(LOW, addr, HIGH, Outbound).unwrap();
let inb = table.admit(LOW, addr, HIGH, Inbound).unwrap();
assert!(out.killed.try_recv().is_ok());
// The loser's guard dropping must not free the winner's slot.
drop(out);
inb.set_established();
assert!(table.admit(LOW, addr, HIGH, Outbound).is_none());
// A different identifier never collides.
assert!(
table
.admit(LOW, addr, Ipv4Addr::new(192, 0, 2, 2), Inbound)
.is_some()
);
// Once the survivor goes away the identifier is free again.
drop(inb);
assert!(table.admit(LOW, addr, HIGH, Outbound).is_some());
}
/// A neighbour reached over IPv4 and over IPv6 announces the same
/// identifier on both sessions. They are two sessions, and neither
/// closes the other. Nor can a connection from elsewhere that claims a
/// peer's identifier take the place of the peer's own.
#[tokio::test]
async fn the_same_identifier_from_another_address_is_another_peer() {
let table = CollisionTable::new(true);
let v4: IpAddr = Ipv4Addr::new(192, 0, 2, 1).into();
let v6: IpAddr = "2001:db8::1".parse().unwrap();
let mut over_v4 = table.admit(LOW, v4, HIGH, Outbound).unwrap();
over_v4.set_established();
let over_v6 = table.admit(LOW, v6, HIGH, Inbound);
assert!(over_v6.is_some());
assert!(over_v4.killed.try_recv().is_err(), "the first is untouched");
}
}
crates/server/src/config.rs
+446 -43
View File
@@ -1,12 +1,22 @@
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::str::FromStr;
use std::time::Duration;
use eyre::{Result, bail, eyre};
use bgp_packet::community;
use bgp_packet::message::OriginPathAttribute;
use bgp_packet::open::AddPathSendRecv;
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use crate::proto::config::BgpConfig;
use crate::error::{ConfigError, OriginateProblem, PeerProblem};
use crate::family::FamilySet;
use crate::proto::config::{self as proto, BgpConfig, OriginatedRoute, Peer};
const DEFAULT_BGP_PORT: u16 = 179;
const DEFAULT_HOLD_TIME_SECS: u16 = 90;
const DEFAULT_CONNECT_RETRY: Duration = Duration::from_secs(5);
const MAX_AS_PATH_PREPEND: u32 = 16;
/// Validated, Rust-native server configuration derived from the proto config.
pub struct Config {
@@ -16,79 +26,472 @@ pub struct Config {
pub hold_time_secs: u16,
pub listen_endpoints: Vec<SocketAddr>,
pub peers: Vec<PeerConfig>,
/// Initial delay between attempts to connect to a configured peer;
/// doubles on each failure.
pub connect_retry: Duration,
/// Resolve collisions against Established sessions by BGP Identifier
/// instead of always closing the new connection (RFC 4271 §8.1.1
/// `CollisionDetectEstablishedState`).
pub collision_detect_established_state: bool,
/// Routes this router originates.
pub originate: Vec<LocalRoute>,
/// Address for the HTTP API; `None` disables it.
pub http_endpoint: Option<SocketAddr>,
}
#[derive(Debug, Clone)]
pub struct PeerConfig {
pub name: String,
pub endpoint: SocketAddr,
pub remote_asn: u32,
pub hold_time_secs: u16,
/// Source address for connections we open to this peer; `None` lets the
/// operating system choose.
pub local_address: Option<IpAddr>,
/// The address families offered to this peer.
pub families: FamilySet,
/// The directions ADD-PATH (RFC 7911) is offered to this peer for, on
/// every one of `families`. `None` offers it for neither.
pub add_path: Option<AddPathSendRecv>,
}
impl Config {
pub fn from_proto(proto: BgpConfig) -> Result<Self> {
/// The configured peer at `address`, if there is one.
pub fn peer_at(&self, address: IpAddr) -> Option<&PeerConfig> {
self.peers.iter().find(|peer| peer.endpoint.ip() == address)
}
pub fn from_proto(proto: BgpConfig) -> Result<Self, ConfigError> {
// Peers refuse an identifier of 0.0.0.0, as we do theirs.
let router_id = Ipv4Addr::from_str(&proto.router_id)
.map_err(|_| eyre!("Invalid router_id {:?}: must be a dotted-quad IPv4 address", proto.router_id))?;
.ok()
.filter(|id| !id.is_unspecified())
.ok_or_else(|| ConfigError::RouterId(proto.router_id.clone()))?;
if proto.local_asn == 0 {
bail!("local_asn must be non-zero");
return Err(ConfigError::LocalAsn);
}
let listen_endpoints = if proto.listen_endpoints.is_empty() {
vec![
SocketAddr::new(IpAddr::V4(Ipv4Addr::UNSPECIFIED), DEFAULT_BGP_PORT),
SocketAddr::new(IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED), DEFAULT_BGP_PORT),
SocketAddr::new(
IpAddr::V6(std::net::Ipv6Addr::UNSPECIFIED),
DEFAULT_BGP_PORT,
),
]
} else {
proto.listen_endpoints
.into_iter()
proto
.listen_endpoints
.iter()
.map(|ep| parse_endpoint(&ep.address, ep.port))
.collect::<Result<_>>()?
.collect::<Result<_, _>>()?
};
let peers = proto.peers
.into_iter()
.map(|p| {
let ep = p.endpoint.as_ref()
.ok_or_else(|| eyre!("Peer {:?} is missing endpoint", p.name))?;
let endpoint = parse_endpoint(&ep.address, ep.port)?;
let mut peers: Vec<PeerConfig> = Vec::with_capacity(proto.peers.len());
for peer in proto.peers {
let name = peer.name.clone();
let rejected = |problem| ConfigError::Peer {
peer: name.clone(),
problem,
};
let peer = parse_peer(peer).map_err(rejected)?;
// A peer is told from the others by its name in what the
// dialer reports, and by its address when it connects to us.
if peers.iter().any(|earlier| earlier.name == peer.name) {
return Err(rejected(PeerProblem::DuplicateName));
}
if peers
.iter()
.any(|earlier| earlier.endpoint.ip() == peer.endpoint.ip())
{
return Err(rejected(PeerProblem::DuplicateAddress));
}
peers.push(peer);
}
if p.remote_asn == 0 {
bail!("Peer {:?} remote_asn must be non-zero", p.name);
}
let mut originate: Vec<LocalRoute> = Vec::with_capacity(proto.originate.len());
for entry in &proto.originate {
let rejected = |problem| ConfigError::Originate {
prefix: entry.prefix.clone(),
problem,
};
let route = parse_originated_route(entry).map_err(rejected)?;
if originate
.iter()
.any(|earlier| earlier.prefix == route.prefix)
{
return Err(rejected(OriginateProblem::Duplicate));
}
originate.push(route);
}
let hold_time_secs = if p.hold_time_seconds == 0 {
DEFAULT_HOLD_TIME_SECS
} else if p.hold_time_seconds < 3 {
bail!("Peer {:?} hold_time_seconds must be 0 or >= 3", p.name);
} else {
p.hold_time_seconds as u16
};
let http_endpoint = match proto.http_endpoint.into_option() {
Some(ep) if ep.port == 0 => return Err(ConfigError::HttpPort),
Some(ep) => Some(parse_endpoint(&ep.address, ep.port)?),
None => None,
};
Ok(PeerConfig {
name: p.name,
endpoint,
remote_asn: p.remote_asn,
hold_time_secs,
})
})
.collect::<Result<_>>()?;
let connect_retry = if proto.connect_retry_seconds == 0 {
DEFAULT_CONNECT_RETRY
} else {
Duration::from_secs(proto.connect_retry_seconds as u64)
};
Ok(Self { router_id, local_asn: proto.local_asn, hold_time_secs: DEFAULT_HOLD_TIME_SECS, listen_endpoints, peers })
Ok(Self {
router_id,
local_asn: proto.local_asn,
hold_time_secs: DEFAULT_HOLD_TIME_SECS,
listen_endpoints,
peers,
connect_retry,
collision_detect_established_state: proto.collision_detect_established_state,
originate,
http_endpoint,
})
}
}
fn parse_endpoint(address: &str, port: u32) -> Result<SocketAddr> {
let port = if port == 0 {
DEFAULT_BGP_PORT
} else if port > u16::MAX as u32 {
bail!("Port {port} is out of range");
} else {
port as u16
fn parse_peer(peer: Peer) -> Result<PeerConfig, PeerProblem> {
let ep = peer.endpoint.as_ref().ok_or(PeerProblem::MissingEndpoint)?;
let endpoint =
parse_endpoint(&ep.address, ep.port).map_err(|e| PeerProblem::Endpoint(Box::new(e)))?;
if peer.remote_asn == 0 {
return Err(PeerProblem::RemoteAsn);
}
let hold_time_secs = match peer.hold_time_seconds {
0 => DEFAULT_HOLD_TIME_SECS,
// RFC 4271 §4.2: zero, or at least three seconds.
1 | 2 => return Err(PeerProblem::HoldTime),
secs => u16::try_from(secs).map_err(|_| PeerProblem::HoldTime)?,
};
let ip = IpAddr::from_str(address)
.map_err(|_| eyre!("Invalid IP address {address:?}"))?;
let local_address = if peer.local_address.is_empty() {
None
} else {
let ip = IpAddr::from_str(&peer.local_address)
.map_err(|_| PeerProblem::LocalAddress(peer.local_address.clone()))?;
if ip.is_ipv4() != endpoint.is_ipv4() {
return Err(PeerProblem::AddressFamilyMismatch);
}
Some(ip)
};
let families = if peer.address_families.is_empty() {
FamilySet::supported()
} else {
peer.address_families
.iter()
.map(|family| match family.enum_value() {
Ok(proto::AddressFamily::IPV4_UNICAST) => Ok(AfiSafi::IPV4_UNICAST),
Ok(proto::AddressFamily::IPV6_UNICAST) => Ok(AfiSafi::IPV6_UNICAST),
Ok(proto::AddressFamily::ADDRESS_FAMILY_UNSPECIFIED) | Err(_) => {
Err(PeerProblem::AddressFamily)
}
})
.collect::<Result<_, _>>()?
};
let add_path = match peer.add_path.enum_value() {
Ok(proto::AddPath::ADD_PATH_OFF) => None,
Ok(proto::AddPath::ADD_PATH_RECEIVE) => Some(AddPathSendRecv::Receive),
Ok(proto::AddPath::ADD_PATH_SEND) => Some(AddPathSendRecv::Send),
Ok(proto::AddPath::ADD_PATH_BOTH) => Some(AddPathSendRecv::Both),
Err(_) => return Err(PeerProblem::AddPath),
};
Ok(PeerConfig {
// A peer without a name goes by its address.
name: if peer.name.is_empty() {
endpoint.ip().to_string()
} else {
peer.name
},
endpoint,
remote_asn: peer.remote_asn,
hold_time_secs,
local_address,
families,
add_path,
})
}
fn parse_endpoint(address: &str, port: u32) -> Result<SocketAddr, ConfigError> {
let port = match port {
0 => DEFAULT_BGP_PORT,
port => u16::try_from(port).map_err(|_| ConfigError::Port(port))?,
};
let ip = IpAddr::from_str(address).map_err(|_| ConfigError::Address(address.to_string()))?;
Ok(SocketAddr::new(ip, port))
}
fn parse_originated_route(route: &OriginatedRoute) -> Result<LocalRoute, OriginateProblem> {
let prefix = IpPrefix::try_from(route.prefix.as_str())?;
let origin = match route.origin.to_ascii_uppercase().as_str() {
"" | "IGP" => OriginPathAttribute::IGP,
"EGP" => OriginPathAttribute::EGP,
"INCOMPLETE" => OriginPathAttribute::INCOMPLETE,
_ => return Err(OriginateProblem::Origin(route.origin.clone())),
};
let prepend = u8::try_from(route.as_path_prepend)
.ok()
.filter(|&n| u32::from(n) <= MAX_AS_PATH_PREPEND)
.ok_or(OriginateProblem::Prepend {
max: MAX_AS_PATH_PREPEND,
})?;
let communities = route
.communities
.iter()
.map(|c| community::parse_standard(c))
.collect::<Result<_, _>>()?;
let large_communities = route
.large_communities
.iter()
.map(|c| community::parse_large(c))
.collect::<Result<_, _>>()?;
Ok(LocalRoute::new(
prefix,
LocalRouteAttrs {
origin,
prepend,
med: route.med,
communities,
large_communities,
},
))
}
#[cfg(test)]
mod tests {
use bgp_primitives::ip_prefix::PrefixError;
use super::*;
fn parse(text: &str) -> Result<Config, ConfigError> {
let proto = protobuf::text_format::parse_from_str::<BgpConfig>(text)
.expect("the test's textproto is well formed");
Config::from_proto(proto)
}
fn rejected(text: &str) -> ConfigError {
parse(text).err().expect(text)
}
const BASE: &str = "router_id: \"192.0.2.1\" local_asn: 65000\n";
#[test]
fn originated_routes_are_parsed_with_their_attributes() {
let config = parse(&format!(
"{BASE}originate {{ prefix: \"192.0.2.0/24\" }}
originate {{
prefix: \"2001:db8::/32\" origin: \"incomplete\" as_path_prepend: 2 med: 0
communities: \"65000:100\" communities: \"NO_EXPORT\"
large_communities: \"65000:1:2\"
}}"
))
.unwrap();
assert_eq!(config.originate.len(), 2);
let plain = &config.originate[0];
assert_eq!(plain.prefix, IpPrefix::try_from("192.0.2.0/24").unwrap());
assert_eq!(plain.attrs, LocalRouteAttrs::default());
let full = &config.originate[1].attrs;
assert_eq!(full.origin, OriginPathAttribute::INCOMPLETE);
assert_eq!(full.prepend, 2);
assert_eq!(
full.med,
Some(0),
"an explicit MED of 0 is not the same as no MED"
);
assert_eq!(full.communities, vec![(65000, 100), (65535, 65281)]);
assert_eq!(full.large_communities.len(), 1);
}
#[test]
fn bad_originated_routes_are_rejected_with_the_entry_and_the_reason() {
let problem = |body: &str| match rejected(&format!("{BASE}originate {{ {body} }}")) {
ConfigError::Originate { problem, .. } => problem,
other => panic!("{body}: {other}"),
};
assert_eq!(
problem("prefix: \"not-a-prefix\""),
OriginateProblem::Prefix(PrefixError::Format("not-a-prefix".into()))
);
assert_eq!(
problem("prefix: \"192.0.2.0/24\" origin: \"static\""),
OriginateProblem::Origin("static".into())
);
assert_eq!(
problem("prefix: \"192.0.2.0/24\" as_path_prepend: 17"),
OriginateProblem::Prepend { max: 16 }
);
assert!(matches!(
problem("prefix: \"192.0.2.0/24\" communities: \"70000:1\""),
OriginateProblem::Community(_)
));
assert!(matches!(
problem("prefix: \"192.0.2.0/24\" large_communities: \"1:2\""),
OriginateProblem::Community(_)
));
let duplicate = rejected(&format!(
"{BASE}originate {{ prefix: \"192.0.2.0/24\" }} originate {{ prefix: \"192.0.2.0/24\" med: 5 }}"
));
assert_eq!(
duplicate,
ConfigError::Originate {
prefix: "192.0.2.0/24".into(),
problem: OriginateProblem::Duplicate
}
);
// The message names the entry, so the operator can find it.
assert_eq!(
duplicate.to_string(),
"originate \"192.0.2.0/24\": listed more than once"
);
}
#[test]
fn a_peer_is_offered_every_address_family_unless_some_are_listed() {
let families = |body: &str| {
let config = parse(&format!(
"{BASE}peers {{ name: \"r1\" endpoint {{ address: \"192.0.2.2\" }} remote_asn: 1 {body} }}"
))
.unwrap();
config.peers[0].families.clone()
};
assert_eq!(families(""), FamilySet::supported());
assert_eq!(
families("address_families: IPV6_UNICAST"),
[AfiSafi::IPV6_UNICAST].into_iter().collect()
);
assert_eq!(
families("address_families: IPV4_UNICAST address_families: IPV6_UNICAST"),
FamilySet::supported()
);
}
#[test]
fn add_path_is_off_unless_a_peer_asks_for_it() {
let add_path = |body: &str| {
let config = parse(&format!(
"{BASE}peers {{ name: \"r1\" endpoint {{ address: \"192.0.2.2\" }} remote_asn: 1 {body} }}"
))
.unwrap();
config.peers[0].add_path
};
assert_eq!(add_path(""), None);
assert_eq!(add_path("add_path: ADD_PATH_OFF"), None);
assert_eq!(
add_path("add_path: ADD_PATH_RECEIVE"),
Some(AddPathSendRecv::Receive)
);
assert_eq!(
add_path("add_path: ADD_PATH_BOTH"),
Some(AddPathSendRecv::Both)
);
}
#[test]
fn bad_peers_and_endpoints_are_rejected_with_the_peer_named() {
assert_eq!(
rejected("router_id: \"nope\" local_asn: 1"),
ConfigError::RouterId("nope".into())
);
assert_eq!(rejected("router_id: \"192.0.2.1\""), ConfigError::LocalAsn);
let problem = |body: &str| match rejected(&format!("{BASE}peers {{ name: \"r1\" {body} }}"))
{
ConfigError::Peer { peer, problem } => {
assert_eq!(peer, "r1");
problem
}
other => panic!("{body}: {other}"),
};
let endpoint = "endpoint { address: \"192.0.2.2\" }";
assert_eq!(problem("remote_asn: 1"), PeerProblem::MissingEndpoint);
assert_eq!(problem(endpoint), PeerProblem::RemoteAsn);
assert_eq!(
problem(&format!("{endpoint} remote_asn: 1 hold_time_seconds: 2")),
PeerProblem::HoldTime
);
assert_eq!(
problem(&format!(
"{endpoint} remote_asn: 1 hold_time_seconds: 70000"
)),
PeerProblem::HoldTime
);
assert_eq!(
problem(&format!("{endpoint} remote_asn: 1 local_address: \"::1\"")),
PeerProblem::AddressFamilyMismatch
);
assert_eq!(
problem(&format!(
"{endpoint} remote_asn: 1 address_families: ADDRESS_FAMILY_UNSPECIFIED"
)),
PeerProblem::AddressFamily
);
assert_eq!(
problem("endpoint { address: \"192.0.2.2\" port: 70000 } remote_asn: 1"),
PeerProblem::Endpoint(Box::new(ConfigError::Port(70000)))
);
assert_eq!(
rejected(&format!("{BASE}http_endpoint {{ address: \"127.0.0.1\" }}")),
ConfigError::HttpPort
);
}
/// Peers are told apart by name where the dialer reports on them and by
/// address when they connect to us; two that share either would be
/// taken for each other.
#[test]
fn peers_must_differ_in_name_and_address() {
let peer = |name: &str, address: &str| {
format!(
"peers {{ name: \"{name}\" endpoint {{ address: \"{address}\" }} remote_asn: 1 }}\n"
)
};
let problem = |peers: String| match rejected(&format!("{BASE}{peers}")) {
ConfigError::Peer { problem, .. } => problem,
other => panic!("{other}"),
};
assert_eq!(
problem(peer("r1", "192.0.2.2") + &peer("r1", "192.0.2.3")),
PeerProblem::DuplicateName
);
assert_eq!(
problem(peer("r1", "192.0.2.2") + &peer("r2", "192.0.2.2")),
PeerProblem::DuplicateAddress
);
// A peer without a name goes by its address, so two of them differ.
let config = parse(&format!(
"{BASE}{}{}",
peer("", "192.0.2.2"),
peer("", "192.0.2.3")
))
.unwrap();
assert_eq!(config.peers[0].name, "192.0.2.2");
assert_eq!(config.peers[1].name, "192.0.2.3");
assert_eq!(
config
.peer_at("192.0.2.3".parse().unwrap())
.map(|p| &*p.name),
Some("192.0.2.3")
);
assert!(config.peer_at("192.0.2.4".parse().unwrap()).is_none());
}
/// A peer would refuse our OPEN.
#[test]
fn a_router_id_of_zero_is_rejected() {
assert_eq!(
rejected("router_id: \"0.0.0.0\" local_asn: 1"),
ConfigError::RouterId("0.0.0.0".into())
);
}
}
crates/server/src/connection.rs
+1039 -8
View File
File diff suppressed because it is too large. Load diff
crates/server/src/dialer.rs
+116
View File
@@ -0,0 +1,116 @@
//! Active side of the FSM: dial each configured peer and keep the session up.
//!
//! One task per configured peer loops forever: connect, run the session,
//! back off, repeat. While a session with the same remote IP and ASN is
//! already established (for example because the peer dialled us first) the
//! dialer stays idle rather than opening a second one. When both sides dial
//! at once the FSM resolves the collision (see [`crate::collision`]); if our
//! connection is the one closed, the next pass finds the surviving session
//! and goes idle.
use std::sync::Arc;
use std::time::{Duration, SystemTime};
use tokio::net::{TcpSocket, TcpStream};
use tokio::task::JoinHandle;
use tracing::{info, warn};
use crate::collision::Direction;
use crate::config::PeerConfig;
use crate::connection::{PeerSettings, ServerContext, run_session};
use crate::session::DialState;
/// Upper bound for exponential backoff between connection attempts.
const MAX_BACKOFF: Duration = Duration::from_secs(120);
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
pub fn spawn_dialers(
peers: &[PeerConfig],
server: &Arc<ServerContext>,
connect_retry: Duration,
) -> Vec<JoinHandle<()>> {
peers
.iter()
.map(|peer| tokio::spawn(dial_loop(peer.clone(), Arc::clone(server), connect_retry)))
.collect()
}
/// Opens the TCP connection, from the configured source address if any.
async fn connect(peer: &PeerConfig) -> std::io::Result<TcpStream> {
let Some(local) = peer.local_address else {
return TcpStream::connect(peer.endpoint).await;
};
let socket = if peer.endpoint.is_ipv4() {
TcpSocket::new_v4()?
} else {
TcpSocket::new_v6()?
};
socket
.bind(std::net::SocketAddr::new(local, 0))
.map_err(|e| {
std::io::Error::new(e.kind(), format!("cannot use local_address {local}: {e}"))
})?;
socket.connect(peer.endpoint).await
}
async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry: Duration) {
let registry = &server.registry;
let mut backoff = connect_retry;
loop {
if registry.is_shutting_down() {
return;
}
if registry.find(peer.endpoint.ip(), peer.remote_asn).is_some() {
// Peer dialled us; nothing to do until that session goes away.
registry.set_dial_state(&peer.name, DialState::Idle);
tokio::time::sleep(connect_retry).await;
continue;
}
registry.set_dial_state(&peer.name, DialState::Connecting);
info!(peer = %peer.name, endpoint = %peer.endpoint, "Connecting to peer");
let ended = match tokio::time::timeout(CONNECT_TIMEOUT, connect(&peer)).await {
Ok(Ok(stream)) => {
info!(peer = %peer.name, endpoint = %peer.endpoint, "TCP connected, starting session");
let end = run_session(
stream,
peer.endpoint,
Direction::Outbound,
Arc::clone(&server),
PeerSettings::from(&peer),
)
.await;
// A session that came up is a peer worth dialling again
// soon. One that connects and is then refused, for its AS
// number say, will be refused again: it is retried ever
// more slowly, like one that cannot be reached at all.
if end.established {
backoff = connect_retry;
}
end.result.map_err(|e| e.to_string())
}
Ok(Err(e)) => Err(format!("connect failed: {e}")),
Err(_) => Err(format!("connect timed out after {CONNECT_TIMEOUT:?}")),
};
let reason = match ended {
Ok(()) => "session closed at our request".to_string(),
Err(reason) => {
warn!(peer = %peer.name, endpoint = %peer.endpoint, err = %reason, "Session attempt ended");
reason
}
};
registry.set_dial_state(
&peer.name,
DialState::Backoff {
until: SystemTime::now() + backoff,
last_error: reason,
},
);
tokio::time::sleep(backoff).await;
backoff = (backoff * 2).min(MAX_BACKOFF);
}
}
crates/server/src/error.rs
+248
View File
@@ -0,0 +1,248 @@
//! Error types of the server crate.
use std::io;
use std::net::SocketAddr;
use std::time::Duration;
use bgp_packet::errors::{BgpError, CeaseSubcode, DecodeError, FsmSubcode, OpenMsgSubcode};
use bgp_packet::message::NotificationMessage;
use bgp_primitives::parser::EncodeError;
use bgp_rib::RibStopped;
use bgp_rib::table::TooManyPeers;
use thiserror::Error;
use crate::family::FamilySet;
use crate::fsm::State;
/// Why a session ended abnormally.
#[derive(Debug, Error)]
pub enum SessionError {
#[error("connection failed: {0}")]
Io(#[from] io::Error),
#[error("received a message that could not be decoded: {0}")]
Decode(#[from] DecodeError),
#[error("could not encode a message: {0}")]
Encode(#[from] EncodeError),
/// The peer sent nothing for as long as the pre-negotiation hold time.
#[error("timed out waiting for a message in the {0} state")]
Timeout(State),
#[error("hold timer expired")]
HoldTimerExpired,
#[error("received {received} in the {state} state")]
UnexpectedMessage {
state: State,
received: &'static str,
},
/// The peer ended the session. Nothing is sent in return.
#[error("peer sent NOTIFICATION: {0}")]
NotifiedByPeer(BgpError),
#[error("refused the peer's OPEN: {0}")]
OpenRefused(BgpError),
#[error("peer announced AS{actual} but AS{expected} was configured")]
WrongPeerAs { expected: u32, actual: u32 },
#[error("closed by connection collision resolution")]
Collision,
/// The routing table has no room for another peer.
#[error(transparent)]
TableFull(#[from] TooManyPeers),
/// A bug: the session's own code panicked. The peer is told the session
/// is being reset, and the session is cleaned up like any other.
#[error("the session panicked: {0}")]
Panicked(String),
/// The peer offers none of the address families configured for it, so
/// the session could carry nothing.
#[error("no address family in common: we offer {ours}, the peer offers {theirs}")]
NoCommonAddressFamily { ours: FamilySet, theirs: FamilySet },
/// The peer took nothing we sent for this long (RFC 9687): it has
/// stopped reading, and there is no telling it so.
#[error("the peer accepted no data for {0:?}")]
SendHoldTimerExpired(Duration),
}
impl SessionError {
/// The NOTIFICATION that tells the peer about this error (RFC 4271 §6),
/// or `None` when there is nothing to say or no way to say it.
pub fn notification(&self) -> Option<NotificationMessage> {
let error = match self {
Self::Io(_) | Self::SendHoldTimerExpired(_) | Self::NotifiedByPeer(_) => return None,
// Something of ours could not be written out, which is a bug
// here: as for a panic, the session is reset.
Self::Encode(_) => BgpError::Cease(CeaseSubcode::AdminReset),
Self::Decode(e) => {
return Some(NotificationMessage {
error: e.notification.clone(),
data: e.data.clone(),
});
}
Self::Timeout(_) | Self::HoldTimerExpired => BgpError::HoldTimer,
Self::UnexpectedMessage { state, .. } => BgpError::FsmError(match state {
State::OpenSent => FsmSubcode::UnexpectedOpenSent,
State::OpenConfirm => FsmSubcode::UnexpectedOpenConfirm,
// A closed machine handles no messages; Established is the
// only other state one can arrive in.
State::Established | State::Closed => FsmSubcode::UnexpectedEstablished,
}),
Self::OpenRefused(error) => error.clone(),
Self::WrongPeerAs { .. } => BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs),
Self::Collision => BgpError::Cease(CeaseSubcode::ConnCollisionResolution),
Self::TableFull(_) => BgpError::Cease(CeaseSubcode::OutOfResources),
// The session will be set up again; nothing is wrong at the
// peer's end.
Self::Panicked(_) => BgpError::Cease(CeaseSubcode::AdminReset),
// RFC 5492 §5: the Data lists the capabilities we need and the
// peer did not offer.
Self::NoCommonAddressFamily { ours, .. } => {
return Some(NotificationMessage {
error: BgpError::OpenMsg(OpenMsgSubcode::UnsupportedCapability),
data: ours.capabilities_wire(),
});
}
};
Some(NotificationMessage {
error,
data: vec![],
})
}
}
/// Why the server could not start, or could not go on.
#[derive(Debug, Error)]
pub enum ServerError {
#[error("failed to bind listener on {addr}: {source}")]
Bind {
addr: SocketAddr,
#[source]
source: io::Error,
},
#[error("listener failed: {0}")]
Io(#[from] io::Error),
/// A RIB's actor died. The server cannot work without its tables.
#[error(transparent)]
RibStopped(#[from] RibStopped),
/// A task the server depends on ended, which none of them do unless
/// they panic.
#[error("the {task} task stopped unexpectedly: {reason}")]
TaskStopped { task: &'static str, reason: String },
}
/// Why a configuration was rejected.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum ConfigError {
#[error("invalid router_id {0:?}: must be a dotted-quad IPv4 address other than 0.0.0.0")]
RouterId(String),
#[error("local_asn must be non-zero")]
LocalAsn,
#[error("invalid IP address {0:?}")]
Address(String),
#[error("port {0} is out of range")]
Port(u32),
#[error("http_endpoint.port is required")]
HttpPort,
#[error("peer {peer:?}: {problem}")]
Peer { peer: String, problem: PeerProblem },
#[error("originate {prefix:?}: {problem}")]
Originate {
prefix: String,
problem: OriginateProblem,
},
}
/// What is wrong with one `peers` entry.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum PeerProblem {
#[error("endpoint is missing")]
MissingEndpoint,
#[error("endpoint is invalid: {0}")]
Endpoint(Box<ConfigError>),
#[error("remote_asn must be non-zero")]
RemoteAsn,
#[error("hold_time_seconds must be 0 or at least 3")]
HoldTime,
#[error("there is another peer of this name")]
DuplicateName,
#[error("there is another peer at this address")]
DuplicateAddress,
#[error("local_address {0:?} is not an IP address")]
LocalAddress(String),
#[error("local_address and endpoint are different address families")]
AddressFamilyMismatch,
#[error("address_families has a value that is not an address family")]
AddressFamily,
#[error("add_path has a value that is not one of its settings")]
AddPath,
}
/// What is wrong with one `originate` entry.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum OriginateProblem {
#[error("{0}")]
Prefix(#[from] bgp_primitives::ip_prefix::PrefixError),
#[error("origin {0:?} must be IGP, EGP or INCOMPLETE")]
Origin(String),
#[error("as_path_prepend must be at most {max}")]
Prepend { max: u32 },
#[error("{0}")]
Community(#[from] bgp_packet::community::CommunityError),
#[error("listed more than once")]
Duplicate,
}
#[cfg(test)]
mod tests {
use bgp_packet::errors::MsgHeaderSubcode;
use super::*;
#[test]
fn each_session_error_names_the_notification_rfc_4271_assigns_it() {
let notification = |e: SessionError| e.notification().map(|n| (n.error, n.data));
assert_eq!(
notification(SessionError::Decode(DecodeError {
notification: BgpError::MsgHeader(MsgHeaderSubcode::BadMessageLength),
data: vec![0x13, 0x88],
detail: "too long".into(),
})),
Some((
BgpError::MsgHeader(MsgHeaderSubcode::BadMessageLength),
vec![0x13, 0x88]
))
);
assert_eq!(
notification(SessionError::Timeout(State::OpenSent)),
Some((BgpError::HoldTimer, vec![]))
);
assert_eq!(
notification(SessionError::UnexpectedMessage {
state: State::OpenConfirm,
received: "update"
}),
Some((
BgpError::FsmError(FsmSubcode::UnexpectedOpenConfirm),
vec![]
))
);
assert_eq!(
notification(SessionError::WrongPeerAs {
expected: 1,
actual: 2
}),
Some((BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs), vec![]))
);
// Something of ours that cannot be written is our fault: the peer
// is told the session is being reset, not left to time it out.
assert_eq!(
notification(SessionError::Encode(EncodeError::TooLong {
what: "path attribute",
len: 70000,
max: 65535
})),
Some((BgpError::Cease(CeaseSubcode::AdminReset), vec![]))
);
// A broken connection cannot carry a NOTIFICATION.
assert_eq!(
notification(SessionError::Io(io::ErrorKind::BrokenPipe.into())),
None
);
}
}
crates/server/src/family.rs
+292
View File
@@ -0,0 +1,292 @@
//! The address families a session can carry, and which of them it does.
//!
//! Each side lists the families it is willing to exchange in its OPEN, as
//! Multiprotocol capabilities (RFC 4760 §8). A family is in use on a session
//! only if both sides listed it.
//!
//! Any family can be named, read from a peer and held in a [`FamilySet`].
//! The ones this speaker can actually exchange routes for are [`SUPPORTED`],
//! and only those are ever offered to a peer.
use std::collections::BTreeSet;
use std::fmt;
use bgp_packet::message::{PathAttribute, UpdateMessage};
use bgp_packet::open::{Capability, OpenMessage};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::IpPrefix;
use bytes::BytesMut;
/// The address families this speaker can exchange routes for. Supporting a
/// new one means being able to parse, store and advertise its routes; then
/// it is added here.
pub const SUPPORTED: [AfiSafi; 2] = [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST];
/// A set of address families.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub struct FamilySet(BTreeSet<AfiSafi>);
impl FamilySet {
/// Every family in [`SUPPORTED`].
pub fn supported() -> Self {
SUPPORTED.into_iter().collect()
}
pub fn insert(&mut self, family: AfiSafi) {
self.0.insert(family);
}
pub fn contains(&self, family: AfiSafi) -> bool {
self.0.contains(&family)
}
/// Whether a unicast route for `prefix` may be exchanged.
pub fn carries(&self, prefix: &IpPrefix) -> bool {
self.contains(AfiSafi::unicast(prefix.address_family()))
}
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
pub fn intersection(&self, other: &Self) -> Self {
Self(self.0.intersection(&other.0).copied().collect())
}
pub fn iter(&self) -> impl Iterator<Item = AfiSafi> + '_ {
self.0.iter().copied()
}
/// The families a peer offered in its OPEN, whether or not we support
/// them.
///
/// A peer that sends no Multiprotocol capability at all is a speaker of
/// plain RFC 4271 BGP, which carries IPv4 unicast and nothing else. One
/// that sends any has listed everything it wants, IPv4 unicast included.
pub fn offered_by(open: &OpenMessage) -> Self {
let listed: Self = open
.capabilities()
.filter_map(|capability| match capability {
Capability::MultiProtocol { afi, safi } => Some(AfiSafi::new(*afi, *safi)),
_ => None,
})
.collect();
if listed.is_empty() {
[AfiSafi::IPV4_UNICAST].into_iter().collect()
} else {
listed
}
}
/// These families as Multiprotocol capabilities in wire form: the Data
/// of an Unsupported Capability NOTIFICATION (RFC 5492 §5).
pub fn capabilities_wire(&self) -> Vec<u8> {
let mut out = BytesMut::new();
for family in self.iter() {
// A Multiprotocol capability always encodes; if that ever
// changed, a NOTIFICATION with less Data is still correct.
if Capability::multiprotocol(family).to_wire(&mut out).is_err() {
return vec![];
}
}
out.to_vec()
}
/// Removes from `update` every route of a family that is not in this
/// set, and returns how many prefixes that was.
///
/// The withdrawn routes and NLRI in the body of an UPDATE are IPv4
/// unicast; every other family travels in MP_REACH_NLRI and
/// MP_UNREACH_NLRI, which name theirs.
pub fn restrict(&self, update: &mut UpdateMessage) -> usize {
let mut removed = 0;
if !self.contains(AfiSafi::IPV4_UNICAST) {
removed += update.withdrawn_routes.len() + update.announced_routes.len();
update.withdrawn_routes.clear();
update.announced_routes.clear();
}
update.path_attributes.retain(|attribute| {
let (family, prefixes) = match attribute {
PathAttribute::MpReachNlri(mp) => (mp.family, mp.prefixes.len()),
PathAttribute::MpUnreachNlri(mp) => (mp.family, mp.prefixes.len()),
_ => return true,
};
let keep = self.contains(family);
if !keep {
removed += prefixes;
}
keep
});
removed
}
}
impl FromIterator<AfiSafi> for FamilySet {
fn from_iter<I: IntoIterator<Item = AfiSafi>>(families: I) -> Self {
Self(families.into_iter().collect())
}
}
impl fmt::Display for FamilySet {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
if self.is_empty() {
return f.write_str("none");
}
for (i, family) in self.iter().enumerate() {
if i > 0 {
f.write_str(", ")?;
}
write!(f, "{family}")?;
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use std::net::{Ipv4Addr, Ipv6Addr};
use bgp_packet::message::{
MpReachNlriPathAttribute, MpUnreachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
};
use bgp_packet::open::OpenOption;
use bgp_primitives::constants::{Afi, Safi};
use super::*;
fn open(capabilities: Vec<Capability>) -> OpenMessage {
OpenMessage {
version: 4,
asn: 65001,
hold_time: 90,
identifier: Ipv4Addr::new(10, 0, 0, 2),
options: if capabilities.is_empty() {
vec![]
} else {
vec![OpenOption::Capabilities(capabilities)]
},
}
}
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn only(family: AfiSafi) -> FamilySet {
[family].into_iter().collect()
}
#[test]
fn a_peer_without_multiprotocol_capabilities_speaks_ipv4_unicast() {
assert_eq!(
FamilySet::offered_by(&open(vec![])),
only(AfiSafi::IPV4_UNICAST)
);
// Other capabilities do not change that.
assert_eq!(
FamilySet::offered_by(&open(vec![Capability::RouteRefresh])),
only(AfiSafi::IPV4_UNICAST)
);
}
#[test]
fn a_peer_that_lists_families_gets_exactly_those() {
assert_eq!(
FamilySet::offered_by(&open(vec![Capability::multiprotocol(
AfiSafi::IPV6_UNICAST
)])),
only(AfiSafi::IPV6_UNICAST),
"IPv4 unicast is not implied once a peer lists its families"
);
// Capabilities may be spread over several optional parameters.
let mut spread = open(vec![Capability::multiprotocol(AfiSafi::IPV4_UNICAST)]);
spread
.options
.push(OpenOption::Capabilities(vec![Capability::multiprotocol(
AfiSafi::IPV6_UNICAST,
)]));
assert_eq!(FamilySet::offered_by(&spread), FamilySet::supported());
// A family we do not support is still what the peer offered. It
// drops out when intersected with what we offer.
let evpn = AfiSafi::new(Afi::L2VPN, Safi::EVPN);
let offered = FamilySet::offered_by(&open(vec![
Capability::multiprotocol(evpn),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
]));
assert_eq!(offered.to_string(), "IPv6 unicast, L2VPN EVPN");
assert_eq!(
FamilySet::supported().intersection(&offered),
only(AfiSafi::IPV6_UNICAST)
);
assert!(
FamilySet::supported()
.intersection(&FamilySet::offered_by(&open(vec![
Capability::multiprotocol(evpn)
])))
.is_empty()
);
}
#[test]
fn sets_intersect_and_describe_themselves() {
let v6 = only(AfiSafi::IPV6_UNICAST);
assert_eq!(FamilySet::supported().intersection(&v6), v6);
assert!(only(AfiSafi::IPV4_UNICAST).intersection(&v6).is_empty());
assert_eq!(
FamilySet::supported().to_string(),
"IPv4 unicast, IPv6 unicast"
);
assert_eq!(FamilySet::default().to_string(), "none");
assert!(v6.carries(&prefix("2001:db8::/32")));
assert!(!v6.carries(&prefix("192.0.2.0/24")));
// One capability each: code 1, length 4, AFI, reserved, SAFI.
assert_eq!(
FamilySet::supported().capabilities_wire(),
[1, 4, 0, 1, 0, 1, 1, 4, 0, 2, 0, 1]
);
}
fn mixed_update() -> UpdateMessage {
UpdateMessage {
withdrawn_routes: vec![prefix("198.51.100.0/24").into()],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::LOCALHOST),
prefixes: vec![
prefix("2001:db8:1::/48").into(),
prefix("2001:db8:2::/48").into(),
],
}),
PathAttribute::MpUnreachNlri(MpUnreachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
prefixes: vec![prefix("2001:db8:3::/48").into()],
}),
],
announced_routes: vec![prefix("192.0.2.0/24").into()],
}
}
#[test]
fn restrict_removes_the_routes_of_families_not_in_the_set() {
let mut update = mixed_update();
assert_eq!(FamilySet::supported().restrict(&mut update), 0);
assert_eq!(update, mixed_update());
let mut update = mixed_update();
assert_eq!(only(AfiSafi::IPV4_UNICAST).restrict(&mut update), 3);
assert_eq!(update.announced_routes, vec![prefix("192.0.2.0/24")]);
assert_eq!(update.withdrawn_routes, vec![prefix("198.51.100.0/24")]);
assert_eq!(
update.path_attributes,
vec![PathAttribute::Origin(OriginPathAttribute::IGP)]
);
let mut update = mixed_update();
assert_eq!(only(AfiSafi::IPV6_UNICAST).restrict(&mut update), 2);
assert!(update.announced_routes.is_empty());
assert!(update.withdrawn_routes.is_empty());
assert_eq!(update.path_attributes.len(), 3);
}
}
crates/server/src/fsm.rs
+989 -344
View File
File diff suppressed because it is too large. Load diff
crates/server/src/lib.rs
+10
View File
@@ -1,7 +1,17 @@
pub mod advertise;
pub mod collision;
pub mod config;
pub mod connection;
pub mod dialer;
pub mod error;
pub mod family;
pub mod fsm;
pub mod metrics;
mod panic;
mod routes;
pub mod server;
pub mod session;
mod sync;
pub mod tracing;
pub mod proto {
crates/server/src/main.rs
-29
View File
@@ -1,29 +0,0 @@
use bgp_server::config::Config;
use bgp_server::proto::config::BgpConfig;
use bgp_server::server::BgpServerBuilder;
use bgp_server::tracing::TracingArgs;
use clap::Parser;
use eyre::{Context, Result};
#[derive(Parser, Debug)]
struct ServerArgs {
/// Path to the textproto configuration file.
config: String,
#[command(flatten)]
tracing: TracingArgs,
}
#[tokio::main]
async fn main() -> Result<()> {
let args = ServerArgs::parse();
args.tracing.install()?;
let config_string =
std::fs::read_to_string(&args.config).wrap_err("Failed to open config file")?;
let proto_config = protobuf::text_format::parse_from_str::<BgpConfig>(&config_string)
.wrap_err("Failed to parse config file")?;
let config = Config::from_proto(proto_config).wrap_err("Invalid configuration")?;
BgpServerBuilder::new(config).run().await
}
crates/server/src/metrics.rs
+124
View File
@@ -0,0 +1,124 @@
//! Process-wide Prometheus metrics, registered on the default registry.
use std::sync::LazyLock;
use bgp_packet::message::Message;
use prometheus::{
IntCounter, IntCounterVec, IntGauge, register_int_counter, register_int_counter_vec,
register_int_gauge,
};
pub static MESSAGES_SENT: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_messages_sent_total",
"BGP messages sent, by type",
&["type"]
)
.expect("metric registration")
});
pub static MESSAGES_RECEIVED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_messages_received_total",
"BGP messages received, by type",
&["type"]
)
.expect("metric registration")
});
pub static UPDATE_ERRORS: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_update_errors_total",
"Errors in received UPDATE messages that did not reset the session, by how they were handled (RFC 7606)",
&["handling"]
)
.expect("metric registration")
});
pub static SESSIONS_ESTABLISHED_TOTAL: LazyLock<IntCounter> = LazyLock::new(|| {
register_int_counter!(
"bgp_sessions_established_total",
"Sessions that reached Established"
)
.expect("metric registration")
});
pub static SESSIONS_ESTABLISHED: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_sessions_established",
"Sessions currently in Established"
)
.expect("metric registration")
});
pub static COLLISIONS: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_connection_collisions_total",
"Connection collisions resolved, by which connection was closed",
&["closed"]
)
.expect("metric registration")
});
pub static LOCAL_ROUTES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!("bgp_local_routes", "Locally originated routes")
.expect("metric registration")
});
pub static RIB_ROUTES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_routes",
"Routes learned from peers, counting each peer's separately"
)
.expect("metric registration")
});
pub static RIB_PREFIXES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_prefixes",
"Prefixes with at least one route, and so with a best route"
)
.expect("metric registration")
});
pub static RIB_ATTRIBUTE_SETS: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_attribute_sets",
"Distinct sets of path attributes held for the routes"
)
.expect("metric registration")
});
pub static RIB_JOURNAL_ENTRIES: LazyLock<IntGauge> = LazyLock::new(|| {
register_int_gauge!(
"bgp_rib_journal_entries",
"Changes of best route waiting for a follower to read them"
)
.expect("metric registration")
});
/// Forces registration of every metric so they appear in scrapes before
/// their first observation.
pub fn init() {
LazyLock::force(&MESSAGES_SENT);
LazyLock::force(&MESSAGES_RECEIVED);
LazyLock::force(&UPDATE_ERRORS);
LazyLock::force(&SESSIONS_ESTABLISHED_TOTAL);
LazyLock::force(&SESSIONS_ESTABLISHED);
LazyLock::force(&COLLISIONS);
LazyLock::force(&LOCAL_ROUTES);
LazyLock::force(&RIB_ROUTES);
LazyLock::force(&RIB_PREFIXES);
LazyLock::force(&RIB_ATTRIBUTE_SETS);
LazyLock::force(&RIB_JOURNAL_ENTRIES);
}
pub fn message_type_name(msg: &Message) -> &'static str {
match msg {
Message::Open(_) => "open",
Message::Update(_) => "update",
Message::Notification(_) => "notification",
Message::KeepAlive => "keepalive",
Message::RouteRefresh(_) => "route_refresh",
}
}
crates/server/src/panic.rs
+60
View File
@@ -0,0 +1,60 @@
//! Containing panics.
//!
//! A panic is a bug. It unwinds the task it happens in, and what that task
//! was responsible for still has to be finished properly: a session tells
//! its peer it is closing and gives up its routes; the server, if it lost
//! something it cannot do without, closes every session and exits.
use std::any::Any;
use std::panic::AssertUnwindSafe;
use futures::FutureExt;
/// Runs `future` to completion. If it panics, the panic is caught and its
/// message returned, so that the caller can clean up after it.
///
/// The caller must not go on using what the future was working on as if
/// nothing had happened: a panic can leave it half changed. Closing it down
/// is what this is for.
pub(crate) async fn contained<F: Future>(future: F) -> Result<F::Output, String> {
AssertUnwindSafe(future)
.catch_unwind()
.await
.map_err(|payload| message(payload.as_ref()))
}
/// The text of a panic, from what `catch_unwind` or a `JoinError` carries.
pub(crate) fn message(payload: &(dyn Any + Send)) -> String {
if let Some(text) = payload.downcast_ref::<&'static str>() {
(*text).to_string()
} else if let Some(text) = payload.downcast_ref::<String>() {
text.clone()
} else {
"a panic that carried no message".to_string()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn a_panic_is_caught_with_its_message() {
assert_eq!(contained(async { 7 }).await, Ok(7));
let literal: Result<(), _> = contained(async { panic!("the table is gone") }).await;
assert_eq!(literal, Err("the table is gone".to_string()));
let formatted: Result<(), _> =
contained(async { panic!("index {} out of range", 9) }).await;
assert_eq!(formatted, Err("index 9 out of range".to_string()));
// After a point where the future had been suspended, too.
let later: Result<(), _> = contained(async {
tokio::task::yield_now().await;
panic!("later")
})
.await;
assert_eq!(later, Err("later".to_string()));
}
}
crates/server/src/routes.rs
+64
View File
@@ -0,0 +1,64 @@
//! Putting the routes of a received UPDATE into the routing table.
use std::sync::Arc;
use bgp_packet::message::{NlriNextHop, PathAttribute, UpdateMessage};
use bgp_primitives::ip_prefix::Nlri;
use bgp_primitives::parser::EncodeError;
use bgp_rib::table::{AttrSet, Outcome, PeerIndex, RibTable};
/// Applies `update`, received from `peer`, to `table`: its withdrawals
/// first and then its announcements, so that a prefix named in both ends up
/// announced (RFC 4271 §4.3).
///
/// An UPDATE names prefixes in four places: withdrawn routes and NLRI in the
/// message body, which are IPv4, and the MP_UNREACH_NLRI and MP_REACH_NLRI
/// attributes for every other family (RFC 4760). Where the peer sends
/// several routes for a prefix (ADD-PATH, RFC 7911), each is named by its
/// prefix and path identifier together, and is a route of its own.
///
/// `attributes_wire` is the attributes as the decoder kept them, if it did
/// (see `DecodedUpdate::attributes_wire`). With it the table finds the
/// attribute set from those bytes, without writing the attributes out again.
pub(crate) fn apply(
table: &RibTable,
peer: PeerIndex,
update: &UpdateMessage,
attributes_wire: Option<&[u8]>,
) -> Result<Outcome, EncodeError> {
let mut outcome = Outcome::default();
if !update.withdrawn_routes.is_empty() {
outcome += table.withdraw(peer, &update.withdrawn_routes);
}
for attribute in &update.path_attributes {
if let PathAttribute::MpUnreachNlri(mp) = attribute {
outcome += table.withdraw(peer, &mp.prefixes);
}
}
let attributes = |next_hop: Option<&NlriNextHop>| -> Result<Arc<AttrSet>, EncodeError> {
if let Some(set) =
attributes_wire.and_then(|wire| table.attrs().intern_received(wire, next_hop))
{
return Ok(set);
}
table.attrs().intern(&update.path_attributes, next_hop)
};
let mut announce = |nlri: &[Nlri], next_hop| -> Result<(), EncodeError> {
if !nlri.is_empty() {
outcome += table.announce(peer, &attributes(next_hop)?, nlri);
}
Ok(())
};
// Routes in the body have their next hop in the NEXT_HOP attribute,
// which is one of the set's attributes; routes in MP_REACH_NLRI bring
// theirs along.
announce(&update.announced_routes, None)?;
for attribute in &update.path_attributes {
if let PathAttribute::MpReachNlri(mp) = attribute {
announce(&mp.prefixes, Some(&mp.next_hop))?;
}
}
Ok(outcome)
}
crates/server/src/server.rs
+240 -46
View File
@@ -1,70 +1,109 @@
//! The server: listeners, the tasks that run for as long as it does, and
//! what happens when one of them stops.
use std::io;
use std::net::SocketAddr;
use std::pin::Pin;
use std::sync::Arc;
use std::time::Duration;
use eyre::{Result, WrapErr};
use futures::stream::{FuturesUnordered, StreamExt};
use tokio::net::{TcpListener, TcpStream};
use tracing::{error, info};
use tokio::net::{TcpListener, TcpSocket, TcpStream};
use tokio::task::{JoinError, JoinHandle};
use tracing::{error, info, warn};
use bgp_rib::AdjRibIn;
use bgp_rib::table::RibTable;
use bgp_rib::{LocalRib, RibStopped};
use crate::advertise::spawn_advertiser;
use crate::collision::{CollisionTable, Direction};
use crate::config::Config;
use crate::connection::handle_connection;
use crate::fsm::SessionConfig;
use crate::connection::{PeerSettings, ServerContext, run_session};
use crate::dialer::spawn_dialers;
use crate::error::ServerError;
use crate::panic;
use crate::session::{ConfiguredPeer, SessionRegistry};
pub struct BgpServerBuilder {
config: Config,
}
/// How long sessions get to send their Cease when the server stops.
const SHUTDOWN_GRACE: Duration = Duration::from_secs(2);
/// A `BgpServerBuilder` that has bound all its listeners and is ready to accept.
/// Call [`BgpServer::run`] to start the accept loop.
/// How long to wait before accepting again after an accept failed. What
/// makes it fail (no file descriptors left, usually) does not pass at once,
/// and trying again in a loop would only fill the log.
const ACCEPT_RETRY: Duration = Duration::from_millis(200);
/// How often attribute sets that no route uses any more are dropped. They
/// are not dropped as their last route goes (see `AttrStore::sweep`), and a
/// table whose routes keep changing attributes collects them.
const SWEEP_INTERVAL: Duration = Duration::from_secs(60);
/// How many connections may wait to be accepted on a listener.
const LISTEN_BACKLOG: u32 = 1024;
/// A BGP speaker that has bound its listeners and is ready to accept.
/// Call [`BgpServer::run`] to start it.
pub struct BgpServer {
config: Config,
listeners: Vec<(SocketAddr, TcpListener)>,
/// Shared RIB handle — created at bind time so callers can clone it before
/// the accept loop starts (e.g. tests that want to introspect the RIB).
pub adj_rib_in: AdjRibIn,
/// The routing table: every route learned from peers, and the best one
/// for each prefix. Created at bind time so that callers can share it
/// before the accept loop starts.
pub rib: Arc<RibTable>,
/// Routes this router originates; announced to every established peer.
pub local_rib: LocalRib,
/// Live view of established sessions.
pub registry: SessionRegistry,
}
impl BgpServerBuilder {
pub fn new(config: Config) -> Self {
Self { config }
}
/// Binds all configured listen endpoints and returns a [`BgpServer`].
/// Separating bind from run lets callers (e.g. tests) inspect the actual
/// bound addresses before the accept loop starts.
pub async fn bind(self) -> Result<BgpServer> {
let mut listeners = Vec::with_capacity(self.config.listen_endpoints.len());
for &addr in &self.config.listen_endpoints {
let listener = TcpListener::bind(addr)
.await
.wrap_err_with(|| format!("Failed to bind listener on {addr}"))?;
impl BgpServer {
/// Binds all configured listen endpoints. Separate from
/// [`BgpServer::run`] so that callers (tests, say) can learn the
/// addresses actually bound before any connection is accepted.
pub async fn bind(config: Config) -> Result<Self, ServerError> {
let mut listeners = Vec::with_capacity(config.listen_endpoints.len());
for &addr in &config.listen_endpoints {
let listener = listen(addr).map_err(|source| ServerError::Bind { addr, source })?;
let bound_addr = listener.local_addr()?;
info!(%bound_addr, "Listening for BGP connections");
listeners.push((bound_addr, listener));
}
// Routes from the configuration go in before any session exists;
// each session is sent the whole table when it comes up.
let local_rib = LocalRib::spawn();
for route in &config.originate {
local_rib.announce(route.clone()).await?;
}
let configured = config
.peers
.iter()
.map(|p| ConfiguredPeer {
name: p.name.clone(),
endpoint: p.endpoint,
remote_asn: p.remote_asn,
})
.collect();
Ok(BgpServer {
config: self.config,
listeners,
adj_rib_in: AdjRibIn::spawn(),
rib: Arc::new(RibTable::new(table_shards())),
local_rib,
registry: SessionRegistry::with_configured(configured),
config,
})
}
/// Convenience method: bind then run immediately.
pub async fn run(self) -> Result<()> {
self.bind().await?.run().await
pub fn config(&self) -> &Config {
&self.config
}
}
impl BgpServer {
/// The addresses actually bound (useful when the config uses port 0).
pub fn local_addrs(&self) -> Vec<SocketAddr> {
self.listeners.iter().map(|(addr, _)| *addr).collect()
}
pub async fn run(self) -> Result<()> {
/// Runs the server. Returns only if something it cannot do without has
/// failed, after closing its sessions.
pub async fn run(self) -> Result<(), ServerError> {
crate::metrics::init();
info!(
router_id = %self.config.router_id,
local_asn = self.config.local_asn,
@@ -72,36 +111,154 @@ impl BgpServer {
"BGP server started"
);
let session_config = Arc::new(SessionConfig {
let context = Arc::new(ServerContext {
router_id: self.config.router_id,
local_asn: self.config.local_asn,
hold_time_secs: self.config.hold_time_secs,
adj_rib_in: self.adj_rib_in.clone(),
rib: Arc::clone(&self.rib),
registry: self.registry.clone(),
collisions: CollisionTable::new(self.config.collision_detect_established_state),
});
// The tasks the server cannot do without. None of them ends while
// the server runs. The advertiser comes first: it has to be
// listening before a dialer brings up the first session.
let mut tasks: FuturesUnordered<NamedTask> = FuturesUnordered::new();
tasks.push(named(
"advertiser",
spawn_advertiser(
self.local_rib.clone(),
self.registry.clone(),
self.config.local_asn,
),
));
tasks.push(named("sweeper", spawn_sweeper(Arc::clone(&self.rib))));
for dialer in spawn_dialers(&self.config.peers, &context, self.config.connect_retry) {
tasks.push(named("dialer", dialer));
}
let mut accepting: FuturesUnordered<_> = self
.listeners
.into_iter()
.map(|(addr, listener)| accept_one(addr, listener))
.collect();
while let Some((listen_addr, result, listener)) = accepting.next().await {
loop {
let (listen_addr, result, listener) = tokio::select! {
Some(accepted) = accepting.next() => accepted,
failure = failed(&self.local_rib, &mut tasks) => {
error!(err = %failure, "Shutting down: the server cannot continue");
close_sessions(&self.registry).await;
return Err(failure);
}
};
match result {
Ok((stream, peer_addr)) => {
tokio::spawn(handle_connection(
stream,
peer_addr,
Arc::clone(&session_config),
));
// A connection from a configured peer is run with what
// is configured for that peer.
let peer = self.config.peer_at(peer_addr.ip()).map_or_else(
|| PeerSettings::unconfigured(self.config.hold_time_secs),
PeerSettings::from,
);
tokio::spawn(serve(stream, peer_addr, Arc::clone(&context), peer));
}
Err(e) => {
error!(%listen_addr, err = %e, "Accept error");
tokio::time::sleep(ACCEPT_RETRY).await;
}
}
accepting.push(accept_one(listen_addr, listener));
}
}
}
Ok(())
/// Closes every session politely and waits a moment for them to go: each
/// peer is told we are going (Cease / Administrative Shutdown), so that it
/// stops using our routes now and not when its hold timer expires.
pub async fn close_sessions(registry: &SessionRegistry) {
let left = registry.shutdown(SHUTDOWN_GRACE).await;
if left > 0 {
warn!(sessions = left, "Some sessions did not close in time");
}
}
/// Opens a listening socket on `addr`.
fn listen(addr: SocketAddr) -> io::Result<TcpListener> {
let socket = match addr {
SocketAddr::V4(_) => TcpSocket::new_v4()?,
SocketAddr::V6(_) => {
let socket = TcpSocket::new_v6()?;
// IPv6 only. Where a system's default is to take IPv4
// connections on such a socket too, listening on both 0.0.0.0
// and [::] would fail on the second, and IPv4 peers would come
// in under IPv4-mapped addresses that match no configured peer.
socket2::SockRef::from(&socket).set_only_v6(true)?;
socket
}
};
socket.set_reuseaddr(true)?;
socket.bind(addr)?;
socket.listen(LISTEN_BACKLOG)
}
/// Runs a session on a connection a peer opened, and logs how it ended.
async fn serve(
stream: TcpStream,
peer_addr: SocketAddr,
context: Arc<ServerContext>,
peer: PeerSettings,
) {
let end = run_session(stream, peer_addr, Direction::Inbound, context, peer).await;
if let Err(e) = end.result {
warn!(%peer_addr, err = %e, "BGP session terminated with error");
}
}
/// Drops, every so often, the attribute sets no route uses any more.
fn spawn_sweeper(rib: Arc<RibTable>) -> JoinHandle<()> {
tokio::spawn(async move {
let mut tick = tokio::time::interval(SWEEP_INTERVAL);
loop {
tick.tick().await;
let rib = Arc::clone(&rib);
// Takes each of the store's locks in turn and looks at every
// set: not for a thread that sessions run on.
let _ = tokio::task::spawn_blocking(move || rib.attrs().sweep()).await;
}
})
}
/// How many shards the routing table has. Sessions write to it from every
/// core at once; a good many more shards than cores keeps two of them from
/// wanting the same one.
fn table_shards() -> usize {
16 * std::thread::available_parallelism().map_or(1, |cores| cores.get())
}
type NamedTask = Pin<Box<dyn Future<Output = (&'static str, Result<(), JoinError>)> + Send>>;
fn named(name: &'static str, task: JoinHandle<()>) -> NamedTask {
Box::pin(async move { (name, task.await) })
}
/// Resolves when something the server depends on has stopped: the local
/// RIB's actor or one of `tasks`. In a healthy server it never resolves.
///
/// Nothing else would notice. A stopped RIB is only found out by whoever
/// next uses it, and a stopped task by nobody at all: the server would go
/// on accepting sessions while storing or announcing nothing.
async fn failed(local_rib: &LocalRib, tasks: &mut FuturesUnordered<NamedTask>) -> ServerError {
tokio::select! {
() = local_rib.stopped() => RibStopped { rib: "local RIB" }.into(),
Some((task, result)) = tasks.next() => ServerError::TaskStopped {
task,
reason: match result {
Ok(()) => "it returned".into(),
Err(e) if e.is_panic() => {
format!("it panicked: {}", panic::message(e.into_panic().as_ref()))
}
Err(e) => e.to_string(),
},
},
}
}
@@ -112,3 +269,40 @@ async fn accept_one(
let result = listener.accept().await;
(addr, result, listener)
}
#[cfg(test)]
mod tests {
use super::*;
/// The advertiser and the dialers run for as long as the server does.
/// If one of them panics, nothing else would ever find out.
#[tokio::test]
async fn a_task_that_panics_is_a_failure_of_the_server() {
let local_rib = LocalRib::spawn();
let mut tasks: FuturesUnordered<NamedTask> = FuturesUnordered::new();
tasks.push(named("dialer", tokio::spawn(std::future::pending())));
tasks.push(named(
"advertiser",
tokio::spawn(async { panic!("no route to announce") }),
));
let failure = failed(&local_rib, &mut tasks).await;
assert_eq!(
failure.to_string(),
"the advertiser task stopped unexpectedly: it panicked: no route to announce"
);
}
#[tokio::test]
async fn a_stopped_rib_is_a_failure_of_the_server() {
let local_rib = LocalRib::spawn();
let mut tasks: FuturesUnordered<NamedTask> = FuturesUnordered::new();
tasks.push(named("advertiser", tokio::spawn(std::future::pending())));
local_rib.stop_for_testing();
let failure = failed(&local_rib, &mut tasks).await;
assert!(
matches!(failure, ServerError::RibStopped(RibStopped { rib }) if rib == "local RIB")
);
}
}
crates/server/src/session.rs
+288
View File
@@ -0,0 +1,288 @@
//! Registry of established BGP sessions.
//!
//! A session registers itself when it reaches Established and is removed when
//! its [`SessionGuard`] drops, so a session that errors out is always
//! deregistered. Other parts of the server (the advertiser, the HTTP API) use
//! the registry to enumerate live sessions and queue outbound messages.
use std::collections::HashMap;
use std::net::{IpAddr, SocketAddr};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, RwLock};
use std::time::{Duration, SystemTime};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::NotificationMessage;
use bgp_packet::message::Message;
use bgp_rib::table::PeerIndex;
use tokio::sync::{broadcast, mpsc};
use crate::collision::Direction;
use crate::family::FamilySet;
use crate::fsm::Negotiated;
use crate::metrics::SESSIONS_ESTABLISHED;
use crate::sync::IgnorePoisonRw;
const EVENT_CHANNEL_CAPACITY: usize = 256;
/// Identifies a session by the peer's TCP address and its AS number.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PeerId {
pub addr: SocketAddr,
pub asn: u32,
}
impl std::fmt::Display for PeerId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}(AS{})", self.addr, self.asn)
}
}
/// Where the dialer is in bringing up a configured peer.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum DialState {
/// Not dialling: the peer connected to us, or the dialer has not started.
Idle,
Connecting,
/// The last attempt failed or the session ended; waiting to retry.
Backoff {
until: SystemTime,
/// How the last attempt or session ended.
last_error: String,
},
}
/// A peer from the configuration file, whether or not it is currently up.
#[derive(Debug, Clone)]
pub struct ConfiguredPeer {
pub name: String,
pub endpoint: SocketAddr,
pub remote_asn: u32,
}
/// Handle to one established session. Cloning is cheap.
#[derive(Debug, Clone)]
pub struct SessionHandle {
pub peer: PeerId,
/// Name from the configuration, if this peer is configured.
pub name: Option<String>,
/// Which side opened the TCP connection.
pub direction: Direction,
/// Our side of the TCP connection; used as the next hop for routes we
/// originate to this peer.
pub local_addr: SocketAddr,
pub established_at: SystemTime,
/// Whether the peer is in our own AS.
pub ibgp: bool,
/// Whether the peer negotiated RFC 6793 four-octet ASNs. Determines how
/// AS_PATH is encoded on the wire for this session.
pub four_octet_asn: bool,
/// The address families negotiated with the peer. Routes of any other
/// family are neither sent to it nor accepted from it.
pub families: FamilySet,
/// The families for which the peer sends us several routes for a
/// prefix (ADD-PATH, RFC 7911), and those for which we send it path
/// identifiers.
pub add_path_receive: FamilySet,
pub add_path_send: FamilySet,
/// The peer's number in the routing table: what its routes there are
/// marked with.
pub rib_peer: PeerIndex,
outbound: mpsc::Sender<Message>,
}
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
#[error("session with {0} has closed")]
pub struct SessionClosed(pub PeerId);
impl SessionHandle {
pub(crate) fn new(
peer: PeerId,
name: Option<String>,
direction: Direction,
local_addr: SocketAddr,
negotiated: &Negotiated,
rib_peer: PeerIndex,
outbound: mpsc::Sender<Message>,
) -> Self {
Self {
peer,
name,
direction,
local_addr,
established_at: SystemTime::now(),
ibgp: negotiated.ibgp,
four_octet_asn: negotiated.four_octet_asn,
families: negotiated.families.clone(),
add_path_receive: negotiated.add_path_receive.clone(),
add_path_send: negotiated.add_path_send.clone(),
rib_peer,
outbound,
}
}
/// Queues a message for transmission on this session. Waits for queue
/// space if the session's writer is behind.
pub async fn send(&self, msg: Message) -> Result<(), SessionClosed> {
self.outbound
.send(msg)
.await
.map_err(|_| SessionClosed(self.peer))
}
}
#[derive(Debug, Clone)]
pub enum SessionEvent {
Up(SessionHandle),
Down(PeerId),
/// The peer sent ROUTE-REFRESH (RFC 2918): it wants everything we
/// advertise to it sent again, typically because its import policy
/// changed.
RefreshRequested(SessionHandle),
}
#[derive(Clone)]
pub struct SessionRegistry {
inner: Arc<RwLock<HashMap<PeerId, SessionHandle>>>,
configured: Arc<Vec<ConfiguredPeer>>,
dial_states: Arc<RwLock<HashMap<String, DialState>>>,
shutting_down: Arc<AtomicBool>,
events: broadcast::Sender<SessionEvent>,
}
impl Default for SessionRegistry {
fn default() -> Self {
Self::new()
}
}
impl SessionRegistry {
pub fn new() -> Self {
Self::with_configured(vec![])
}
pub fn with_configured(configured: Vec<ConfiguredPeer>) -> Self {
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let dial_states = configured
.iter()
.map(|p| (p.name.clone(), DialState::Idle))
.collect();
Self {
inner: Arc::new(RwLock::new(HashMap::new())),
configured: Arc::new(configured),
dial_states: Arc::new(RwLock::new(dial_states)),
shutting_down: Arc::new(AtomicBool::new(false)),
events,
}
}
/// Peers from the configuration file, each with the dialer's current
/// state for it.
pub fn configured_peers(&self) -> Vec<(ConfiguredPeer, DialState)> {
let states = self.dial_states.reading();
self.configured
.iter()
.map(|p| {
(
p.clone(),
states.get(&p.name).cloned().unwrap_or(DialState::Idle),
)
})
.collect()
}
pub(crate) fn set_dial_state(&self, name: &str, state: DialState) {
self.dial_states.writing().insert(name.to_string(), state);
}
/// An established session with this remote IP and ASN, regardless of
/// which side opened the TCP connection.
pub fn find(&self, ip: IpAddr, asn: u32) -> Option<SessionHandle> {
self.inner
.reading()
.values()
.find(|h| h.peer.addr.ip() == ip && h.peer.asn == asn)
.cloned()
}
/// Adds a session and emits [`SessionEvent::Up`]. The returned guard
/// removes the session again when dropped.
pub(crate) fn register(&self, handle: SessionHandle) -> SessionGuard {
let peer = handle.peer;
self.inner.writing().insert(peer, handle.clone());
SESSIONS_ESTABLISHED.inc();
let _ = self.events.send(SessionEvent::Up(handle));
SessionGuard {
registry: self.clone(),
peer,
}
}
/// All currently established sessions, in no particular order.
pub fn list(&self) -> Vec<SessionHandle> {
self.inner.reading().values().cloned().collect()
}
/// Called by a session when its peer asks for a route refresh.
pub(crate) fn request_refresh(&self, peer: &PeerId) {
let handle = self.inner.reading().get(peer).cloned();
if let Some(handle) = handle {
let _ = self.events.send(SessionEvent::RefreshRequested(handle));
}
}
/// True once [`shutdown`](Self::shutdown) has begun; dialers stop
/// reconnecting.
pub fn is_shutting_down(&self) -> bool {
self.shutting_down.load(Ordering::Relaxed)
}
/// Closes every session politely: each peer is sent Cease /
/// Administrative Shutdown (RFC 4486) so it can drop the session at once
/// instead of discovering a dead TCP connection. Waits up to `grace` for
/// the sessions to finish; returns how many were still open.
pub async fn shutdown(&self, grace: Duration) -> usize {
self.shutting_down.store(true, Ordering::Relaxed);
for session in self.list() {
// Not waited for: a session whose queue is full is one whose
// peer is not reading, and it would hold up the goodbye to all
// the others. It ends when the process does.
let _ = session
.outbound
.try_send(Message::Notification(NotificationMessage {
error: BgpError::Cease(CeaseSubcode::AdminShutdown),
data: vec![],
}));
}
let deadline = tokio::time::Instant::now() + grace;
while !self.list().is_empty() && tokio::time::Instant::now() < deadline {
tokio::time::sleep(Duration::from_millis(20)).await;
}
self.list().len()
}
pub fn subscribe(&self) -> broadcast::Receiver<SessionEvent> {
self.events.subscribe()
}
fn remove(&self, peer: &PeerId) {
let removed = self.inner.writing().remove(peer);
if removed.is_some() {
SESSIONS_ESTABLISHED.dec();
let _ = self.events.send(SessionEvent::Down(*peer));
}
}
}
/// Removes the session from its registry on drop.
pub struct SessionGuard {
registry: SessionRegistry,
peer: PeerId,
}
impl Drop for SessionGuard {
fn drop(&mut self) {
self.registry.remove(&self.peer);
}
}
crates/server/src/sync.rs
+66
View File
@@ -0,0 +1,66 @@
//! Locks that survive a panic.
//!
//! A standard lock is poisoned when a thread panics while holding it, and
//! every later `lock().unwrap()` then panics too. For the tables shared by
//! all sessions that would turn a bug in one session into the end of every
//! session, one panic at a time.
//!
//! The tables here are maps that are changed by single inserts and removes,
//! so a panic part-way through an update cannot leave one in a state that
//! is unsafe to read. The poison is therefore ignored: a session that
//! panics is cleaned up by its guards like one that ended any other way.
use std::sync::{Mutex, MutexGuard, PoisonError, RwLock, RwLockReadGuard, RwLockWriteGuard};
pub(crate) trait IgnorePoison<T> {
/// Locks, whether or not an earlier holder panicked.
fn locked(&self) -> MutexGuard<'_, T>;
}
impl<T> IgnorePoison<T> for Mutex<T> {
fn locked(&self) -> MutexGuard<'_, T> {
self.lock().unwrap_or_else(PoisonError::into_inner)
}
}
pub(crate) trait IgnorePoisonRw<T> {
/// Locks for reading, whether or not an earlier holder panicked.
fn reading(&self) -> RwLockReadGuard<'_, T>;
/// Locks for writing, whether or not an earlier holder panicked.
fn writing(&self) -> RwLockWriteGuard<'_, T>;
}
impl<T> IgnorePoisonRw<T> for RwLock<T> {
fn reading(&self) -> RwLockReadGuard<'_, T> {
self.read().unwrap_or_else(PoisonError::into_inner)
}
fn writing(&self) -> RwLockWriteGuard<'_, T> {
self.write().unwrap_or_else(PoisonError::into_inner)
}
}
#[cfg(test)]
mod tests {
use std::sync::Arc;
use super::*;
#[test]
fn a_lock_whose_holder_panicked_can_still_be_taken() {
let mutex = Arc::new(Mutex::new(1));
let rwlock = Arc::new(RwLock::new(1));
let (m, r) = (mutex.clone(), rwlock.clone());
let panicked = std::thread::spawn(move || {
let _m = m.lock().unwrap();
let _r = r.write().unwrap();
panic!("while holding both");
})
.join();
assert!(panicked.is_err());
assert!(mutex.is_poisoned() && rwlock.is_poisoned());
*mutex.locked() += 1;
*rwlock.writing() += 1;
assert_eq!((*mutex.locked(), *rwlock.reading()), (2, 2));
}
}
crates/server/src/tracing.rs
+7 -9
View File
@@ -1,26 +1,24 @@
use clap::Args;
use eyre::Result;
use tracing::Level;
use tracing_subscriber::EnvFilter;
use tracing_subscriber::filter::ParseError;
/// Default targets that we enable logging for.
const DEFAULT_TARGETS: [&str; 1] = ["bgp_server"];
const DEFAULT_TARGETS: [&str; 3] = ["bgp", "bgp_server", "bgp_web"];
#[derive(Debug, Args)]
pub struct TracingArgs {
/// Set the minimum log level.
#[arg(short, long, global = true, default_value_t = Level::TRACE, help_heading = "Tracing")]
/// Set the minimum log level. At `debug` and below there is a line for
/// every UPDATE and every route announced.
#[arg(short, long, global = true, default_value_t = Level::INFO, help_heading = "Tracing")]
level: Level,
/// Disable logging to stdout.
#[arg(long, global = true, default_value_t = false, help_heading = "Tracing")]
quiet: bool,
}
impl TracingArgs {
pub fn install(&self) -> Result<()> {
pub fn install(&self) -> Result<(), ParseError> {
let filter = DEFAULT_TARGETS.into_iter().try_fold(
EnvFilter::builder().from_env_lossy(),
|filter, target| -> Result<_> {
|filter, target| -> Result<_, ParseError> {
Ok(filter.add_directive(format!("{}={}", target, self.level).parse()?))
},
)?;
crates/web/Cargo.toml
+25
View File
@@ -0,0 +1,25 @@
[package]
name = "bgp-web"
description = "HTTP API, event stream and metrics for the BGP daemon"
authors.workspace = true
edition.workspace = true
homepage.workspace = true
license.workspace = true
repository.workspace = true
version.workspace = true
[dependencies]
axum.workspace = true
bgp-packet.workspace = true
bgp-primitives.workspace = true
bgp-rib.workspace = true
bgp-server.workspace = true
futures.workspace = true
prometheus.workspace = true
serde.workspace = true
tokio.workspace = true
tokio-stream.workspace = true
tracing.workspace = true
[lints]
workspace = true
crates/web/src/api.rs
+406
View File
@@ -0,0 +1,406 @@
//! JSON handlers and their view types. Everything here is read-only: the
//! daemon is configured through its configuration file.
use std::time::{SystemTime, UNIX_EPOCH};
use axum::Json;
use std::collections::HashMap;
use axum::extract::{Query, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use bgp_packet::community;
use bgp_packet::message::{NlriNextHop, OriginPathAttribute, PathAttribute};
use bgp_rib::table::{PeerIndex, Route};
use bgp_rib::{LocalRoute, RibStopped};
use bgp_server::session::{ConfiguredPeer, DialState, PeerId, SessionHandle, SessionRegistry};
use serde::{Deserialize, Serialize};
use crate::AppState;
// ---------------------------------------------------------------------------
// Views
// ---------------------------------------------------------------------------
fn unix_secs(t: SystemTime) -> u64 {
t.duration_since(UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
#[derive(Debug, Serialize)]
pub struct StatusView {
pub router_id: String,
pub local_asn: u32,
pub version: String,
pub hostname: String,
pub started_at_unix: u64,
pub uptime_secs: u64,
pub sessions: usize,
pub local_routes: usize,
/// Routes learned from peers, counting each peer's separately.
pub received_routes: usize,
/// Prefixes with at least one such route, and so with a best route.
pub received_prefixes: usize,
}
#[derive(Debug, Clone, Serialize)]
pub struct PeerView {
/// Name from the configuration file; absent for unconfigured inbound peers.
pub name: Option<String>,
pub peer_addr: String,
pub peer_asn: u32,
/// `Established`, `Connecting`, `Backoff` or `Idle`.
pub state: String,
/// `inbound` or `outbound`: which side opened the live TCP connection.
pub direction: Option<String>,
pub local_addr: Option<String>,
pub established_at_unix: Option<u64>,
pub uptime_secs: Option<u64>,
pub four_octet_asn: Option<bool>,
/// The address families negotiated with the peer, e.g. `IPv4 unicast`.
pub address_families: Option<Vec<String>>,
/// The families for which the peer sends us several routes for a
/// prefix (ADD-PATH, RFC 7911), and those for which we send it path
/// identifiers.
pub add_path_receive: Option<Vec<String>>,
pub add_path_send: Option<Vec<String>>,
/// Why the last connection attempt or session ended, while in Backoff.
pub last_error: Option<String>,
pub retry_at_unix: Option<u64>,
}
impl From<&SessionHandle> for PeerView {
fn from(s: &SessionHandle) -> Self {
Self {
name: s.name.clone(),
peer_addr: s.peer.addr.to_string(),
peer_asn: s.peer.asn,
state: "Established".into(),
direction: Some(s.direction.as_str().into()),
local_addr: Some(s.local_addr.to_string()),
established_at_unix: Some(unix_secs(s.established_at)),
uptime_secs: Some(s.established_at.elapsed().map(|d| d.as_secs()).unwrap_or(0)),
four_octet_asn: Some(s.four_octet_asn),
address_families: Some(s.families.iter().map(|f| f.to_string()).collect()),
add_path_receive: Some(s.add_path_receive.iter().map(|f| f.to_string()).collect()),
add_path_send: Some(s.add_path_send.iter().map(|f| f.to_string()).collect()),
last_error: None,
retry_at_unix: None,
}
}
}
impl PeerView {
fn from_configured(peer: &ConfiguredPeer, state: &DialState) -> Self {
let (state_name, last_error, retry_at_unix) = match state {
DialState::Idle => ("Idle", None, None),
DialState::Connecting => ("Connecting", None, None),
DialState::Backoff { until, last_error } => {
("Backoff", Some(last_error.clone()), Some(unix_secs(*until)))
}
};
Self {
name: Some(peer.name.clone()),
peer_addr: peer.endpoint.to_string(),
peer_asn: peer.remote_asn,
state: state_name.into(),
direction: None,
local_addr: None,
established_at_unix: None,
uptime_secs: None,
four_octet_asn: None,
address_families: None,
add_path_receive: None,
add_path_send: None,
last_error,
retry_at_unix,
}
}
}
/// Configured peers first (with their live session when established), then
/// any unconfigured inbound sessions.
pub fn peer_views(registry: &SessionRegistry) -> Vec<PeerView> {
let sessions = registry.list();
let mut claimed = std::collections::HashSet::new();
let mut views: Vec<PeerView> = registry
.configured_peers()
.iter()
.map(|(peer, state)| {
match sessions
.iter()
.find(|s| s.peer.addr.ip() == peer.endpoint.ip() && s.peer.asn == peer.remote_asn)
{
Some(session) => {
claimed.insert(session.peer);
PeerView {
name: Some(peer.name.clone()),
..PeerView::from(session)
}
}
None => PeerView::from_configured(peer, state),
}
})
.collect();
let mut rest: Vec<PeerView> = sessions
.iter()
.filter(|s| !claimed.contains(&s.peer))
.map(PeerView::from)
.collect();
rest.sort_by(|a, b| a.peer_addr.cmp(&b.peer_addr));
views.extend(rest);
views
}
#[derive(Debug, Clone, Serialize)]
pub struct LocalRouteView {
pub prefix: String,
pub origin: String,
pub prepend: u8,
pub med: Option<u32>,
pub communities: Vec<String>,
pub large_communities: Vec<String>,
pub announced_at_unix: u64,
}
impl From<&LocalRoute> for LocalRouteView {
fn from(r: &LocalRoute) -> Self {
Self {
prefix: r.prefix.to_string(),
origin: origin_name(r.attrs.origin).to_string(),
prepend: r.attrs.prepend,
med: r.attrs.med,
communities: r
.attrs
.communities
.iter()
.copied()
.map(community::format_standard)
.collect(),
large_communities: r
.attrs
.large_communities
.iter()
.map(community::format_large)
.collect(),
announced_at_unix: unix_secs(r.announced_at),
}
}
}
/// Human-oriented digest of a path attribute list.
#[derive(Debug, Clone, Default, Serialize)]
pub struct AttrSummary {
pub origin: Option<String>,
/// AS_PATH flattened in order; AS_SETs are included in place.
pub as_path: Vec<u32>,
pub next_hop: Option<String>,
pub med: Option<u32>,
pub local_pref: Option<u32>,
pub communities: Vec<String>,
pub large_communities: Vec<String>,
}
pub fn summarize(attributes: &[PathAttribute], next_hop: Option<&NlriNextHop>) -> AttrSummary {
let mut s = AttrSummary {
// A multiprotocol route has its next hop here; an IPv4 route has it
// in the NEXT_HOP attribute below.
next_hop: next_hop.map(ToString::to_string),
..Default::default()
};
for attr in attributes {
match attr {
PathAttribute::Origin(o) => s.origin = Some(origin_name(*o).to_string()),
PathAttribute::AsPath(p) => {
s.as_path = p
.segments
.iter()
.flat_map(|seg| seg.path.iter().copied())
.collect()
}
PathAttribute::NextHop(nh) => s.next_hop = Some(nh.0.to_string()),
PathAttribute::MultiExitDisc(m) => s.med = Some(m.0),
PathAttribute::LocalPref(l) => s.local_pref = Some(l.0),
PathAttribute::Communities(c) => {
s.communities =
c.0.iter()
.copied()
.map(community::format_standard)
.collect()
}
PathAttribute::LargeCommunities(l) => {
s.large_communities = l.communities.iter().map(community::format_large).collect()
}
_ => {}
}
}
s
}
#[derive(Debug, Clone, Serialize)]
pub struct ReceivedRouteView {
pub peer_addr: String,
pub peer_asn: u32,
pub prefix: String,
/// The peer's identifier for the route: zero unless the peer sends
/// several routes for a prefix (ADD-PATH, RFC 7911).
pub path_id: u32,
/// Whether this is the route chosen for its prefix.
pub best: bool,
pub summary: AttrSummary,
/// The decoded attribute list. MP_REACH_NLRI is not in it: its next hop
/// is in the summary and its prefixes are routes of their own.
pub attributes: Vec<PathAttribute>,
}
/// Who the routes in the table are from: the table knows peers by number,
/// the sessions know which number is whose.
pub(crate) struct PeerNames(HashMap<PeerIndex, PeerId>);
impl PeerNames {
pub(crate) fn of(registry: &SessionRegistry) -> Self {
Self(
registry
.list()
.into_iter()
.map(|session| (session.rib_peer, session.peer))
.collect(),
)
}
/// A view of one route. `None` if the route's session has just ended:
/// the route is on its way out of the table.
pub(crate) fn view(&self, route: &Route) -> Option<ReceivedRouteView> {
let peer = self.0.get(&route.peer)?;
// What a session stored was accepted by the decoder, and parses.
let (attributes, next_hop) = route.attrs.attributes().unwrap_or_default();
Some(ReceivedRouteView {
peer_addr: peer.addr.to_string(),
peer_asn: peer.asn,
prefix: route.prefix.to_string(),
path_id: route.path_id,
best: route.best,
summary: summarize(&attributes, next_hop.as_ref()),
attributes,
})
}
}
fn origin_name(o: OriginPathAttribute) -> &'static str {
match o {
OriginPathAttribute::IGP => "IGP",
OriginPathAttribute::EGP => "EGP",
OriginPathAttribute::INCOMPLETE => "INCOMPLETE",
}
}
// ---------------------------------------------------------------------------
// Handlers
// ---------------------------------------------------------------------------
/// A table could not be read because its actor has stopped. The daemon is
/// on its way down; saying so beats answering with an empty table.
pub struct Unavailable(RibStopped);
impl From<RibStopped> for Unavailable {
fn from(e: RibStopped) -> Self {
Self(e)
}
}
impl IntoResponse for Unavailable {
fn into_response(self) -> Response {
(StatusCode::SERVICE_UNAVAILABLE, self.0.to_string()).into_response()
}
}
pub async fn status(State(state): State<AppState>) -> Result<Json<StatusView>, Unavailable> {
Ok(Json(StatusView {
router_id: state.info.router_id.to_string(),
local_asn: state.info.local_asn,
version: state.info.version.clone(),
hostname: state.info.hostname.clone(),
started_at_unix: unix_secs(state.info.started_at),
uptime_secs: state.info.started.elapsed().as_secs(),
sessions: state.registry.list().len(),
local_routes: state.local_rib.len().await?,
received_routes: state.rib.len(),
received_prefixes: state.rib.prefix_count(),
}))
}
/// The configuration file the daemon was started with.
#[derive(Debug, Clone, Serialize)]
pub struct ConfigView {
pub path: Option<String>,
/// Raw file contents as read at startup; later edits to the file are not
/// reflected until the daemon restarts.
pub text: Option<String>,
}
pub async fn config(State(state): State<AppState>) -> Json<ConfigView> {
let (path, text) = state.info.config.clone().unzip();
Json(ConfigView { path, text })
}
pub async fn peers(State(state): State<AppState>) -> Json<Vec<PeerView>> {
Json(peer_views(&state.registry))
}
pub async fn local_routes(
State(state): State<AppState>,
) -> Result<Json<Vec<LocalRouteView>>, Unavailable> {
let mut routes: Vec<LocalRouteView> = state
.local_rib
.snapshot()
.await?
.iter()
.map(LocalRouteView::from)
.collect();
routes.sort_by(|a, b| a.prefix.cmp(&b.prefix));
Ok(Json(routes))
}
/// How many routes [`received_routes`] returns unless asked for another
/// number. The table can hold hundreds of millions.
const DEFAULT_ROUTE_LIMIT: usize = 1000;
/// The most routes [`received_routes`] returns, whatever is asked for:
/// each is copied out of the table and written as JSON.
const MAX_ROUTE_LIMIT: usize = 100_000;
#[derive(Debug, Deserialize)]
pub struct RouteQuery {
/// At most this many routes.
limit: Option<usize>,
}
/// Some of the routes learned from peers: all the routes of as many
/// prefixes as fit in the limit. `/api/status` has the totals.
pub async fn received_routes(
State(state): State<AppState>,
Query(query): Query<RouteQuery>,
) -> Result<Json<Vec<ReceivedRouteView>>, StatusCode> {
let limit = query
.limit
.unwrap_or(DEFAULT_ROUTE_LIMIT)
.min(MAX_ROUTE_LIMIT);
let names = PeerNames::of(&state.registry);
// Copying, parsing and sorting many routes takes long enough to keep
// off the threads that sessions run on.
let routes = tokio::task::spawn_blocking(move || {
let mut routes: Vec<ReceivedRouteView> = state
.rib
.some_routes(limit)
.iter()
.filter_map(|route| names.view(route))
.collect();
routes.sort_by(|a, b| (&a.prefix, &a.peer_addr).cmp(&(&b.prefix, &b.peer_addr)));
routes
})
.await
// Only if making the views panicked.
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
Ok(Json(routes))
}
crates/web/src/events.rs
+198
View File
@@ -0,0 +1,198 @@
//! Server-sent event stream merging session, local-route and best-route
//! changes into one feed the UI can follow.
use std::convert::Infallible;
use std::sync::Arc;
use axum::extract::State;
use axum::response::sse::{Event, KeepAlive, Sse};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::LocalRibEvent;
use bgp_rib::table::{self, Route};
use bgp_server::session::SessionEvent;
use futures::stream::{Stream, StreamExt};
use serde::Serialize;
use tokio::sync::mpsc;
use tokio_stream::wrappers::errors::BroadcastStreamRecvError;
use tokio_stream::wrappers::{BroadcastStream, ReceiverStream};
use crate::AppState;
use crate::api::{LocalRouteView, PeerNames, PeerView, ReceivedRouteView};
#[derive(Debug, Serialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum WireEvent {
SessionUp {
peer: PeerView,
},
SessionDown {
peer_addr: String,
peer_asn: u32,
},
/// The peer asked us to send all our routes again (ROUTE-REFRESH).
RouteRefreshRequested {
peer: PeerView,
},
LocalAnnounced {
route: LocalRouteView,
},
LocalWithdrawn {
prefix: String,
},
/// The best route for a prefix is now this one.
BestChanged {
route: ReceivedRouteView,
},
/// A prefix has no route any more.
BestWithdrawn {
prefix: String,
},
/// The client fell behind and missed events on `stream` (`missed` of
/// them, where that is known); it should refetch the corresponding
/// table.
Lagged {
stream: &'static str,
missed: u64,
},
}
impl WireEvent {
fn from_session(ev: Result<SessionEvent, BroadcastStreamRecvError>) -> Self {
match ev {
Ok(SessionEvent::Up(h)) => Self::SessionUp {
peer: PeerView::from(&h),
},
Ok(SessionEvent::Down(p)) => Self::SessionDown {
peer_addr: p.addr.to_string(),
peer_asn: p.asn,
},
Ok(SessionEvent::RefreshRequested(h)) => Self::RouteRefreshRequested {
peer: PeerView::from(&h),
},
Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
stream: "sessions",
missed: n,
},
}
}
fn from_local(ev: Result<LocalRibEvent, BroadcastStreamRecvError>) -> Self {
match ev {
Ok(LocalRibEvent::Announced(r)) => Self::LocalAnnounced {
route: LocalRouteView::from(&r),
},
Ok(LocalRibEvent::Withdrawn { prefix }) => Self::LocalWithdrawn {
prefix: prefix.to_string(),
},
Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
stream: "local_routes",
missed: n,
},
}
}
}
/// The table's changes of best route as events, from when the client
/// connects. Each client follows the table's journal with a reader of its
/// own, so one that reads slowly holds nobody else up.
fn best_routes(state: &AppState) -> impl Stream<Item = WireEvent> + use<> {
let (events, stream) = mpsc::channel(BEST_ROUTE_BUFFER);
let registry = state.registry.clone();
// Taken before this returns, so that nothing that happens after the
// client is answered is missed.
let mut reader = state.rib.reader_from_now();
tokio::spawn(async move {
// If the reader falls too far behind it is given the whole table
// again. That is not news: the client is told it missed something
// and gets its table from the API.
let mut syncing = false;
loop {
tokio::select! {
() = reader.changed() => {}
() = events.closed() => return,
}
loop {
// Only what is needed is taken while the table is held;
// the views are made afterwards.
let mut changes = Vec::new();
let progress = reader.poll(BEST_ROUTE_BUFFER, |event| match event {
table::Event::Reset => {
syncing = true;
changes.push(Change::Lagged);
}
table::Event::Synced => syncing = false,
table::Event::Best {
prefix,
peer,
path_id,
attrs,
} if !syncing => {
changes.push(Change::Best(Route {
prefix,
peer,
path_id,
attrs: Arc::clone(attrs),
best: true,
}));
}
table::Event::Withdrawn { prefix } if !syncing => {
changes.push(Change::Withdrawn(prefix));
}
_ => {}
});
let names = PeerNames::of(&registry);
for change in changes {
let event = match change {
Change::Best(route) => match names.view(&route) {
Some(route) => WireEvent::BestChanged { route },
None => continue,
},
Change::Withdrawn(prefix) => WireEvent::BestWithdrawn {
prefix: prefix.to_string(),
},
Change::Lagged => WireEvent::Lagged {
stream: "best_routes",
missed: 0,
},
};
if events.send(event).await.is_err() {
return;
}
}
if progress.caught_up {
break;
}
}
}
});
ReceiverStream::new(stream)
}
/// How many best-route events are read from the journal, and queued for a
/// client, at a time.
const BEST_ROUTE_BUFFER: usize = 256;
enum Change {
Best(Route),
Withdrawn(IpPrefix),
/// The client's reader fell too far behind the table.
Lagged,
}
pub async fn stream(
State(state): State<AppState>,
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
let sessions = BroadcastStream::new(state.registry.subscribe()).map(WireEvent::from_session);
let local = BroadcastStream::new(state.local_rib.subscribe()).map(WireEvent::from_local);
let received = best_routes(&state);
let merged =
futures::stream::select(sessions, futures::stream::select(local, received)).map(|ev| {
Ok(Event::default()
.event("bgp")
.json_data(&ev)
.unwrap_or_else(|e| Event::default().event("error").data(e.to_string())))
});
Sse::new(merged).keep_alive(KeepAlive::default())
}
crates/web/src/lib.rs
+127
View File
@@ -0,0 +1,127 @@
//! HTTP surface of the daemon: a read-only JSON API for inspecting state, a
//! status page, a server-sent event stream, and Prometheus metrics.
//!
//! Everything here is a thin layer over handles owned by the BGP server
//! ([`SessionRegistry`], [`LocalRib`], [`RibTable`]); no routing state lives
//! in this crate.
mod api;
mod events;
mod metrics;
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::time::{Instant, SystemTime};
use axum::Router;
use axum::http::header;
use axum::response::{Html, IntoResponse, Redirect};
use axum::routing::get;
use bgp_rib::LocalRib;
use bgp_rib::table::RibTable;
use bgp_server::session::SessionRegistry;
use tokio::net::TcpListener;
use tracing::info;
pub use api::{AttrSummary, ConfigView, LocalRouteView, PeerView, ReceivedRouteView, StatusView};
/// Static facts about the router shown in status output.
#[derive(Debug, Clone)]
pub struct RouterInfo {
pub router_id: Ipv4Addr,
pub local_asn: u32,
pub version: String,
pub started: Instant,
/// Wall-clock start time, for display.
pub started_at: SystemTime,
/// Name of the machine we run on, for display.
pub hostname: String,
/// The configuration file as read at startup (path and raw text), shown
/// on the status page. `None` when the server was configured in code.
pub config: Option<(String, String)>,
}
impl RouterInfo {
pub fn new(router_id: Ipv4Addr, local_asn: u32, version: impl Into<String>) -> Self {
Self {
router_id,
local_asn,
version: version.into(),
started: Instant::now(),
started_at: SystemTime::now(),
hostname: hostname(),
config: None,
}
}
/// Records the configuration file so the status page can show it.
pub fn with_config(mut self, path: impl Into<String>, text: impl Into<String>) -> Self {
self.config = Some((path.into(), text.into()));
self
}
}
/// Best-effort host name without pulling in a dependency: ask the `hostname`
/// utility, which exists on every platform we run on.
fn hostname() -> String {
std::process::Command::new("hostname")
.output()
.ok()
.filter(|o| o.status.success())
.and_then(|o| String::from_utf8(o.stdout).ok())
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "unknown host".into())
}
#[derive(Clone)]
pub struct AppState {
pub info: Arc<RouterInfo>,
pub registry: SessionRegistry,
pub local_rib: LocalRib,
/// The routing table: routes learned from peers and the best of them.
pub rib: Arc<RibTable>,
}
/// The status page, embedded so the binary is self-contained.
const STATUSZ_HTML: &str = include_str!("../static/statusz.html");
async fn statusz() -> impl IntoResponse {
(
[(header::CONTENT_TYPE, "text/html; charset=utf-8")],
Html(STATUSZ_HTML),
)
}
/// Builds the router. Exposed so tests can drive it without a socket.
pub fn router(state: AppState) -> Router {
Router::new()
.route("/", get(|| async { Redirect::temporary("/statusz") }))
.route("/statusz", get(statusz))
.route("/api/status", get(api::status))
.route("/api/config", get(api::config))
.route("/api/peers", get(api::peers))
.route("/api/routes/local", get(api::local_routes))
.route("/api/routes/received", get(api::received_routes))
.route("/api/events", get(events::stream))
.route("/metricz", get(metrics::metricz))
.with_state(state)
}
/// Binds `listen` and serves the API until the task is dropped.
pub async fn serve(state: AppState, listen: SocketAddr) -> std::io::Result<()> {
let listener = TcpListener::bind(listen).await.map_err(|e| {
std::io::Error::new(
e.kind(),
format!("failed to bind HTTP listener on {listen}: {e}"),
)
})?;
serve_on(state, listener).await
}
/// Serves the API on an already-bound listener (lets tests use port 0).
pub async fn serve_on(state: AppState, listener: TcpListener) -> std::io::Result<()> {
info!(addr = %listener.local_addr()?, "HTTP API listening");
axum::serve(listener, router(state)).await?;
Ok(())
}
crates/web/src/metrics.rs
+38
View File
@@ -0,0 +1,38 @@
//! Prometheus scrape endpoint. Counters are updated by the server crate as
//! messages flow; table-size gauges are refreshed at scrape time.
use axum::extract::State;
use axum::http::{StatusCode, header};
use axum::response::IntoResponse;
use bgp_server::metrics::{
LOCAL_ROUTES, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES, RIB_ROUTES,
};
use prometheus::{Encoder, TextEncoder};
use crate::AppState;
pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
// A scrape that cannot read a table fails. Serving the last known
// size would hide that the daemon is broken from the one system whose
// job is to notice.
let gauge = |count: usize| i64::try_from(count).unwrap_or(i64::MAX);
match state.local_rib.len().await {
Ok(local) => LOCAL_ROUTES.set(gauge(local)),
Err(e) => return (StatusCode::SERVICE_UNAVAILABLE, e.to_string()).into_response(),
}
RIB_ROUTES.set(gauge(state.rib.len()));
RIB_PREFIXES.set(gauge(state.rib.prefix_count()));
RIB_ATTRIBUTE_SETS.set(gauge(state.rib.attrs().len()));
RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len()));
let mut buf = Vec::new();
match TextEncoder::new().encode(&prometheus::gather(), &mut buf) {
Ok(()) => (
StatusCode::OK,
[(header::CONTENT_TYPE, "text/plain; version=0.0.4")],
buf,
)
.into_response(),
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
}
}
crates/web/static/statusz.html
+260
View File
@@ -0,0 +1,260 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Status for ferrix_bgp</title>
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16'%3E%3Crect width='16' height='16' fill='%23eef'/%3E%3Ctext x='8' y='12' font-size='11' font-family='sans-serif' font-weight='bold' fill='%2300c' text-anchor='middle'%3EB%3C/text%3E%3C/svg%3E">
<!--
A read-only status page in the manner of the classic /statusz pages: plain
sans-serif HTML, default links, dense bordered tables, and one full-width
light blue (#eef) heading bar per section. Deliberately no dark theme, no
framework, no external resources. Everything shown comes from /api/*.
-->
<style>
body { font-family: sans-serif; font-size: 13px; color: #000; background: #fff; margin: 8px; }
h1 { clear: both; width: 100%; text-align: center; font-size: 120%; background: #eef; margin: 14px 0 6px; padding: 2px 0; }
h1.top { font-size: 150%; margin-top: 0; }
.lefthand { float: left; width: 70%; }
.righthand { text-align: right; }
a { color: #00c; } a:visited { color: #551a8b; }
table.data { border-collapse: collapse; margin: 4px 0; }
table.data th, table.data td { border: 1px solid #99a; padding: 2px 6px; text-align: left; vertical-align: top; }
table.data th { background: #dde; font-weight: bold; }
.mono, table.data td.mono { font-family: monospace; }
.ok { color: #080; font-weight: bold; } .bad { color: #c00; font-weight: bold; } .warn { color: #a60; font-weight: bold; }
.dim { color: #666; }
#event-log { font-family: monospace; font-size: 12px; max-height: 22em; overflow-y: auto; }
#event-log div { white-space: pre-wrap; }
details { clear: both; margin: 8px 0 0; }
summary { cursor: pointer; color: #00c; }
summary:hover { text-decoration: underline; }
#config-text { border: 1px solid #99a; padding: 4px 6px; margin: 4px 0; font-size: 12px; overflow-x: auto; }
address { font-size: smaller; margin-top: 2em; color: #444; }
</style>
</head>
<body>
<h1 class="top">Status for ferrix_bgp</h1>
<div>
<div class="lefthand">
Started: <span id="s-started">...</span> (up <span id="s-uptime">...</span>)<br>
Version <span id="s-version">...</span><br>
Router ID <b id="s-router-id">...</b>, local AS <b id="s-asn">...</b><br>
</div>
<div class="righthand">
Running on <span id="s-hostname">...</span><br>
View: <a href="/metricz">variables</a>, <a href="/api/status">status json</a>, <a href="/api/peers">peers json</a>, <a href="/api/routes/local">originated routes json</a>, <a href="/api/routes/received">received routes json</a><br>
<span id="refresh-ctl"></span><br>
</div>
</div>
<details id="config-zippy" hidden>
<summary>Configuration <span class="dim" id="config-path"></span></summary>
<pre id="config-text"></pre>
</details>
<h1 id="peers">Peers (<span id="peers-count">...</span>)</h1>
<table class="data">
<thead><tr><th>name</th><th>peer</th><th>AS</th><th>state</th><th>opened by</th><th>local address (next hop)</th><th>established</th><th>uptime</th><th>4-byte AS</th><th>address families</th><th>detail</th></tr></thead>
<tbody id="peers-body"></tbody>
</table>
<h1 id="local">Originated routes (<span id="local-count">...</span>)</h1>
<table class="data">
<thead><tr><th>prefix</th><th>origin</th><th>AS path</th><th>MED</th><th>communities</th><th>large communities</th><th>since</th></tr></thead>
<tbody id="local-body"></tbody>
</table>
<h1 id="received">Received routes (<span id="received-count">...</span>)</h1>
<table class="data">
<thead><tr><th>prefix</th><th>from peer</th><th>AS path</th><th>next hop</th><th>origin</th><th>MED</th><th>local pref</th><th>communities</th><th>large communities</th><th>best</th></tr></thead>
<tbody id="received-body"></tbody>
</table>
<h1 id="events">Event log</h1>
<div id="log-state" class="dim"></div>
<div id="event-log"></div>
<address>ferrix_bgp &middot; <a href="/statusz">/statusz</a> <a href="/metricz">/metricz</a> <a href="/api/events">/api/events</a></address>
<script>
(() => {
// Well-known standard communities (RFC 1997, RFC 7999, RFC 8326), shown by
// name with the number as a tooltip.
const WELL_KNOWN = {
'65535:0': 'GRACEFUL_SHUTDOWN', '65535:666': 'BLACKHOLE', '65535:65281': 'NO_EXPORT',
'65535:65282': 'NO_ADVERTISE', '65535:65283': 'NO_EXPORT_SUBCONFED', '65535:65284': 'NOPEER',
};
const $ = id => document.getElementById(id);
const esc = s => String(s).replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
const state = { status: null, peers: [], local: [], received: [] };
const DASH = '<span class="dim">-</span>';
// ---- helpers -----------------------------------------------------------
const fmtDur = s => {
s = Math.max(0, Math.floor(s));
const d = Math.floor(s / 86400), h = Math.floor(s % 86400 / 3600), m = Math.floor(s % 3600 / 60), sec = s % 60;
return d ? `${d}d ${h}h` : h ? `${h}h ${m}m` : m ? `${m}m ${sec}s` : `${sec}s`;
};
const now = () => Math.floor(Date.now() / 1000);
// Tables are only rewritten when their markup actually changes, so a
// refresh with unchanged data never touches the DOM (no flicker, and
// selections survive). Anything that changes with the clock is a
// placeholder that tick() fills in as plain text.
function setHTML(el, html) { if (el._html !== html) { el.innerHTML = html; el._html = html; tick(); } }
const age = unix => `<span data-since="${unix}"></span>`;
const until = unix => `<span data-until="${unix}"></span>`;
function tick() {
const t = now();
document.querySelectorAll('[data-since]').forEach(e => { const v = fmtDur(t - e.dataset.since); if (e.textContent !== v) e.textContent = v; });
document.querySelectorAll('[data-until]').forEach(e => { const v = fmtDur(e.dataset.until - t); if (e.textContent !== v) e.textContent = v; });
if (state.status) $('s-uptime').textContent = fmtDur(t - state.status.started_at_unix);
}
const fmtTime = unix => new Date(unix * 1000).toLocaleTimeString();
const commList = (list, named) => list.length ? list.map(c =>
named && WELL_KNOWN[c] ? `<span title="${esc(c)}">${WELL_KNOWN[c]}</span>` : esc(c)).join(' ') : DASH;
const asPath = p => p.length ? p.join(' ') : '<span class="dim">(empty)</span>';
async function getJSON(url) {
const r = await fetch(url);
if (!r.ok) throw new Error(`${url}: ${r.status} ${await r.text()}`);
return r.json();
}
// ---- renderers ---------------------------------------------------------
function renderStatus() {
const s = state.status; if (!s) return;
$('s-router-id').textContent = s.router_id; $('s-asn').textContent = s.local_asn;
$('s-version').textContent = s.version; $('s-hostname').textContent = s.hostname;
$('s-started').textContent = new Date(s.started_at_unix * 1000).toString().replace(/ \(.*\)$/, '');
tick();
}
const STATE_CLASS = { Established: 'ok', Connecting: 'warn', Backoff: 'bad', Idle: 'dim' };
// Where several routes for a prefix are exchanged (ADD-PATH), and which way.
function addPath(p) {
const rx = p.add_path_receive || [], tx = p.add_path_send || [];
if (!rx.length && !tx.length) return '';
const parts = [];
if (rx.length) parts.push(`receive ${esc(rx.join(', '))}`);
if (tx.length) parts.push(`send ${esc(tx.join(', '))}`);
return ` <span class="dim">· add-path: ${parts.join('; ')}</span>`;
}
function renderPeers() {
const b = $('peers-body');
$('peers-count').textContent = `${state.peers.filter(p => p.state === 'Established').length} of ${state.peers.length} established`;
if (!state.peers.length) { setHTML(b, '<tr><td colspan="11" class="dim">No peers configured and none connected.</td></tr>'); return; }
setHTML(b, state.peers.map(p => {
const up = p.state === 'Established';
const detail = p.state === 'Backoff' ? `${esc(p.last_error || '')}; retry in ${until(p.retry_at_unix)}` : '';
return `<tr>
<td>${p.name ? esc(p.name) : '<span class="dim">(inbound)</span>'}</td><td class="mono">${esc(p.peer_addr)}</td><td>${p.peer_asn}</td>
<td><span class="${STATE_CLASS[p.state] || ''}">${esc(p.state)}</span></td><td>${up ? (p.direction === 'outbound' ? 'us' : 'peer') : DASH}</td>
<td class="mono">${up ? esc(p.local_addr) : DASH}</td><td>${up ? fmtTime(p.established_at_unix) : DASH}</td>
<td>${up ? age(p.established_at_unix) : DASH}</td><td>${up ? (p.four_octet_asn ? 'yes' : 'no') : DASH}</td>
<td>${up ? esc((p.address_families || []).join(', ')) + addPath(p) : DASH}</td><td>${detail}</td></tr>`;
}).join(''));
}
function renderLocal() {
const b = $('local-body'); $('local-count').textContent = state.local.length;
if (!state.local.length) { setHTML(b, '<tr><td colspan="7" class="dim">This router originates no routes.</td></tr>'); return; }
const asn = state.status ? state.status.local_asn : '?';
setHTML(b, state.local.map(r => `<tr>
<td class="mono">${esc(r.prefix)}</td><td>${esc(r.origin)}</td>
<td class="mono">${Array(1 + r.prepend).fill(asn).join(' ')}</td><td>${r.med ?? DASH}</td>
<td class="mono">${commList(r.communities, true)}</td><td class="mono">${commList(r.large_communities)}</td>
<td>${age(r.announced_at_unix)}</td></tr>`).join(''));
}
function renderReceived() {
const b = $('received-body'), st = state.status;
$('received-count').textContent = st && st.received_routes > state.received.length
? `showing ${state.received.length} of ${st.received_routes}, for ${st.received_prefixes} prefixes`
: `${state.received.length}${st ? `, for ${st.received_prefixes} prefixes` : ''}`;
if (!state.received.length) { setHTML(b, '<tr><td colspan="10" class="dim">No routes received from peers.</td></tr>'); return; }
setHTML(b, state.received.map(r => { const s = r.summary; return `<tr>
<td class="mono">${esc(r.prefix)}</td><td class="mono">${esc(r.peer_addr)} <span class="dim">AS${r.peer_asn}${r.path_id ? ` path ${r.path_id}` : ''}</span></td><td class="mono">${asPath(s.as_path)}</td>
<td class="mono">${s.next_hop ? esc(s.next_hop) : DASH}</td><td>${s.origin ? esc(s.origin) : DASH}</td><td>${s.med ?? DASH}</td><td>${s.local_pref ?? DASH}</td>
<td class="mono">${commList(s.communities, true)}</td><td class="mono">${commList(s.large_communities)}</td><td>${r.best ? '<span class="ok">best</span>' : ''}</td></tr>`; }).join(''));
}
// ---- data loading ------------------------------------------------------
async function loadStatus() { state.status = await getJSON('/api/status'); renderStatus(); renderLocal(); renderReceived(); }
async function loadPeers() { state.peers = await getJSON('/api/peers'); renderPeers(); }
async function loadLocal() { state.local = await getJSON('/api/routes/local'); renderLocal(); }
async function loadReceived() { state.received = await getJSON('/api/routes/received'); renderReceived(); }
async function loadAll() { await Promise.all([loadStatus(), loadPeers(), loadLocal(), loadReceived()]); }
// ---- events ------------------------------------------------------------
function logEvent(html) {
const box = $('event-log'), d = document.createElement('div');
d.innerHTML = `${new Date().toLocaleTimeString()} ${html}`;
box.prepend(d); while (box.children.length > 200) box.lastChild.remove();
}
const names = list => list.map(c => WELL_KNOWN[c] || c).map(esc).join(' ');
function describe(ev) {
switch (ev.type) {
case 'session_up': return `<span class="ok">session UP</span> with ${esc(ev.peer.name || ev.peer.peer_addr)} AS${ev.peer.peer_asn} (opened by ${ev.peer.direction === 'outbound' ? 'us' : 'peer'})`;
case 'route_refresh_requested': return `${esc(ev.peer.name || ev.peer.peer_addr)} AS${ev.peer.peer_asn} asked for a route refresh; re-sent all originated routes`;
case 'session_down': return `<span class="bad">session DOWN</span> with ${esc(ev.peer_addr)} AS${ev.peer_asn}`;
case 'local_announced': { const r = ev.route; return `originating ${esc(r.prefix)}` +
(r.communities.length ? ` [${names(r.communities)}]` : '') + (r.large_communities.length ? ` [${names(r.large_communities)}]` : '') +
(r.prepend ? ` prepend ${r.prepend}` : '') + (r.med != null ? ` med ${r.med}` : ''); }
case 'local_withdrawn': return `<span class="warn">no longer originating</span> ${esc(ev.prefix)}`;
case 'best_changed': return `best route for ${esc(ev.route.prefix)} is now from AS${ev.route.peer_asn} path [${ev.route.summary.as_path.join(' ')}]`;
case 'best_withdrawn': return `${esc(ev.prefix)} has <span class="warn">no route</span> any more`;
case 'lagged': return `<span class="bad">missed ${ev.missed} events</span> on ${esc(ev.stream)}`;
default: return esc(JSON.stringify(ev));
}
}
function connect() {
const st = $('log-state');
const es = new EventSource('/api/events');
es.onopen = () => { st.textContent = 'Live: connected to the event stream.'; };
es.onerror = () => { st.textContent = 'Reconnecting to the event stream...'; };
// Events only feed the log. How often the tables reload is decided by
// ?refresh=, so the page never changes faster than the reader asked for.
es.addEventListener('bgp', e => logEvent(describe(JSON.parse(e.data))));
}
// ---- auto-refresh, opt-in through ?refresh=<seconds> --------------------
const refreshParam = parseInt(new URLSearchParams(location.search).get('refresh'), 10);
const REFRESH = Number.isFinite(refreshParam) && refreshParam >= 1 ? Math.min(refreshParam, 3600) : 0;
function refreshLink(seconds, text) {
const url = new URL(location.href);
if (seconds) url.searchParams.set('refresh', seconds); else url.searchParams.delete('refresh');
return `<a href="${esc(url.pathname + url.search + url.hash)}">${text}</a>`;
}
function renderRefreshControl() {
$('refresh-ctl').innerHTML = (REFRESH
? `Auto-refresh: <b>every ${fmtDur(REFRESH)}</b> (${refreshLink(0, 'turn off')})`
: `Auto-refresh: off (turn on: ${[2, 5, 30].map(n => refreshLink(n, n + 's')).join(', ')})`) +
` &middot; <a href="#" id="reload-now">reload now</a>`;
$('reload-now').addEventListener('click', e => { e.preventDefault(); loadAll().catch(console.error); });
}
// ---- boot --------------------------------------------------------------
getJSON('/api/config').then(c => {
if (!c.text) return;
$('config-path').textContent = c.path ? `(${c.path}, as read at startup)` : '';
$('config-text').textContent = c.text;
$('config-zippy').hidden = false;
}).catch(console.error);
renderRefreshControl();
loadAll().catch(e => logEvent(`<span class="bad">${esc(e.message)}</span>`));
if (REFRESH) {
connect();
setInterval(() => loadAll().catch(console.error), REFRESH * 1000);
setInterval(tick, 1000);
} else {
// A snapshot: nothing moves until the reader reloads or turns refresh on.
$('log-state').textContent = 'The event log is live only while auto-refresh is on.';
}
})();
</script>
</body>
</html>
docs/RFCs.md
+14 -4
View File
@@ -25,9 +25,17 @@ The base protocol specification.
| §4.4 | KEEPALIVE message | ✅ |
| §4.5 | NOTIFICATION message (all error codes and subcodes) | ✅ |
| §8 | FSM — passive (inbound) side: OpenSent → OpenConfirm → Established | ✅ |
| §8 | FSM — active (outbound) side: Idle → Connect → Active | ❌ |
| §8 | FSM — active (outbound) side: dial configured peers, exponential retry | ✅ |
| §6.2 | Bad Peer AS check against configured ASN | ✅ |
| §6.7 / RFC 4486 | Cease / Administrative Shutdown sent to every peer on SIGINT or SIGTERM | ✅ |
| §6.8 | Connection collision detection (BGP Identifier comparison, Cease / Connection Collision Resolution) | ✅ |
| §8.1.1 | Optional `CollisionDetectEstablishedState` (`collision_detect_established_state`) | ✅ |
| §6.2 | Bad BGP Identifier check (peer using our identifier, or 0.0.0.0) | ✅ |
| §9 | Decision process / best-path selection | ❌ |
| §9 | RIB (Adj-RIB-In, Loc-RIB, Adj-RIB-Out) | ❌ |
| §9 | Adj-RIB-In (per-peer table of received routes) | ✅ |
| §9 | Locally originated routes, advertised to all established peers | ✅ |
| §9 | Loc-RIB / Adj-RIB-Out for routes learned from peers (re-advertisement) | ❌ |
| §9.2 | Hold timer reset on any received message (not only KEEPALIVE) | ✅ |
---
@@ -59,7 +67,7 @@ https://datatracker.ietf.org/doc/html/rfc2918
| `RouteRefresh` capability advertisement | ✅ |
| `ROUTE-REFRESH` message type (5) framing | ✅ |
| AFI/SAFI payload parsing | ❌ |
| Acting on received ROUTE-REFRESH (re-advertising routes) | ❌ |
| Acting on received ROUTE-REFRESH (re-advertising all local routes, regardless of the AFI/SAFI asked for) | ✅ |
### RFC 7313 — Enhanced Route Refresh Capability for BGP-4
https://datatracker.ietf.org/doc/html/rfc7313
@@ -100,7 +108,7 @@ https://datatracker.ietf.org/doc/html/rfc2858
| Feature | Status |
|---------|--------|
| `MultiProtocol` capability advertisement (IPv4 unicast) | ✅ |
| `MultiProtocol` capability advertisement (IPv4 and IPv6 unicast) | ✅ |
| `MP_REACH_NLRI` path attribute (parse, serialise) | ✅ |
| `MP_UNREACH_NLRI` path attribute (parse, serialise) | ✅ |
| IPv4 and IPv6 next-hop encoding | ✅ |
@@ -122,6 +130,7 @@ https://datatracker.ietf.org/doc/html/rfc1997
| Feature | Status |
|---------|--------|
| `COMMUNITIES` path attribute (parse, serialise) | ✅ |
| Attach to locally originated routes | ✅ |
### RFC 4360 — BGP Extended Communities Attribute
https://datatracker.ietf.org/doc/html/rfc4360
@@ -137,6 +146,7 @@ https://datatracker.ietf.org/doc/html/rfc8092
| Feature | Status |
|---------|--------|
| `LARGE_COMMUNITIES` path attribute (parse, serialise) | ✅ |
| Attach to locally originated routes | ✅ |
---
fuzz/.gitignore
+6
View File
@@ -0,0 +1,6 @@
# What a fuzzing run produces. Seeds worth keeping live in seeds/.
target/
corpus/
artifacts/
coverage/
Cargo.lock
fuzz/Cargo.toml
+54
View File
@@ -0,0 +1,54 @@
# Coverage-guided fuzz targets, run with cargo-fuzz on a nightly toolchain.
# See README.md in this directory.
[package]
name = "bgp-fuzz"
version = "0.0.0"
edition = "2024"
license = "Apache-2.0"
publish = false
[package.metadata]
cargo-fuzz = true
# Not part of the main workspace: libfuzzer needs nightly, the workspace
# builds on stable.
[workspace]
[dependencies]
bgp-packet = { path = "../crates/packet" }
bgp-primitives = { path = "../crates/primitives" }
bgp-server = { path = "../crates/server" }
bytes = "1.10.1"
libfuzzer-sys = "0.4"
protobuf = "3.7.2"
[lib]
path = "src/lib.rs"
[[bin]]
name = "message"
path = "fuzz_targets/message.rs"
test = false
doc = false
bench = false
[[bin]]
name = "message_body"
path = "fuzz_targets/message_body.rs"
test = false
doc = false
bench = false
[[bin]]
name = "prefix_text"
path = "fuzz_targets/prefix_text.rs"
test = false
doc = false
bench = false
[[bin]]
name = "config"
path = "fuzz_targets/config.rs"
test = false
doc = false
bench = false
fuzz/README.md
+47
View File
@@ -0,0 +1,47 @@
# Fuzz targets
Coverage-guided fuzzing of everything that reads input we do not control.
The property tests (`cargo test`, e.g. `crates/packet/tests/properties.rs`)
check the same things on generated input and run on every build; these
targets go further by letting libFuzzer follow code coverage for as long as
you leave them running.
| Target | Input |
|----------------|------------------------------------------------------------------|
| `message` | Raw bytes as they come off a TCP connection |
| `message_body` | A message body behind a correct header (reaches the body parsers) |
| `prefix_text` | A prefix as text |
| `config` | The configuration file |
For the two message targets a finding is either a panic, or a message the
decoder accepted that cannot be written and read back as itself.
## Running
libFuzzer needs a nightly toolchain and `cargo-fuzz`:
```bash
rustup toolchain install nightly
cargo install cargo-fuzz
```
Then, from this directory:
```bash
cargo +nightly fuzz run message_body corpus/message_body seeds/message_body -- -max_total_time=300
```
The first directory is the working corpus, which the fuzzer adds to and
which is not checked in. `seeds/` holds a few valid inputs to start from.
Leave out `-max_total_time` to run until interrupted.
## When it finds something
The input is saved under `artifacts/<target>/`. Reproduce it with
```bash
cargo +nightly fuzz run message_body artifacts/message_body/crash-...
```
Fix the bug, and add a unit test with the input next to the code it broke,
so that `cargo test` keeps it fixed without anyone having to run the fuzzer.
fuzz/fuzz_targets/config.rs
+13
View File
@@ -0,0 +1,13 @@
//! The configuration file. It is written by the operator and not by a peer,
//! but a typo in it should be an error message, never a panic at startup.
#![no_main]
use bgp_server::config::Config;
use bgp_server::proto::config::BgpConfig;
use libfuzzer_sys::fuzz_target;
fuzz_target!(|text: &str| {
if let Ok(proto) = protobuf::text_format::parse_from_str::<BgpConfig>(text) {
let _ = Config::from_proto(proto);
}
});
fuzz/fuzz_targets/message.rs
+12
View File
@@ -0,0 +1,12 @@
//! Raw bytes, as they come off a TCP connection. The first byte picks what the
//! session settled: see `bgp_fuzz::Session`.
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
let Some((selector, wire)) = data.split_first() else {
return;
};
bgp_fuzz::decode_and_check(bgp_fuzz::Session::from_bits(*selector), wire);
});
fuzz/fuzz_targets/message_body.rs
+25
View File
@@ -0,0 +1,25 @@
//! A message body behind a correct header, so that the fuzzer spends its
//! time in the OPEN, UPDATE and NOTIFICATION parsers and not on guessing
//! the marker and length. The first byte picks the message type and what
//! the session settled: see `bgp_fuzz::Session`.
#![no_main]
use libfuzzer_sys::fuzz_target;
/// The longest body that fits a BGP message (4096 less the 19-byte header).
const MAX_BODY: usize = 4096 - 19;
fuzz_target!(|data: &[u8]| {
let Some((selector, body)) = data.split_first() else {
return;
};
let body = &body[..body.len().min(MAX_BODY)];
// Bits 1 to 3; the others pick the session.
let msg_type = 1 + ((selector >> 1) & 0x07) % 5;
let mut wire = vec![0xff; 16];
wire.extend_from_slice(&((19 + body.len()) as u16).to_be_bytes());
wire.push(msg_type);
wire.extend_from_slice(body);
bgp_fuzz::decode_and_check(bgp_fuzz::Session::from_bits(*selector), &wire);
});
fuzz/fuzz_targets/prefix_text.rs
+12
View File
@@ -0,0 +1,12 @@
//! Prefixes as text, as they appear in the configuration file.
#![no_main]
use bgp_primitives::ip_prefix::IpPrefix;
use libfuzzer_sys::fuzz_target;
fuzz_target!(|text: &str| {
if let Ok(prefix) = IpPrefix::try_from(text) {
let shown = prefix.to_string();
assert_eq!(IpPrefix::try_from(shown.as_str()), Ok(prefix));
}
});
fuzz/seeds/config/00
+38
View File
@@ -0,0 +1,38 @@
# A worked example; the fields are documented in
# crates/server/proto/config.proto.
router_id: "10.0.0.1"
local_asn: 65000
# Where to accept BGP connections. Port 179 needs privileges; pick another
# port to run unprivileged against a peer that dials a custom port.
listen_endpoints { address: "127.0.0.1" port: 179 }
listen_endpoints { address: "::1" port: 179 }
# A peer we connect to (and accept connections from).
peers {
name: "r1"
endpoint { address: "127.0.0.1" port: 1791 }
remote_asn: 65001
hold_time_seconds: 30
# Offer only these families to the peer. Left out, all are offered.
address_families: IPV4_UNICAST
address_families: IPV6_UNICAST
}
connect_retry_seconds: 5
# Let a restarted peer replace a stale Established session immediately
# (RFC 4271 CollisionDetectEstablishedState). Default false.
collision_detect_established_state: false
# Routes this router originates, announced to every peer.
originate { prefix: "10.100.0.0/16" }
originate {
prefix: "2001:db8:100::/48"
med: 50
communities: "65000:100"
large_communities: "65000:1:2"
}
# Read-only status page, JSON API and metrics. It has no authentication:
# keep it on loopback or behind a reverse proxy.
http_endpoint { address: "127.0.0.1" port: 8080 }
fuzz/seeds/message/00
BIN
View File
Binary file not shown.
fuzz/seeds/message/01
BIN
View File
Binary file not shown.
fuzz/seeds/message/02
BIN
View File
Binary file not shown.
Loaded 100 of 137 files, more files were not shown because too many files have changed in this diff. Show more