rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: rayhaan/ferrix:v0.2.0
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
...
compare: rayhaan/ferrix:d7d7d39bb9a315df39bdf14b438e4414322259c1
rayhaan/ferrix:main rayhaan/ferrix:speaker-cleanup rayhaan/ferrix:netsec_2026
rayhaan/ferrix:v0.2.0
43 Commits
v0.2.0 ... d7d7d39bb9
Author SHA1 Message Date
Rayhaan JaufeerallyandClaude Fable 5.1 d7d7d39bb9 metrics: export each session's TCP connection figures
ci / docs (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The kernel's view of a session's TCP connection, which the status page
shows on request since the last change, could not be looked at after
the fact: a feed that crawled overnight left no record of whether the
peer's window or our own side held it up.

This change samples it at every scrape of `/metricz`: for each
established session, one `NETLINK_SOCK_DIAG` round trip, and twenty
gauges labelled by peer, `bgp_peer_tcp_*`: smoothed, least and variance
of the round-trip time; the congestion window and segment size; the
peer's receive window and ours; the send queue, the part of it not yet
sent, the receive queue and the segments in flight; retransmitted and
lost segments; the busy time and the time held up by the peer's window
and by our send buffer, cumulative, so that their rates give the share
of time a session is held up; octets sent and received; and the
delivery and pacing rates. A session the kernel has no connection for
loses its series, as does one that ends.

`SessionHandle::metric_label` is the one place a peer's metric label is
made (its name, or its address), used by the advertiser's gauge and here.

Documentation: the metrics reference lists them; the monitoring guide
gives the query for the share of time a session is held up by the
neighbor's receive window, which is the question the overnight feed
raised.

Testing: a daemon in the Linux VM with one session shows the series and
drops them when the session ends; the lab's basics phase checks the
scrape for the FRR session's round-trip time; the workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 12:12:53 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 99a47da2af status: a session's TCP connection as the kernel sees it; a route in full 
When a session advertised a full table to a route collector at two
thousand routes a second, nothing on the status page said whether the
daemon or the peer set that pace. The answer is in the kernel: the
peer's receive window, the queues, and how long the connection was held
up by the window or by the send buffer, which `ss -ti` shows.

This change asks the kernel the same way `ss` does, over the socket
diagnostics netlink family (`NETLINK_SOCK_DIAG`, `SOCK_DIAG_BY_FAMILY`
with the connection's addresses and ports), and reads `tcp_info` and
the congestion control name from the answer. No external program is
run and no unsafe code is needed: the request and the answer are byte
layouts of the kernel's ABI, read field by field, with fields an older
kernel does not send left at zero. A socket on a link-local address is
found by its interface, which the request names.

* `bgp_fib::sockdiag`: the request, the parser, `TcpConnection`, and
  `tcp_connection(local, peer)`; `Ok(None)` where there is no such
  connection or the system is not Linux.
* `GET /api/tcp?peer=ADDRESS` in the HTTP API: the connection of the
  session with that peer, with times in milliseconds, the two windows
  named by what they allow, and the held-up times also as shares of the
  busy time.
* The status page: a "tcp" link on each established session, which
  expands a row below it with the figures, re-asked on every refresh
  while open. The group rows' span is corrected for the "sent" column.

Also on the status page: the attributes of a received route that have
no column of their own were badges with their value in a tooltip, which
is hard to read. Clicking the attrs cell now expands a row under the
route with every attribute by its name (ORIGIN, AS_PATH, NEXT_HOP, MED,
LOCAL_PREF, COMMUNITIES, LARGE_COMMUNITIES, AGGREGATOR and the rest),
the path identifier, the selection rank and the RPKI and ASPA states;
the badges stay. Both expansions use the terms of `ss` and the RFCs
rather than sentences.

Documentation: the endpoint in the HTTP API reference, and the link and
the expansion in the monitoring guide.

Testing: unit tests of the request and of the parser on a reply built
field by field, and of a "no such socket" and a refused answer; a
Linux-only test that asks the kernel about a loopback connection of the
test's own and about one that does not exist; the lab's basics phase
checks the endpoint for the FRR session; the page was checked by hand
against the daemon in the Linux VM.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 11:41:47 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 efde277fd8 Test: both families reach a collector over one IPv6 session 
A collector at an exchange is fed over one IPv6 session that carries
both families, with ADD-PATH for both and IPv4 routes with IPv6 next
hops (RFC 8950). The integration test for that has the collector on
IPv6 loopback with those capabilities and two feeders sending 20,000
IPv4 and 1,000 IPv6 prefixes each, and checks that every route of both
families arrives with its identifier. The test helper now reads the
prefixes that come and go in the multiprotocol attributes too.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:37:57 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9b211b61e9 export: count routes as they go out, and say when the table is walked again 
The number of routes advertised to a peer was written once a flush was
over, after every UPDATE of it had been handed to the session. The first
flush to a route collector is the whole table, two million routes or
more to a peer that reads at the pace of the Internet, so the status
page showed the peer holding one route for as long as that took, which
read as nothing going out.

Now the count is written after each batch of a flush, so that the page
and `bgp_peer_advertised_routes` show the table going out.

Also, an exporter that had the whole table and is given it again,
because the table changed faster than the exporter read it and the
journal let the oldest changes go, now logs that it is going through the
table again, at info. Until now that was invisible, and a collector
session during a full-table load from several peers is where it can
happen: the exporter is held up sending while the shards' journals,
cut back to twice their rows, run past its place.

Testing: a new integration test has a collector connected before two
feeders send 40,000 prefixes each, and checks it is told all 80,000
routes with their identifiers, which takes the exporter's initial walk
of the table through many batches; the advertiser and ADD-PATH tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:31:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 ec5c9be454 export: advertise every route for a prefix with ADD-PATH (all_paths) 
Before this change, a session on which ADD-PATH (RFC 7911) was
negotiated in the sending direction was sent path identifiers, but only
the best route for each prefix. A route collector such as bgp.tools
wants every route a router has accepted, from every neighbor, and from
one session: without this the collector saw one path per prefix, nearly
always the same neighbor's.

This change adds `all_paths` to the export policy. On a session with it,
and ADD-PATH send negotiated for a family, every accepted route for a
prefix of that family is advertised, each with a path identifier of its
own; for a family without ADD-PATH send the best route goes as before.
`all_paths` needs `learned_routes` (a configuration error otherwise) and
the peer's `add_path` to offer sending. The export policy, the role
rules, the well-known communities and the export limits apply to every
route alike, and the export limits count every route.

Implementation, in the advertiser:

* An exporter with `all_paths` follows the table's journal of all routes
  (`RibTable::reader_of_all_routes`, until now unused) instead of the
  best-route journal, and is given all the routes of a prefix whenever
  one of them changes (`News::Routes`).
* What the peer holds is `Announced`: the set of prefixes for a
  best-route session, as before; or for an all-paths session, for each
  prefix the routes announced, keyed by the peer they came from and that
  peer's own identifier, with the identifier they went out with and the
  form they went out in. A route keeps its identifier for as long as it
  is announced, goes out again only if it changed, and is withdrawn by
  its identifier; a new route gets the lowest identifier not in use for
  the prefix, from 1.
* Verdicts carry either kind (`Verdict::Best`, `Verdict::All`), and a
  reset of the table puts what was announced in doubt until heard of
  again, as before, with the same identifiers kept across the reset.
* The record for an all-paths session is about 50 bytes a route, some
  tens of megabytes for a full table from several neighbors; the
  best-route bookkeeping is unchanged in size.

Also, so that a collector session can be watched: the number of routes
advertised to each peer and not withdrawn since, every route counted
where several are sent for a prefix. The exporter keeps it in the
session's counters after every change; it is `advertised_routes` in
`/api/peers`, the "sent" column of the status page's neighbor table,
and the gauge `bgp_peer_advertised_routes{peer}`.

Documentation: a "Feeding a Route Collector" section in the guide to
passing routes on, with the configuration for a collector; `all_paths`
in the configuration reference; the ADD-PATH sections of the neighbor
guide and the standards reference; the limitation that was removed;
the new field, column and metric in the HTTP API, monitoring and
metrics references.

Testing: an integration test with two feeders and a collector that
negotiated ADD-PATH receive, which checks the two routes with their
identifiers, the withdrawal of one by its identifier, a changed route
keeping its identifier and a new route taking the free one; a
configuration test for `all_paths` without `learned_routes`; a new
`collector` phase of the interoperability lab in which FRR, as a route
collector, is told both of BIRD's routes for a prefix from BIRD's two
sessions and holds them apart, and loses one when its session is
disabled, with the advertised count checked at each step. The workspace tests, the kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 01:16:48 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 03f4f58ecb rib: give a row room for every peer's route at its first growth 
A row's routes are in a vector that grew by one route at a time, so with
four peers sending full tables every row's vector was reallocated three
times as their routes arrived, and the reallocations were 7% of the
daemon's CPU time in a profile of a four-peer load.

This change has the first growth of a row take it to as many routes as
the table has peers, at most 8 (`FIRST_ROOM`), since a prefix that one
full table has, the others nearly always have too; after that it grows
by a quarter as before. A table with one peer never grows a row and is
unchanged. A route server's clients do not all announce every prefix,
hence the cap: a row of a prefix that fewer peers announce has at most
seven spare slots of 16 bytes.

Measured in the Linux VM with four peers sending the synthetic table of
1.23 million prefixes: the kernel is in step 1.9 s after the first route
instead of 2.2 s, the daemon's CPU time over the load is 5% lower, and
it is 604 MiB resident instead of about 630, since there are no
reallocation leftovers. With one peer nothing changes. Memory per route
in the benchmarks, allocations counted: 77 bytes before and after with
the synthetic table from four peers; 64 before and 65 after with the
RIPE RIS dump's eight largest peers, whose load went from 2.8 s to
1.8 s on one thread, the allocation counting making every reallocation
dearer than it is in the daemon.

Also tried and not adopted, with the new `JEMALLOC_CONF` setting of the
profiling script: transparent huge pages for the allocator
(`thp:always,metadata_thp:always`). Over three pairs of runs with one
peer the table arrived in 0.66, 1.17 and 0.70 s with them against 0.87,
0.75 and 0.87 s without, the CPU time was within the noise, and the
daemon was 60 to 100 MiB larger. An arena for the routes would not help
either: jemalloc's size classes already give rows exact 16, 32, 48 and
64 byte allocations, and the cost was the number of reallocations, not
the allocator.

Testing: the table tests, including the one that grows a row past
65,535 routes, pass; the kernel tests and the interoperability lab pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:34:23 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 48da8cc7d2 rib: size the routing table for the routes the peers are expected to send 
The routing table's shards started empty and grew as routes came in,
each growth moving every row of the shard into a table twice the size.
In a profile of the daemon taking in a full table, the growth showed as
up to a seventh of the session's time, and the daemon's resident memory
had the old and the new table of each shard side by side for a moment.

This change adds `RibTable::expect_prefixes`, which reserves room for
that many prefixes spread over the shards, and has the server call it at
startup with the largest `expected_routes` among the configured peers:
the prefixes of several full tables are mostly the same ones. Without
`expected_routes` nothing changes. The `full_table` benchmark does the
same unless `BENCH_PRESIZE=0`.

Measured, the saving in time is within the run-to-run noise: in the
Linux VM one peer's table is in the routing table in 0.81 s instead of
0.89 s, and the kernel is in step at about the same time as before;
three pairs of runs of the benchmark on a Mac average 0.33 s against
0.35 s. The profile's attribution to the growth was mostly sampling skid.
The memory saving is real: 442 MiB resident with one peer's table in
instead of 483 MiB, and 626 MiB instead of 632 MiB with four.

Also: the profiling script printed nothing for the kernel's change
counts because it asked `/metrics` for them; the endpoint is `/metricz`.
With that fixed, a four-peer load shows 1.25 million installs for 1.23
million prefixes: the journal already gives the kernel task only the
latest change of a prefix when it lags, so no further coalescing of
installs is called for.

Documentation: `expected_routes` in the configuration reference, the
neighbor guide and the proto comment say that the table is sized by it;
the testing guide's VM results are brought up to date.

Testing: a table test checks that the room is there and is kept; the
workspace tests pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:13:02 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 9e9e4f90ef fib: take best routes from the table 16384 at a time 
The kernel task read changes of best route from the routing table 1024
at a time. A poll of the table does two binary searches and takes a read
lock in every shard, however few events the shard has; with the 128
shards of an 8-core machine, that overhead was paid for every 8 events a
shard, and the profile of a full-table load showed 8% to 9% of the
daemon's CPU time in the reader's searches and locks.

This change raises the batch to 16384. In the Linux VM, with four peers
sending the synthetic table of 1.23 million prefixes, the kernel is in
step 2.2 s after the first route instead of 2.9 s, and the daemon's CPU
time over the load falls 13%; the reader's share drops from 9% to under
1%. With one peer the kernel-side share falls the same way and the CPU
time 3%. A batch holds up to 16384 small records between polls, which is
about a megabyte.

Testing: the fib unit tests pass; measured with
scripts/profile/full-table.sh with one and four peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 5915c8e562 profile: configure the peers and report the kernel's change counts 
The full-table profiling script took its peers as unconfigured ones under
the permissive policy, which is not how a router is set up. Now it writes
a peer block for each of them, with `expected_routes` set to the table's
size, so that the default import policy and anything that sizes itself by
the expected routes are in the measurement. The few bogon prefixes in the
synthetic table are refused by that policy, as the output says.

After the table is in, the script also prints
`bgp_kernel_route_changes_total`, so that a run shows how many installs
and removals the kernel was asked for and not only how many routes it
ended up with.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:07:46 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1bc71aaac5 fib, bench: profile the daemon taking in a full table; count blackholes instead of scanning 
Before this change there was no way to see where the daemon's CPU time
goes while it receives a full Internet routing table: the benchmarks in
crates/bench time the routing table in-process, without the sessions,
the policy, the kernel task or the production allocator. Profiling the
daemon itself showed that a third of its CPU time during a full-table
load went into `Plan::blackholes()`, which walked every installed route
to count the blackhole ones each time the kernel task published its
status, after every batch of changes; with 1.2 million routes
installed that was 3.9 seconds from the first route to a synced kernel
instead of 2.0.

This change:

* Keeps counts of blackhole routes and of routes kept unconfirmed from
  an earlier run in `Plan`, in step with the map through `Plan::put`
  and `Plan::take`, so that `Plan::blackholes()` and `Plan::kept()`
  cost nothing. `kept()` had the same scan; it ran only while routes of
  an earlier run were being kept, which is exactly during a full-table
  load after a restart.
* Adds `scripts/profile/full-table.sh`, which builds the daemon with the
  new `profiling` Cargo profile (release with line tables) and frame
  pointers, runs it in a network namespace joined by a veth pair to a
  second namespace, feeds it the synthetic table from there as one or
  more neighbors, samples it with `perf`, and writes a flame graph
  (inferno), flat profiles by self and inclusive time, the time until
  the routes are in the table and in the kernel, and the daemon's
  resident memory. `PEERS`, `KERNEL`, `SCALE_DOWN`, `HOLD_SECS`, `FREQ`
  and `OUT` control it. From macOS it runs in the Linux VM through
  scripts/linux-vm.sh.
* Adds the load generator `crates/bench/examples/feed.rs`: it connects
  from the peers' addresses, completes the handshakes, writes each
  neighbor's table, drains what the daemon sends back and holds the
  sessions open with KEEPALIVEs. `TestPeer::into_stream` gives it the
  connection after the handshake.
* Updates the testing guide: the current `full_table` numbers, a
  "Profiling" section, and the daemon's numbers in the VM.

Measured in the Linux VM (8 vCPUs) with the synthetic table of 1.23
million prefixes: one neighbor's table is in the routing table after
0.7 s and in the kernel after 2.0 s (before: 3.9 s); four neighbors'
4.9 million routes are in the table after 1.1 s and the kernel is
synced after 3.0 s; 520 MB resident with one table, 680 MB with four.
Without the kernel one session takes in 2.0 million routes per second
on one core. A profile also attributed 14% of the session's time to
`Journal::tidy`; inlining its early return changed nothing in an A/B
run of the benchmark, so that was a sampling artefact and is left as it
is.

Testing: a unit test checks the counts against a scan of the entries
through adopt, best, reset, synced, release, withdrawal, refusal and
clear; the kernel test suite and the workspace tests pass; the script
was run with one and four neighbors, with and without the kernel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-09 00:03:43 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 6d8f89e509 policy, fib: blackhole routing for routes with the BLACKHOLE community
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
Before this change, a route carrying the BLACKHOLE community (65535:666,
RFC 7999) from an EBGP neighbor was always rejected with the reason
"blackhole request", and nothing discarded traffic. A customer under a
denial-of-service attack had no way to ask this router to drop traffic
to an address.

This change adds `accept_blackholes` to the import policy. When it is
set, a route with the BLACKHOLE community is accepted if:

* its prefix is a single address, a /32 or a /128 (otherwise it is
  rejected with the new reason "blackhole for more than one address");
  the `max_ipv4_prefix_length` and `max_ipv6_prefix_length` limits do
  not apply, as RFC 7999 section 3.2 recommends;
* the address is inside a prefix the neighbor may announce, if
  `prefixes` is configured, regardless of the entry's `max_length`
  (new `PrefixSet::covers`); without a prefix list, an address inside
  the local AS's own prefixes is rejected as before ("prefix is our
  own");
* the AS path, origin and RPKI checks that apply to every route pass.

The next hop of an accepted blackhole route is not checked for
third-party addresses and is not resolved, because traffic is never
forwarded to it. The route takes part in best path selection as usual;
a host route is normally the active route for its address.

The kernel task installs the active route as a kernel blackhole route
(`RTN_BLACKHOLE`, shown by `ip route` as `blackhole PREFIX proto bgp`).
Blackhole routes left by an earlier run are adopted at startup like
other routes of ours. The route is advertised to other neighbors as the
export policy and export limits permit, with the community intact, so a
provider that honors BLACKHOLE can discard the traffic upstream.

Implementation:

* `plan::Target` becomes an enum: `Via { gateway, interface }` or
  `Discard`. `plan::resolve` takes a `blackhole` flag and returns
  `Discard` for such routes without looking at the next hop (the
  own-network check still applies). No nexthop object is made for a
  discard target.
* `AttrSet::is_blackhole` reads the community off the stored attribute
  bytes, like `next_hop` and `as_path`, so the kernel task, the
  revalidator and the RPKI revalidation can ask it of every route.
* `ImportPolicy::is_blackhole_request` and `check_blackhole_prefix`
  carry the import side; the import flow (`routes.rs`) computes once per
  UPDATE whether its routes are an accepted blackhole request and skips
  the next hop checks for them.
* `Status::blackholes` and the `blackholes` field of `/api/kernel` count
  the installed blackhole routes; the status page shows the count next
  to the installed routes.

Not covered: a prefix this router originates with the BLACKHOLE
community is advertised but not installed as a blackhole route. The
documentation says to add a kernel blackhole route by hand for that
case.

Documentation: the "Blackhole Routes" section of the import policy
guide replaces "Blackhole Requests", the kernel guide gains a "Blackhole
Routes" section, the configuration reference describes
`accept_blackholes`, the HTTP API reference describes `blackholes`, and
the RFC 7999 rows of the standards reference are updated.

Testing: unit tests for the attribute check, the prefix checks and the
plan; integration tests that a blackhole request is refused by default
and, with `accept_blackholes`, taken for one address inside the peer's
space and refused otherwise, with the recorded reasons; a kernel test
that a blackhole request becomes a kernel blackhole route whatever its
next hop, is removed on withdrawal and is adopted after a restart. The
existing kernel tests, the interoperability lab and the workspace tests
pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 23:08:50 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 01ac3e2cd8 Test peer: no bail! in expression position 
Newer compilers warn of the macro's trailing semicolon there, and CI fails
on warnings.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 22:23:52 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3608897245 fib: resolve next hops through the kernel's routing table 
Before this change, Ferrix only installed a route if its BGP next hop was
directly connected: an address in the subnet of an interface that is up.
A route with any other next hop was held at import and never installed.
This made routes from iBGP neighbors unusable unless the neighbor set
itself as the next hop, and it is why the route reflector rewrites the
next hop on every reflected route.

Ferrix now resolves a next hop that is not directly connected through the
kernel's own routes, as other BGP implementations do:

* The route is looked up in one kernel table, the "resolution table"
  (`kernel { resolution_table: N }`, default 254 = main). The longest
  prefix that contains the next hop wins; of several routes for that
  prefix, the one with the lowest metric.
* Because the Linux kernel does not resolve gateways recursively, the BGP
  route is installed with the gateway and interface of the kernel route
  that covers the next hop. A kernel route without a gateway (a route
  onto a point-to-point or tunnel interface) makes the next hop itself
  the gateway on that interface. A kernel route that names a nexthop
  object is resolved through the object.
* Routes with protocol `bgp` never resolve a next hop, whether Ferrix's
  own or another daemon's. A next hop that is reachable only through a
  BGP route depends on another BGP next hop, and such chains are not
  predictable.
* The default route does not resolve a next hop unless
  `kernel { resolve_via_default_route: true }` is set. On a router the
  default route points at a transit provider, and a next hop that is
  reachable only through it is not usable.
* Blackhole, unreachable and prohibit routes make the next hop
  unresolvable, as does a route whose interface is down.

The same resolution is used by the import policy and the revalidator, so
a route whose next hop cannot be resolved is still held rather than
selected (RFC 4271 section 9.1.2.1), and is taken when a kernel route
makes its next hop resolvable.

Implementation:

* `bgp_fib::routes::KernelRoutes` holds the routes of the resolution
  table and answers longest-prefix lookups.
* `bgp_fib::topology::Topology` bundles the interfaces, the kernel routes
  and the kernel's nexthop objects into one value. The kernel task
  publishes it through `Fib::topology()` (formerly `Fib::interfaces()`),
  and `plan::resolve` takes it in place of `Interfaces`.
* The kernel task learns the resolution table with a dump of that table
  alone, using NETLINK_GET_STRICT_CHK (Linux 4.20 and later; older
  kernels get a dump of every table, filtered in user space). It asks
  again, debounced, whenever a dedicated netlink socket that is a member
  of the route multicast groups receives anything. That socket is given a
  64 KiB receive buffer on purpose: the kernel reports Ferrix's own route
  installs on it too, up to a million at a time, and an ENOBUFS overflow
  is treated as the same hint as a message. Its contents are never
  parsed. An earlier attempt to receive route notifications on the
  socket that writes routes failed, because the kernel only excludes the
  requesting socket from a notification when NLM_F_ECHO was requested.
* `Unresolved::NotConnected` is replaced by `NoRoute`, `OnlyDefault` and
  `Discarded`, which appear as the reasons in `/api/kernel` and on the
  status page.

Configuration: two new `kernel` fields, `resolution_table` and
`resolve_via_default_route`, documented in the configuration reference
and in the kernel guide, whose "Next Hop Resolution" section is rewritten.
Existing configurations behave as before except that routes with far
next hops may now be installed where a static or connected route covers
the next hop.

Testing: unit tests for `KernelRoutes`, `Topology` and resolution
through static, tunnel, nexthop-object, blackhole and default routes; a
kernel test (`a_far_next_hop_goes_the_way_the_kernels_routes_say`) that
checks the installed routes follow static, tunnel and nexthop-object
routes as they are added, replaced and removed; the existing kernel
tests, the interoperability lab with FRR and BIRD, and the workspace
tests, all passing. Installing 200,000 routes still runs at about
186,000 routes per second.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-08 22:23:27 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 430addcf20 Status page: IPv6 first in a neighbour's pair of sessions 
Of <stem>-v6 and <stem>-v4, the IPv6 session is listed first and carries
the name in full; IPv4 is under it. The same order in the lookup's list of
peers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 23:19:56 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 277b973f1a Status page: what moved, what is missing, and a dark room 
The attention strip's names lead to the peer's row, which lights up for a
moment. The IPv4 and IPv6 sessions of one neighbour (NAME-v4, NAME-v6) are
listed together, joined down the side, the name said once in full. A route
count that moved by more than 2% between refreshes carries ▲ or ▼ for two
minutes; a row whose state or count changed while the tab was in the
background carries ● until it is clicked. A compact mode drops the next
hop and session columns, and is remembered by the browser. Colours are
variables, with a second set for prefers-color-scheme: dark.

A new per-peer `expected_routes` says about how many routes a peer sends
when all is well. The daemon only passes it to the API; the page lists an
established peer with under half of it as needing attention, a minute
after the session came up, with the expected count beside the actual one.
An established session with next to no routes looked, until now, exactly
like one that was fine.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 23:16:01 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 7b6547d3c9 Docs: say what the forge cannot render, and two headings by the style guide 
The documentation sources are Zola Markdown: front matter between +++
lines, shortcodes that insert the example configurations, and @/ internal
links. A code forge shows all three unrendered, which the README used to
deny. A docs/README.md now explains each, and the README points to it.

"Configuring BGP Neighbors" becomes "BGP Neighbors", as the other feature
pages are titled; "When a Neighbor Restarts" becomes "Neighbor Restart", a
noun phrase as the style guide asks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:18:18 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3a1e42e594 README for a public release
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details 
What Ferrix is and does, that it routes AS210036, how to build it and try
it without a router, the security points an operator must know, and a note
that the code is written with an AI coding agent under the maintainer's
review.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:09:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 35ce2e2e57 Status page: routes in order of selection, lookup by origin AS 
Received routes are listed from the best down: the RIB gains ranked_routes,
which runs selection again without the winners, so the second route is the
one that would be installed were the best withdrawn. The API gives each
route its rank, and the page shows it as a numbered badge after the star.

The lookup box moves to the received-routes table, which is what it filters,
and takes an AS number: a new `origin` query parameter finds every route
whose path ends in that AS, with an empty path counting as the local AS.

Long cells are clamped to a few lines with a control to open them, and an
AS repeated along a path (prepending) is shown once with a ×N badge. Peer
names are bold and dark, not underlined links, so a row can be found by
its name among the addresses and numbers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-07 22:09:07 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 1ce7eef27e Status page: what needs attention first, glyphs for what repeats 
The page opened with every session in bold green, the normal state, and
the brightest numbers on it were the rejection counts, which RPKI doing
its job produces by the thousand. The two sessions that were actually
down were in the same red, thirty rows down. Now:

- A status line at the top says what needs attention, by name and with
  the reason: sessions down, peers restarting, the kernel behind or
  refusing routes, RPKI data missing or about to expire. When nothing
  does, it says so. A second line gives the normal state in grey.
- Colour means one thing each. Red: failed, needs a hand. Amber:
  degraded, worth watching. Green, set small: a system doing its job.
  Grey: inactive and expected. Normal is black; "Established" is no
  longer green, and a rejection count no longer red.
- Peers that need attention come first, then the others grouped by what
  they are to us, with a "problems only" switch. Numbers are right
  aligned with thousands separators; the established time and uptime
  are one cell; the session's properties are glyphs (v4, v6, GR, ENH,
  AP, 2B) with the meaning as a tooltip, as are the roles.
- Received routes: the best route leads, marked ★; attributes without a
  column are glyphs (OTC, AA, AGG, AS4, OID, CL); validity is ✓ ✗ –.
- Rejected routes: those alike (same prefix, reason and path) are one
  row with a count and the last time; each peer's reasons are a bar.
- Communities with a configured name (community_names, new) are badges
  with that name, the number as the tooltip: "comrack", "SwissIX",
  "interconnection 2011" rather than 210036:100:2 and so on. A name for
  a pattern shows the open part after it. The daemon does nothing with
  the names but pass them to the page.
- One lookup box at the top, which takes a peer's name too; a peer's
  name anywhere on the page leads to its routes; table headers and the
  status line stay in view while scrolling.
- The host name comes from gethostname(2), not from running `hostname`,
  which a static binary on a bare machine could not find: comrack said
  "unknown host".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 17:49:06 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 23e8b4aadb One word for a route that is not taken: rejected
ci / check (push) Canceled after 0s
Details
ci / docs (push) Canceled after 0s
Details
ci / interop (push) Canceled after 0s
Details
ci / fuzz (push) Canceled after 0s
Details 
The API, the metrics, the status page and the documentation say a route
is "rejected" (/api/routes/rejected, rejected_routes,
bgp_routes_rejected_total, "Rejected routes"), and the reason type is
Rejection. The code that keeps the record said "refused": RefusalLog,
PeerRefusals, RefusedRoute, RefusalsView, RefusedRouteView, the refusals
module. Those are RejectionLog, PeerRejections, RejectedRoute,
RejectionsView, RejectedRouteView and rejections now, and the log line
says "Rejected a route". Nothing the API or the metrics expose changes.

Refusing stays what it was: a connection turned away at the door
(server::Refusal), a kernel that would not take a route, an OPEN we
would not have.

The two aliases of SendReceive, AddPathSendRecv and OrfSendRecv, are
gone; it is one type.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 12:10:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c218ea4d85 Integration tests: one copy of each helper 
The test files had grown their own copies of the same helpers: prefix()
twelve times, established() seven, an UPDATE builder six, a seventeen
field PeerConfig literal in fifteen places, a timeout constant in twenty
one. Each is in crates/integration-tests/src once now: peer_config() and
TestServer::with_peer for a configured peer, Announcement for building
UPDATEs, established(), the waits (wait_until, wait_for, wait_for_routes,
wait_for_sessions), get_json, and TIMEOUT. A new field on PeerConfig is
one line to add, not fifteen.

TestPeer::recv has a deadline, so a test that gets nothing fails instead
of hanging; is_closed, which read one byte and took a KEEPALIVE for an
open connection, is closed_within and closed_unanswered.

Net 788 lines fewer; every test checks what it did.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:54:49 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 673920d48b Names that said the same thing twice, or two things once 
- PathAttribute::UnknownPathAttribute and Capability::UnknownCapability
  are PathAttribute::Unknown and Capability::Unknown: the enum already
  says what kind of thing is unknown.
- The capability entry types are all named XxxEntry now
  (ExtendedNextHopEntry, LongLivedGracefulRestartEntry), rather than two
  of them sharing a name with the variant that holds a list of them; and
  MultiLabelCompat is MultiLabel, since nothing explained the "Compat".
- Two different things were both called Reflection: what a session needs
  to reflect routes (export.rs, still Reflection) and the identifiers a
  reflected route must not carry back to us (policy.rs, now
  ReflectorIds).
- well_known_allow's peer_is_internal is ibgp, as everywhere else.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:35:12 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 0829f873f2 Server: the long functions in named steps 
Five functions did several things each in one body of 120 to 190 lines.
Each is now a list of steps with a name and a doc comment of their own,
with nothing changed in what they do:

- run_session: hold_down_after, keep_for and release_routes.
- Session::establish: session_handle, validation and route_source. A
  session keeps the peer's settings as one PeerSettings rather than
  eight fields copied out of it one by one; the sweeps' two channel ends
  are named fields, not .0 and .1; whether the peer can be asked to send
  its routes again is a bool, not a copy of the families.
- Fsm::on_open: negotiate_role and negotiate_add_path.
- BgpServer::run: log_startup and spawn_tasks.
- Config::from_proto: parse_listen_endpoints, parse_peers, check_roles,
  parse_originate and Ours::tell_peers, with the three per-peer defaults
  in one PeerDefaults.
- Egress::outgoing: may_go, rewrite and next_hop_for.
- Exporter::follow: Learned::drain, judge and flush, with the three
  things called "announced" told apart (own_announced, learned.announced,
  batches) and the verdicts given a type.

The hold-downs are a module of their own, holddown.rs; policy.rs is
about policy again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:32:20 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 724bb99f11 Kernel: read what the table has before letting kept routes go 
Once every peer had sent all its routes, the kernel task let go of the
routes kept from the earlier run that the table did not have, at once.
But "the table has" and "the kernel task has read" are not the same
moment: a route a peer sent just before its End-of-RIB could still be
unread in the table's journal, in which case it was taken out of the
kernel and put back a moment later. The interop lab saw this now and
then as "N routes were taken out of the kernel over the restart".

The task now reads the table to the end before it decides what is
unconfirmed; follow() says whether the reader is caught up.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:21:53 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 175a13d506 Hygiene: scripts, CI, docs and manifests 
- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
  it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
  in the namespace rebuilt what kernel-tests.sh had just built. Each is
  passed on only where set.
- The README's list of unsupported features still had route reflection
  and graceful restart, both of which exist; the command-line reference
  lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
  what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
  which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
  three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
  unused variable; small things in the web crate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 e5341ea639 Packet: the wire format's shared pieces in one module 
What the parsers and writers of messages, attributes and capabilities
had each written for themselves is one module (wire.rs): the shape of a
parser's result and its error, reading a list to the end, checking that
nothing was left over, a non-empty list of fixed-size records (the
community attributes and CLUSTER_LIST, which now fail alike), an AS
number at the session's width, an AFI/SAFI pair, and a length written
before what it counts. open.rs loses its fail! macro and attributes.rs
its own copies of the same.

Tests for what had none: AS4_AGGREGATOR kept opaque and discarded when
malformed; IPv4 routes with IPv6 next hops (RFC 8950) in the property
tests; and what a part-level parser accepts re-encodes to what it read,
but for the fields the decoders normalise.

Nlri::wire_len had no callers; the mask of a prefix's last octet is one
function; a test's name is spelled right; a comment is on the test it
describes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 b467b136e4 Kernel and RPKI: say what went wrong, and once 
The RPKI client had one Timeout error whose text said "did not answer
within 60s" whatever had run out: the connect timeout, the time for a
whole answer, or the data's lifetime. These are what /api/rpki shows as
the last error; each is its own now. The protocol version shown is the
one the cache answered in, not the one being tried. The VRP set, hashed
by the hundred thousand with every answer, uses the quick hash the ASPA
table already did.

In the kernel installer the writer task's error was worked out in three
places, and the next-hop resolution checked for our own networks twice
on every cache miss. A refusal was logged twice, once with the status
lock held. The interfaces are listed in order of index by the structure
that holds them, from a function that can be tested on any OS, and is.
EIO is with the other errnos; Plan has no installed_prefixes (no
callers) and clears its records in one place, with the order of
Unresolved::ALL asserted at compile time since indexing depends on it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 3b22e318a2 Server: one way to work on the table off-thread, and smaller tidying 
Removing a peer's routes, sweeping its stale ones and sweeping the
attribute sets were each written out as their own spawn_blocking, and
each dealt with a failed task differently: the periodic sweep dropped a
panic on the floor, and a restarting peer whose kept routes could not be
removed was logged as "removed 0 routes". off_thread does it once, and
every caller logs a failure.

Also:
- the hard-reset wrapping of a NOTIFICATION is one function in the FSM;
- the two ways to close every session are one, and the registry has one
  leave() rather than leave() and shutdown();
- the dialer says why a session ended with SessionEnd::reason, and no
  longer turns a result into a result;
- as_path_prepend is parsed in one place, and the limits in error texts
  come from the constants rather than being written out;
- router_id and cluster_id are parsed by one function;
- the exporter builds its Egress in one place;
- the fields a session keeps only to hand to its handle are fewer:
  route_refresh is a bool, not a copy of the families;
- counting the stale routes removed cannot underflow;
- docs that had gone stale: the proto still said there was no route
  reflection, the advertiser's module doc said it announced only our own
  routes, from_server listed two of its five sources, and so on.

Tests for what had none: which session ends count as restarts, the Hard
Reset wrapping, the graceful restart settings, the dialer's jitter,
withdrawals in MP_UNREACH_NLRI, and AS_TRANS on a two-octet session.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 c69aa38090 Routing table: a reader waits out a change under way instead of spinning 
A reader does not read past the earliest change still under way, which
may not have reached every shard. A change to every shard, as when a
peer's whole table goes, is under way for a while; changes others made
meanwhile are beyond it. A reader that found only those saw "more to
read" with nothing it could read, and asked again at once, round and
round, taking every shard's read lock each time, until the change ended:
every exporter and the kernel installer, spinning for as long as a full
table took to remove.

Now a pass that moves no cursor though there is more says it is caught
up, and waits. And a change wakes the readers when it ends if any other
change woke them while it was under way, since what they were woken for
is only readable now. The three per-shard change loops are one
(RibTable::change), which drops the change before waking, as only one of
them did.

Also: the add_peer doc was attached to peer(); some_routes had no
callers; count_stale is for tests.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 11:15:58 +02:00
Rayhaan JaufeerallyandClaude Fable 5.1 befc292215 Tidy the export limits, passive peers and status page code 
Readability, no change in behaviour, but for one log line:

- The routes a peer holds from us were counted twice over, differently,
  in the exporter: once per learned-route batch and once per originated
  route. Learned::held_by_peer counts the learned ones in one place, and
  ExportLimits::exceeded_by returns the limit that was passed instead of a
  bool, so that ending the session no longer has to dig it out again with
  a default that could not happen.
- The two hold-downs for a limit passed, ours or the peer's, were two
  near-identical blocks; they are one match.
- How a session ended is put into words in one place
  (SessionEnd::reason), for the dialer's backoff and the passive peer's
  state alike.
- parse_export_limits parses one block; what a missing block means (no
  limits at the top, the top's limits for a peer) is said where it
  differs, instead of Option in and Option out.
- The startup warnings about export limits are a function of their own,
  and the peers without limits are named in one line rather than one
  line each: a router with eighty sessions does not need eighty warnings.
  The warning about a peer without a role is gone: the configuration now
  refuses one.
- Hex formatting of attribute octets is one function; the text for an
  unknown attribute is built in three plain steps.
- The exporter's test fixture returns a named struct, not a tuple.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-06 10:05:31 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 2dfc00862b Release 0.2.5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 00:06:06 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 c9751883c4 Passive peers: configured, and left to connect 
A session could only be configured for a peer we would then dial, every
few seconds to two minutes, for as long as it was not up. A new peer
whose side was not ready yet filled the log with failed attempts, and a
peer that opens its sessions itself was dialled for nothing.

A peer with `passive: true` is not dialled. A connection from its address
is taken as for any configured peer, and the session comes up with all
that is configured for it: role, import policy, export limits,
hold-downs. Until then the status page and /api/peers show it as
`Passive`, waiting for the peer to connect; after a session, with how
that session ended, which nothing else would say since nobody dials it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-06 00:05:54 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 910dd258fb Release 0.2.4 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:11:03 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 9cfd3f67f0 Export limits, and a role for every peer where routes are passed on 
What other networks were told of rested on one thing: the Only to
Customer mark that routes from providers, peers and route servers get as
they come in. A route without it, from a session without a role or by a
mistake anywhere along the way, went to every peer, from every router of
ours, and nothing of ours stood in the way.

export_limits are a second check, independent of the roles, the marks
and the export policy. For peers above or beside us (PROVIDER, PEER,
ROUTE_SERVER, and any in another AS without a role):

- prefixes: only routes for these go out, learned or our own. With our
  and our customers' space listed, a leak of anyone else's routes goes
  nowhere above us.
- max_prefixes: if more routes were to go out than this, none of them
  does. The session ends with a Cease / Administrative Shutdown whose
  Shutdown Communication says why (RFC 9003), and the peer is held down
  for its import.hold_down_seconds, so that it is told nothing rather
  than too much. The whole table is counted before any of it is sent.

They are set at the top and hold for every such peer that has none of its
own; unlike `export`, a peer's own export policy does not drop them. A
peer's own export_limits take their place, and an empty one lifts them.
Customers, route server clients and our own routers are not limited.

A configuration in which learned routes are passed on and a peer in
another AS has no role is now refused, as is accept_unconfigured_peers
with learned routes: without a role, a peer's routes go to everyone and
it is told of everything. This used to be a warning.

At startup, a warning for each peer the limits apply to that has none,
and for each originated route the limits leave out. A metric,
bgp_export_limit_exceeded_total, counts the sessions ended for it.

The interop lab has a phase in which a customer leaks: a route outside
its space never reaches BIRD, and when more were to go than the limit,
BIRD's session ends and its log has our reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:10:43 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 17045a3634 Status page: tables for the kernel, communities by AS, attributes as text 
The kernel section was one run-on line of interfaces and addresses. It is
now a small table of the installation state, and a table of interfaces
with their IPv4, IPv6 and link-local addresses one per line; ports that
are down and have no addresses share one line at the bottom.

Communities ran together into one block that wrapped anywhere. Each AS's
communities are now a line of their own, kept whole so that they copy,
with a hanging indent where a line wraps; pointing at a line names the AS.
Where control communities are on (now in /api/status), those of ours that
ask for something say what as a tooltip.

A rejected route's details were a fold-out that repeated the summary and
then dumped the decoded attributes as JSON. Rejected routes now have the
same columns as received ones, and both have an "other attributes" column
for what the summary had no field for, as text: aggregator, extended
communities (route targets and origins by name), AS4_PATH with its sets in
braces, originator ID, cluster list, and unknown attributes in hex. The
API has them as summary.other; the JSON stays in `attributes`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 23:09:02 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 89e0c3a3e2 Release 0.2.3 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:15:28 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 b16227c51f Route reflection, with ourselves as next hop 
A route learned from one router of our AS went to no other (RFC 4271
§9.2.1), so every router needed a session with every other. A router at
the end of a single link, with one session to its neighbour inside the AS,
heard only of what that neighbour learned itself, and nothing of what the
neighbour had from the routers behind it.

A peer in our own AS can now be made a route reflector client
(`route_reflector_client`, RFC 4456). What a client sends goes to all our
routers; what the others send goes to the clients. A reflected route gets
ORIGINATOR_ID (where it started, if it does not say) and our cluster ID at
the front of CLUSTER_LIST; it is not sent back to the router it started
at, nor round to a cluster it has been through, and one that comes back
anyway is refused as "route reflected back to us". The cluster ID is the
router ID unless `cluster_id` says otherwise. The two attributes are now
decoded as such rather than kept as bytes, and are never sent to other
networks.

Unlike a standard reflector, this one is the next hop of what it reflects,
as it is of everything it announces: there is no resolving a next hop
that is not a neighbour, so a client reaches what lies behind another
router through the reflector.

Only peers in our own AS can be clients; the configuration refuses the
setting for any other. Nothing changes in what other networks are told:
Only to Customer marks go along with reflected routes, and every
receiving session keeps its own max_prefixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 18:10:35 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 164b1099f1 Release 0.2.2 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:39:06 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 4e945b278f Import rules: preference and communities for some routes only 
The import policy did the same to every route of a session. Now `rules`
can do more to some of them: those with certain communities, those for
certain prefixes, or both. Each rule can set the local preference and put
communities on; each route takes the first rule that holds for it.

Rules see a route as the session's own statements leave it, with our
communities taken off and the session's marks put on. So a router can act
on where another router of ours learned a route, by the mark that router
put on it, and a customer's single prefix can be marked for a prepend.

Routes of one UPDATE that different rules hold for go into the table with
attribute sets of their own. An UPDATE that no rule touches is taken as
before, without its attributes being written out again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:34:17 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 4d13336874 Release 0.2.1 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:16:45 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 72444c2706 Control communities: session groups, prepends, and only ours to give 
Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:

  L:11:P .. L:13:P   our AS once to three times more towards AS P
                     (P 0: towards every peer in another AS)
  L:20:G / L:21:G    not to / only to the sessions of group G
  L:31:G .. L:33:G   our AS once to three times more towards group G

A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.

They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.

And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.

This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:12:05 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 3609451bdc Ignore our own prefixes from our own routers, and say why a route for a network of this machine is held 
Two routers of an AS that both originate a prefix (anycast) each
announce it to the other over their internal session. Each took the
other's route, could not install it in the kernel, since the prefix is
on one of its own interfaces, and so held it, counted as rejected with
"next hop unreachable": a permanent rejection on a healthy session, for
a next hop that was in fact fine.

A route from a peer in our own AS for a prefix this router originates
is now ignored: not taken, not held, not counted. Only that prefix
itself; what is inside it is judged as usual. From other networks such
routes are refused as our own prefixes, as before.

A route that is held because its prefix is a network of this machine,
or inside one (an exchange's LAN prefix, say), is now counted as "prefix
is a network of this router", apart from routes whose next hop cannot
be reached.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:11:53 +02:00
Rayhaan JaufeerallyandClaude Opus 5.5 52593bb2c2 RPKI: ask a cache without data yet again soon 
A validator that is still running its first validation answers a query
with No Data Available (error code 2). That ended the session, and the
next attempt came after the retry interval: 600 seconds when the cache
had not yet said otherwise, which it cannot have before it has data.
For those ten minutes after a validator started, nothing was validated.

No Data Available is now asked again after 30 seconds, or after
retry_seconds if that is shorter, and logged as information rather than
as a warning. Other failures wait as before, after trying the next
cache if there is one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 16:08:19 +02:00
116 changed files with 12340 additions and 3785 deletions
Expand all files Collapse all files

No files matched your search

.github/workflows/ci.yml
+4
View File
@@ -24,6 +24,10 @@ jobs:
run: cargo fmt --all -- --check
- name: Lint
run: cargo clippy --locked --workspace --all-targets -- -D warnings
# The memory-counting allocator of the benchmarks is behind a feature
# that the workspace build does not turn on.
- name: Lint the benchmarks with memory counting
run: cargo clippy --locked -p bgp-bench --all-targets --features count-memory -- -D warnings
- name: Test
run: cargo test --locked --workspace
# A link in the documentation to something that is not there.
Cargo.lock
Generated
+11 -12
View File
@@ -157,7 +157,7 @@ checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]]
name = "bgp"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-server",
"bgp-web",
@@ -172,7 +172,7 @@ dependencies = [
[[package]]
name = "bgp-bench"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-integration-tests",
"bgp-packet",
@@ -183,13 +183,11 @@ dependencies = [
"flate2",
"stats_alloc",
"tokio",
"tracing",
"tracing-subscriber",
]
[[package]]
name = "bgp-fib"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-packet",
"bgp-primitives",
@@ -207,7 +205,7 @@ dependencies = [
[[package]]
name = "bgp-integration-tests"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-fib",
"bgp-packet",
@@ -226,7 +224,7 @@ dependencies = [
[[package]]
name = "bgp-packet"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-primitives",
"bitfield",
@@ -242,7 +240,7 @@ dependencies = [
[[package]]
name = "bgp-primitives"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bytes",
"nom",
@@ -254,7 +252,7 @@ dependencies = [
[[package]]
name = "bgp-rib"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-packet",
"bgp-primitives",
@@ -269,7 +267,7 @@ dependencies = [
[[package]]
name = "bgp-rpki"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-primitives",
"bytes",
@@ -282,7 +280,7 @@ dependencies = [
[[package]]
name = "bgp-server"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"bgp-fib",
"bgp-packet",
@@ -304,7 +302,7 @@ dependencies = [
[[package]]
name = "bgp-web"
version = "0.2.0"
version = "0.2.5"
dependencies = [
"axum",
"bgp-fib",
@@ -314,6 +312,7 @@ dependencies = [
"bgp-rpki",
"bgp-server",
"futures",
"nix",
"prometheus",
"serde",
"tokio",
Cargo.toml
+8 -1
View File
@@ -12,7 +12,7 @@ edition = "2024"
homepage = "https://rayhaan.ch"
license = "Apache-2.0"
repository = "https://git.rayhaan.net/rayhaan/ferrix"
version = "0.2.0"
version = "0.2.5"
# Every crate opts in with `[lints] workspace = true`. CI turns warnings into
# errors, so a lint added here is enforced.
@@ -65,3 +65,10 @@ tokio-stream = { version = "0.1.19", features = ["sync"] }
reqwest = { version = "0.13.5", default-features = false, features = ["json"] }
# The daemon's memory allocator; see bin/bgp/src/main.rs.
tikv-jemallocator = "0.7.0"
# `cargo build --profile profiling`: the release build, with the line tables
# a profiler needs to name functions. See scripts/profile/.
[profile.profiling]
inherits = "release"
debug = "line-tables-only"
strip = "none"
README.md
+142 -41
View File
@@ -1,45 +1,101 @@
# Ferrix BGP
An implementation of the Border Gateway Protocol, which is used to exchange routing and control plane information between Autonomous Systems.
Ferrix is a BGP daemon for Linux routers, written in Rust. It exchanges
routes with neighbors, filters them by policy and the RPKI, selects the
best path for each prefix, installs it in the kernel, and passes routes on
as BGP roles and communities permit. It aims to be small, safe, observable,
and configured from one file that can be read by someone who did not write
it.
The aim of this project is to provide a modern, high performance, secure, programmatic and customizable software package that can be useful to network operators, researchers, and people interested in Internet.
## Status
## Current status
Ferrix routes the author's network, AS210036: three routers carrying full
tables from transit providers, route servers at several Internet exchanges,
and bilateral peers, with routes reflected between the routers over iBGP.
It is young software all the same. Read the [limitations](#not-supported)
before relying on it, and treat version `0.x` as meaning that configuration
and APIs may still change between releases.
The daemon establishes BGP sessions with configured neighbors, evaluates received routes against import policy, validates them with the RPKI (route origin validation and ASPA), selects the best path for each prefix, and, on Linux, installs it in the kernel. It advertises the routes that it originates and, to neighbors configured for it, the best paths learned from other neighbors, as BGP roles (RFC 9234) and communities permit.
What it does:
Ferrix has not been deployed in a production network.
- BGP-4 over IPv4 and IPv6, with four-octet AS numbers, multiprotocol
reach for both families, IPv4 routes with IPv6 next hops (RFC 8950),
route refresh, graceful restart and ADD-PATH receive.
- Import policy per neighbor: prefix limits, prefix and origin AS lists,
bogon checks, AS path checks, and communities added on the way in.
- Route origin validation and ASPA path verification against an RTR cache.
- BGP roles (RFC 9234) and the Only to Customer attribute, with export
limits as a second line of defence against route leaks.
- Best path selection and installation of the result in the Linux kernel
over netlink, in a routing table of your choice.
- Route reflection between routers of the same AS, and passive neighbors.
- A status page, a JSON API, server-sent events, and Prometheus metrics.
- Graceful restart of the daemon itself for upgrades, with routes left in
the kernel while it is away.
### Unsupported Features
### Not supported
- Route reflection, per-client best path selection for route servers, and ADD-PATH for advertised routes
- Per-client best path selection for route servers, and ADD-PATH for
advertised routes
- TCP-AO, TCP MD5, and GTSM
- Graceful restart, BFD, and configuration changes without a restart
- Recursive resolution of next hops that are not directly connected
- Long-Lived Graceful Restart, BFD, and configuration changes without a
restart
- Address families other than IPv4 and IPv6 unicast
### Layout
## Building
| Crate | What is in it |
|-------|---------------|
| `crates/primitives` | Prefixes, address families, parsing helpers |
| `crates/packet` | The wire format: messages, attributes, capabilities |
| `crates/rib` | The routing table, best route selection, the journal of changes |
| `crates/rpki` | The RTR client; origin validation and ASPA path verification |
| `crates/fib` | The best routes in the Linux kernel, over netlink |
| `crates/server` | Sessions, the state machine, policy, configuration |
| `crates/web` | The status page, HTTP API and metrics |
| `bin/bgp` | The daemon |
You need a stable Rust toolchain from [rustup.rs](https://rustup.rs), and a
C compiler and `make` for the jemalloc allocator the daemon uses (for
example `build-essential` on Debian and Ubuntu, `base-devel` on Arch Linux,
or the Xcode command line tools on macOS). The Protocol Buffers compiler is
not needed: the configuration schema is compiled in pure Rust.
```bash
git clone https://git.rayhaan.net/rayhaan/ferrix.git
cd ferrix
cargo build --release --locked -p bgp
```
The binary is `target/release/bgp`. `--locked` builds with the dependency
versions recorded in `Cargo.lock`, which are the versions CI tested.
To try it without touching any router, run the quick-start configuration.
It listens on loopback, needs no privileges, and serves the status page at
<http://127.0.0.1:8080>; the [quick start
guide](docs/content/docs/getting-started/quick-start.md) adds a test peer
that announces routes to it.
```bash
target/release/bgp --check docs/examples/quick-start.textpb
target/release/bgp docs/examples/quick-start.textpb
```
Other ways to build:
- **A static Linux executable** that runs on any x86_64 distribution with
nothing else installed: `scripts/build-static.sh`. It needs the
`x86_64-unknown-linux-musl` target and a musl C compiler; the script's
header says how to install them, including on macOS.
- **A container image** with the provided `Dockerfile`:
`docker build -t ferrix .`
Kernel route installation needs Linux and `CAP_NET_ADMIN`; everywhere else
Ferrix is a BGP speaker only. The [installation
guide](docs/content/docs/getting-started/installation.md) and the
[deployment guide](docs/content/docs/operations/deployment.md) cover
privileges, the systemd unit, and upgrading without withdrawing routes.
## Documentation
The documentation is a website built from [`docs/`](docs/) with
[Zola](https://www.getzola.org):
[Zola](https://www.getzola.org): `zola --root docs serve`, then open
<http://127.0.0.1:1111>. The sources are Markdown with Zola front matter,
shortcodes and internal links, so the forge shows them with those parts
unrendered; [`docs/README.md`](docs/README.md) explains what each is. The
pages:
```
zola --root docs serve # http://127.0.0.1:1111
```
- [Overview](docs/content/docs/getting-started/introduction.md) and
- [Overview](docs/content/docs/getting-started/introduction.md),
[installation](docs/content/docs/getting-started/installation.md) and
[quick start](docs/content/docs/getting-started/quick-start.md)
- User guide: [BGP neighbors](docs/content/docs/guides/peers.md),
[import policy](docs/content/docs/guides/import-policy.md),
@@ -47,46 +103,91 @@ zola --root docs serve # http://127.0.0.1:1111
[RPKI](docs/content/docs/guides/rpki.md),
[kernel route installation](docs/content/docs/guides/kernel.md),
[route advertisement](docs/content/docs/guides/passing-routes-on.md),
[configuration examples](docs/examples/)
[graceful restart](docs/content/docs/guides/graceful-restart.md),
[configuration examples](docs/content/docs/guides/examples.md)
- [Deployment](docs/content/docs/operations/deployment.md) and
[monitoring](docs/content/docs/operations/monitoring.md)
- Reference: [configuration](docs/content/docs/reference/configuration.md),
[command line](docs/content/docs/reference/command-line.md),
[HTTP API](docs/content/docs/reference/http-api.md),
[metrics](docs/content/docs/reference/metrics.md),
[supported standards](docs/content/docs/reference/rfcs.md)
- [Architecture](docs/content/docs/development/architecture.md) and
[testing](docs/content/docs/development/testing.md)
- [Architecture](docs/content/docs/development/architecture.md),
[testing](docs/content/docs/development/testing.md) and
[contributing](docs/content/docs/development/contributing.md)
Complete configurations for an edge router, an exchange router and a
transit customer are in [`docs/examples/`](docs/examples/); CI checks that
each of them loads.
## Security
- The status page and HTTP API are not authenticated and show the whole
configuration and routing table. Bind `http_endpoint` to a loopback
address, or put it behind a proxy that authenticates.
- Ferrix does not implement TCP-MD5 or TCP-AO. Filter TCP port 179 in the
host firewall so that only configured neighbors can connect.
- Everything read from the network, the RPKI cache, and the kernel is
parsed without panicking; each parser has property tests and a fuzz
target. `unsafe` code is forbidden throughout.
If you find a vulnerability, please contact the maintainer directly rather
than opening a public issue.
## Layout
| Crate | What is in it |
| -------------------------- | -------------------------------------------------------------------------- |
| `crates/primitives` | Prefixes, address families, parsing helpers |
| `crates/packet` | The wire format: messages, attributes, capabilities |
| `crates/rib` | The routing table, best route selection, the journal of changes |
| `crates/rpki` | The RTR client; origin validation and ASPA path verification |
| `crates/fib` | The best routes in the Linux kernel, over netlink |
| `crates/server` | Sessions, the state machine, policy, configuration |
| `crates/web` | The status page, HTTP API and metrics |
| `crates/integration-tests` | A test server and test peers, and the tests that run the daemon end to end |
| `crates/bench` | Benchmarks at the scale of the Internet routing table |
| `bin/bgp` | The daemon |
## Development
CI runs these on every push, and fails on any warning:
```
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --document-private-items
cargo clippy --locked --workspace --all-targets -- -D warnings
cargo test --locked --workspace
RUSTDOCFLAGS="-D warnings" cargo doc --locked --workspace --no-deps --document-private-items
zola --root docs build
```
It also runs the tests that need a real Linux kernel
(`scripts/kernel-tests.sh`) and the lab with FRR, BIRD and Routinator
(`scripts/interop/lab.sh`); `scripts/linux-vm.sh` runs either from macOS.
Lints are configured once, under `[workspace.lints]` in the root `Cargo.toml`.
`unsafe` code is forbidden throughout.
(`scripts/kernel-tests.sh`) and a lab that peers the daemon with FRR, BIRD
and Routinator (`scripts/interop/lab.sh`); `scripts/linux-vm.sh` runs
either from macOS in a Lima VM.
`cargo test` includes property tests: generated messages must survive being
written and read back, and no input, however damaged, may make a parser
panic. They take about a second; for a longer hunt:
```
```bash
PROPTEST_CASES=1000000 cargo test --release -p bgp-packet --test properties
```
Coverage-guided fuzz targets for the same parsers are in `fuzz/`; see
`fuzz/README.md`. CI runs each of them for a minute.
[`fuzz/README.md`](fuzz/README.md). CI runs each of them for a minute.
### Development with AI agents
Much of Ferrix is written with an AI coding agent (Claude Code) working
alongside the maintainer, who directs the work, reviews every change, and
runs the result on real routers. Nothing is merged on the agent's say-so:
every change passes the checks above, the kernel tests and the
interoperability lab, and the changes that matter are tried on the network
first. Commits made with an agent carry a `Co-Authored-By` trailer that
says so. If you contribute, you are welcome to work the same way, or not;
what is reviewed is the change, not how it was produced.
## License
Apache License 2.0; see `LICENSE`.
Apache License 2.0; see [`LICENSE`](LICENSE).
bin/bgp/src/main.rs
+15 -3
View File
@@ -5,7 +5,7 @@ use std::sync::Arc;
use bgp_server::config::Config;
use bgp_server::proto::config::BgpConfig;
use bgp_server::server::{BgpServer, leave_for_restart, shut_down};
use bgp_web::{AppState, RouterInfo};
use bgp_web::{AppState, CommunityName, RouterInfo};
use clap::Parser;
use eyre::{Context, Result};
use tracing::{error, info};
@@ -79,12 +79,24 @@ async fn main() -> Result<ExitCode> {
server.config().local_asn,
env!("CARGO_PKG_VERSION"),
)
.with_config(args.config.to_string_lossy(), &config_string),
.with_config(args.config.to_string_lossy(), &config_string)
.with_control_communities(server.config().control.is_some())
.with_community_names(
server
.config()
.community_names
.iter()
.map(|(community, name)| CommunityName {
community: community.clone(),
name: name.clone(),
})
.collect(),
),
),
registry: server.registry.clone(),
local_rib: server.local_rib.clone(),
rib: Arc::clone(&server.rib),
refusals: server.refusals.clone(),
rejections: server.rejections.clone(),
rpki: server.rpki.clone(),
kernel: server.kernel.clone(),
restarts: Arc::clone(&server.restarts),
crates/bench/Cargo.toml
-2
View File
@@ -21,8 +21,6 @@ bgp-integration-tests.workspace = true
bgp-rib.workspace = true
bgp-server.workspace = true
tokio.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
[features]
# Counts every allocation, to report how much memory a table holds. The
crates/bench/benches/full_table.rs
+7 -9
View File
@@ -3,7 +3,8 @@
//!
//! Run with `cargo bench -p bgp-bench --bench full_table`. Set
//! `BENCH_PEERS` (default 4) and `BENCH_SCALE_DOWN` (default 1; 10 makes the
//! table ten times smaller) to change the size. The table is synthetic (see
//! table ten times smaller) to change the size. `BENCH_PRESIZE=0` leaves
//! the table to grow as it fills, as it does without `expected_routes`. The table is synthetic (see
//! the crate documentation); `real_table` runs on a real one.
//!
//! This is a stopwatch around whole operations at full size, not a
@@ -17,7 +18,7 @@ use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::memory::{Region, describe};
use bgp_bench::{Prefixes, Shape, announcements, encoded, wire_context};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{Message, PathAttribute};
@@ -25,13 +26,6 @@ use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
use bytes::BytesMut;
fn env(name: &str, default: usize) -> usize {
std::env::var(name)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
@@ -155,6 +149,9 @@ fn main() {
fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) {
let region = Region::begin();
let table = RibTable::new(threads * 16);
if env("BENCH_PRESIZE", 1) == 1 {
table.expect_prefixes(routes / wire.len());
}
let peers: Vec<PeerIndex> = (0..wire.len())
.map(|peer| {
table
@@ -164,6 +161,7 @@ fn into_table(wire: &[Vec<BytesMut>], routes: usize, threads: usize) {
address: Ipv4Addr::new(192, 0, 2, peer as u8 + 1).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.expect("room for the peer")
})
crates/bench/benches/real_table.rs
+1
View File
@@ -283,6 +283,7 @@ fn add_peers(table: &RibTable, dump: &Dump, peers: &[usize]) -> Vec<PeerIndex> {
address: dump.peers[peer].address,
interface: 0,
ibgp: false,
reflector_client: false,
})
.expect("room for the peer")
})
crates/bench/benches/transit.rs
+2 -8
View File
@@ -7,7 +7,7 @@ use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::{Duration, Instant};
use bgp_bench::{Prefixes, Shape, announcements, encoded};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::PathAttribute;
@@ -15,13 +15,6 @@ use bgp_server::marks::{Control, Marks};
use bgp_server::policy::ExportPolicy;
use bytes::BytesMut;
fn env(name: &str, default: usize) -> usize {
std::env::var(name)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
/// How a server passes routes on in one run.
#[derive(Clone, Copy)]
enum Passing {
@@ -49,6 +42,7 @@ fn run(
let server = TestServer::start_with(4_200_000_000, |config| {
config.export = Arc::new(ExportPolicy {
learned_routes: !matches!(passing, Passing::Not),
all_paths: false,
marks: match passing {
Passing::Dressed => Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
_ => Marks::default(),
crates/bench/examples/feed.rs
+158
View File
@@ -0,0 +1,158 @@
//! Feeds the synthetic Internet routing table (see the crate documentation)
//! to a running daemon over TCP, as one or more peers would, and holds the
//! sessions open afterwards. This is the load for profiling the daemon as
//! it takes in a full table; `scripts/profile/full-table.sh` runs it under
//! `perf`.
//!
//! Settings, from the environment:
//!
//! - `FEED_TO`: where the daemon listens (default `127.0.0.1:1790`).
//! - `FEED_PEERS`: how many peers send the table (default 1).
//! - `FEED_SCALE_DOWN`: as `BENCH_SCALE_DOWN`; 10 makes the table ten
//! times smaller (default 1).
//! - `FEED_FROM`: the first three octets of an IPv4 /24, "192.0.2". Peer N
//! connects from .N, which is also the next hop of its IPv4 routes.
//! Without it, peers connect from wherever the system puts them.
//! - `FEED_ASN`: the first peer's AS number (default 64513); the others
//! follow on from it.
//! - `FEED_HOLD_SECS`: how long the sessions stay open after the table is
//! sent (default 30).
//!
//! What is printed: when the sessions are up and how long the writes took.
//! The writes end when the kernel's socket buffers have taken the last
//! bytes, which is before the daemon has read them: when the table is in
//! is for the daemon's API to say.
use std::net::{Ipv4Addr, SocketAddr};
use std::time::{Duration, Instant};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
use bgp_integration_tests::test_peer::TestPeer;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpSocket;
use tokio::net::tcp::OwnedWriteHalf;
/// A KEEPALIVE message: the marker, the length (19) and the type (4).
const KEEPALIVE: [u8; 19] = {
let mut bytes = [0xff; 19];
bytes[16] = 0;
bytes[17] = 19;
bytes[18] = 4;
bytes
};
/// How often KEEPALIVEs go out while the sessions are held open: well
/// inside the 90 s hold time.
const KEEPALIVE_EVERY: Duration = Duration::from_secs(20);
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
#[tokio::main]
async fn main() {
let to: SocketAddr = std::env::var("FEED_TO")
.unwrap_or_else(|_| "127.0.0.1:1790".into())
.parse()
.expect("FEED_TO is an address and port");
let peers = env("FEED_PEERS", 1) as u32;
let shape = Shape::INTERNET.scaled_down(env("FEED_SCALE_DOWN", 1));
let from = std::env::var("FEED_FROM").ok();
let first_asn = env("FEED_ASN", 64513) as u32;
let hold = Duration::from_secs(env("FEED_HOLD_SECS", 30) as u64);
let started = Instant::now();
let prefixes = Prefixes::generate(shape);
let tables: Vec<Vec<u8>> = (1..=peers)
.map(|peer| {
encoded(&announcements(&prefixes, shape, peer))
.iter()
.flat_map(|message| message.iter().copied())
.collect()
})
.collect();
let routes = shape.prefixes() * peers as usize;
let bytes: usize = tables.iter().map(Vec::len).sum();
println!(
"table: {} IPv4 + {} IPv6 prefixes from each of {peers} peer(s): {routes} routes, {:.1} MiB on the wire, made in {:.2} s",
shape.ipv4_prefixes,
shape.ipv6_prefixes,
mib(bytes),
started.elapsed().as_secs_f64()
);
let mut sessions = Vec::new();
for peer in 1..=peers {
let socket = TcpSocket::new_v4().expect("a socket");
if let Some(from) = &from {
let address: Ipv4Addr = format!("{from}.{peer}")
.parse()
.expect("FEED_FROM is the first three octets of an IPv4 address");
socket
.bind(SocketAddr::new(address.into(), 0))
.expect("the peer's address is ours to connect from");
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(
first_asn + peer - 1,
Ipv4Addr::new(10, 0, 0, peer as u8),
90,
)
.await
.expect("the daemon takes the session");
sessions.push(session.into_stream());
}
println!("{peers} session(s) established; sending");
let started = Instant::now();
let senders: Vec<_> = sessions
.into_iter()
.zip(tables)
.map(|(stream, table)| {
tokio::spawn(async move {
let (mut reader, mut writer) = stream.into_split();
// Whatever the daemon sends back, KEEPALIVEs as a rule, is
// read and dropped, so that its writer is never held up.
tokio::spawn(async move {
let mut sink = [0u8; 65536];
while let Ok(n) = reader.read(&mut sink).await {
if n == 0 {
break;
}
}
});
let started = Instant::now();
writer.write_all(&table).await.expect("the daemon reads");
(writer, started.elapsed())
})
})
.collect();
let mut writers: Vec<OwnedWriteHalf> = Vec::new();
for (index, sender) in senders.into_iter().enumerate() {
let (writer, took) = sender.await.expect("the sender runs");
println!(
"peer {}: table written in {:.3} s",
index + 1,
took.as_secs_f64()
);
writers.push(writer);
}
let elapsed = started.elapsed();
println!(
"all {routes} routes written in {:.3} s: {:.2} M routes/s into the sockets",
elapsed.as_secs_f64(),
routes as f64 / elapsed.as_secs_f64() / 1e6
);
let deadline = Instant::now() + hold;
while Instant::now() < deadline {
tokio::time::sleep(KEEPALIVE_EVERY.min(deadline - Instant::now())).await;
for writer in &mut writers {
// A session the daemon closed is left alone.
let _ = writer.write_all(&KEEPALIVE).await;
}
}
println!("sessions closed after {} s", hold.as_secs());
}
crates/bench/src/lib.rs
+15 -5
View File
@@ -1,9 +1,10 @@
//! A synthetic copy of the Internet routing table, as the UPDATE messages a
//! peer would send to announce it.
//!
//! It is made up, not captured, so that the benchmarks need no data files
//! and give the same input every run. It is shaped like the real thing where
//! that matters for cost:
//! It is made up, not captured, so that the benchmarks of a synthetic table
//! need no data files and give the same input every run; the one of a real
//! table reads an MRT dump instead, see [`mrt`]. It is shaped like the real
//! thing where that matters for cost:
//!
//! - about a million IPv4 prefixes, most of them /24s, and about a quarter
//! of a million IPv6 prefixes, most of them /48s;
@@ -163,7 +164,7 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
let hops = 2 + rng.below(4) as usize;
let mut path = vec![peer_asn];
path.extend((0..hops).map(|_| 1 + rng.below(400_000) as u32));
let mut attributes = vec![
let attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
PathAttribute::AsPath(AsPathAttribute {
segments: vec![AsPathSegment {
@@ -199,7 +200,7 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
} else {
None
};
(std::mem::take(&mut attributes), later, large)
(attributes, later, large)
};
let mut updates = Vec::new();
@@ -283,3 +284,12 @@ pub fn wire_context() -> ParserContext {
add_path: Vec::new(),
}
}
/// The value of the environment variable `name` as a number, or `default`
/// when it is not set or is not one: how the benchmarks take their size.
pub fn env(name: &str, default: usize) -> usize {
std::env::var(name)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
crates/fib/src/interfaces.rs
+49 -3
View File
@@ -2,12 +2,13 @@
//! them, and what they say about a next hop: whether it can be reached
//! directly, and from which interface.
use std::collections::{HashMap, HashSet};
use std::collections::{BTreeMap, HashSet};
use std::net::IpAddr;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::InterfaceStatus;
use crate::netlink::{Address, Link, Message};
/// The interface a neighbor is reached on.
@@ -22,7 +23,8 @@ pub struct Connected {
/// The interfaces there are and the addresses they have.
#[derive(Debug, Clone, Default)]
pub struct Interfaces {
links: HashMap<u32, Link>,
/// By index, which is the order they are given out in too.
links: BTreeMap<u32, Link>,
addresses: HashSet<Address>,
}
@@ -64,11 +66,31 @@ impl Interfaces {
}
}
/// The interfaces, by index.
/// The interfaces, in order of index.
pub fn links(&self) -> impl Iterator<Item = &Link> {
self.links.values()
}
/// The interfaces and their addresses, for showing: in order of index,
/// each with its addresses in order.
pub fn status(&self) -> Vec<InterfaceStatus> {
self.links()
.map(|link| {
let mut addresses: Vec<(IpAddr, u8)> = self
.addresses_of(link.index)
.map(|address| (address.local, address.network.length()))
.collect();
addresses.sort();
InterfaceStatus {
index: link.index,
name: link.name.clone(),
up: link.up,
addresses,
}
})
.collect()
}
pub fn link(&self, index: u32) -> Option<&Link> {
self.links.get(&index)
}
@@ -237,4 +259,28 @@ mod tests {
// What is not about interfaces changes nothing.
assert!(!interfaces.apply(&Message::Done));
}
#[test]
fn the_interfaces_are_shown_in_order() {
let mut interfaces = some();
// Learned last, listed in its place.
assert!(interfaces.apply(&link(0, "dummy0", false)));
let shown = interfaces.status();
let indexes: Vec<u32> = shown.iter().map(|interface| interface.index).collect();
assert_eq!(indexes, [0, 1, 2, 3, 4]);
assert_eq!(
shown[2],
InterfaceStatus {
index: 2,
name: "eth0".into(),
up: true,
addresses: vec![
("192.0.2.1".parse().unwrap(), 24),
("2001:db8::1".parse().unwrap(), 64),
],
}
);
assert_eq!((shown[4].up, shown[4].addresses.len()), (false, 1));
assert!(shown[0].addresses.is_empty());
}
}
crates/fib/src/kernel.rs
+256 -118
View File
@@ -3,7 +3,6 @@
use std::collections::{HashMap, HashSet};
use std::io;
use std::net::IpAddr;
use std::sync::{Arc, LazyLock};
use std::time::Duration;
@@ -19,22 +18,27 @@ use prometheus::{
register_int_gauge,
};
use tokio::sync::{mpsc, watch};
use tokio::task::JoinHandle;
use tokio::task::{JoinError, JoinHandle};
use tokio::time::{Instant, sleep_until};
use tracing::{debug, info, warn};
use crate::interfaces::Interfaces;
use crate::netlink::{
self, Change, Dump, KIND_UNICAST, Message, PROTOCOL_BGP, Route, decode, group,
self, Change, Dump, KIND_BLACKHOLE, KIND_UNICAST, KIND_UNSPEC, Message, PROTOCOL_BGP, Route,
decode, group,
};
use crate::plan::{
HashMap as FastHashMap, Neighbor, Op, Plan, Target, Unresolved, resolve_next_hop,
};
use crate::routes::KernelRoutes;
use crate::topology::Topology;
use crate::{Fib, FibConfig, FibError, Stop};
/// "Input/output error": for what should not happen.
const EIO: i32 = 5;
use crate::plan::{Neighbor, Op, Plan, Target, Unresolved, resolve};
use crate::{Fib, FibConfig, FibError, InterfaceStatus, Stop};
/// How many best routes are taken from the routing table at a time.
const BATCH: usize = 1024;
/// How many best routes are taken from the routing table at a time. A
/// poll of the table costs two binary searches and a lock in every shard,
/// however few events it brings back; with a few hundred shards, a batch
/// of a thousand had that overhead for every eight events a shard.
const BATCH: usize = 16384;
/// How many requests are sent to the kernel in one write.
const WRITE_BATCH: usize = 512;
@@ -72,11 +76,18 @@ const RECEIVE_BUFFER: usize = 8 * 1024 * 1024;
/// not at all.
const ANSWER_TIMEOUT: Duration = Duration::from_secs(10);
/// How much room the socket that hints at route changes has: enough for
/// a few hundred messages, and no more, because what it hears is never
/// read. See `Sockets::hints`.
const HINT_BUFFER: usize = 64 * 1024;
/// The errors of the kernel that have a meaning here.
mod errno {
pub const EPERM: i32 = 1;
pub const ENOENT: i32 = 2;
pub const ESRCH: i32 = 3;
/// "Input/output error": for what should not happen.
pub const EIO: i32 = 5;
pub const ENOBUFS: i32 = 105;
}
@@ -128,20 +139,21 @@ struct Socket {
impl Socket {
/// A socket that is sent the notifications of `groups`.
fn open(groups: u32) -> io::Result<Self> {
Self::open_with_room(groups, RECEIVE_BUFFER)
}
/// Opens a socket with `room` octets of receive buffer, as far as the
/// kernel allows.
fn open_with_room(groups: u32, room: usize) -> io::Result<Self> {
let mut socket = TokioSocket::new(NETLINK_ROUTE)?;
socket.socket_mut().bind(&SocketAddr::new(0, groups))?;
// Room for the answers to a whole write, and answers that do not
// repeat the request. Both are for the better and neither is
// needed: what is lost for want of room is noticed and made good.
let raw = socket.socket_ref();
if nix::sys::socket::setsockopt(
raw,
nix::sys::socket::sockopt::RcvBufForce,
&RECEIVE_BUFFER,
)
.is_err()
if nix::sys::socket::setsockopt(raw, nix::sys::socket::sockopt::RcvBufForce, &room).is_err()
{
let _ = raw.set_rx_buf_sz(RECEIVE_BUFFER);
let _ = raw.set_rx_buf_sz(room);
}
let _ = raw.set_cap_ack(true);
Ok(Self {
@@ -220,7 +232,7 @@ impl Socket {
0 => Ok(()),
errno => Err(FibError::Dump {
what: match what {
Dump::Routes(_) => "its routes",
Dump::Routes(_) | Dump::RoutesIn { .. } => "its routes",
Dump::Addresses => "its addresses",
Dump::Links => "its interfaces",
Dump::Nexthops => "its nexthop objects",
@@ -307,6 +319,17 @@ pub(crate) struct Sockets {
/// For what the kernel tells unasked: changes to the interfaces,
/// their addresses, and nexthop objects.
news: Socket,
/// For asking which routes the resolution table has, and only that:
/// with strict checking on, the kernel answers with that table alone
/// rather than with every table it has, which may be a million routes
/// of ours. `None` on a kernel from before it could (Linux 4.20).
resolution: Option<Socket>,
/// Told of every change to any route, as a hint that the resolution
/// table may have changed: what it says is never read, only that it
/// said something. The kernel tells of our own routes going in too, a
/// million at a time, so this socket has little room and is expected
/// to overflow; an overflow is as good a hint as a message.
hints: Socket,
}
impl Sockets {
@@ -318,10 +341,18 @@ impl Sockets {
.socket
.socket_ref()
.add_membership(netlink::GROUP_NEXTHOP);
let resolution = Socket::open(0)?;
let strict = resolution
.socket
.socket_ref()
.set_netlink_get_strict_chk(true)
.is_ok();
Ok(Self {
control: Socket::open(0)?,
routes: Socket::open(0)?,
news,
resolution: strict.then_some(resolution),
hints: Socket::open_with_room(group::IPV4_ROUTE | group::IPV6_ROUTE, HINT_BUFFER)?,
})
}
}
@@ -417,6 +448,8 @@ enum News {
prefix: IpPrefix,
peer: PeerIndex,
next_hop: Option<NlriNextHop>,
/// Whether the route asks for its packets to be dropped.
blackhole: bool,
},
Withdrawn(IpPrefix),
Synced,
@@ -427,23 +460,28 @@ struct Task {
fib: Fib,
config: FibConfig,
control: Socket,
/// See `Sockets::resolution`.
resolution: Option<Socket>,
/// Where changes to routes go to be written; `None` once they have
/// all been handed over, at the end.
queue: Option<mpsc::Sender<Batch>>,
writer: JoinHandle<Result<(), FibError>>,
interfaces: Interfaces,
/// The kernel's view of the machine, as far as next hops are
/// concerned: interfaces, the routes that say the way to a next hop,
/// and nexthop objects.
topology: Topology,
/// What each next hop seen so far resolves to, by the next hop, the
/// IP version of the routes, and for a link-local one the peer.
/// There are as many next hops as neighbors, and a million routes
/// share them; looking through the interfaces for each route would
/// take as long as there are addresses. Emptied when the interfaces
/// change.
resolved: HashMap<Resolving, Result<Target, Unresolved>, foldhash::fast::RandomState>,
/// share them; looking through the interfaces and routes for each
/// route would take as long as there are of those. Emptied when the
/// topology changes.
resolved: FastHashMap<Resolving, Result<Target, Unresolved>>,
plan: Plan,
/// The kernel's nexthop objects that our routes name, by where they
/// lead; `None` if routes carry their own gateway instead, because
/// the kernel has no such objects or the configuration says so.
nexthops: Option<HashMap<Target, u32, foldhash::fast::RandomState>>,
nexthops: Option<FastHashMap<Target, u32>>,
/// The numbers of the nexthop objects that are ours: made by this run,
/// or named by routes of ours that an earlier run left. Others of
/// BGP's are another daemon's, and are neither used nor removed:
@@ -470,7 +508,9 @@ impl Task {
&& route.metric == self.config.metric
}
/// Learns the machine's interfaces and addresses anew.
/// Learns the machine's interfaces and addresses anew. The routes
/// and nexthop objects are learned where the kernel is asked for them
/// anyway: `kernel_routes` and `kernel_nexthops`.
async fn learn_interfaces(&mut self) -> Result<(), FibError> {
let mut interfaces = Interfaces::default();
for what in [Dump::Links, Dump::Addresses] {
@@ -480,18 +520,18 @@ impl Task {
})
.await?;
}
self.interfaces = interfaces;
self.topology.interfaces = interfaces;
self.resolved.clear();
Ok(())
}
/// Asks which nexthop objects of BGP's the kernel has, and returns
/// those that lead somewhere, by number.
/// Asks which nexthop objects the kernel has. Every one goes into
/// the topology, where a route of the resolution table may name it;
/// those of BGP's that lead somewhere are returned by number, for
/// finding our routes among the kernel's.
async fn kernel_nexthops(&mut self) -> Result<HashMap<u32, Target>, FibError> {
let mut by_id = HashMap::new();
if self.nexthops.is_none() {
return Ok(by_id);
}
let mut all = FastHashMap::default();
let asked = self
.control
.dump(Dump::Nexthops, |message| {
@@ -499,10 +539,14 @@ impl Task {
present: true,
nexthop,
} = message
&& nexthop.protocol == PROTOCOL_BGP
&& let (Some(gateway), Some(interface)) = (nexthop.gateway, nexthop.interface)
{
by_id.insert(nexthop.id, Target { gateway, interface });
all.insert(nexthop.id, nexthop);
if nexthop.protocol == PROTOCOL_BGP
&& let (Some(gateway), Some(interface)) =
(nexthop.gateway, nexthop.interface)
{
by_id.insert(nexthop.id, Target::Via { gateway, interface });
}
}
})
.await;
@@ -510,14 +554,18 @@ impl Task {
Ok(()) => {}
// A kernel from before it had them (Linux 5.3).
Err(FibError::Dump { errno, .. }) => {
info!(
errno,
"The kernel has no nexthop objects; routes will carry their own gateway"
);
if self.nexthops.is_some() {
info!(
errno,
"The kernel has no nexthop objects; routes will carry their own gateway"
);
}
self.nexthops = None;
}
Err(e) => return Err(e),
}
self.topology.nexthops = all;
self.resolved.clear();
Ok(by_id)
}
@@ -553,6 +601,36 @@ impl Task {
Ok(found)
}
/// Asks the kernel for the routes of the resolution table anew, and
/// returns whether they differ from those known. Asked of the kernel
/// with strict checking where it has that, which gives the one table;
/// otherwise every table is asked for and all but the one is thrown
/// away.
async fn learn_routes(&mut self) -> Result<bool, FibError> {
let table = self.topology.routes.table();
let mut routes = KernelRoutes::new(table);
for version in [IpVersion::Ipv4, IpVersion::Ipv6] {
let each = |message: Message| {
routes.apply(&message);
};
match &mut self.resolution {
Some(socket) => socket.dump(Dump::RoutesIn { version, table }, each).await?,
None => self.control.dump(Dump::Routes(version), each).await?,
}
}
if routes == self.topology.routes {
return Ok(false);
}
debug!(
table,
routes = routes.len(),
"The routes that say the way to next hops changed"
);
self.topology.routes = routes;
self.resolved.clear();
Ok(true)
}
/// Finds the routes of ours that the kernel has from before.
async fn adopt_routes(
&mut self,
@@ -570,8 +648,9 @@ impl Task {
nexthops.get(&id).copied()
}
(None, Some(gateway), Some(interface)) if route.kind == KIND_UNICAST => {
Some(Target { gateway, interface })
Some(Target::Via { gateway, interface })
}
(None, None, _) if route.kind == KIND_BLACKHOLE => Some(Target::Discard),
_ => None,
};
self.plan.adopt(route.prefix, target, keep);
@@ -589,6 +668,9 @@ impl Task {
if let Some(id) = known.get(&target) {
return Ok(Ok(*id));
}
let Target::Via { gateway, interface } = target else {
unreachable!("a route that discards names no nexthop object");
};
// One at a time, and waited for: there is one for each neighbor,
// not for each route, and a route must not name one that is not
// there yet.
@@ -597,7 +679,7 @@ impl Task {
.control
.ask(
|out, sequence| {
netlink::nexthop_request(out, sequence, &target.gateway, target.interface);
netlink::nexthop_request(out, sequence, &gateway, interface);
},
|message| {
if let Message::Nexthop {
@@ -612,7 +694,7 @@ impl Task {
.await?;
Ok(match (errno, made) {
(0, Some(id)) => {
debug!(id, gateway = %target.gateway, "Made a nexthop object");
debug!(id, %gateway, "Made a nexthop object");
self.owned.insert(id);
if let Some(known) = &mut self.nexthops {
known.insert(target, id);
@@ -620,7 +702,7 @@ impl Task {
Ok(id)
}
// Made, and the kernel did not say under what number.
(0, None) => Err(EIO),
(0, None) => Err(errno::EIO),
(errno, _) => Err(errno),
})
}
@@ -639,16 +721,29 @@ impl Task {
kind: KIND_UNICAST,
};
let change = match op {
Op::Remove { .. } => Change::Remove,
// Whatever kind of route of ours there is for the prefix: the
// kernel removes a route of the kind asked for only, and a
// blackhole is not a unicast route.
Op::Remove { .. } => {
route.kind = KIND_UNSPEC;
Change::Remove
}
Op::Install {
target: Target::Discard,
..
} => {
route.kind = KIND_BLACKHOLE;
Change::Replace
}
Op::Install { target, .. } => {
if self.nexthops.is_some() {
match self.nexthop(*target).await? {
Ok(id) => route.nexthop = Some(id),
Err(errno) => return Ok(Err(errno)),
}
} else {
route.gateway = Some(target.gateway);
route.interface = Some(target.interface);
} else if let Target::Via { gateway, interface } = target {
route.gateway = Some(*gateway);
route.interface = Some(*interface);
}
Change::Replace
}
@@ -693,18 +788,16 @@ impl Task {
let queue = self.queue.as_ref().expect("nothing is sent after the end");
if queue.send(batch).await.is_err() {
// The writer has ended, which it does only by failing.
return Err(self.writer_failed().await);
return Err(writer_error((&mut self.writer).await));
}
Ok(())
}
/// Why the task that writes to the kernel ended.
async fn writer_failed(&mut self) -> FibError {
match (&mut self.writer).await {
Ok(Err(e)) => e,
Ok(Ok(())) => io::Error::other("the task that writes to the kernel ended").into(),
Err(e) => io::Error::other(e).into(),
}
/// Lets the writer know there is no more to come, and waits until it
/// has written what there is.
async fn finish_writer(&mut self) -> Result<(), FibError> {
self.queue = None;
(&mut self.writer).await.map_err(io::Error::other)?
}
/// Deals with the kernel having refused `op`. Returns what to ask of
@@ -719,37 +812,47 @@ impl Task {
}
let error = io::Error::from_raw_os_error(errno);
let what = match op {
Op::Install { prefix, target } => format!(
"install {prefix} via {} on {}",
target.gateway,
self.interfaces.link(target.interface).map_or_else(
|| format!("interface {}", target.interface),
Op::Install {
prefix,
target: Target::Via { gateway, interface },
} => format!(
"install {prefix} via {gateway} on {}",
self.topology.interfaces.link(*interface).map_or_else(
|| format!("interface {interface}"),
|link| link.name.clone()
)
),
Op::Install {
prefix,
target: Target::Discard,
} => format!("install a blackhole for {prefix}"),
Op::Remove { prefix } => format!("remove {prefix}"),
};
// Counted always, said once in a while: a table's worth of
// refusals is a table's worth of lines.
// Counted always, said out loud once in a while: a table's worth
// of rejections is a table's worth of lines.
ERRORS.inc();
let said = format!("the kernel refused to {what}: {error}");
debug!("{said}");
self.fib.update(|status| {
if status.errors == 0 || status.errors.is_power_of_two() {
warn!(refusals = status.errors + 1, "{said}");
}
let before = self.fib.update(|status| {
let before = status.errors;
status.errors += 1;
status.last_error = Some(said);
status.last_error = Some(said.clone());
before
});
if before == 0 || before.is_power_of_two() {
warn!(rejections = before + 1, "{said}");
} else {
debug!(rejections = before + 1, "{said}");
}
Ok(self.plan.refused_by_kernel(op))
}
/// Takes the next changes of best route from the routing table and
/// hands over what brings the kernel in step with them.
async fn follow(&mut self, reader: &mut Reader) -> Result<(), FibError> {
/// hands over what brings the kernel in step with them. Returns whether
/// the reader has read everything the table has for it.
async fn follow(&mut self, reader: &mut Reader) -> Result<bool, FibError> {
let mut news = Vec::with_capacity(BATCH);
// Only what is needed is taken while the table is held.
reader.poll(BATCH, |event| {
let progress = reader.poll(BATCH, |event| {
news.push(match event {
Event::Reset => News::Reset,
Event::Synced => News::Synced,
@@ -762,6 +865,7 @@ impl Task {
prefix,
peer,
next_hop: attrs.next_hop(),
blackhole: attrs.is_blackhole(),
},
Event::Withdrawn { prefix } => News::Withdrawn(prefix),
// Not given to a follower of the best routes.
@@ -784,6 +888,7 @@ impl Task {
prefix,
peer,
next_hop,
blackhole,
} => {
// Whose link a link-local next hop is on is asked only
// where there is one.
@@ -798,17 +903,24 @@ impl Task {
address: peer.address,
interface: peer.interface,
});
let interfaces = &self.interfaces;
let topology = &self.topology;
// Asked of every route, and not remembered with the
// next hop: it is about the prefix.
let target = if interfaces.has_network_around(&prefix) {
let target = if topology.interfaces.has_network_around(&prefix) {
Err(Unresolved::OwnNetwork)
} else if blackhole {
Ok(Target::Discard)
} else {
*self
.resolved
.entry((next_hop, prefix.address_family(), peer))
.or_insert_with(|| {
resolve(interfaces, &prefix, next_hop.as_ref(), peer)
resolve_next_hop(
topology,
prefix.address_family(),
next_hop.as_ref(),
peer,
)
})
};
ops.extend(self.plan.best(prefix, target));
@@ -816,7 +928,8 @@ impl Task {
News::Withdrawn(prefix) => ops.extend(self.plan.withdrawn(prefix)),
}
}
self.send(ops, whole_table).await
self.send(ops, whole_table).await?;
Ok(progress.caught_up)
}
/// Takes in what the writer says became of what it was given.
@@ -878,7 +991,11 @@ impl Task {
while !rest.is_empty() {
let (_, message, used) = decode(rest)?;
rest = &rest[used..];
if self.interfaces.apply(&message) {
// A nexthop object of BGP's is one of ours, or another
// daemon's: never the way to a next hop, and made by the
// hundred as a table goes in. Only the others matter here.
let of_bgp = matches!(&message, Message::Nexthop { nexthop, .. } if nexthop.protocol == PROTOCOL_BGP);
if self.topology.apply(&message) && !of_bgp {
self.resolved.clear();
changed = true;
}
@@ -909,6 +1026,7 @@ impl Task {
self.flush(heard).await?;
self.synced = false;
self.plan = Plan::default();
self.learn_routes().await?;
let nexthops = self.kernel_nexthops().await?;
// While routes of an earlier run are being kept for the peers to
// announce again, they are kept through this too.
@@ -926,7 +1044,7 @@ impl Task {
Some(report) => {
self.hear(report).await?;
}
None => return Err(self.writer_failed().await),
None => return Err(writer_error((&mut self.writer).await)),
}
}
Ok(())
@@ -939,9 +1057,7 @@ impl Task {
routes = self.plan.installed(),
"Leaving our routes in the kernel for the next run"
);
self.queue = None;
(&mut self.writer).await.map_err(io::Error::other)??;
Ok(())
self.finish_writer().await
}
/// Takes everything of ours out of the kernel.
@@ -976,11 +1092,7 @@ impl Task {
}
let ops = self.plan.clear();
self.send(ops, false).await?;
// No more to come: the writer ends when it has written what there
// is.
self.queue = None;
(&mut self.writer).await.map_err(io::Error::other)??;
Ok(())
self.finish_writer().await
}
/// Says how things stand, for whoever asks.
@@ -989,32 +1101,14 @@ impl Task {
UNRESOLVED.set(self.plan.unresolved() as i64);
REFUSED.set(self.plan.refused() as i64);
let interfaces = interfaces_changed.then(|| {
self.fib.publish_interfaces(&self.interfaces);
let mut interfaces: Vec<InterfaceStatus> = self
.interfaces
.links()
.map(|link| {
let mut addresses: Vec<(IpAddr, u8)> = self
.interfaces
.addresses_of(link.index)
.map(|address| (address.local, address.network.length()))
.collect();
addresses.sort();
InterfaceStatus {
index: link.index,
name: link.name.clone(),
up: link.up,
addresses,
}
})
.collect();
interfaces.sort_by_key(|interface| interface.index);
interfaces
self.fib.publish_topology(&self.topology);
self.topology.interfaces.status()
});
self.fib.update(|status| {
status.synced = self.synced;
status.installed = self.plan.installed();
status.kept = if self.keeping { self.plan.kept() } else { 0 };
status.blackholes = self.plan.blackholes();
status.refused = self.plan.refused();
status.unresolved = self
.plan
@@ -1028,6 +1122,16 @@ impl Task {
}
}
/// Why the task that writes to the kernel ended, from what joining it
/// gave. It ends of its own accord only by failing.
fn writer_error(joined: Result<Result<(), FibError>, JoinError>) -> FibError {
match joined {
Ok(Err(e)) => e,
Ok(Ok(())) => io::Error::other("the task that writes to the kernel ended").into(),
Err(e) => io::Error::other(e).into(),
}
}
/// Keeps the kernel's routes in step with the best routes of `rib` until
/// told to stop, and then takes them out.
pub(crate) async fn run(
@@ -1046,6 +1150,8 @@ pub(crate) async fn run(
control,
routes,
mut news,
resolution,
mut hints,
} = sockets;
let (queue, batches) = mpsc::channel(QUEUE);
let (reports, mut heard) = mpsc::unbounded_channel();
@@ -1055,12 +1161,13 @@ pub(crate) async fn run(
rib,
fib,
control,
resolution,
queue: Some(queue),
writer: tokio::spawn(write(routes, batches, reports)),
interfaces: Interfaces::default(),
resolved: HashMap::default(),
topology: Topology::new(config.resolution_table, config.resolve_via_default),
resolved: FastHashMap::default(),
plan: Plan::default(),
nexthops: config.nexthop_objects.then(HashMap::default),
nexthops: config.nexthop_objects.then(FastHashMap::default),
owned: HashSet::new(),
awaiting: None,
marks: 0,
@@ -1071,6 +1178,7 @@ pub(crate) async fn run(
// The socket for news was bound before this: what changes while the
// interfaces are being learned is told there.
task.learn_interfaces().await?;
task.learn_routes().await?;
let nexthops = task.kernel_nexthops().await?;
// Kept for a while, if there is to be any keeping: our table is empty
// until the peers have sent theirs again.
@@ -1097,7 +1205,10 @@ pub(crate) async fn run(
table = config.table,
metric = config.metric,
nexthop_objects = task.nexthops.is_some(),
interfaces = task.interfaces.links().count(),
interfaces = task.topology.interfaces.links().count(),
resolution_table = config.resolution_table,
routes_to_next_hops = task.topology.routes.len(),
filtered_dumps = task.resolution.is_some(),
"Installing best routes in the kernel"
);
@@ -1105,17 +1216,44 @@ pub(crate) async fn run(
// kernel in step; starting a new one does it over.
let mut reader = task.rib.reader();
let mut again = Again::default();
// When the routes that say the way to next hops are next to be asked
// for again, if they are.
let mut routes_again = Again::default();
loop {
tokio::select! {
hint = hints.read() => {
match hint {
// Something about some route changed, or so much did
// that the socket overflowed: either way, ask.
Ok(_) => routes_again.soon(),
Err(e) if e.raw_os_error() == Some(errno::ENOBUFS) => routes_again.soon(),
Err(e) => return Err(e.into()),
}
}
() = at(routes_again.due()), if routes_again.due().is_some() => {
routes_again = Again::default();
if task.learn_routes().await? {
// The way to a next hop may be another now: every
// route is worked out again, which asks the kernel
// only for what comes out different.
info!(
routes = task.topology.routes.len(),
"The kernel's routes changed; working out next hops again"
);
reader = task.rib.reader();
task.publish(true);
}
}
() = reader.changed() => {
task.follow(&mut reader).await?;
task.publish(false);
}
Some(report) = heard.recv() => {
if task.hear(report).await? {
let changed = task.hear(report).await?;
if changed {
again.soon();
}
task.publish(false);
task.publish(changed);
}
read = news.read() => {
if task.on_news(read).await? {
@@ -1130,6 +1268,7 @@ pub(crate) async fn run(
info!("Working the kernel's routes out again");
task.relearn(&mut heard).await?;
reader = task.rib.reader();
task.publish(true);
}
_ = async {
tokio::select! {
@@ -1139,6 +1278,11 @@ pub(crate) async fn run(
}
}, if release_at.is_some() => {
release_at = None;
// What the table has by now, the kernel is to have before
// anything is let go of: a route a peer sent just before
// its End-of-RIB may not have been read yet, and would be
// taken out only to be put back.
while !task.follow(&mut reader).await? {}
task.keeping = false;
let stale = task.plan.release();
info!(
@@ -1148,13 +1292,7 @@ pub(crate) async fn run(
task.send(stale, false).await?;
task.publish(false);
}
ended = &mut task.writer => {
return Err(match ended {
Ok(Err(e)) => e,
Ok(Ok(())) => io::Error::other("the task that writes to the kernel ended").into(),
Err(e) => io::Error::other(e).into(),
});
}
ended = &mut task.writer => return Err(writer_error(ended)),
_ = stopping.changed() => {
if *stopping.borrow() == Some(Stop::Keep) {
task.leave().await?;
crates/fib/src/lib.rs
+45 -21
View File
@@ -9,7 +9,9 @@
//!
//! - [`netlink`] is the kernel's messages, as bytes and back.
//! - [`interfaces`] is the machine's interfaces and what is reachable on
//! them.
//! them; [`routes`] the kernel's own routes, which say the way to what
//! is not; [`topology`] the two together with the kernel's nexthop
//! objects, as one view.
//! - [`plan`] works out what to ask the kernel for.
//! - The rest, which only Linux has, asks it.
//!
@@ -25,17 +27,24 @@
//!
//! # What is installed
//!
//! A best route whose next hop is a neighbor: an address in the network
//! of one of the machine's interfaces. A route whose next hop is further
//! away than that, as with routes from a peer in our own AS that did not
//! set itself as next hop, would need the way to the next hop looked up in
//! turn, which is not done: the prefix then has no route in the kernel.
//! A best route whose next hop can be reached: a neighbor, which is an
//! address in the network of one of the machine's interfaces, or an
//! address the kernel's own routes know the way to. In the second case
//! the route goes into the kernel with the gateway and interface of the
//! route that knows the way, since the kernel does not look a gateway up
//! in turn: a route whose next hop is a router of our own AS, reached over
//! a tunnel by a static route, is installed as a route into the tunnel.
//! Which of the kernel's routes count for this, and which never do, is the
//! business of [`routes`].
pub mod interfaces;
#[cfg(target_os = "linux")]
mod kernel;
pub mod netlink;
pub mod plan;
pub mod routes;
pub mod sockdiag;
pub mod topology;
use std::net::IpAddr;
use std::sync::{Arc, Mutex, PoisonError};
@@ -71,6 +80,12 @@ pub struct FibConfig {
/// everything at once would stop the machine forwarding meanwhile.
/// After this long, what has not been announced again goes.
pub stale_time: Duration,
/// The kernel's table whose routes say the way to a next hop that is
/// not a neighbor: normally `main`, with the connected and static
/// routes. BGP's own routes in it never count; see [`routes`].
pub resolution_table: u32,
/// Whether the default route counts as the way to a next hop.
pub resolve_via_default: bool,
}
impl Default for FibConfig {
@@ -80,6 +95,8 @@ impl Default for FibConfig {
metric: DEFAULT_METRIC,
nexthop_objects: true,
stale_time: DEFAULT_STALE_TIME,
resolution_table: netlink::TABLE_MAIN,
resolve_via_default: false,
}
}
}
@@ -123,6 +140,9 @@ pub struct Status {
/// How many of them are kept from an earlier run though the routing
/// table does not have them (yet); see [`FibConfig::stale_time`].
pub kept: usize,
/// How many of them are blackhole routes, which drop the packets for
/// their prefix.
pub blackholes: usize,
/// Prefixes whose best route is not in the kernel because its next
/// hop is not usable, by why.
pub unresolved: Vec<(&'static str, usize)>,
@@ -135,9 +155,10 @@ pub struct Status {
pub interfaces: Vec<InterfaceStatus>,
}
/// The machine's interfaces as the kernel last described them; `None`
/// until it has. Whoever holds one is told when they change.
pub type InterfacesWatch = watch::Receiver<Option<Arc<interfaces::Interfaces>>>;
/// The machine's interfaces, routes and nexthop objects as the kernel
/// last described them, as far as next hops are concerned; `None` until
/// it has. Whoever holds one is told when they change.
pub type TopologyWatch = watch::Receiver<Option<Arc<topology::Topology>>>;
/// How the task that keeps the kernel's routes is to end.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -151,7 +172,7 @@ enum Stop {
struct Shared {
config: FibConfig,
status: Mutex<Status>,
interfaces: watch::Sender<Option<Arc<interfaces::Interfaces>>>,
topology: watch::Sender<Option<Arc<topology::Topology>>>,
stop: watch::Sender<Option<Stop>>,
/// How many routes of an earlier run were found in the kernel at the
/// start, once that has been looked at.
@@ -171,7 +192,8 @@ pub struct Fib {
impl Fib {
/// Starts keeping the kernel's routes in step with the best routes of
/// `rib`, and returns a handle with the task that does it. The task
/// ends only by failing, or when [`Fib::stop`] is called.
/// ends only by failing, or when [`Fib::stop`] or [`Fib::leave`] is
/// called.
///
/// Must be called from within a Tokio runtime. Fails at once where
/// there is no kernel to talk to in this way.
@@ -184,7 +206,7 @@ impl Fib {
shared: Arc::new(Shared {
config,
status: Mutex::default(),
interfaces: watch::channel(None).0,
topology: watch::channel(None).0,
stop,
adopted: watch::channel(None).0,
release: watch::channel(false).0,
@@ -199,17 +221,18 @@ impl Fib {
self.shared.config
}
/// The machine's interfaces, for those who want to know whether a
/// next hop can be reached before the kernel is asked to use it.
pub fn interfaces(&self) -> InterfacesWatch {
self.shared.interfaces.subscribe()
/// The kernel's view of the machine, for those who want to know
/// whether a next hop can be reached before the kernel is asked to use
/// it.
pub fn topology(&self) -> TopologyWatch {
self.shared.topology.subscribe()
}
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
fn publish_interfaces(&self, interfaces: &interfaces::Interfaces) {
fn publish_topology(&self, topology: &topology::Topology) {
self.shared
.interfaces
.send_replace(Some(Arc::new(interfaces.clone())));
.topology
.send_replace(Some(Arc::new(topology.clone())));
}
pub fn status(&self) -> Status {
@@ -270,15 +293,16 @@ impl Fib {
self.shared.release.send_replace(true);
}
/// Changes the status, and gives back what `change` does.
#[cfg_attr(not(target_os = "linux"), allow(dead_code))]
fn update(&self, change: impl FnOnce(&mut Status)) {
fn update<R>(&self, change: impl FnOnce(&mut Status) -> R) -> R {
change(
&mut self
.shared
.status
.lock()
.unwrap_or_else(PoisonError::into_inner),
);
)
}
}
crates/fib/src/netlink.rs
+72 -1
View File
@@ -23,7 +23,11 @@ pub const TABLE_MAIN: u32 = 254;
/// A route that forwards to a gateway or out of an interface
/// (`RTN_UNICAST`), as opposed to one that drops, say.
/// `RTN_UNSPEC`: in a request to remove a route, a route of any kind.
pub const KIND_UNSPEC: u8 = 0;
pub const KIND_UNICAST: u8 = 1;
/// `RTN_BLACKHOLE`: a route that discards the packets for its prefix.
pub const KIND_BLACKHOLE: u8 = 6;
const HEADER_LEN: usize = 16;
@@ -62,7 +66,9 @@ mod flag {
pub mod group {
pub const LINK: u32 = 0x1;
pub const IPV4_ADDRESS: u32 = 0x10;
pub const IPV4_ROUTE: u32 = 0x40;
pub const IPV6_ADDRESS: u32 = 0x100;
pub const IPV6_ROUTE: u32 = 0x400;
}
/// The number of the group that is told of changes to nexthop objects
@@ -467,7 +473,15 @@ pub enum Change {
/// What to ask for all of.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Dump {
/// Every route of the IP version, of every table.
Routes(IpVersion),
/// The routes of one table. The kernel honours the table only for a
/// socket with strict checking on (`NETLINK_GET_STRICT_CHK`, Linux
/// 4.20 and later); otherwise it answers as for [`Dump::Routes`].
RoutesIn {
version: IpVersion,
table: u32,
},
Addresses,
Links,
Nexthops,
@@ -619,8 +633,24 @@ pub fn nexthop_removal(out: &mut BytesMut, sequence: u32, id: u32) {
/// Writes the request for all of `what` to the end of `out`. The answer is
/// a message for each, and then [`Message::Done`].
pub fn dump_request(out: &mut BytesMut, sequence: u32, what: Dump) {
if let Dump::RoutesIn { version, table } = what {
let start = begin(out, message_type::GET_ROUTE, flag::DUMP, sequence);
// rtmsg: family, destination length, source length, TOS, table,
// protocol, scope, type, and four octets of flags; the table in
// its octet if it fits, else as an attribute.
out.put_u8(family_of(version));
out.put_bytes(0, 3);
out.put_u8(u8::try_from(table).unwrap_or(TABLE_IN_ATTRIBUTE));
out.put_bytes(0, 7);
if u8::try_from(table).is_err() {
put_attribute(out, route_attribute::TABLE, &table.to_ne_bytes());
}
finish(out, start);
return;
}
let (message_type, family, fixed_length) = match what {
Dump::Routes(version) => (message_type::GET_ROUTE, family_of(version), 12),
Dump::RoutesIn { .. } => unreachable!("dealt with above"),
Dump::Addresses => (message_type::GET_ADDRESS, 0, 8),
Dump::Links => (message_type::GET_LINK, 0, 16),
Dump::Nexthops => (message_type::GET_NEXTHOP, 0, 8),
@@ -883,6 +913,42 @@ mod tests {
}
}
/// A dump of one table names the table in its octet, or, for a number
/// the octet cannot hold, in an attribute.
#[test]
fn a_dump_of_one_table_names_the_table() {
let mut wire = BytesMut::new();
dump_request(
&mut wire,
3,
Dump::RoutesIn {
version: IpVersion::Ipv4,
table: TABLE_MAIN,
},
);
assert_eq!(wire.len(), 28);
assert_eq!(wire[16], 2, "AF_INET");
assert_eq!(wire[20], TABLE_MAIN as u8);
let mut wire = BytesMut::new();
dump_request(
&mut wire,
3,
Dump::RoutesIn {
version: IpVersion::Ipv6,
table: 1000,
},
);
// The fixed part, then an attribute of four octets with its
// four-octet header.
assert_eq!(wire.len(), 36);
assert_eq!(wire[16], 10, "AF_INET6");
assert_eq!(wire[20], TABLE_IN_ATTRIBUTE);
assert_eq!(wire[28..30], 8u16.to_ne_bytes());
assert_eq!(wire[30..32], route_attribute::TABLE.to_ne_bytes());
assert_eq!(wire[32..36], 1000u32.to_ne_bytes());
}
/// A message as the kernel would send it: header, fixed part,
/// attributes.
fn message(message_type: u16, sequence: u32, fixed: &[u8], attrs: &[(u16, &[u8])]) -> Vec<u8> {
@@ -1091,9 +1157,14 @@ mod tests {
}
/// Noise behind a header that could be a message's, to get past it.
/// The types of nexthop messages lie far from the rest.
#[test]
fn noise_behind_a_plausible_header_is_never_more_than_an_error(
message_type in 0u16..30,
message_type in prop_oneof![
0u16..30,
Just(message_type::NEW_NEXTHOP),
Just(message_type::DEL_NEXTHOP),
],
body in proptest::collection::vec(any::<u8>(), 0..100),
) {
let wire = message(message_type, 1, &body, &[]);
crates/fib/src/plan.rs
+526 -97
View File
@@ -9,18 +9,25 @@ use bgp_packet::message::NlriNextHop;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::interfaces::Interfaces;
use crate::netlink::KIND_UNICAST;
use crate::topology::Topology;
// Asked for every route: a quick hash matters here.
type HashMap<K, V> = std::collections::HashMap<K, V, foldhash::fast::RandomState>;
type HashSet<K> = std::collections::HashSet<K, foldhash::fast::RandomState>;
pub(crate) type HashMap<K, V> = std::collections::HashMap<K, V, foldhash::fast::RandomState>;
pub(crate) type HashSet<K> = std::collections::HashSet<K, foldhash::fast::RandomState>;
/// Where the kernel is to send packets for a prefix.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct Target {
pub gateway: IpAddr,
/// The index of the interface the gateway is on.
pub interface: u32,
pub enum Target {
/// To a gateway, out of an interface.
Via {
gateway: IpAddr,
/// The index of the interface the gateway is on.
interface: u32,
},
/// Nowhere: the packets are dropped. For a route that asks for that
/// with the BLACKHOLE community (RFC 7999).
Discard,
}
/// Why a route's next hop is not something the kernel can be given.
@@ -32,10 +39,17 @@ pub enum Unresolved {
OtherFamily,
/// The next hop is one of this machine's own addresses.
Ourselves,
/// The next hop is not in the network of any interface that is up.
/// Finding the way to such a next hop through other routes is not
/// done.
NotConnected,
/// The next hop is not in the network of any interface that is up,
/// and none of the kernel's routes that count says the way to it
/// (see `crate::routes`).
NoRoute,
/// The only route to the next hop is the default route, which is not
/// taken as the way to a next hop unless asked for
/// (`Topology::via_default`).
OnlyDefault,
/// The route to the next hop is one that discards packets: a
/// blackhole, unreachable or prohibit route.
Discarded,
/// The route is for a network this machine is on itself, or for part
/// of one. The kernel reaches that directly; a route from a peer that
/// said otherwise would send the neighbors' traffic, and the sessions
@@ -44,11 +58,13 @@ pub enum Unresolved {
}
impl Unresolved {
pub const ALL: [Self; 5] = [
pub const ALL: [Self; 7] = [
Self::Missing,
Self::OtherFamily,
Self::Ourselves,
Self::NotConnected,
Self::NoRoute,
Self::OnlyDefault,
Self::Discarded,
Self::OwnNetwork,
];
@@ -57,12 +73,24 @@ impl Unresolved {
Self::Missing => "no next hop",
Self::OtherFamily => "IPv4 next hop for an IPv6 route",
Self::Ourselves => "next hop is this machine",
Self::NotConnected => "next hop not directly connected",
Self::NoRoute => "no route to the next hop",
Self::OnlyDefault => "next hop reachable only by the default route",
Self::Discarded => "the route to the next hop discards packets",
Self::OwnNetwork => "inside a network of this machine",
}
}
}
// A reason's place in `ALL` is its number as declared, which is what
// [`Plan`] counts them by.
const _: () = {
let mut at = 0;
while at < Unresolved::ALL.len() {
assert!(Unresolved::ALL[at] as usize == at);
at += 1;
}
};
/// The peer a route was learned from, as far as its next hop is concerned.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct Neighbor {
@@ -83,21 +111,76 @@ impl From<IpAddr> for Neighbor {
}
/// Works out where the kernel is to send packets for a route to `prefix`
/// with the next hop `next_hop`, learned from `peer`.
/// with the next hop `next_hop`, learned from `peer`. `blackhole` says
/// the route asks for its packets to be dropped (the BLACKHOLE community,
/// RFC 7999): then they go nowhere, and the next hop is not looked at.
///
/// Only a next hop that is a neighbor on one of our interfaces is
/// resolved. A link-local next hop means something only on one link, and
/// that is taken to be the link the peer itself is on.
/// A next hop that is a neighbor on one of our interfaces is where the
/// packets go. One that is not is looked up in the kernel's own routes,
/// and the packets go where the route that covers it sends them: to its
/// gateway, or onto its interface. A link-local next hop means something
/// only on one link, and that is taken to be the link the peer itself is
/// on.
pub fn resolve(
interfaces: &Interfaces,
topology: &Topology,
prefix: &IpPrefix,
next_hop: Option<&NlriNextHop>,
peer: Option<Neighbor>,
blackhole: bool,
) -> Result<Target, Unresolved> {
if interfaces.has_network_around(prefix) {
if topology.interfaces.has_network_around(prefix) {
return Err(Unresolved::OwnNetwork);
}
resolve_next_hop(interfaces, prefix.address_family(), next_hop, peer)
if blackhole {
return Ok(Target::Discard);
}
resolve_next_hop(topology, prefix.address_family(), next_hop, peer)
}
/// The way to `gateway`, which is not a neighbor, by the kernel's routes
/// that count: where the route that covers it sends packets. One step
/// only: the kernel sees to it that a route's own gateway is a neighbor.
fn through_routes(
topology: &Topology,
routes: IpVersion,
gateway: IpAddr,
) -> Result<Target, Unresolved> {
let route = topology.routes.lookup(gateway).ok_or(Unresolved::NoRoute)?;
if route.prefix.length() == 0 && !topology.via_default {
return Err(Unresolved::OnlyDefault);
}
if route.kind != KIND_UNICAST {
return Err(Unresolved::Discarded);
}
let (hop, interface) = match route.nexthop {
Some(id) => {
// A group of nexthops has no gateway or interface of its own;
// which of its members packets take is the kernel's to say,
// and nothing to give it as a route.
let nexthop = topology.nexthops.get(&id).ok_or(Unresolved::NoRoute)?;
(
nexthop.gateway,
nexthop.interface.ok_or(Unresolved::NoRoute)?,
)
}
None => (route.gateway, route.interface.ok_or(Unresolved::NoRoute)?),
};
if !topology
.interfaces
.link(interface)
.is_some_and(|link| link.up)
{
return Err(Unresolved::NoRoute);
}
// A route without a gateway puts the destination on its interface:
// the next hop itself is the neighbor there.
let gateway = hop.unwrap_or(gateway);
// An IPv4 route may go by an IPv6 gateway (RFC 8950, and the kernel
// allows it); an IPv6 route by an IPv4 gateway is nothing.
if routes == IpVersion::Ipv6 && gateway.is_ipv4() {
return Err(Unresolved::NoRoute);
}
Ok(Target::Via { gateway, interface })
}
/// The part of [`resolve`] that is the same for every route of IP version
@@ -105,11 +188,12 @@ pub fn resolve(
/// those who ask it of many such routes, and ask of each only whether it
/// is for one of the machine's own networks.
pub fn resolve_next_hop(
interfaces: &Interfaces,
topology: &Topology,
routes: IpVersion,
next_hop: Option<&NlriNextHop>,
peer: Option<Neighbor>,
) -> Result<Target, Unresolved> {
let interfaces = &topology.interfaces;
let next_hop = next_hop.ok_or(Unresolved::Missing)?;
let version = match next_hop {
NlriNextHop::Ipv4(_) => IpVersion::Ipv4,
@@ -125,17 +209,17 @@ pub fn resolve_next_hop(
if interfaces.is_local(gateway) {
return Err(Unresolved::Ourselves);
}
interfaces
.connected(gateway)
.map(|connected| Target {
match interfaces.connected(gateway) {
Some(connected) => Ok(Target::Via {
gateway,
interface: connected.interface,
})
.ok_or(Unresolved::NotConnected)
}),
None => through_routes(topology, routes, gateway),
}
};
// The interface the peer is a neighbor on, for a link-local gateway.
let on_peers_link = |gateway: Ipv6Addr| {
let interface = match peer.ok_or(Unresolved::NotConnected)? {
let interface = match peer.ok_or(Unresolved::NoRoute)? {
// The session is on a link-local address itself, and so says
// which link: every interface has fe80::/64, and looking the
// peer up there would find any of them.
@@ -147,14 +231,14 @@ pub fn resolve_next_hop(
.connected(address)
.map(|connected| connected.interface),
}
.ok_or(Unresolved::NotConnected)?;
.ok_or(Unresolved::NoRoute)?;
if interfaces
.addresses_of(interface)
.any(|ours| ours.local == IpAddr::V6(gateway))
{
return Err(Unresolved::Ourselves);
}
Ok(Target {
Ok(Target::Via {
gateway: gateway.into(),
interface,
})
@@ -172,8 +256,11 @@ pub fn resolve_next_hop(
{
on_peers_link(link_local)
}
// Where there is no way to the global one, the link-local one is
// tried; a way that is there and is bad (ourselves, a blackhole)
// is an answer as it stands.
NlriNextHop::Ipv6WithLl { global, link_local } => match direct(global.into()) {
Err(Unresolved::NotConnected) => on_peers_link(link_local),
Err(Unresolved::NoRoute | Unresolved::OnlyDefault) => on_peers_link(link_local),
resolved => resolved,
},
}
@@ -230,6 +317,14 @@ pub struct Plan {
unresolved_counts: [usize; Unresolved::ALL.len()],
/// Prefixes whose route the kernel refused.
refused: HashSet<IpPrefix>,
/// How many entries of `installed` are blackhole routes, and how many
/// are kept from an earlier run without having been confirmed. Kept in
/// step with the map by [`Plan::put`] and [`Plan::take`], so that
/// asking costs nothing however large the table: the kernel task asks
/// after every batch of changes, and a scan of a million entries each
/// time was a third of its work.
blackholes: usize,
kept_unconfirmed: usize,
}
impl Plan {
@@ -238,6 +333,42 @@ impl Plan {
self.installed.len()
}
/// How many of the routes the kernel has drop their packets.
pub fn blackholes(&self) -> usize {
self.blackholes
}
/// What `entry` counts for: one if it is a blackhole route, and one if
/// it is kept from an earlier run without having been confirmed.
fn weigh(entry: &Entry) -> (usize, usize) {
(
usize::from(entry.target == Some(Target::Discard)),
usize::from(entry.kept && !entry.confirmed),
)
}
/// Puts `entry` in as the route of `prefix`, in place of what was
/// there, keeping the counts in step.
fn put(&mut self, prefix: IpPrefix, entry: Entry) {
let (blackhole, kept) = Self::weigh(&entry);
self.blackholes += blackhole;
self.kept_unconfirmed += kept;
if let Some(old) = self.installed.insert(prefix, entry) {
let (blackhole, kept) = Self::weigh(&old);
self.blackholes -= blackhole;
self.kept_unconfirmed -= kept;
}
}
/// Takes the route of `prefix` out, keeping the counts in step.
fn take(&mut self, prefix: &IpPrefix) -> Option<Entry> {
let old = self.installed.remove(prefix)?;
let (blackhole, kept) = Self::weigh(&old);
self.blackholes -= blackhole;
self.kept_unconfirmed -= kept;
Some(old)
}
/// How many prefixes have a best route that could not be given to the
/// kernel for want of a usable next hop.
pub fn unresolved(&self) -> usize {
@@ -259,12 +390,19 @@ impl Plan {
Some(reason) => self.unresolved.insert(prefix, reason),
None => self.unresolved.remove(&prefix),
};
for (reason, by) in [(before, -1isize), (reason, 1)] {
if let Some(reason) = reason {
let count = &mut self.unresolved_counts[reason as usize];
*count = count.wrapping_add_signed(by);
}
if let Some(before) = before {
self.unresolved_counts[before as usize] -= 1;
}
if let Some(reason) = reason {
self.unresolved_counts[reason as usize] += 1;
}
}
/// Forgets which prefixes are unresolved or refused, and why.
fn forget_unresolved_and_refused(&mut self) {
self.unresolved.clear();
self.unresolved_counts = [0; Unresolved::ALL.len()];
self.refused.clear();
}
/// How many prefixes have a best route that the kernel refused.
@@ -272,11 +410,6 @@ impl Plan {
self.refused.len()
}
/// The prefixes of the routes the kernel has.
pub fn installed_prefixes(&self) -> impl Iterator<Item = &IpPrefix> {
self.installed.keys()
}
/// Notes a route of ours that the kernel already has. It stays only if
/// the routing table turns out to have the same.
///
@@ -286,7 +419,7 @@ impl Plan {
/// have sent theirs again, and to remove everything meanwhile would
/// stop the machine forwarding for as long as that takes.
pub fn adopt(&mut self, prefix: IpPrefix, target: Option<Target>, keep: bool) {
self.installed.insert(
self.put(
prefix,
Entry {
target,
@@ -299,36 +432,38 @@ impl Plan {
/// How many routes are being kept from an earlier run without the
/// routing table having confirmed them.
pub fn kept(&self) -> usize {
self.installed
.values()
.filter(|entry| entry.kept && !entry.confirmed)
.count()
self.kept_unconfirmed
}
/// The time for keeping unconfirmed routes of an earlier run is over:
/// those the routing table still does not have go.
pub fn release(&mut self) -> Vec<Op> {
let mut stale = Vec::new();
let mut blackholes_gone = 0;
self.installed.retain(|prefix, entry| {
let goes = entry.kept && !entry.confirmed;
entry.kept = false;
if goes {
stale.push(Op::Remove { prefix: *prefix });
blackholes_gone += usize::from(entry.target == Some(Target::Discard));
}
!goes
});
// What stays was confirmed, and is now not kept either.
self.kept_unconfirmed = 0;
self.blackholes -= blackholes_gone;
stale
}
/// The whole table follows: nothing is known to be wanted until it is
/// heard of again.
pub fn reset(&mut self) {
self.kept_unconfirmed = 0;
for entry in self.installed.values_mut() {
entry.confirmed = false;
self.kept_unconfirmed += usize::from(entry.kept);
}
self.unresolved.clear();
self.unresolved_counts = [0; Unresolved::ALL.len()];
self.refused.clear();
self.forget_unresolved_and_refused();
}
/// The best route for `prefix` is now one that resolves as `target`.
@@ -347,20 +482,26 @@ impl Plan {
confirmed: true,
kept: false,
});
let before = Self::weigh(entry);
entry.confirmed = true;
if entry.target == Some(target) {
return None;
}
entry.target = Some(target);
Some(Op::Install { prefix, target })
let op = if entry.target == Some(target) {
None
} else {
entry.target = Some(target);
Some(Op::Install { prefix, target })
};
let after = Self::weigh(entry);
self.blackholes += after.0;
self.blackholes -= before.0;
self.kept_unconfirmed += after.1;
self.kept_unconfirmed -= before.1;
op
}
Err(reason) => {
self.set_unresolved(prefix, Some(reason));
// Packets must not go on following a route that is no
// longer the best.
self.installed
.remove(&prefix)
.map(|_| Op::Remove { prefix })
self.take(&prefix).map(|_| Op::Remove { prefix })
}
}
}
@@ -369,9 +510,7 @@ impl Plan {
pub fn withdrawn(&mut self, prefix: IpPrefix) -> Option<Op> {
self.set_unresolved(prefix, None);
self.refused.remove(&prefix);
self.installed
.remove(&prefix)
.map(|_| Op::Remove { prefix })
self.take(&prefix).map(|_| Op::Remove { prefix })
}
/// The whole table has been given: what was not in it goes.
@@ -385,7 +524,7 @@ impl Plan {
stale
.into_iter()
.map(|prefix| {
self.installed.remove(&prefix);
self.take(&prefix);
Op::Remove { prefix }
})
.collect()
@@ -400,7 +539,7 @@ impl Plan {
if self.installed.get(prefix)?.target != Some(*target) {
return None;
}
self.installed.remove(prefix);
self.take(prefix);
self.refused.insert(*prefix);
// What the kernel refused to replace, it still has.
Some(Op::Remove { prefix: *prefix })
@@ -412,9 +551,9 @@ impl Plan {
/// Everything goes: the changes that empty the kernel of our routes.
pub fn clear(&mut self) -> Vec<Op> {
self.unresolved.clear();
self.unresolved_counts = [0; Unresolved::ALL.len()];
self.refused.clear();
self.forget_unresolved_and_refused();
self.blackholes = 0;
self.kept_unconfirmed = 0;
self.installed
.drain()
.map(|(prefix, _)| Op::Remove { prefix })
@@ -427,27 +566,27 @@ mod tests {
use proptest::prelude::*;
use super::*;
use crate::netlink::{Address, Link, Message};
use crate::netlink::{Address, Link, Message, Nexthop, PROTOCOL_BGP, Route, TABLE_MAIN};
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn target(gateway: &str, interface: u32) -> Target {
Target {
Target::Via {
gateway: gateway.parse().unwrap(),
interface,
}
}
fn interfaces() -> Interfaces {
let mut interfaces = Interfaces::default();
fn interfaces() -> Topology {
let mut topology = Topology::new(TABLE_MAIN, false);
for (index, local, network) in [
(2, "192.0.2.1", "192.0.2.0/24"),
(2, "2001:db8::1", "2001:db8::/64"),
(3, "198.51.100.1", "198.51.100.0/24"),
] {
interfaces.apply(&Message::Link {
topology.apply(&Message::Link {
present: true,
link: Link {
index,
@@ -455,7 +594,7 @@ mod tests {
up: true,
},
});
interfaces.apply(&Message::Address {
topology.apply(&Message::Address {
present: true,
address: Address {
interface: index,
@@ -464,7 +603,28 @@ mod tests {
},
});
}
interfaces
topology
}
/// A route of the kernel's main table, as a static route is.
fn kernel_route(prefix_text: &str, gateway: Option<&str>, interface: Option<u32>) -> Route {
Route {
prefix: prefix(prefix_text),
table: TABLE_MAIN,
protocol: 4,
metric: 0,
gateway: gateway.map(|g| g.parse().unwrap()),
interface,
nexthop: None,
kind: KIND_UNICAST,
}
}
fn with_route(topology: &mut Topology, route: Route) {
assert!(topology.apply(&Message::Route {
present: true,
route,
}));
}
fn neighbor(address: &str) -> Neighbor {
@@ -481,27 +641,30 @@ mod tests {
let v6_hop = |s: &str| NlriNextHop::Ipv6(s.parse().unwrap());
assert_eq!(
resolve(&interfaces, &v4, Some(&v4_hop("192.0.2.2")), peer),
resolve(&interfaces, &v4, Some(&v4_hop("192.0.2.2")), peer, false),
Ok(target("192.0.2.2", 2))
);
// A neighbor other than the peer: a route server's routes.
assert_eq!(
resolve(&interfaces, &v4, Some(&v4_hop("198.51.100.9")), peer),
resolve(&interfaces, &v4, Some(&v4_hop("198.51.100.9")), peer, false),
Ok(target("198.51.100.9", 3))
);
assert_eq!(
resolve(&interfaces, &v6, Some(&v6_hop("2001:db8::2")), peer),
resolve(&interfaces, &v6, Some(&v6_hop("2001:db8::2")), peer, false),
Ok(target("2001:db8::2", 2))
);
for (hop, why) in [
(Some(v4_hop("10.9.9.9")), Unresolved::NotConnected),
(Some(v4_hop("10.9.9.9")), Unresolved::NoRoute),
(Some(v4_hop("192.0.2.1")), Unresolved::Ourselves),
(None, Unresolved::Missing),
] {
assert_eq!(resolve(&interfaces, &v4, hop.as_ref(), peer), Err(why));
assert_eq!(
resolve(&interfaces, &v4, hop.as_ref(), peer, false),
Err(why)
);
}
assert_eq!(
resolve(&interfaces, &v6, Some(&v4_hop("192.0.2.2")), peer),
resolve(&interfaces, &v6, Some(&v4_hop("192.0.2.2")), peer, false),
Err(Unresolved::OtherFamily)
);
// A route for part of a network we are on is not for the kernel,
@@ -518,7 +681,7 @@ mod tests {
IpVersion::Ipv6 => v6_hop("2001:db8::2"),
};
assert_eq!(
resolve(&interfaces, &ours, Some(&hop), peer),
resolve(&interfaces, &ours, Some(&hop), peer, false),
Err(Unresolved::OwnNetwork),
"{ours}"
);
@@ -528,14 +691,15 @@ mod tests {
&interfaces,
&prefix("192.0.0.0/22"),
Some(&v4_hop("192.0.2.2")),
peer
peer,
false
),
Ok(target("192.0.2.2", 2))
);
// An IPv4 route by way of an IPv6 neighbor: a link with only IPv6
// addresses (RFC 8950).
assert_eq!(
resolve(&interfaces, &v4, Some(&v6_hop("2001:db8::2")), peer),
resolve(&interfaces, &v4, Some(&v6_hop("2001:db8::2")), peer, false),
Ok(target("2001:db8::2", 2))
);
assert_eq!(
@@ -547,6 +711,7 @@ mod tests {
link_local: "fe80::2".parse().unwrap(),
}),
Some(neighbor("2001:db8::2")),
false
),
Ok(target("fe80::2", 2))
);
@@ -564,26 +729,44 @@ mod tests {
};
// The global address is a neighbor: it is used.
assert_eq!(
resolve(&interfaces, &v6, Some(&both("2001:db8::2")), peer),
resolve(&interfaces, &v6, Some(&both("2001:db8::2")), peer, false),
Ok(target("2001:db8::2", 2))
);
// It is not: the link-local one, on the interface the peer is on.
assert_eq!(
resolve(&interfaces, &v6, Some(&both("2001:db8:ffff::2")), peer),
resolve(
&interfaces,
&v6,
Some(&both("2001:db8:ffff::2")),
peer,
false
),
Ok(target("fe80::2", 2))
);
assert_eq!(
resolve(&interfaces, &v6, Some(&NlriNextHop::Ipv6(link_local)), peer),
resolve(
&interfaces,
&v6,
Some(&NlriNextHop::Ipv6(link_local)),
peer,
false
),
Ok(target("fe80::2", 2))
);
// A peer that is not a neighbor says nothing about the link.
let far = Some(neighbor("2001:db8:ffff::9"));
assert_eq!(
resolve(&interfaces, &v6, Some(&NlriNextHop::Ipv6(link_local)), far),
Err(Unresolved::NotConnected)
resolve(
&interfaces,
&v6,
Some(&NlriNextHop::Ipv6(link_local)),
far,
false
),
Err(Unresolved::NoRoute)
);
assert_eq!(
resolve(&interfaces, &v6, Some(&both("2001:db8::1")), peer),
resolve(&interfaces, &v6, Some(&both("2001:db8::1")), peer, false),
Err(Unresolved::Ourselves)
);
}
@@ -628,7 +811,7 @@ mod tests {
for interface in [2, 3] {
for route in [&v4, &v6] {
assert_eq!(
resolve(&interfaces, route, Some(&next_hop), on(interface)),
resolve(&interfaces, route, Some(&next_hop), on(interface), false),
Ok(target("fe80::2", interface)),
"{next_hop:?} on {interface}"
);
@@ -636,28 +819,220 @@ mod tests {
}
// An interface that is not there, or is down.
assert_eq!(
resolve(&interfaces, &v6, Some(&next_hop), on(9)),
Err(Unresolved::NotConnected)
resolve(&interfaces, &v6, Some(&next_hop), on(9), false),
Err(Unresolved::NoRoute)
);
}
// Our own address on that link, which on another is a neighbor's.
let ours = NlriNextHop::Ipv6("fe80::a".parse().unwrap());
assert_eq!(
resolve(&interfaces, &v6, Some(&ours), on(2)),
resolve(&interfaces, &v6, Some(&ours), on(2), false),
Err(Unresolved::Ourselves)
);
assert_eq!(
resolve(&interfaces, &v6, Some(&ours), on(3)),
resolve(&interfaces, &v6, Some(&ours), on(3), false),
Ok(target("fe80::a", 3))
);
// A global next hop on such a session is looked up as any other.
let global = NlriNextHop::Ipv6("2001:db8::2".parse().unwrap());
assert_eq!(
resolve(&interfaces, &v6, Some(&global), on(3)),
resolve(&interfaces, &v6, Some(&global), on(3), false),
Ok(target("2001:db8::2", 2))
);
}
/// A next hop that is no neighbor is reached the way the kernel's
/// own routes say, and the route goes into the kernel with that way's
/// gateway and interface.
#[test]
fn a_far_next_hop_resolves_by_the_kernels_routes() {
let mut topology = interfaces();
let v4 = prefix("203.0.113.0/24");
let v6 = prefix("2001:db8:100::/48");
let peer = Some(neighbor("192.0.2.2"));
let v4_hop = |s: &str| NlriNextHop::Ipv4(s.parse().unwrap());
let v6_hop = |s: &str| NlriNextHop::Ipv6(s.parse().unwrap());
let far = v4_hop("10.9.9.9");
// Nothing says the way.
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Err(Unresolved::NoRoute)
);
// The default route does not count unless asked for.
with_route(
&mut topology,
kernel_route("0.0.0.0/0", Some("192.0.2.254"), Some(2)),
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Err(Unresolved::OnlyDefault)
);
topology.via_default = true;
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Ok(target("192.0.2.254", 2))
);
topology.via_default = false;
// A static route with a gateway: the gateway is where packets go.
with_route(
&mut topology,
kernel_route("10.9.0.0/16", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Ok(target("198.51.100.9", 3))
);
// A more specific route onto an interface, as a tunnel's is: the
// next hop itself is the neighbor there.
topology.apply(&Message::Link {
present: true,
link: Link {
index: 7,
name: "wg0".into(),
up: true,
},
});
with_route(&mut topology, kernel_route("10.9.9.0/24", None, Some(7)));
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Ok(target("10.9.9.9", 7))
);
// The tunnel goes down: the kernel keeps such routes, we do not
// use them.
topology.apply(&Message::Link {
present: true,
link: Link {
index: 7,
name: "wg0".into(),
up: false,
},
});
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Err(Unresolved::NoRoute)
);
// A route that names a nexthop object goes where the object says.
with_route(
&mut topology,
Route {
nexthop: Some(77),
..kernel_route("10.9.9.9/32", None, None)
},
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Err(Unresolved::NoRoute),
"an object the kernel has not told of"
);
topology.apply(&Message::Nexthop {
present: true,
nexthop: Nexthop {
id: 77,
protocol: 4,
gateway: Some("192.0.2.77".parse().unwrap()),
interface: Some(2),
},
});
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Ok(target("192.0.2.77", 2))
);
// A route that discards packets is a way to nowhere.
with_route(
&mut topology,
Route {
kind: 6, // blackhole
..kernel_route("10.9.9.9/32", None, None)
},
);
assert_eq!(
resolve(&topology, &v4, Some(&far), peer, false),
Err(Unresolved::Discarded)
);
// Routes of BGP's never say the way, be the prefix ever so long.
assert!(!topology.apply(&Message::Route {
present: true,
route: Route {
protocol: PROTOCOL_BGP,
metric: 20,
..kernel_route("10.9.9.9/32", Some("198.51.100.9"), Some(3))
},
}));
// An IPv6 next hop by an IPv4 gateway is no way for an IPv6 route,
// but is one for an IPv4 route (RFC 8950 the other way about is
// what the kernel allows).
with_route(
&mut topology,
kernel_route("2001:db8:9::/48", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v6, Some(&v6_hop("2001:db8:9::9")), peer, false),
Err(Unresolved::NoRoute)
);
with_route(
&mut topology,
kernel_route("2001:db8:9::/48", Some("2001:db8::9"), Some(2)),
);
assert_eq!(
resolve(&topology, &v6, Some(&v6_hop("2001:db8:9::9")), peer, false),
Ok(target("2001:db8::9", 2))
);
assert_eq!(
resolve(&topology, &v4, Some(&v6_hop("2001:db8:9::9")), peer, false),
Ok(target("2001:db8::9", 2))
);
// A neighbor is a neighbor, whatever the routes say.
with_route(
&mut topology,
kernel_route("192.0.2.2/32", Some("198.51.100.9"), Some(3)),
);
assert_eq!(
resolve(&topology, &v4, Some(&v4_hop("192.0.2.2")), peer, false),
Ok(target("192.0.2.2", 2))
);
}
/// A route that asks for its packets to be dropped goes nowhere,
/// whatever its next hop; one for a network of ours does not.
#[test]
fn a_blackhole_route_resolves_to_nowhere() {
let topology = interfaces();
let far = NlriNextHop::Ipv4("10.9.9.9".parse().unwrap());
let peer = Some(neighbor("192.0.2.2"));
assert_eq!(
resolve(
&topology,
&prefix("203.0.113.9/32"),
Some(&far),
peer,
false
),
Err(Unresolved::NoRoute)
);
assert_eq!(
resolve(&topology, &prefix("203.0.113.9/32"), Some(&far), peer, true),
Ok(Target::Discard)
);
assert_eq!(
resolve(&topology, &prefix("203.0.113.9/32"), None, peer, true),
Ok(Target::Discard)
);
assert_eq!(
resolve(&topology, &prefix("192.0.2.9/32"), Some(&far), peer, true),
Err(Unresolved::OwnNetwork)
);
let mut plan = Plan::default();
assert_eq!(
plan.best(prefix("203.0.113.9/32"), Ok(Target::Discard)),
Some(Op::Install {
prefix: prefix("203.0.113.9/32"),
target: Target::Discard
})
);
assert_eq!((plan.installed(), plan.blackholes()), (1, 1));
}
#[test]
fn the_kernel_is_asked_only_for_what_changes() {
let mut plan = Plan::default();
@@ -684,13 +1059,13 @@ mod tests {
assert_eq!(plan.installed(), 1);
// A best route that cannot be installed takes the old one out.
assert_eq!(
plan.best(p, Err(Unresolved::NotConnected)),
plan.best(p, Err(Unresolved::NoRoute)),
Some(Op::Remove { prefix: p })
);
assert_eq!((plan.installed(), plan.unresolved()), (0, 1));
assert_eq!(
plan.unresolved_by_reason().collect::<Vec<_>>(),
[(Unresolved::NotConnected, 1)]
[(Unresolved::NoRoute, 1)]
);
assert_eq!(plan.withdrawn(p), None);
assert_eq!(plan.unresolved_by_reason().count(), 0);
@@ -834,6 +1209,60 @@ mod tests {
]
}
/// What the counts would be if the entries were scanned.
fn scanned(plan: &Plan) -> (usize, usize) {
plan.installed
.values()
.map(Plan::weigh)
.fold((0, 0), |(b, k), (eb, ek)| (b + eb, k + ek))
}
#[test]
fn the_counts_keep_step_with_the_routes() {
let mut plan = Plan::default();
let p = |n: u8| prefix(&format!("203.0.113.{n}/32"));
let via = target("192.0.2.2", 2);
let check = |plan: &Plan| assert_eq!((plan.blackholes(), plan.kept()), scanned(plan));
// Adopted from an earlier run: two kept, one a blackhole.
plan.adopt(p(1), Some(Target::Discard), true);
plan.adopt(p(2), Some(via), true);
plan.adopt(p(3), Some(via), false);
check(&plan);
// Heard of again, as a blackhole and as a route by way of a
// gateway; one changes kind.
plan.best(p(1), Ok(Target::Discard));
plan.best(p(2), Ok(Target::Discard));
plan.best(p(4), Ok(Target::Discard));
check(&plan);
plan.best(p(4), Ok(via));
check(&plan);
// The whole table again: everything unconfirmed, then confirmed.
plan.reset();
check(&plan);
plan.best(p(1), Ok(Target::Discard));
plan.synced();
check(&plan);
plan.reset();
plan.release();
check(&plan);
// Gone for want of a next hop, withdrawn, refused, adopted over.
plan.best(p(1), Err(Unresolved::Missing));
plan.best(p(5), Ok(Target::Discard));
plan.withdrawn(p(5));
plan.best(p(6), Ok(Target::Discard));
plan.refused_by_kernel(&Op::Install {
prefix: p(6),
target: Target::Discard,
});
plan.adopt(p(7), Some(Target::Discard), true);
plan.adopt(p(7), Some(via), true);
check(&plan);
let had = plan.installed();
assert_eq!(plan.clear().len(), had);
assert_eq!((plan.blackholes(), plan.kept()), (0, 0));
}
proptest! {
/// Whatever happens to the best routes, and whatever an earlier
/// run left behind, a kernel that does as it is asked ends up with
@@ -841,7 +1270,7 @@ mod tests {
#[test]
fn the_kernel_ends_up_with_the_best_routes_that_resolve(
left_behind in proptest::collection::vec((0u8..16, 0u8..3), 0..6),
steps in proptest::collection::vec(any_step(), 0..60),
steps in proptest::collection::vec(any_step(), 0..60)
) {
let p = |n: u8| IpPrefix::new(IpVersion::Ipv4, [10, n, 0, 0], 16).unwrap();
let t = |n: u8| target(&format!("192.0.2.{}", n + 2), 2);
@@ -858,7 +1287,7 @@ mod tests {
Some(Op::Remove { prefix }) => { kernel.remove(&prefix); }
None => {}
};
let resolved = |gateway: Option<u8>| gateway.map(t).ok_or(Unresolved::NotConnected);
let resolved = |gateway: Option<u8>| gateway.map(t).ok_or(Unresolved::NoRoute);
// A new reader is first given the whole table.
let steps = std::iter::once(Step::Resync).chain(steps);
for step in steps {
crates/fib/src/routes.rs
+266
View File
@@ -0,0 +1,266 @@
//! The kernel's own routes, as far as next hops are concerned: the way to
//! a next hop that is not a neighbor on any of our interfaces.
//!
//! Only the routes of one table count, the resolution table, which is
//! normally `main`: the connected and static routes, and an IGP's if there
//! is one. Routes put there by BGP, ours or another daemon's, never count:
//! a next hop that is reached only by way of a BGP route is reached by way
//! of something that may itself depend on a next hop, and a table that
//! leans on itself like that is one that falls over in a way nobody can
//! predict. The default route counts only if asked for: on a router, the
//! default points at a transit provider, and a next hop that is "reached"
//! only by sending the packets to the Internet at large is not reached.
use std::net::IpAddr;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::netlink::{Message, PROTOCOL_BGP, Route};
use crate::plan::HashMap;
/// The routes of the resolution table that may say the way to a next hop.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct KernelRoutes {
table: u32,
/// By prefix, the routes there are for it: usually one, but the kernel
/// keeps one for each metric.
routes: HashMap<IpPrefix, Vec<Route>>,
/// How many prefixes there are of each length, by IP version: the
/// lengths a lookup has to try.
lengths: [[u32; 129]; 2],
}
/// `address` as the prefix that is only it.
pub(crate) fn host(address: IpAddr) -> IpPrefix {
match address {
IpAddr::V4(v4) => IpPrefix::new(IpVersion::Ipv4, v4.octets(), 32),
IpAddr::V6(v6) => IpPrefix::new(IpVersion::Ipv6, v6.octets(), 128),
}
.expect("an address is a prefix of its own length")
}
fn version_index(version: IpVersion) -> usize {
match version {
IpVersion::Ipv4 => 0,
IpVersion::Ipv6 => 1,
}
}
impl KernelRoutes {
pub fn new(table: u32) -> Self {
Self {
table,
routes: HashMap::default(),
lengths: [[0; 129]; 2],
}
}
/// The table the routes are of.
pub fn table(&self) -> u32 {
self.table
}
/// Whether `route` is one that may say the way to a next hop.
fn counts(&self, route: &Route) -> bool {
route.table == self.table && route.protocol != PROTOCOL_BGP
}
/// Takes in what a message from the kernel says about routes. Returns
/// whether anything is different for it.
pub fn apply(&mut self, message: &Message) -> bool {
let Message::Route { present, route } = message else {
return false;
};
if !self.counts(route) {
return false;
}
let version = version_index(route.prefix.address_family());
let length = usize::from(route.prefix.length());
if *present {
let routes = self.routes.entry(route.prefix).or_default();
// A route for a prefix with a metric there is one for already
// takes that one's place: the kernel keeps one per metric.
match routes.iter_mut().find(|known| known.metric == route.metric) {
Some(known) if known == route => false,
Some(known) => {
*known = *route;
true
}
None => {
if routes.is_empty() {
self.lengths[version][length] += 1;
}
// In order of metric, so that two tables with the
// same routes compare equal however they were told.
let at = routes.partition_point(|known| known.metric < route.metric);
routes.insert(at, *route);
true
}
}
} else {
let Some(routes) = self.routes.get_mut(&route.prefix) else {
return false;
};
let before = routes.len();
routes.retain(|known| known.metric != route.metric);
let (after, none_left) = (routes.len(), routes.is_empty());
if none_left {
self.routes.remove(&route.prefix);
self.lengths[version][length] -= 1;
}
before != after
}
}
/// How many prefixes have a route.
pub fn len(&self) -> usize {
self.routes.len()
}
pub fn is_empty(&self) -> bool {
self.routes.is_empty()
}
/// The route the kernel would send packets for `address` by, from
/// this table: the one for the longest prefix that covers it, and of
/// several for that prefix the one with the lowest metric. The
/// default route too, if there is one and nothing longer covers the
/// address.
pub fn lookup(&self, address: IpAddr) -> Option<&Route> {
let whole = host(address);
let lengths = &self.lengths[version_index(whole.address_family())];
(0..=whole.length())
.rev()
.filter(|length| lengths[usize::from(*length)] > 0)
.find_map(|length| self.routes.get(&whole.shortened(length)))
.and_then(|routes| routes.iter().min_by_key(|route| route.metric))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::netlink::{KIND_UNICAST, TABLE_MAIN};
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(prefix_text: &str, gateway: Option<&str>, interface: u32, metric: u32) -> Route {
Route {
prefix: prefix(prefix_text),
table: TABLE_MAIN,
protocol: 4, // static
metric,
gateway: gateway.map(|g| g.parse().unwrap()),
interface: Some(interface),
nexthop: None,
kind: KIND_UNICAST,
}
}
fn present(route: Route) -> Message {
Message::Route {
present: true,
route,
}
}
fn gone(route: Route) -> Message {
Message::Route {
present: false,
route,
}
}
fn lookup(routes: &KernelRoutes, address: &str) -> Option<(String, Option<String>, u32)> {
routes.lookup(address.parse().unwrap()).map(|route| {
(
route.prefix.to_string(),
route.gateway.map(|g| g.to_string()),
route.metric,
)
})
}
#[test]
fn the_longest_prefix_with_the_lowest_metric_is_the_way() {
let mut routes = KernelRoutes::new(TABLE_MAIN);
for r in [
route("0.0.0.0/0", Some("192.0.2.1"), 2, 100),
route("10.0.0.0/8", Some("192.0.2.9"), 2, 0),
route("10.1.0.0/16", None, 7, 0),
route("10.1.0.0/16", Some("198.51.100.1"), 3, 50),
route("2001:db8::/32", Some("2001:db8::1"), 2, 1024),
] {
assert!(routes.apply(&present(r)));
}
assert_eq!(routes.len(), 4);
assert_eq!(
lookup(&routes, "10.2.3.4"),
Some(("10.0.0.0/8".into(), Some("192.0.2.9".into()), 0))
);
// Two routes for the prefix: the lower metric wins.
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.1.0.0/16".into(), None, 0))
);
assert_eq!(
lookup(&routes, "203.0.113.1"),
Some(("0.0.0.0/0".into(), Some("192.0.2.1".into()), 100))
);
assert_eq!(
lookup(&routes, "2001:db8:1::1"),
Some(("2001:db8::/32".into(), Some("2001:db8::1".into()), 1024))
);
assert_eq!(lookup(&routes, "2001:db9::1"), None);
// Told again what is known: nothing changes.
assert!(!routes.apply(&present(route("10.0.0.0/8", Some("192.0.2.9"), 2, 0))));
// The same prefix and metric with another gateway: replaced.
assert!(routes.apply(&present(route("10.0.0.0/8", Some("192.0.2.8"), 2, 0))));
assert_eq!(
lookup(&routes, "10.2.3.4"),
Some(("10.0.0.0/8".into(), Some("192.0.2.8".into()), 0))
);
// The better of the two for 10.1/16 goes: the other is the way.
assert!(routes.apply(&gone(route("10.1.0.0/16", None, 7, 0))));
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.1.0.0/16".into(), Some("198.51.100.1".into()), 50))
);
assert!(routes.apply(&gone(route("10.1.0.0/16", Some("198.51.100.1"), 3, 50))));
assert_eq!(
lookup(&routes, "10.1.2.3"),
Some(("10.0.0.0/8".into(), Some("192.0.2.8".into()), 0))
);
// Gone already: nothing changes.
assert!(!routes.apply(&gone(route("10.1.0.0/16", None, 7, 0))));
assert_eq!(routes.len(), 3);
}
#[test]
fn routes_of_other_tables_and_of_bgp_do_not_count() {
let mut routes = KernelRoutes::new(TABLE_MAIN);
let other_table = Route {
table: 200,
..route("10.0.0.0/8", Some("192.0.2.9"), 2, 0)
};
let bgp = Route {
protocol: PROTOCOL_BGP,
..route("10.0.0.0/8", Some("192.0.2.9"), 2, 20)
};
assert!(!routes.apply(&present(other_table)));
assert!(!routes.apply(&present(bgp)));
assert!(routes.is_empty());
assert_eq!(lookup(&routes, "10.2.3.4"), None);
// What is not about routes changes nothing either.
assert!(!routes.apply(&Message::Done));
let mut table_200 = KernelRoutes::new(200);
assert!(table_200.apply(&present(other_table)));
assert_eq!(table_200.table(), 200);
assert_eq!(table_200.len(), 1);
}
}
crates/fib/src/sockdiag.rs
+426
View File
@@ -0,0 +1,426 @@
//! What the kernel knows about one of our TCP connections: the round-trip
//! time, the windows, the queues and what has held the connection up,
//! asked over the socket diagnostics netlink family (`NETLINK_SOCK_DIAG`,
//! as `ss -ti` asks). For the status page, when someone wants to know
//! why a session is slow.
use std::io;
use std::net::{IpAddr, SocketAddr};
/// A TCP connection as the kernel sees it. Times are in microseconds,
/// sizes in octets, rates in octets per second, unless said otherwise.
/// Fields the running kernel does not report are zero.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct TcpConnection {
/// The TCP state, by its number (1 is ESTABLISHED); see
/// [`TcpConnection::state_name`].
pub state: u8,
/// The congestion control algorithm, if the kernel named it.
pub congestion: Option<String>,
/// Octets received and not yet read by us.
pub recv_queue: u32,
/// Octets written by us and not yet acknowledged by the peer.
pub send_queue: u32,
pub rtt: u32,
pub rtt_variance: u32,
pub min_rtt: u32,
pub rto: u32,
pub mss: u32,
pub pmtu: u32,
/// The congestion window, in segments.
pub cwnd: u32,
/// The slow start threshold, in segments; `None` if there is none.
pub ssthresh: Option<u32>,
/// What the peer lets us send: its receive window.
pub send_window: u32,
/// What we let the peer send.
pub receive_window: u32,
pub receive_space: u32,
pub receive_ssthresh: u32,
pub send_window_scale: u8,
pub receive_window_scale: u8,
/// Segments sent and not acknowledged.
pub unacked: u32,
/// Segments retransmitted and still out.
pub retrans: u32,
pub lost: u32,
pub total_retrans: u32,
pub bytes_retrans: u64,
/// Octets in the socket's buffer that are not sent yet.
pub not_sent: u32,
pub pacing_rate: u64,
pub delivery_rate: u64,
pub bytes_sent: u64,
pub bytes_acked: u64,
pub bytes_received: u64,
pub segments_out: u32,
pub segments_in: u32,
/// How long the connection had something to send.
pub busy: u64,
/// Of that, how long it was held up by the peer's receive window.
pub limited_by_receive_window: u64,
/// And how long by our own send buffer being full.
pub limited_by_send_buffer: u64,
pub delivered: u32,
pub reordering: u32,
/// Milliseconds since data was last sent, received, and since the
/// last acknowledgement came.
pub last_data_sent_ms: u32,
pub last_data_received_ms: u32,
pub last_ack_received_ms: u32,
}
impl TcpConnection {
pub fn state_name(&self) -> &'static str {
match self.state {
1 => "established",
2 => "syn-sent",
3 => "syn-received",
4 => "fin-wait-1",
5 => "fin-wait-2",
6 => "time-wait",
7 => "close",
8 => "close-wait",
9 => "last-ack",
10 => "listen",
11 => "closing",
_ => "unknown",
}
}
}
const SOCK_DIAG_BY_FAMILY: u16 = 20;
const NLMSG_ERROR: u16 = 2;
const NLM_F_REQUEST: u16 = 1;
const IPPROTO_TCP: u8 = 6;
const AF_INET: u8 = 2;
const AF_INET6: u8 = 10;
const INET_DIAG_INFO: u16 = 2;
const INET_DIAG_CONG: u16 = 4;
/// `idiag_ext` bits asking for those two.
const EXTENSIONS: u8 = (1 << (INET_DIAG_INFO - 1)) | (1 << (INET_DIAG_CONG - 1));
const NO_COOKIE: u32 = u32::MAX;
const HEADER_LEN: usize = 16;
const REQUEST_LEN: usize = HEADER_LEN + 56;
const DIAG_MSG_LEN: usize = 72;
/// The request for the connection from `local` to `peer`, with `sequence`.
pub fn request(local: SocketAddr, peer: SocketAddr, sequence: u32) -> Vec<u8> {
let family = match local {
SocketAddr::V4(_) => AF_INET,
SocketAddr::V6(_) => AF_INET6,
};
let mut buf = Vec::with_capacity(REQUEST_LEN);
buf.extend_from_slice(&(REQUEST_LEN as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&NLM_F_REQUEST.to_ne_bytes());
buf.extend_from_slice(&sequence.to_ne_bytes());
buf.extend_from_slice(&0u32.to_ne_bytes());
// inet_diag_req_v2
buf.extend_from_slice(&[family, IPPROTO_TCP, EXTENSIONS, 0]);
buf.extend_from_slice(&u32::MAX.to_ne_bytes()); // any state
// inet_diag_sockid: ports in network order, addresses as 16 octets.
buf.extend_from_slice(&local.port().to_be_bytes());
buf.extend_from_slice(&peer.port().to_be_bytes());
buf.extend_from_slice(&address_octets(local.ip()));
buf.extend_from_slice(&address_octets(peer.ip()));
// The interface, which a socket on a link-local address is bound to
// and is found by.
let interface = match local {
SocketAddr::V6(v6) => v6.scope_id(),
SocketAddr::V4(_) => 0,
};
buf.extend_from_slice(&interface.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
buf.extend_from_slice(&NO_COOKIE.to_ne_bytes());
debug_assert_eq!(buf.len(), REQUEST_LEN);
buf
}
fn address_octets(address: IpAddr) -> [u8; 16] {
let mut octets = [0u8; 16];
match address {
IpAddr::V4(v4) => octets[..4].copy_from_slice(&v4.octets()),
IpAddr::V6(v6) => octets = v6.octets(),
}
octets
}
/// What the kernel answered.
#[derive(Debug, PartialEq, Eq)]
pub enum Answer {
Connection(Box<TcpConnection>),
/// No such connection.
None,
/// The kernel refused, with this error number.
Refused(i32),
}
/// Reads the kernel's answer to a [`request`].
pub fn parse(buf: &[u8]) -> io::Result<Answer> {
let header = buf
.get(..HEADER_LEN)
.ok_or_else(|| io::Error::other("short netlink answer"))?;
let length = ne_u32(&header[0..4]) as usize;
let message_type = ne_u16(&header[4..6]);
let message = buf
.get(HEADER_LEN..length.min(buf.len()))
.ok_or_else(|| io::Error::other("short netlink answer"))?;
if message_type == NLMSG_ERROR {
let errno = -(message
.first_chunk::<4>()
.map_or(0, |e| i32::from_ne_bytes(*e)));
// ENOENT: no such socket.
return Ok(if errno == 2 {
Answer::None
} else {
Answer::Refused(errno)
});
}
if message_type != SOCK_DIAG_BY_FAMILY || message.len() < DIAG_MSG_LEN {
return Err(io::Error::other("not a socket diagnostic"));
}
let mut connection = TcpConnection {
state: message[1],
recv_queue: ne_u32(&message[56..60]),
send_queue: ne_u32(&message[60..64]),
..TcpConnection::default()
};
let mut attributes = &message[DIAG_MSG_LEN..];
while attributes.len() >= 4 {
let length = usize::from(ne_u16(&attributes[0..2]));
let kind = ne_u16(&attributes[2..4]);
if length < 4 || length > attributes.len() {
break;
}
let value = &attributes[4..length];
match kind {
INET_DIAG_INFO => fill_from_tcp_info(&mut connection, value),
INET_DIAG_CONG => {
let name = value.split(|b| *b == 0).next().unwrap_or(&[]);
connection.congestion = Some(String::from_utf8_lossy(name).into_owned());
}
_ => {}
}
attributes = &attributes[(length + 3) & !3..];
}
Ok(Answer::Connection(Box::new(connection)))
}
/// Fills in what `tcp_info` (linux/tcp.h) has, as far as the kernel sent
/// it: the structure has grown over the years, and a field that is not
/// there is left at zero.
fn fill_from_tcp_info(c: &mut TcpConnection, info: &[u8]) {
let u32_at = |at: usize| info.get(at..at + 4).map_or(0, ne_u32);
let u64_at = |at: usize| info.get(at..at + 8).map_or(0, ne_u64);
c.send_window_scale = info.get(6).map_or(0, |b| b & 0x0f);
c.receive_window_scale = info.get(6).map_or(0, |b| b >> 4);
c.rto = u32_at(8);
c.mss = u32_at(16);
c.unacked = u32_at(24);
c.lost = u32_at(32);
c.retrans = u32_at(36);
c.last_data_sent_ms = u32_at(44);
c.last_data_received_ms = u32_at(52);
c.last_ack_received_ms = u32_at(56);
c.pmtu = u32_at(60);
c.receive_ssthresh = u32_at(64);
c.rtt = u32_at(68);
c.rtt_variance = u32_at(72);
c.ssthresh = Some(u32_at(76)).filter(|t| *t < 0x7fff_ffff);
c.cwnd = u32_at(80);
c.reordering = u32_at(88);
c.receive_space = u32_at(96);
c.total_retrans = u32_at(100);
c.pacing_rate = u64_at(104);
c.bytes_acked = u64_at(120);
c.bytes_received = u64_at(128);
c.segments_out = u32_at(136);
c.segments_in = u32_at(140);
c.not_sent = u32_at(144);
c.min_rtt = u32_at(148);
c.delivery_rate = u64_at(160);
c.busy = u64_at(168);
c.limited_by_receive_window = u64_at(176);
c.limited_by_send_buffer = u64_at(184);
c.delivered = u32_at(192);
c.bytes_sent = u64_at(200);
c.bytes_retrans = u64_at(208);
c.send_window = u32_at(228);
c.receive_window = u32_at(232);
}
fn ne_u16(bytes: &[u8]) -> u16 {
u16::from_ne_bytes([bytes[0], bytes[1]])
}
fn ne_u32(bytes: &[u8]) -> u32 {
u32::from_ne_bytes([bytes[0], bytes[1], bytes[2], bytes[3]])
}
fn ne_u64(bytes: &[u8]) -> u64 {
let mut array = [0u8; 8];
array.copy_from_slice(bytes);
u64::from_ne_bytes(array)
}
/// Asks the kernel about the TCP connection from `local` to `peer`.
/// `Ok(None)` if there is no such connection, or on an operating system
/// that cannot be asked.
#[cfg(target_os = "linux")]
pub async fn tcp_connection(
local: SocketAddr,
peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
use netlink_sys::protocols::NETLINK_SOCK_DIAG;
use netlink_sys::{AsyncSocket, AsyncSocketExt, TokioSocket};
let mut socket = TokioSocket::new(NETLINK_SOCK_DIAG)?;
socket
.socket_mut()
.bind(&netlink_sys::SocketAddr::new(0, 0))?;
let sequence = 1;
socket.send(&request(local, peer, sequence)).await?;
let mut buf = bytes::BytesMut::with_capacity(4096);
tokio::time::timeout(std::time::Duration::from_secs(2), socket.recv(&mut buf))
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "the kernel did not answer"))??;
match parse(&buf)? {
Answer::Connection(connection) => Ok(Some(*connection)),
Answer::None => Ok(None),
Answer::Refused(errno) => Err(io::Error::from_raw_os_error(errno)),
}
}
#[cfg(not(target_os = "linux"))]
pub async fn tcp_connection(
_local: SocketAddr,
_peer: SocketAddr,
) -> io::Result<Option<TcpConnection>> {
Ok(None)
}
#[cfg(test)]
mod tests {
use super::*;
/// A reply as the kernel writes it, for an established connection
/// with a `tcp_info` of the full modern length.
fn reply() -> Vec<u8> {
let mut info = vec![0u8; 248];
info[0] = 1; // established
info[6] = 0x7a; // snd_wscale 10, rcv_wscale 7
let put32 = |info: &mut Vec<u8>, at: usize, v: u32| {
info[at..at + 4].copy_from_slice(&v.to_ne_bytes())
};
let put64 = |info: &mut Vec<u8>, at: usize, v: u64| {
info[at..at + 8].copy_from_slice(&v.to_ne_bytes())
};
put32(&mut info, 16, 1448); // mss
put32(&mut info, 24, 12); // unacked
put32(&mut info, 68, 2300); // rtt
put32(&mut info, 76, 0x7fff_ffff); // no ssthresh
put32(&mut info, 80, 42); // cwnd
put32(&mut info, 144, 65536); // notsent
put64(&mut info, 176, 900_000); // rwnd limited
put64(&mut info, 168, 1_000_000); // busy
put32(&mut info, 228, 131072); // snd_wnd
put32(&mut info, 232, 262144); // rcv_wnd
let cong = b"cubic\0";
let mut message = vec![0u8; DIAG_MSG_LEN];
message[1] = 1;
message[60..64].copy_from_slice(&80_000u32.to_ne_bytes()); // wqueue
// attributes: INFO, then CONG
let mut attrs = Vec::new();
attrs.extend_from_slice(&((4 + info.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_INFO.to_ne_bytes());
attrs.extend_from_slice(&info);
attrs.extend_from_slice(&((4 + cong.len()) as u16).to_ne_bytes());
attrs.extend_from_slice(&INET_DIAG_CONG.to_ne_bytes());
attrs.extend_from_slice(cong);
while attrs.len() % 4 != 0 {
attrs.push(0);
}
let length = HEADER_LEN + message.len() + attrs.len();
let mut buf = Vec::new();
buf.extend_from_slice(&(length as u32).to_ne_bytes());
buf.extend_from_slice(&SOCK_DIAG_BY_FAMILY.to_ne_bytes());
buf.extend_from_slice(&[0, 0, 1, 0, 0, 0, 0, 0, 0, 0]);
buf.extend_from_slice(&message);
buf.extend_from_slice(&attrs);
buf
}
#[test]
fn a_reply_is_read_field_by_field() {
let Answer::Connection(c) = parse(&reply()).unwrap() else {
panic!("a connection");
};
assert_eq!(c.state_name(), "established");
assert_eq!(c.congestion.as_deref(), Some("cubic"));
assert_eq!(c.send_queue, 80_000);
assert_eq!((c.send_window_scale, c.receive_window_scale), (10, 7));
assert_eq!((c.mss, c.unacked, c.rtt, c.cwnd), (1448, 12, 2300, 42));
assert_eq!(c.ssthresh, None);
assert_eq!(c.not_sent, 65536);
assert_eq!((c.busy, c.limited_by_receive_window), (1_000_000, 900_000));
assert_eq!((c.send_window, c.receive_window), (131072, 262144));
}
#[test]
fn no_such_connection_and_a_refusal_are_told_apart() {
let error = |errno: i32| {
let mut buf = vec![0u8; 20];
buf[0..4].copy_from_slice(&20u32.to_ne_bytes());
buf[4..6].copy_from_slice(&NLMSG_ERROR.to_ne_bytes());
buf[16..20].copy_from_slice(&(-errno).to_ne_bytes());
buf
};
assert_eq!(parse(&error(2)).unwrap(), Answer::None);
assert_eq!(parse(&error(13)).unwrap(), Answer::Refused(13));
}
#[test]
fn a_request_names_the_connection() {
let local: SocketAddr = "[2001:db8::1]:179".parse().unwrap();
let peer: SocketAddr = "[2001:db8::2]:40000".parse().unwrap();
let request = request(local, peer, 7);
assert_eq!(request.len(), REQUEST_LEN);
assert_eq!(request[16], AF_INET6);
assert_eq!(&request[24..26], &179u16.to_be_bytes());
assert_eq!(&request[26..28], &40000u16.to_be_bytes());
assert_eq!(
&request[28..44],
&"2001:db8::1"
.parse::<std::net::Ipv6Addr>()
.unwrap()
.octets()
);
}
#[cfg(target_os = "linux")]
#[tokio::test]
async fn the_kernel_tells_of_a_connection_of_ours() {
use tokio::io::AsyncWriteExt;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let mut client = tokio::net::TcpStream::connect(listener.local_addr().unwrap())
.await
.unwrap();
let (_server, _) = listener.accept().await.unwrap();
client.write_all(&[0u8; 1000]).await.unwrap();
let connection = tcp_connection(client.local_addr().unwrap(), client.peer_addr().unwrap())
.await
.unwrap()
.expect("the kernel knows it");
assert_eq!(connection.state_name(), "established");
assert!(connection.mss > 0 && connection.cwnd > 0);
assert!(connection.bytes_sent >= 1000 || connection.bytes_acked >= 1000);
let none = tcp_connection(
"127.0.0.1:1".parse().unwrap(),
"127.0.0.1:2".parse().unwrap(),
)
.await
.unwrap();
assert!(none.is_none());
}
}
crates/fib/src/topology.rs
+123
View File
@@ -0,0 +1,123 @@
//! What the kernel knows that says whether, and how, a next hop can be
//! reached: the machine's interfaces and addresses, the routes of the
//! resolution table, and the kernel's nexthop objects that those routes
//! may name. One value, replaced as a whole when any of it changes, so
//! that everyone judging next hops judges them by one moment's view.
use crate::interfaces::Interfaces;
use crate::netlink::{Message, Nexthop};
use crate::plan::HashMap;
use crate::routes::KernelRoutes;
/// The kernel's view of the machine, as far as next hops are concerned.
#[derive(Debug, Clone)]
pub struct Topology {
pub interfaces: Interfaces,
/// The routes a next hop that is not a neighbor is reached by.
pub routes: KernelRoutes,
/// Every nexthop object the kernel has, by number, whoever made it:
/// a route of the resolution table may name one instead of carrying
/// its own gateway.
pub nexthops: HashMap<u32, Nexthop>,
/// Whether the default route counts as the way to a next hop.
pub via_default: bool,
}
impl Topology {
/// Nothing known yet, with next hops to be resolved by the routes of
/// `resolution_table`.
pub fn new(resolution_table: u32, via_default: bool) -> Self {
Self {
interfaces: Interfaces::default(),
routes: KernelRoutes::new(resolution_table),
nexthops: HashMap::default(),
via_default,
}
}
/// Takes in what a message from the kernel says. Returns whether
/// anything is different for it: then what any next hop resolves to
/// may be different too.
pub fn apply(&mut self, message: &Message) -> bool {
match message {
Message::Nexthop {
present: true,
nexthop,
} => self.nexthops.insert(nexthop.id, *nexthop) != Some(*nexthop),
Message::Nexthop {
present: false,
nexthop,
} => self.nexthops.remove(&nexthop.id).is_some(),
Message::Route { .. } => self.routes.apply(message),
_ => self.interfaces.apply(message),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::netlink::{KIND_UNICAST, Link, PROTOCOL_BGP, Route, TABLE_MAIN};
use bgp_primitives::ip_prefix::IpPrefix;
#[test]
fn each_kind_of_news_goes_where_it_belongs() {
let mut topology = Topology::new(TABLE_MAIN, false);
let nexthop = Nexthop {
id: 7,
protocol: 4,
gateway: Some("192.0.2.1".parse().unwrap()),
interface: Some(2),
};
let present = |nexthop| Message::Nexthop {
present: true,
nexthop,
};
assert!(topology.apply(&present(nexthop)));
assert!(!topology.apply(&present(nexthop)));
assert!(topology.apply(&present(Nexthop {
interface: Some(3),
..nexthop
})));
assert_eq!(topology.nexthops[&7].interface, Some(3));
assert!(topology.apply(&Message::Nexthop {
present: false,
nexthop,
}));
assert!(topology.nexthops.is_empty());
let route = Route {
prefix: IpPrefix::try_from("10.0.0.0/8").unwrap(),
table: TABLE_MAIN,
protocol: 4,
metric: 0,
gateway: Some("192.0.2.1".parse().unwrap()),
interface: Some(2),
nexthop: None,
kind: KIND_UNICAST,
};
assert!(topology.apply(&Message::Route {
present: true,
route,
}));
assert_eq!(topology.routes.len(), 1);
// BGP's routes are not the way to anything.
assert!(!topology.apply(&Message::Route {
present: true,
route: Route {
protocol: PROTOCOL_BGP,
..route
},
}));
assert!(topology.apply(&Message::Link {
present: true,
link: Link {
index: 2,
name: "eth0".into(),
up: true,
},
}));
assert!(topology.interfaces.link(2).is_some());
assert!(!topology.apply(&Message::Done));
}
}
crates/fib/tests/kernel.rs
+149 -5
View File
@@ -15,8 +15,8 @@ use std::time::{Duration, Instant};
use bgp_fib::{Fib, FibConfig, FibError};
use bgp_packet::message::{
AsPathAttribute, MultiExitDiscPathAttribute, NextHopPathAttribute, NlriNextHop,
OriginPathAttribute, PathAttribute,
AsPathAttribute, CommunitiesPathAttribute, MultiExitDiscPathAttribute, NextHopPathAttribute,
NlriNextHop, OriginPathAttribute, PathAttribute,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{AttrSet, PeerIndex, PeerInfo, RibTable};
@@ -109,6 +109,7 @@ impl Machine {
metric: 20,
nexthop_objects: self.nexthop_objects,
stale_time: self.stale_time,
..FibConfig::default()
},
)
.expect("the kernel can be talked to")
@@ -122,6 +123,7 @@ impl Machine {
address: self.neighbor(host).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap()
}
@@ -293,7 +295,7 @@ async fn the_kernel_has_the_best_route_of_each_prefix() {
machine.shown("2001:db8:100::/48", v6_neighbor),
]);
status_becomes(&fib, "one unresolved", |status| {
status.installed == 2 && status.unresolved == [("next hop not directly connected", 1)]
status.installed == 2 && status.unresolved == [("no route to the next hop", 1)]
});
// A session ends: its routes go.
@@ -366,7 +368,7 @@ async fn routes_follow_the_machines_interfaces() {
.announce(peer, &via, &[prefix("203.0.113.0/24")]);
let (fib, task) = machine.start();
status_becomes(&fib, "unresolved", |status| {
status.synced && status.unresolved == [("next hop not directly connected", 1)]
status.synced && status.unresolved == [("no route to the next hop", 1)]
});
assert!(machine.kernel_routes().is_empty());
@@ -415,6 +417,147 @@ async fn routes_follow_the_machines_interfaces() {
machine.kernel_becomes(&[]);
}
/// A next hop that is no neighbor goes the way the kernel's own routes
/// say: a static route's gateway, a tunnel's interface, a nexthop object.
/// When those routes change, so do ours; a route that discards packets is
/// the way to nowhere.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn a_far_next_hop_goes_the_way_the_kernels_routes_say() {
let machine = Machine::new(10);
let interface = machine.interface();
let peer = machine.peer(2);
// Next hops that are not in any network of ours.
let far = machine.via("172.31.10.9".parse().unwrap(), 0);
let farther = machine.via("172.31.11.9".parse().unwrap(), 0);
machine
.rib
.announce(peer, &far, &[prefix("203.0.113.0/24")]);
machine
.rib
.announce(peer, &farther, &[prefix("198.18.0.0/15")]);
let (fib, task) = machine.start();
status_becomes(&fib, "no way to either", |status| {
status.synced && status.unresolved == [("no route to the next hop", 2)]
});
assert!(machine.kernel_routes().is_empty());
// A static route with a gateway says the way to one: the route goes
// in with that gateway.
ip(&format!(
"route add 172.31.10.0/24 via 10.10.0.9 dev {interface}"
));
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", "10.10.0.9")]);
// A route onto the interface, as a tunnel's is, says the way to the
// other: the next hop itself is the neighbor there.
ip(&format!("route add 172.31.11.0/24 dev {interface}"));
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.9"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
status_becomes(&fib, "both resolved", |status| {
status.unresolved.is_empty() && status.installed == 2
});
// The way changes: so does the route. Then it goes, and so does the
// route.
ip(&format!(
"route replace 172.31.10.0/24 via 10.10.0.8 dev {interface}"
));
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.8"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
ip("route del 172.31.10.0/24");
machine.kernel_becomes(&[machine.shown("198.18.0.0/15", "172.31.11.9")]);
status_becomes(&fib, "one unresolved again", |status| {
status.unresolved == [("no route to the next hop", 1)]
});
// A route that names a nexthop object goes where the object says.
ip(&format!(
"nexthop add id 1010 via 10.10.0.7 dev {interface}"
));
ip("route add 172.31.10.0/24 nhid 1010");
machine.kernel_becomes(&[
machine.shown("203.0.113.0/24", "10.10.0.7"),
machine.shown("198.18.0.0/15", "172.31.11.9"),
]);
// The tunnel's route becomes a blackhole: the way to nowhere.
ip("route replace blackhole 172.31.11.0/24");
machine.kernel_becomes(&[machine.shown("203.0.113.0/24", "10.10.0.7")]);
status_becomes(&fib, "discarded", |status| {
status.unresolved == [("the route to the next hop discards packets", 1)]
});
ip("route del 172.31.10.0/24");
ip("route del blackhole 172.31.11.0/24");
ip("nexthop del id 1010");
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
/// A route that asks for its packets to be dropped goes into the kernel
/// as a blackhole route, whatever its next hop, and comes out again with
/// the request.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
async fn a_blackhole_request_becomes_a_blackhole_route() {
let machine = Machine::new(13);
let peer = machine.peer(2);
// Asked for from a next hop nothing is a neighbor of, and with one
// ordinary route beside it to show the two apart.
let mut attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65002]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(172, 31, 11, 9))),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65002, 1), (0xFFFF, 0x029A)])),
];
let blackhole = machine.rib.attrs().intern(&attributes, None).unwrap();
attributes.pop();
attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(vec![
(65002, 1),
])));
let plain = machine.rib.attrs().intern(&attributes, None).unwrap();
machine.rib.announce(
peer,
&blackhole,
&[prefix("203.0.113.9/32"), prefix("2001:db8:100::9/128")],
);
machine
.rib
.announce(peer, &plain, &[prefix("198.18.0.0/15")]);
let (fib, task) = machine.start();
// The kernel puts an IPv6 blackhole route on lo, and says so.
machine.kernel_becomes(&[
"blackhole 203.0.113.9 proto bgp metric 20".into(),
"blackhole 2001:db8:100::9 dev lo proto bgp metric 20".into(),
]);
status_becomes(&fib, "two blackholes, one unresolved", |status| {
status.synced
&& status.installed == 2
&& status.blackholes == 2
&& status.unresolved == [("no route to the next hop", 1)]
});
// The request is withdrawn: the blackhole goes.
machine.rib.withdraw(peer, &[prefix("203.0.113.9/32")]);
machine.kernel_becomes(&["blackhole 2001:db8:100::9 dev lo proto bgp metric 20".into()]);
// A blackhole left by an earlier run is found and kept like any other.
fib.leave().await;
task.await.unwrap().unwrap();
let (fib, task) = machine.start();
status_becomes(&fib, "the blackhole is adopted", |status| {
status.synced && status.installed == 1 && status.blackholes == 1
});
fib.stop().await;
task.await.unwrap().unwrap();
machine.kernel_becomes(&[]);
}
/// A table's worth of routes goes in, and comes out, in reasonable time.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
@@ -462,7 +605,7 @@ async fn many_routes_are_installed_and_removed() {
/// Someone empties the table behind our back (`ip route flush`), and then
/// the peers withdraw everything: the kernel refuses every removal, a
/// table's worth of refusals at once. That is more than the socket holds
/// table's worth of rejections at once. That is more than the socket holds
/// answers for, and must be no more than a reason to look again.
#[tokio::test(flavor = "multi_thread")]
#[ignore = "changes the machine's routes; run scripts/kernel-tests.sh"]
@@ -721,6 +864,7 @@ async fn an_ipv4_route_goes_by_way_of_an_ipv6_neighbor() {
address: neighbor,
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap();
// Both families by way of the same neighbor.
crates/integration-tests/Cargo.toml
+4 -1
View File
@@ -17,11 +17,14 @@ bgp-rpki.workspace = true
bgp-server.workspace = true
bgp-web.workspace = true
bytes.workspace = true
clap.workspace = true
eyre.workspace = true
reqwest.workspace = true
serde_json.workspace = true
tokio.workspace = true
# Used by the examples only.
[dev-dependencies]
clap.workspace = true
[lints]
workspace = true
crates/integration-tests/examples/fake_peer.rs
+14 -35
View File
@@ -16,11 +16,9 @@
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::message::{Message, PathAttribute, UpdateMessage};
use bgp_primitives::ip_prefix::IpPrefix;
use clap::Parser;
use eyre::Result;
@@ -88,8 +86,8 @@ async fn run(args: &Args, mut peer: TestPeer, addr: std::net::SocketAddr) -> Res
for p in &args.announce {
let prefix = IpPrefix::try_from(p.as_str())?;
let update = announce(prefix, vec![args.asn], vec![(args.asn as u16, 1)]);
peer.send(&Message::Update(update)).await?;
peer.announce(announce(prefix, vec![args.asn], vec![(args.asn as u16, 1)]))
.await?;
println!("announced {prefix}");
}
@@ -98,7 +96,8 @@ async fn run(args: &Args, mut peer: TestPeer, addr: std::net::SocketAddr) -> Res
loop {
tokio::select! {
_ = ticker.tick() => peer.send(&Message::KeepAlive).await?,
msg = peer.recv() => match msg? {
// However long the server is quiet: this peer stays on the line.
msg = peer.recv_untimed() => match msg? {
Message::KeepAlive => {}
Message::Update(u) => {
for w in &u.withdrawn_routes {
@@ -121,22 +120,10 @@ async fn run(args: &Args, mut peer: TestPeer, addr: std::net::SocketAddr) -> Res
}
}
fn announce(prefix: IpPrefix, as_path: Vec<u32>, communities: Vec<(u16, u16)>) -> UpdateMessage {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 2))),
];
if !communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
communities,
)));
}
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: vec![prefix.into()],
}
fn announce(prefix: IpPrefix, as_path: Vec<u32>, communities: Vec<(u16, u16)>) -> Announcement {
Announcement::of([prefix.into()], &as_path)
.with_next_hop(Ipv4Addr::new(10, 0, 0, 2))
.with_communities(&communities)
}
async fn echo_back(peer: &mut TestPeer, u: &UpdateMessage, asn: u32) -> Result<()> {
@@ -157,21 +144,13 @@ async fn echo_back(peer: &mut TestPeer, u: &UpdateMessage, asn: u32) -> Result<(
}
for nlri in &u.announced_routes {
let prefix = nlri.prefix;
peer.send(&Message::Update(announce(
prefix,
as_path.clone(),
communities.clone(),
)))
.await?;
peer.announce(announce(prefix, as_path.clone(), communities.clone()))
.await?;
println!(" echoed {prefix} back with path {as_path:?}");
}
if !u.withdrawn_routes.is_empty() {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: u.withdrawn_routes.clone(),
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
peer.announce(withdrawal(u.withdrawn_routes.iter().cloned()))
.await?;
println!(
" echoed withdraw of {:?}",
u.withdrawn_routes
crates/integration-tests/src/announcement.rs
+110
View File
@@ -0,0 +1,110 @@
//! The UPDATEs a test peer sends, built from what a test has to say about
//! them and nothing more.
use std::net::Ipv4Addr;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::Nlri;
use crate::prefix;
/// An UPDATE announcing IPv4 routes as the peers of these tests do: with
/// ORIGIN IGP, the AS path given, and 192.0.2.1 as the next hop unless
/// another is set. Attributes are written in order of type code.
#[derive(Debug, Clone)]
pub struct Announcement {
routes: Vec<Nlri>,
path: Vec<u32>,
next_hop: Ipv4Addr,
only_to_customer: Option<u32>,
communities: Vec<(u16, u16)>,
more: Vec<PathAttribute>,
}
/// Announces `prefixes`, written as text, with the AS path `path`.
pub fn announcement(prefixes: &[&str], path: &[u32]) -> Announcement {
Announcement::of(prefixes.iter().map(|p| prefix(p).into()), path)
}
/// Withdraws `routes`.
pub fn withdrawal(routes: impl IntoIterator<Item = Nlri>) -> UpdateMessage {
UpdateMessage {
withdrawn_routes: routes.into_iter().collect(),
path_attributes: vec![],
announced_routes: vec![],
}
}
impl Announcement {
/// Announces `routes`, prefixes with or without path identifiers, with
/// the AS path `path`.
pub fn of(routes: impl IntoIterator<Item = Nlri>, path: &[u32]) -> Self {
Self {
routes: routes.into_iter().collect(),
path: path.to_vec(),
next_hop: Ipv4Addr::new(192, 0, 2, 1),
only_to_customer: None,
communities: Vec::new(),
more: Vec::new(),
}
}
pub fn with_next_hop(mut self, next_hop: Ipv4Addr) -> Self {
self.next_hop = next_hop;
self
}
/// With a COMMUNITIES attribute carrying `communities`, if there are
/// any.
pub fn with_communities(mut self, communities: &[(u16, u16)]) -> Self {
self.communities = communities.to_vec();
self
}
/// Marked as only to go to customers, by `asn` (RFC 9234).
pub fn with_only_to_customer(mut self, asn: u32) -> Self {
self.only_to_customer = Some(asn);
self
}
/// With `attributes` after the mandatory ones, as given.
pub fn with_attributes(mut self, attributes: impl IntoIterator<Item = PathAttribute>) -> Self {
self.more.extend(attributes);
self
}
}
impl From<Announcement> for UpdateMessage {
fn from(announcement: Announcement) -> Self {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(announcement.path),
PathAttribute::NextHop(NextHopPathAttribute(announcement.next_hop)),
];
if !announcement.communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
announcement.communities,
)));
}
path_attributes.extend(
announcement
.only_to_customer
.map(PathAttribute::OnlyToCustomer),
);
path_attributes.extend(announcement.more);
UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: announcement.routes,
}
}
}
impl From<Announcement> for Message {
fn from(announcement: Announcement) -> Self {
Message::Update(announcement.into())
}
}
crates/integration-tests/src/lib.rs
+88
View File
@@ -1,13 +1,101 @@
pub mod announcement;
pub mod test_cache;
pub mod test_peer;
pub mod test_server;
use std::time::Duration;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::address::Endpoint;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use eyre::{Result, bail};
use serde_json::Value;
/// How long a test waits for something the server does on its own: a
/// route reaching the table, a session coming up, a message arriving.
/// Everything happens on the loopback, so it is generous.
pub const TIMEOUT: Duration = Duration::from_secs(5);
/// The policy a peer has with nothing configured, but for one thing: the
/// peers of these tests are all at 127.0.0.1, which no route can have as
/// its next hop, so their routes name another.
pub fn default_policy() -> bgp_server::policy::ImportPolicy {
bgp_server::policy::ImportPolicy {
accept_third_party_next_hop: true,
accept_blackholes: false,
..Default::default()
}
}
/// A prefix written as text. The tests' prefixes are literals, so a bad
/// one is a mistake in the test.
pub fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A configured peer with nothing said about it but where it is and what
/// AS it is in: the default hold time, every family, the default policy of
/// these tests, and no role, limits, groups or ADD-PATH. A test says what
/// else with struct update syntax: `PeerConfig { role: Some(..),
/// ..peer_config(..) }`.
pub fn peer_config(name: &str, endpoint: impl Into<Endpoint>, remote_asn: u32) -> PeerConfig {
PeerConfig {
name: name.into(),
endpoint: endpoint.into(),
remote_asn,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: default_policy(),
role: None,
require_role: false,
export: Default::default(),
export_limits: None,
graceful_restart: Default::default(),
groups: Vec::new(),
reflector_client: false,
passive: false,
expected_routes: None,
}
}
/// Waits until `done` holds, looking every few milliseconds, or fails
/// with `what` once [`TIMEOUT`] has passed.
pub async fn wait_until(what: &str, mut done: impl FnMut() -> bool) -> Result<()> {
let waited = tokio::time::timeout(TIMEOUT, async {
while !done() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await;
if waited.is_err() {
bail!("timed out waiting until {what}");
}
Ok(())
}
/// Waits until `get` has something, and returns it; fails with `what`
/// once [`TIMEOUT`] has passed.
pub async fn wait_for<T>(what: &str, mut get: impl FnMut() -> Option<T>) -> Result<T> {
let mut got = None;
wait_until(what, || {
got = get();
got.is_some()
})
.await?;
Ok(got.expect("the wait ended with something"))
}
/// `GET`s `url` and reads the JSON it answers with. An error status is an
/// error.
pub async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
}
crates/integration-tests/src/test_peer.rs
+78 -10
View File
@@ -1,13 +1,16 @@
use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_packet::message::{BGP_HEADER_LEN, Message, UpdateMessage};
use bgp_packet::message::{BGP_HEADER_LEN, Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{Capability, OpenMessage, OpenOption};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::ParserContext;
use bytes::BytesMut;
use eyre::{Result, bail, eyre};
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
use tokio::net::{TcpListener, TcpStream};
use crate::TIMEOUT;
/// A minimal BGP peer for use in tests. Connects over TCP and can exchange
/// framed BGP messages using the `bgp-packet` wire format.
@@ -30,6 +33,13 @@ impl TestPeer {
Ok(Self::from_stream(stream))
}
/// The next connection to `listener`, as the server dials it. It is
/// waited for at most [`TIMEOUT`].
pub async fn accept(listener: &TcpListener) -> Result<Self> {
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
Ok(Self::from_stream(stream))
}
/// Wraps an already-connected stream, e.g. one accepted from a server
/// that dialled us.
pub fn from_stream(stream: TcpStream) -> Self {
@@ -41,6 +51,17 @@ impl TestPeer {
}
}
/// The connection itself, for a test or a tool that goes on without
/// the framing, once the handshake is done.
pub fn into_stream(self) -> TcpStream {
self.stream
}
/// Our end of the connection: what the server sees the peer as.
pub fn local_addr(&self) -> Result<SocketAddr> {
Ok(self.stream.local_addr()?)
}
/// Send a BGP message.
pub async fn send(&mut self, msg: &Message) -> Result<()> {
let send_ctx = ParserContext {
@@ -53,8 +74,32 @@ impl TestPeer {
Ok(())
}
/// Read exactly one BGP message.
/// Send an UPDATE: an [`Announcement`](crate::announcement::Announcement),
/// a [`withdrawal`](crate::announcement::withdrawal), or one built by
/// hand.
pub async fn announce(&mut self, update: impl Into<UpdateMessage>) -> Result<()> {
self.send(&Message::Update(update.into())).await
}
/// Read exactly one BGP message, waiting at most [`TIMEOUT`] for it. A
/// test that expects nothing to arrive says for how long with
/// [`TestPeer::recv_within`].
pub async fn recv(&mut self) -> Result<Message> {
self.recv_within(TIMEOUT).await
}
/// Read exactly one BGP message, waiting at most `wait` for it.
pub async fn recv_within(&mut self, wait: Duration) -> Result<Message> {
match tokio::time::timeout(wait, self.recv_untimed()).await {
Ok(message) => message,
Err(_) => Err(eyre::eyre!("no message from the server within {wait:?}")),
}
}
/// Read exactly one BGP message, however long it takes to come. For a
/// peer that is to stay on the line, like the `fake_peer` example;
/// tests want [`TestPeer::recv`].
pub async fn recv_untimed(&mut self) -> Result<Message> {
let mut header = [0u8; BGP_HEADER_LEN as usize];
self.stream.read_exact(&mut header).await?;
@@ -84,6 +129,18 @@ impl TestPeer {
}
}
/// Read messages until a NOTIFICATION arrives, skipping KEEPALIVEs. Any
/// other message type is an error.
pub async fn expect_notification(&mut self) -> Result<NotificationMessage> {
loop {
match self.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => return Err(eyre::eyre!("expected NOTIFICATION, got {other:?}")),
}
}
}
/// Receive and unwrap as `OpenMessage`, failing if a different type arrives.
pub async fn recv_open(&mut self) -> Result<OpenMessage> {
match self.recv().await? {
@@ -207,12 +264,23 @@ impl TestPeer {
Ok(())
}
/// Returns true if the remote end has closed the connection.
pub async fn is_closed(&mut self) -> bool {
let mut buf = [0u8; 1];
match self.stream.read(&mut buf).await {
Ok(0) | Err(_) => true,
Ok(_) => false,
}
/// Whether the server closes the connection within `wait`. Whatever it
/// still sends first, KEEPALIVEs as a rule, is read past: the end of
/// the stream, or anything unreadable, is what counts.
pub async fn closed_within(&mut self, wait: Duration) -> bool {
tokio::time::timeout(wait, async { while self.recv_untimed().await.is_ok() {} })
.await
.is_ok()
}
/// Whether the server closes the connection within [`TIMEOUT`] without
/// having sent anything: what a peer that is not let in sees. A server
/// that sends its OPEN first, or that leaves the connection open, is
/// answering.
pub async fn closed_unanswered(&mut self) -> bool {
matches!(
tokio::time::timeout(TIMEOUT, self.recv_untimed()).await,
Ok(Err(_))
)
}
}
crates/integration-tests/src/test_server.rs
+145 -13
View File
@@ -1,29 +1,37 @@
use std::collections::BTreeSet;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use bgp_fib::Fib;
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_packet::open::Capability;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::LocalRib;
use bgp_rib::table::{Event, PeerIndex, Reader, RibTable};
use bgp_rpki::Rpki;
use bgp_server::config::Config;
use bgp_server::config::{Config, PeerConfig};
use bgp_server::error::ServerError;
use bgp_server::refusals::RefusalLog;
use bgp_server::rejections::RejectionLog;
use bgp_server::restart::Restarts;
use bgp_server::server::BgpServer;
use bgp_server::session::SessionRegistry;
use bgp_server::session::{SessionHandle, SessionRegistry};
use bgp_web::{AppState, RouterInfo};
use eyre::Result;
use eyre::{Result, eyre};
use tokio::net::TcpListener;
use tokio::task::JoinHandle;
use crate::test_peer::TestPeer;
use crate::{peer_config, wait_for, wait_until};
/// A BGP server started on an ephemeral port for testing.
/// Aborts the server task when dropped.
pub struct TestServer {
handle: JoinHandle<Result<(), ServerError>>,
/// The single IPv4 loopback address the server is bound to.
pub addr: SocketAddr,
/// Every address it is bound to, in the order configured: `addr`
/// first.
pub addrs: Vec<SocketAddr>,
/// The server's routing table.
pub rib: Arc<RibTable>,
/// Handle to the server's locally originated routes.
@@ -31,7 +39,7 @@ pub struct TestServer {
/// Live view of the server's established sessions.
pub registry: SessionRegistry,
/// What peers announced that was not taken.
pub refusals: RefusalLog,
pub rejections: RejectionLog,
/// The data for route origin validation, if the server validates.
pub rpki: Option<Rpki>,
/// What keeps the best routes in the kernel, if the server does.
@@ -57,6 +65,7 @@ impl TestServer {
pub async fn start_with(local_asn: u32, customize: impl FnOnce(&mut Config)) -> Result<Self> {
let mut config = Config {
router_id: Ipv4Addr::new(127, 0, 0, 1),
cluster_id: Ipv4Addr::new(127, 0, 0, 1),
local_asn,
hold_time_secs: 90,
listen_endpoints: vec![SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0)],
@@ -71,21 +80,20 @@ impl TestServer {
rpki: None,
kernel: None,
export: Default::default(),
export_limits: None,
community_names: Vec::new(),
control: None,
graceful_restart: Default::default(),
};
customize(&mut config);
let bound = BgpServer::bind(config).await?;
let addr = bound
.local_addrs()
.into_iter()
.next()
.expect("at least one listener");
let addrs = bound.local_addrs();
let addr = *addrs.first().expect("at least one listener");
let rib = Arc::clone(&bound.rib);
let local_rib = bound.local_rib.clone();
let registry = bound.registry.clone();
let refusals = bound.refusals.clone();
let rejections = bound.rejections.clone();
let rpki = bound.rpki.clone();
let kernel = bound.kernel.clone();
let restarts = Arc::clone(&bound.restarts);
@@ -94,16 +102,140 @@ impl TestServer {
Ok(Self {
handle,
addr,
addrs,
rib,
local_rib,
registry,
refusals,
rejections,
rpki,
kernel,
restarts,
local_asn,
})
}
/// A server in `local_asn` that takes connections from one configured
/// peer only: "r1", in `remote_asn`, at the loopback address the test
/// connects from. `customize` edits the configuration and the peer's
/// before the server starts. The server dials the peer too; where that
/// goes is returned with it, for the test to answer or to leave alone.
pub async fn with_peer(
local_asn: u32,
remote_asn: u32,
customize: impl FnOnce(&mut Config, &mut PeerConfig),
) -> Result<ServerWithPeer> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = Self::start_with(local_asn, |config| {
config.accept_unconfigured_peers = false;
let mut peer = peer_config("r1", endpoint, remote_asn);
customize(config, &mut peer);
config.peers = vec![peer];
})
.await?;
Ok(ServerWithPeer { server, listener })
}
}
/// A server with one configured peer, and the listener at the peer's
/// address, which the server's dialling goes to.
pub struct ServerWithPeer {
pub server: TestServer,
pub listener: TcpListener,
}
impl TestServer {
/// A session with the server from a peer in `asn` with `router_id`,
/// offering the capabilities of a typical dual-stack speaker. Returns
/// once the server lists the session as established. Distinct peers
/// need distinct identifiers, or the second connection is (correctly)
/// treated as a collision with the first.
pub async fn established(&self, asn: u32, router_id: Ipv4Addr) -> Result<TestPeer> {
let mut peer = TestPeer::connect(self.addr).await?;
peer.handshake(asn, router_id, 90).await?;
self.listed(&peer).await?;
Ok(peer)
}
/// As [`TestServer::established`], with an OPEN carrying exactly
/// `capabilities`.
pub async fn established_with(
&self,
asn: u32,
router_id: Ipv4Addr,
capabilities: Vec<Capability>,
) -> Result<TestPeer> {
let mut peer = TestPeer::connect(self.addr).await?;
peer.handshake_with(asn, router_id, 90, capabilities)
.await?;
self.listed(&peer).await?;
Ok(peer)
}
/// Waits until the registry lists `peer`'s session.
async fn listed(&self, peer: &TestPeer) -> Result<()> {
let from = peer.local_addr()?;
wait_until("the session is established", || {
self.registry
.list()
.iter()
.any(|session| session.peer.addr == from)
})
.await
}
/// Waits until the registry lists `n` established sessions.
pub async fn wait_for_sessions(&self, n: usize) -> Result<()> {
wait_until(&format!("{n} sessions are established"), || {
self.registry.list().len() == n
})
.await
}
/// Waits until a session is established, and returns it.
pub async fn wait_for_session(&self) -> Result<SessionHandle> {
wait_for("a session is established", || {
self.registry.list().into_iter().next()
})
.await
}
/// Waits until the table holds `routes` routes.
pub async fn wait_for_routes(&self, routes: usize) -> Result<()> {
wait_until(&format!("the table holds {routes} routes"), || {
self.rib.len() == routes
})
.await
}
/// Waits until the table holds exactly the prefixes `expected` and the
/// peer has `held` routes held aside.
pub async fn table_becomes(&self, expected: &[&str], held: u64) -> Result<()> {
let expected: BTreeSet<String> = expected.iter().map(|p| p.to_string()).collect();
let table = || -> BTreeSet<String> {
self.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect()
};
let held_now = || {
self.registry
.list()
.first()
.map_or(0, |session| session.counters.held())
};
wait_until("the table is as expected", || {
table() == expected && held_now() == held
})
.await
.map_err(|_| {
eyre!(
"the table has {:?} with {} held; expected {expected:?} with {held} held",
table(),
held_now()
)
})
}
}
/// A route in the server's table, as a test wants to look at it.
@@ -218,7 +350,7 @@ impl TestServer {
registry: self.registry.clone(),
local_rib: self.local_rib.clone(),
rib: Arc::clone(&self.rib),
refusals: self.refusals.clone(),
rejections: self.rejections.clone(),
rpki: self.rpki.clone(),
kernel: self.kernel.clone(),
restarts: Arc::clone(&self.restarts),
crates/integration-tests/tests/add_path.rs
+401 -88
View File
@@ -1,57 +1,46 @@
//! End-to-end tests of ADD-PATH (RFC 7911): a peer sending several routes
//! for one prefix, told apart by path identifiers.
use std::net::{Ipv4Addr, SocketAddr};
use std::collections::BTreeMap;
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{peer_config, prefix, wait_until};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::open::{AddPathEntry, AddPathSendRecv, Capability, OpenMessage};
use bgp_packet::open::{AddPathEntry, Capability, ExtendedNextHopEntry, OpenMessage, SendReceive};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::constants::{Afi, Safi};
use bgp_primitives::ip_prefix::Nlri;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::{FamilySet, SUPPORTED};
use bgp_server::policy::ImportPolicy;
use bgp_server::family::SUPPORTED;
use bgp_server::policy::{ExportPolicy, ImportPolicy};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A server on which connections from this machine are those of a
/// configured peer, with ADD-PATH offered to it as `add_path` says.
async fn server_offering(
add_path: Option<AddPathSendRecv>,
originate: &[&str],
) -> Result<TestServer> {
async fn server_offering(add_path: Option<SendReceive>, originate: &[&str]) -> Result<TestServer> {
let originate: Vec<LocalRoute> = originate
.iter()
.map(|p| LocalRoute::new(prefix(p), LocalRouteAttrs::default()))
.collect();
TestServer::start_with(LOCAL_ASN, |c| {
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "route-server-client".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
..peer_config("route-server-client", at, PEER_ASN)
}];
c.originate = originate;
// Keep the dialer out of the way.
@@ -64,7 +53,7 @@ async fn server_offering(
/// IPv4 unicast. Returns the peer and the server's OPEN.
async fn peer_offering(
server: &TestServer,
offer: Option<AddPathSendRecv>,
offer: Option<SendReceive>,
) -> Result<(TestPeer, OpenMessage)> {
let mut capabilities = vec![Capability::FourByteAsn { asn: PEER_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
@@ -81,16 +70,11 @@ async fn peer_offering(
let open = peer
.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90, capabilities)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
server.wait_for_session().await?;
Ok((peer, open))
}
fn add_path_offered(open: &OpenMessage) -> Vec<(AfiSafi, AddPathSendRecv)> {
fn add_path_offered(open: &OpenMessage) -> Vec<(AfiSafi, SendReceive)> {
open.capabilities()
.filter_map(|capability| match capability {
Capability::AddPath { entries } => Some(entries),
@@ -101,26 +85,8 @@ fn add_path_offered(open: &OpenMessage) -> Vec<(AfiSafi, AddPathSendRecv)> {
.collect()
}
fn announcement(nlri: Nlri, as_path: &[u32]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: vec![nlri],
})
}
async fn wait_for_routes(server: &TestServer, routes: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != routes {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(())
fn announcement(nlri: Nlri, as_path: &[u32]) -> Announcement {
Announcement::of([nlri], as_path)
}
/// The routes the server holds, as (path identifier, AS path, best).
@@ -148,14 +114,14 @@ fn held(server: &TestServer) -> Vec<(u32, Vec<u32>, bool)> {
/// of them is chosen, and each is withdrawn on its own.
#[tokio::test]
async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
let server = server_offering(Some(AddPathSendRecv::Receive), &[]).await?;
let server = server_offering(Some(SendReceive::Receive), &[]).await?;
let (base, _http) = server.serve_http().await?;
let (mut peer, open) = peer_offering(&server, Some(AddPathSendRecv::Send)).await?;
let (mut peer, open) = peer_offering(&server, Some(SendReceive::Send)).await?;
// The server offered to receive, for every family it offers.
assert_eq!(
add_path_offered(&open),
SUPPORTED.map(|family| (family, AddPathSendRecv::Receive))
SUPPORTED.map(|family| (family, SendReceive::Receive))
);
// In use for the one family the peer sends for, and in that direction.
let session = &server.registry.list()[0];
@@ -168,11 +134,11 @@ async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
let p = prefix("203.0.113.0/24");
let path = |path_id| Nlri { prefix: p, path_id };
peer.send(&announcement(path(1), &[PEER_ASN, 64999, 64998]))
peer.announce(announcement(path(1), &[PEER_ASN, 64999, 64998]))
.await?;
peer.send(&announcement(path(2), &[PEER_ASN, 64998]))
peer.announce(announcement(path(2), &[PEER_ASN, 64998]))
.await?;
wait_for_routes(&server, 2).await?;
server.wait_for_routes(2).await?;
assert_eq!(
held(&server),
[
@@ -205,34 +171,24 @@ async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
);
// Announcing a path again replaces that path only.
peer.send(&announcement(path(1), &[PEER_ASN])).await?;
tokio::time::timeout(TIMEOUT, async {
while held(&server)[0].1.len() != 1 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
peer.announce(announcement(path(1), &[PEER_ASN])).await?;
wait_until("the path is replaced", || held(&server)[0].1.len() == 1).await?;
assert_eq!(
held(&server),
[(1, vec![PEER_ASN], true), (2, vec![PEER_ASN, 64998], false),]
);
// And withdrawing one leaves the other.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![path(1)],
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
wait_for_routes(&server, 1).await?;
peer.announce(withdrawal([path(1)])).await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(2, vec![PEER_ASN, 64998], true)]);
// When the session ends, all of its routes go.
peer.send(&announcement(path(3), &[PEER_ASN, 1, 2, 3]))
peer.announce(announcement(path(3), &[PEER_ASN, 1, 2, 3]))
.await?;
wait_for_routes(&server, 2).await?;
server.wait_for_routes(2).await?;
drop(peer);
wait_for_routes(&server, 0).await?;
server.wait_for_routes(0).await?;
Ok(())
}
@@ -242,24 +198,24 @@ async fn test_several_routes_for_a_prefix_from_one_peer() -> Result<()> {
#[tokio::test]
async fn test_add_path_is_not_used_unless_both_sides_offer_it() -> Result<()> {
let server = server_offering(None, &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, Some(AddPathSendRecv::Both)).await?;
let (mut peer, open) = peer_offering(&server, Some(SendReceive::Both)).await?;
assert!(add_path_offered(&open).is_empty());
let session = &server.registry.list()[0];
assert!(session.add_path_receive.is_empty() && session.add_path_send.is_empty());
// Plain prefixes both ways.
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
peer.send(&announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
peer.announce(announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
wait_for_routes(&server, 1).await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
// And the other way round: offered by the server, not by the peer.
let server = server_offering(Some(AddPathSendRecv::Both), &["10.100.0.0/16"]).await?;
let server = server_offering(Some(SendReceive::Both), &["10.100.0.0/16"]).await?;
let (mut peer, open) = peer_offering(&server, None).await?;
assert_eq!(add_path_offered(&open).len(), SUPPORTED.len());
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
Ok(())
}
@@ -269,8 +225,8 @@ async fn test_add_path_is_not_used_unless_both_sides_offer_it() -> Result<()> {
/// understands them.
#[tokio::test]
async fn test_prefixes_are_sent_with_path_identifiers_when_agreed() -> Result<()> {
let server = server_offering(Some(AddPathSendRecv::Both), &["10.100.0.0/16"]).await?;
let (mut peer, _) = peer_offering(&server, Some(AddPathSendRecv::Receive)).await?;
let server = server_offering(Some(SendReceive::Both), &["10.100.0.0/16"]).await?;
let (mut peer, _) = peer_offering(&server, Some(SendReceive::Receive)).await?;
let session = &server.registry.list()[0];
assert_eq!(
session.add_path_send,
@@ -280,15 +236,372 @@ async fn test_prefixes_are_sent_with_path_identifiers_when_agreed() -> Result<()
// Read as agreed: with identifiers.
peer.ctx.add_path = vec![AfiSafi::IPV4_UNICAST];
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes.len(), 1);
assert_eq!(update.announced_routes[0].prefix, prefix("10.100.0.0/16"));
// The peer did not offer to send, so the server reads its prefixes
// plain.
peer.send(&announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
peer.announce(announcement(prefix("203.0.113.0/24").into(), &[PEER_ASN]))
.await?;
wait_for_routes(&server, 1).await?;
server.wait_for_routes(1).await?;
assert_eq!(held(&server), [(0, vec![PEER_ASN], true)]);
Ok(())
}
const COLLECTOR_ASN: u32 = 65100;
/// What a collector holds: for each prefix and path identifier, the
/// route's AS path.
type Held = BTreeMap<(String, u32), Vec<u32>>;
/// Reads what the collector is told, into `held`, until `done` says so.
async fn told(
collector: &mut TestPeer,
held: &mut Held,
done: impl Fn(&Held) -> bool,
) -> Result<()> {
while !done(held) {
let update = collector.recv_update().await?;
let path = update
.path_attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::AsPath(path) => Some(path.segments[0].path.clone()),
_ => None,
})
.unwrap_or_default();
for nlri in &update.withdrawn_routes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
for nlri in &update.announced_routes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
// And what came and went in the multiprotocol attributes.
for attribute in &update.path_attributes {
match attribute {
PathAttribute::MpReachNlri(reach) => {
for nlri in &reach.prefixes {
held.insert((nlri.prefix.to_string(), nlri.path_id), path.clone());
}
}
PathAttribute::MpUnreachNlri(unreach) => {
for nlri in &unreach.prefixes {
held.remove(&(nlri.prefix.to_string(), nlri.path_id));
}
}
_ => {}
}
}
}
Ok(())
}
/// The routes held for 203.0.113.0/24, by path identifier.
fn paths(held: &Held) -> Vec<(u32, Vec<u32>)> {
held.iter()
.map(|((prefix, id), path)| {
assert_eq!(prefix, "203.0.113.0/24");
(*id, path.clone())
})
.collect()
}
/// A collector that agreed to be sent several routes for a prefix, and
/// whose export policy says to send it every route, is told of every
/// peer's route for a prefix, each with a path identifier of its own that
/// stays with the route; a route that goes is withdrawn by its identifier.
#[tokio::test]
async fn test_every_route_goes_to_a_collector_told_apart_by_path_identifiers() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// The collector is the configured peer at the IPv4 loopback
// address; the feeders come over IPv6 loopback as peers of no
// account.
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let route = |path: &[u32]| Announcement::of([prefix("203.0.113.0/24").into()], path);
let feeder = async |n: u32| -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
Ok(peer)
};
let mut a = feeder(0).await?;
let mut b = feeder(1).await?;
server.wait_for_sessions(2).await?;
a.announce(route(&[PEER_ASN, 64510])).await?;
b.announce(route(&[PEER_ASN + 1, 64510])).await?;
server.wait_for_routes(2).await?;
let mut capabilities = vec![Capability::FourByteAsn { asn: COLLECTOR_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: vec![AddPathEntry {
afi: AfiSafi::IPV4_UNICAST.afi,
safi: AfiSafi::IPV4_UNICAST.safi,
send_recv: SendReceive::Receive,
}],
});
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
capabilities,
)
.await?;
collector.ctx.add_path = vec![AfiSafi::IPV4_UNICAST];
// Both routes, told apart.
let mut held = Held::new();
told(&mut collector, &mut held, |held| held.len() == 2).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 1, 64510]),
]
);
// The second peer's route goes: withdrawn by its identifier, the
// first's untouched.
b.announce(withdrawal([prefix("203.0.113.0/24").into()]))
.await?;
told(&mut collector, &mut held, |held| held.len() == 1).await?;
assert_eq!(paths(&held), [(1, vec![LOCAL_ASN, PEER_ASN, 64510])]);
// The first peer's route changes: it goes out again with the same
// identifier. A third peer's route gets the lowest free one.
a.announce(route(&[PEER_ASN, 64511, 64510])).await?;
let mut c = feeder(2).await?;
server.wait_for_sessions(4).await?;
c.announce(route(&[PEER_ASN + 2, 64510])).await?;
let changed = |held: &Held| {
held.len() == 2
&& held.get(&("203.0.113.0/24".to_string(), 1))
== Some(&vec![LOCAL_ASN, PEER_ASN, 64511, 64510])
};
told(&mut collector, &mut held, changed).await?;
assert_eq!(
paths(&held),
[
(1, vec![LOCAL_ASN, PEER_ASN, 64511, 64510]),
(2, vec![LOCAL_ASN, PEER_ASN + 2, 64510]),
]
);
Ok(())
}
/// A collector that is connected before the table arrives is told the
/// whole table, however large: the walk of the table that a new follower
/// starts with takes many batches with a table of some size, and goes on
/// while the feeders keep sending.
#[tokio::test]
async fn test_a_large_table_reaches_a_collector_that_was_there_first() -> Result<()> {
const PREFIXES: u32 = 40_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut capabilities = vec![Capability::FourByteAsn { asn: COLLECTOR_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: vec![AddPathEntry {
afi: AfiSafi::IPV4_UNICAST.afi,
safi: AfiSafi::IPV4_UNICAST.safi,
send_recv: SendReceive::Receive,
}],
});
let mut collector = TestPeer::connect(server.addr).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
capabilities,
)
.await?;
collector.ctx.add_path = vec![AfiSafi::IPV4_UNICAST];
server.wait_for_sessions(1).await?;
// Two feeders, each sending the same prefixes a hundred to an UPDATE.
let nth = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addrs[1]).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..PREFIXES).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| nth(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * PREFIXES as usize
})
.await?;
let ids: std::collections::BTreeSet<u32> = held.keys().map(|(_, id)| *id).collect();
assert_eq!(ids.into_iter().collect::<Vec<_>>(), [1, 2]);
Ok(())
}
/// As a router at an exchange feeds a collector: one IPv6 session that
/// carries both families, with ADD-PATH for both and IPv4 routes with
/// IPv6 next hops (RFC 8950), and a table of some size from two feeders.
#[tokio::test]
async fn test_both_families_reach_a_collector_over_one_ipv6_session() -> Result<()> {
const V4: u32 = 20_000;
const V6: u32 = 1_000;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.listen_endpoints
.push(SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 0));
c.peers = vec![PeerConfig {
add_path: Some(SendReceive::Send),
import: ImportPolicy::permissive(),
export: Arc::new(ExportPolicy {
learned_routes: true,
all_paths: true,
..ExportPolicy::default()
}),
..peer_config(
"collector",
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), 1),
COLLECTOR_ASN,
)
}];
c.connect_retry = Duration::from_secs(3600);
})
.await?;
let mut capabilities = vec![Capability::FourByteAsn { asn: COLLECTOR_ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST]
.into_iter()
.map(|family| AddPathEntry {
afi: family.afi,
safi: family.safi,
send_recv: SendReceive::Receive,
})
.collect(),
});
capabilities.push(Capability::ExtendedNextHop {
entries: vec![ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
}],
});
let mut collector = TestPeer::connect(server.addrs[1]).await?;
collector
.handshake_with(
COLLECTOR_ASN,
Ipv4Addr::new(10, 0, 0, 100),
90,
capabilities,
)
.await?;
collector.ctx.add_path = vec![AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST];
server.wait_for_sessions(1).await?;
let session = &server.registry.list()[0];
assert!(session.extended_next_hop, "RFC 8950 agreed");
assert_eq!(session.add_path_send.iter().count(), 2);
let v4 = |n: u32| {
let [_, b, c, _] = (0x0a00_0000u32 + n * 256).to_be_bytes();
prefix(&format!("10.{b}.{c}.0/24"))
};
let v6 = |n: u32| prefix(&format!("2001:db8:{:x}::/48", 0x1000 + n));
let mut feeders = Vec::new();
for n in 0..2u32 {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN + n, Ipv4Addr::new(10, 0, 0, n as u8 + 1), 90)
.await?;
feeders.push(peer);
}
server.wait_for_sessions(3).await?;
for start in (0..V4).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
feeder
.announce(Announcement::of(
(start..start + 100).map(|i| v4(i).into()),
&[PEER_ASN + n as u32, 64510],
))
.await?;
}
}
for start in (0..V6).step_by(100) {
for (n, feeder) in feeders.iter_mut().enumerate() {
let update = UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN + n as u32, 64510]),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6("2001:db8::9".parse().unwrap()),
prefixes: (start..start + 100).map(|i| v6(i).into()).collect(),
}),
],
announced_routes: vec![],
};
feeder.send(&Message::Update(update)).await?;
}
}
let mut held = Held::new();
told(&mut collector, &mut held, |held| {
held.len() == 2 * (V4 + V6) as usize
})
.await?;
let v6_routes = held
.keys()
.filter(|(prefix, _)| prefix.contains(':'))
.count();
assert_eq!(v6_routes, 2 * V6 as usize);
Ok(())
}
crates/integration-tests/tests/admission.rs
+18 -44
View File
@@ -5,38 +5,14 @@ use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, peer_config};
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn peer_at(endpoint: SocketAddr) -> PeerConfig {
PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
}
}
/// Whether the server closed the connection without sending anything.
async fn closed_unanswered(peer: &mut TestPeer) -> bool {
matches!(tokio::time::timeout(TIMEOUT, peer.recv()).await, Ok(Err(_)))
}
/// A host that is not a configured peer gets nothing: no OPEN to answer,
/// no session, no routes taken from it. Any host used to be able to open a
@@ -46,12 +22,16 @@ async fn test_a_host_that_is_not_a_configured_peer_is_refused() -> Result<()> {
// The only peer configured is somewhere else (documentation space).
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
c.peers = vec![peer_at("192.0.2.9:179".parse().unwrap())];
let elsewhere: SocketAddr = "192.0.2.9:179".parse().unwrap();
c.peers = vec![PeerConfig {
import: ImportPolicy::permissive(),
..peer_config("r1", elsewhere, PEER_ASN)
}];
})
.await?;
let mut stranger = TestPeer::connect(server.addr).await?;
assert!(closed_unanswered(&mut stranger).await);
assert!(stranger.closed_unanswered().await);
assert!(server.registry.list().is_empty());
Ok(())
}
@@ -60,24 +40,18 @@ async fn test_a_host_that_is_not_a_configured_peer_is_refused() -> Result<()> {
/// configured for that peer.
#[tokio::test]
async fn test_a_configured_peer_is_let_in() -> Result<()> {
// Somewhere for the server's own dialling to go; never answered.
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
c.peers = vec![peer_at(endpoint)];
// The server's own dialling goes to the listener; it is never answered.
let ServerWithPeer {
server,
listener: _listener,
} = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |_, peer| {
peer.import = ImportPolicy::permissive();
})
.await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
let _peer = server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2))
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
assert_eq!(server.registry.list()[0].name.as_deref(), Some("r1"));
Ok(())
}
@@ -93,11 +67,11 @@ async fn test_one_address_may_open_only_a_few_connections() -> Result<()> {
let mut held = Vec::new();
for _ in 0..4 {
let mut peer = TestPeer::connect(server.addr).await?;
tokio::time::timeout(TIMEOUT, peer.recv_open()).await??;
peer.recv_open().await?;
held.push(peer);
}
let mut one_too_many = TestPeer::connect(server.addr).await?;
assert!(closed_unanswered(&mut one_too_many).await);
assert!(one_too_many.closed_unanswered().await);
// When one of them goes, there is room again.
held.pop();
crates/integration-tests/tests/announce.rs
+23 -59
View File
@@ -4,25 +4,21 @@
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
LocalPrefPathAttribute, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::session::SessionEvent;
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn route(p: &str, attrs: LocalRouteAttrs) -> LocalRoute {
LocalRoute::new(prefix(p), attrs)
@@ -32,27 +28,12 @@ async fn established_peer(server: &TestServer) -> Result<TestPeer> {
established_peer_with_id(server, Ipv4Addr::new(10, 0, 0, 1)).await
}
/// Distinct routers need distinct BGP Identifiers, or the second connection
/// is (correctly) treated as a collision with the first.
async fn established_peer_with_id(server: &TestServer, router_id: Ipv4Addr) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, router_id, 90).await?;
Ok(peer)
server.established(PEER_ASN, router_id).await
}
async fn next_update(peer: &mut TestPeer) -> Result<UpdateMessage> {
tokio::time::timeout(TIMEOUT, peer.recv_update()).await?
}
/// Waits until the registry reports `n` established sessions.
async fn wait_for_sessions(server: &TestServer, n: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().len() != n {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
peer.recv_update().await
}
// ---------------------------------------------------------------------------
@@ -97,7 +78,7 @@ async fn test_registry_tracks_session_lifecycle() -> Result<()> {
async fn test_announce_after_session_up() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
@@ -174,7 +155,7 @@ async fn test_announce_reaches_all_sessions() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
wait_for_sessions(&server, 2).await?;
server.wait_for_sessions(2).await?;
server
.local_rib
@@ -192,7 +173,7 @@ async fn test_announce_reaches_all_sessions() -> Result<()> {
async fn test_withdraw_sends_withdrawn_route() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
@@ -213,7 +194,7 @@ async fn test_withdraw_sends_withdrawn_route() -> Result<()> {
async fn test_reannounce_with_prepend_and_med() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
@@ -258,7 +239,7 @@ async fn test_keepalives_continue_alongside_updates() -> Result<()> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 3)
.await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
@@ -289,22 +270,12 @@ async fn test_received_routes_removed_when_session_closes() -> Result<()> {
let mut rib_events = server.route_changes();
let mut session_events = server.registry.subscribe();
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
peer.send(&bgp_packet::message::Message::Update(
bgp_packet::message::UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 1))),
],
announced_routes: vec![
prefix("203.0.113.0/24").into(),
prefix("198.51.100.0/24").into(),
],
},
))
peer.announce(
announcement(&["203.0.113.0/24", "198.51.100.0/24"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 1)),
)
.await?;
for _ in 0..2 {
let ev = tokio::time::timeout(TIMEOUT, rib_events.recv()).await??;
@@ -351,7 +322,7 @@ const ROUTE_REFRESH_IPV4: &[u8] = &[
async fn test_route_refresh_resends_local_routes() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = established_peer(&server).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
server
.local_rib
@@ -386,33 +357,26 @@ async fn test_route_refresh_resends_local_routes() -> Result<()> {
#[tokio::test]
async fn test_shutdown_sends_cease_to_every_peer() -> Result<()> {
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
let server = TestServer::start(LOCAL_ASN).await?;
let mut a = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 1)).await?;
let mut b = established_peer_with_id(&server, Ipv4Addr::new(10, 0, 0, 2)).await?;
wait_for_sessions(&server, 2).await?;
server.wait_for_sessions(2).await?;
let left = server.registry.shutdown(Duration::from_secs(2)).await;
let left = server
.registry
.leave(bgp_server::restart::Leave::Stop, Duration::from_secs(2))
.await;
assert_eq!(left, 0, "sessions still open after shutdown");
assert!(server.registry.is_shutting_down());
for peer in [&mut a, &mut b] {
let notif = tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {}
other => eyre::bail!("unexpected {other:?}"),
}
}
})
.await??;
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::Cease(CeaseSubcode::AdminShutdown)),
"{notif:?}"
);
assert!(peer.is_closed().await);
assert!(peer.closed_within(TIMEOUT).await);
}
Ok(())
}
@@ -467,7 +431,7 @@ async fn test_a_peer_in_our_own_as_gets_an_empty_path_and_a_local_pref() -> Resu
// The peer reads as a router in the server's AS does: one outside it
// would drop the LOCAL_PREF.
peer.ctx.ibgp = true;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
assert!(server.registry.list()[0].ibgp);
server
crates/integration-tests/tests/aspa.rs
+46 -124
View File
@@ -6,37 +6,20 @@ use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::default_policy;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_cache::TestCache;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_packet::message::PathAttribute;
use bgp_rpki::{CacheState, RpkiConfig};
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::{ImportPolicy, PeerRole};
use eyre::Result;
use serde_json::Value;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// What a test holds on to: the server, and the listener its dialling
/// goes to.
struct Setup {
server: TestServer,
_listener: TcpListener,
}
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server that validates against `cache`, with one configured peer at
/// the address the test connects from, which is `role` to it. Returns once
@@ -45,104 +28,33 @@ async fn server_with(
cache: &TestCache,
role: Option<PeerRole>,
import: ImportPolicy,
) -> Result<Setup> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
) -> Result<ServerWithPeer> {
let cache = cache.address();
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
let setup = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.rpki = Some(RpkiConfig {
caches: vec![cache],
retry: Some(Duration::from_millis(100)),
..RpkiConfig::default()
});
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import,
role,
require_role: false,
export: Default::default(),
}];
peer.import = import;
peer.role = role;
})
.await?;
let rpki = server.rpki.clone().expect("validation is configured");
tokio::time::timeout(TIMEOUT, async {
while rpki.status().state != CacheState::Synchronised {
tokio::time::sleep(Duration::from_millis(5)).await;
}
let rpki = setup.server.rpki.clone().expect("validation is configured");
wait_until("the server has the cache's data", || {
rpki.status().state == CacheState::Synchronised
})
.await?;
Ok(Setup {
server,
_listener: listener,
})
Ok(setup)
}
async fn established(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(peer)
server.established(PEER_ASN, PEER_ID).await
}
/// Announces `prefixes` with the AS path `path`.
async fn announce(peer: &mut TestPeer, prefixes: &[&str], path: &[u32]) -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
}))
.await
}
/// Waits until the table holds exactly `expected` and the peer has `held`
/// routes held aside.
async fn table_becomes(server: &TestServer, expected: &[&str], held: u64) -> Result<()> {
let expected: BTreeSet<String> = expected.iter().map(|p| p.to_string()).collect();
let table = || -> BTreeSet<String> {
server
.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect()
};
let held_now = || {
server
.registry
.list()
.first()
.map_or(0, |session| session.counters.held())
};
tokio::time::timeout(TIMEOUT, async {
while table() != expected || held_now() != held {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.map_err(|_| {
eyre::eyre!(
"the table has {:?} with {} held; expected {expected:?} with {held} held",
table(),
held_now()
)
})
peer.announce(announcement(prefixes, path)).await
}
/// AS 15169 buys from 3356 and says so; 3356 and the peer, 3333, have no
@@ -157,7 +69,8 @@ fn aspas() -> Vec<(u32, Vec<u32>)> {
#[tokio::test]
async fn test_a_leaked_route_is_refused_and_the_rest_are_taken() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let Setup { server, .. } = server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
@@ -167,7 +80,9 @@ async fn test_a_leaked_route_is_refused_and_the_rest_are_taken() -> Result<()> {
announce(&mut peer, &["193.0.0.0/21"], &[PEER_ASN]).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121]).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
table_becomes(&server, &["193.0.0.0/21", "194.0.0.0/24"], 1).await?;
server
.table_becomes(&["193.0.0.0/21", "194.0.0.0/24"], 1)
.await?;
let client = reqwest::Client::new();
let get = |path: &str| {
@@ -213,13 +128,13 @@ async fn test_a_route_that_has_crossed_a_peering_is_taken_from_a_provider_only()
(None, true),
] {
let cache = TestCache::start_with(vec![], aspas()).await;
let Setup { server, .. } = server_with(&cache, role, default_policy()).await?;
let ServerWithPeer { server, .. } = server_with(&cache, role, default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
let expected = if taken {
table_becomes(&server, &["8.8.8.0/24"], 0).await
server.table_becomes(&["8.8.8.0/24"], 0).await
} else {
table_becomes(&server, &[], 1).await
server.table_becomes(&[], 1).await
};
expected.map_err(|e| eyre::eyre!("from a peer that is {role:?}: {e}"))?;
}
@@ -236,14 +151,15 @@ async fn test_paths_from_a_route_server_do_not_begin_with_its_as() -> Result<()>
accept_any_first_as: true,
..default_policy()
};
let Setup { server, .. } = server_with(&cache, Some(PeerRole::RouteServer), import).await?;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::RouteServer), import).await?;
let mut peer = established(&server).await?;
// 3356's customer route, as 3356 gave it to the route server.
announce(&mut peer, &["8.8.8.0/24"], &[3356, 15169]).await?;
// The same route leaked to the route server by another member.
announce(&mut peer, &["8.8.4.0/24"], &[2121, 3356, 15169]).await?;
table_becomes(&server, &["8.8.8.0/24"], 1).await
server.table_becomes(&["8.8.8.0/24"], 1).await
}
/// Validity changes with the ASPAs, and the routes follow: one that was
@@ -252,20 +168,21 @@ async fn test_paths_from_a_route_server_do_not_begin_with_its_as() -> Result<()>
#[tokio::test]
async fn test_routes_follow_the_aspas() -> Result<()> {
let cache = TestCache::start_with(vec![], aspas()).await;
let Setup { server, .. } = server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121]).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
table_becomes(&server, &["194.0.0.0/24"], 1).await?;
server.table_becomes(&["194.0.0.0/24"], 1).await?;
// 2121 says that its only provider is 1101: its route by way of 3333
// is now invalid, and goes.
let mut changed = aspas();
changed.push((2121, vec![1101]));
cache.set_aspas(changed);
table_becomes(&server, &[], 2).await?;
let refused = &server.refusals.peers()[0].1;
server.table_becomes(&[], 2).await?;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 2);
assert_eq!(refused.recent()[0].prefix, prefix("194.0.0.0/24"));
assert_eq!(&refused.recent()[0].as_path[..], [PEER_ASN, 2121]);
@@ -278,7 +195,9 @@ async fn test_routes_follow_the_aspas() -> Result<()> {
(PEER_ASN, vec![0]),
(2121, vec![1101, PEER_ASN]),
]);
table_becomes(&server, &["194.0.0.0/24", "8.8.8.0/24"], 0).await?;
server
.table_becomes(&["194.0.0.0/24", "8.8.8.0/24"], 0)
.await?;
let paths: BTreeSet<Vec<u32>> = server
.received_routes()
.iter()
@@ -300,9 +219,11 @@ async fn test_routes_follow_the_aspas() -> Result<()> {
(3356, vec![0]),
(PEER_ASN, vec![0]),
]);
table_becomes(&server, &["194.0.0.0/24"], 1).await?;
server.table_becomes(&["194.0.0.0/24"], 1).await?;
cache.set_aspas(vec![]);
table_becomes(&server, &["194.0.0.0/24", "8.8.8.0/24"], 0).await
server
.table_becomes(&["194.0.0.0/24", "8.8.8.0/24"], 0)
.await
}
/// A peer can be exempt: its routes with invalid paths are taken, and
@@ -314,18 +235,18 @@ async fn test_a_peer_can_be_exempt_from_path_verification() -> Result<()> {
accept_aspa_invalid: true,
..default_policy()
};
let Setup { server, .. } = server_with(&cache, Some(PeerRole::Peer), import).await?;
let ServerWithPeer { server, .. } = server_with(&cache, Some(PeerRole::Peer), import).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
table_becomes(&server, &["8.8.8.0/24"], 0).await?;
server.table_becomes(&["8.8.8.0/24"], 0).await?;
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
assert_eq!(routes["routes"][0]["aspa"], "invalid");
assert!(server.refusals.peers().is_empty());
assert!(server.rejections.peers().is_empty());
Ok(())
}
@@ -338,16 +259,17 @@ async fn test_a_route_is_held_until_origin_and_path_are_both_in_order() -> Resul
asn,
};
let cache = TestCache::start_with(vec![google(13335)], aspas()).await;
let Setup { server, .. } = server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let ServerWithPeer { server, .. } =
server_with(&cache, Some(PeerRole::Peer), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["8.8.8.0/24"], &[PEER_ASN, 3356, 15169]).await?;
table_becomes(&server, &[], 1).await?;
server.table_becomes(&[], 1).await?;
// The origin is put right; the path is still a leak.
cache.set(vec![google(15169)]);
tokio::time::sleep(Duration::from_millis(500)).await;
table_becomes(&server, &[], 1).await?;
server.table_becomes(&[], 1).await?;
cache.set_aspas(vec![(15169, vec![3356]), (3356, vec![PEER_ASN])]);
table_becomes(&server, &["8.8.8.0/24"], 0).await
server.table_becomes(&["8.8.8.0/24"], 0).await
}
crates/integration-tests/tests/basic_connection.rs
+10 -38
View File
@@ -4,6 +4,7 @@ use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::RouteChange;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode, FsmSubcode, MsgHeaderSubcode, UpdateMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, MpUnreachNlriPathAttribute,
@@ -14,8 +15,6 @@ use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::IpPrefix;
use eyre::Result;
const TIMEOUT: Duration = Duration::from_secs(5);
/// The attributes every announcement of IPv4 routes must carry. An UPDATE
/// without them has its routes withdrawn instead (RFC 7606).
fn mandatory_attributes() -> Vec<PathAttribute> {
@@ -98,20 +97,6 @@ async fn test_bad_marker_does_not_crash_server() -> Result<()> {
Ok(())
}
/// Reads until a NOTIFICATION arrives, skipping KEEPALIVEs.
async fn expect_notification(peer: &mut TestPeer) -> Result<NotificationMessage> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => eyre::bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
/// A header with the given length and type, followed by `body`.
fn framed(length: u16, msg_type: u8, body: &[u8]) -> Vec<u8> {
let mut buf = vec![0xFF; 16];
@@ -170,7 +155,7 @@ async fn test_malformed_messages_are_answered_with_the_matching_notification() -
peer.handshake(65001, Ipv4Addr::new(10, 0, 1, i as u8 + 1), 90)
.await?;
peer.send_raw(&bytes).await?;
let notification = expect_notification(&mut peer).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
(notification.error, notification.data),
(error.clone(), data.clone()),
@@ -181,7 +166,7 @@ async fn test_malformed_messages_are_answered_with_the_matching_notification() -
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send_raw(&bytes).await?;
let notification = expect_notification(&mut peer).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
(notification.error, notification.data),
(error, data),
@@ -194,7 +179,7 @@ async fn test_malformed_messages_are_answered_with_the_matching_notification() -
let mut peer = TestPeer::connect(server.addr).await?;
let _ = peer.recv_open().await?;
peer.send(&Message::KeepAlive).await?;
let notification = expect_notification(&mut peer).await?;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::FsmError(FsmSubcode::UnexpectedOpenSent)
@@ -228,13 +213,10 @@ async fn test_message_split_across_a_keepalive_tick_is_still_one_message() -> Re
tokio::time::sleep(Duration::from_millis(1300)).await;
peer.send_raw(&update[10..]).await?;
tokio::time::timeout(TIMEOUT, async {
while !server.received_routes().iter().any(|e| e.prefix == prefix) {
tokio::time::sleep(Duration::from_millis(20)).await;
}
wait_until("the split UPDATE reaches the RIB", || {
server.received_routes().iter().any(|e| e.prefix == prefix)
})
.await
.map_err(|_| eyre::eyre!("the split UPDATE never reached the RIB"))?;
.await?;
assert_eq!(server.registry.list().len(), 1, "session was dropped");
Ok(())
}
@@ -266,7 +248,7 @@ async fn test_server_sends_keepalives() -> Result<()> {
peer.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 3).await?;
// We should receive a KEEPALIVE within ~1 second (hold/3).
let msg = tokio::time::timeout(Duration::from_secs(3), peer.recv()).await??;
let msg = peer.recv_within(Duration::from_secs(3)).await?;
assert!(
matches!(msg, Message::KeepAlive),
"Expected KEEPALIVE, got {:?}",
@@ -286,16 +268,7 @@ async fn test_hold_timer_expires() -> Result<()> {
// Don't send any keepalives. The server will send keepalives until its own
// hold timer fires and it sends a NOTIFICATION. Drain messages until we
// see it (or the outer timeout fires).
let notif = tokio::time::timeout(Duration::from_secs(6), async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {} // server keepalive — keep waiting
other => eyre::bail!("Unexpected message: {:?}", other),
}
}
})
.await??;
let notif = tokio::time::timeout(Duration::from_secs(6), peer.expect_notification()).await??;
assert!(
matches!(notif.error, BgpError::HoldTimer),
"Expected HoldTimer NOTIFICATION, got {:?}",
@@ -319,8 +292,7 @@ async fn test_peer_sends_notification() -> Result<()> {
.await?;
// Server should close the connection cleanly.
tokio::time::sleep(Duration::from_millis(100)).await;
assert!(peer.is_closed().await);
assert!(peer.closed_within(TIMEOUT).await);
Ok(())
}
crates/integration-tests/tests/best_route.rs
+41 -70
View File
@@ -2,66 +2,34 @@
//! peers, which of them the server chooses, and how that is shown.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, withdrawal as withdrawal_of};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{RouteChange, TestServer};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_integration_tests::{TIMEOUT, get_json, prefix};
use bgp_packet::message::{PathAttribute, UpdateMessage};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use eyre::Result;
use serde_json::Value;
const LOCAL_ASN: u32 = 65000;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A session from the peer numbered `number`, in AS 65000 + `number`.
async fn peer(server: &TestServer, number: u8) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(
LOCAL_ASN + u32::from(number),
Ipv4Addr::new(10, 0, 0, number),
90,
)
.await?;
Ok(peer)
server
.established(
LOCAL_ASN + u32::from(number),
Ipv4Addr::new(10, 0, 0, number),
)
.await
}
fn announcement(prefixes: &[IpPrefix], as_path: &[u32]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: Nlri::plain(prefixes.iter().copied()),
})
fn announcement(prefixes: &[IpPrefix], as_path: &[u32]) -> Announcement {
Announcement::of(Nlri::plain(prefixes.iter().copied()), as_path)
}
fn withdrawal(prefixes: &[IpPrefix]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: Nlri::plain(prefixes.iter().copied()),
path_attributes: vec![],
announced_routes: vec![],
})
}
/// Waits until the table holds `routes` routes.
async fn wait_for_routes(server: &TestServer, routes: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != routes {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(())
fn withdrawal(prefixes: &[IpPrefix]) -> UpdateMessage {
withdrawal_of(Nlri::plain(prefixes.iter().copied()))
}
/// The AS path of the route the server holds as best for `prefix`.
@@ -78,16 +46,6 @@ fn best_path(server: &TestServer, prefix: &IpPrefix) -> Option<Vec<u32>> {
})
}
async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
}
/// Two peers offer the same prefix. The route with the shorter AS path is
/// chosen; when it goes, the other is.
#[tokio::test]
@@ -99,14 +57,14 @@ async fn test_the_best_route_is_chosen_and_replaced_when_it_goes() -> Result<()>
let p = prefix("203.0.113.0/24");
let mut far = peer(&server, 1).await?;
far.send(&announcement(&[p], &[65001, 64999, 64998]))
far.announce(announcement(&[p], &[65001, 64999, 64998]))
.await?;
wait_for_routes(&server, 1).await?;
server.wait_for_routes(1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
let mut near = peer(&server, 2).await?;
near.send(&announcement(&[p], &[65002, 64998])).await?;
wait_for_routes(&server, 2).await?;
near.announce(announcement(&[p], &[65002, 64998])).await?;
server.wait_for_routes(2).await?;
assert_eq!(best_path(&server, &p), Some(vec![65002, 64998]));
// Both routes are kept and shown; one is marked.
@@ -123,14 +81,27 @@ async fn test_the_best_route_is_chosen_and_replaced_when_it_goes() -> Result<()>
assert_eq!(routes.as_array().unwrap().len(), 2);
assert_eq!(best.len(), 1);
assert_eq!(best[0]["peer_asn"], 65002);
// Listed from the best down, each with its place in selection.
let ranks: Vec<(u64, u64)> = routes
.as_array()
.unwrap()
.iter()
.map(|route| {
(
route["rank"].as_u64().unwrap(),
route["peer_asn"].as_u64().unwrap(),
)
})
.collect();
assert_eq!(ranks, vec![(1, 65002), (2, 65001)]);
// The session with the best route ends: the other route takes over.
drop(near);
wait_for_routes(&server, 1).await?;
server.wait_for_routes(1).await?;
assert_eq!(best_path(&server, &p), Some(vec![65001, 64999, 64998]));
far.send(&withdrawal(&[p])).await?;
wait_for_routes(&server, 0).await?;
far.announce(withdrawal(&[p])).await?;
server.wait_for_routes(0).await?;
assert_eq!(best_path(&server, &p), None);
// A follower of the best routes is told of each change: the first
@@ -157,9 +128,9 @@ async fn test_changes_of_best_route_are_on_the_event_stream() -> Result<()> {
// A route that is there before the client connects is not an event.
let mut first = peer(&server, 1).await?;
first
.send(&announcement(&[prefix("198.51.100.0/24")], &[65001]))
.announce(announcement(&[prefix("198.51.100.0/24")], &[65001]))
.await?;
wait_for_routes(&server, 1).await?;
server.wait_for_routes(1).await?;
let mut stream = reqwest::Client::new()
.get(format!("{base}/api/events"))
@@ -167,10 +138,10 @@ async fn test_changes_of_best_route_are_on_the_event_stream() -> Result<()> {
.await?
.error_for_status()?;
first.send(&announcement(&[p], &[65001, 64998])).await?;
wait_for_routes(&server, 2).await?;
first.send(&withdrawal(&[p])).await?;
wait_for_routes(&server, 1).await?;
first.announce(announcement(&[p], &[65001, 64998])).await?;
server.wait_for_routes(2).await?;
first.announce(withdrawal(&[p])).await?;
server.wait_for_routes(1).await?;
let mut text = String::new();
let mut events: Vec<Value> = Vec::new();
@@ -223,8 +194,8 @@ async fn test_received_routes_are_returned_up_to_a_limit() -> Result<()> {
let prefixes: Vec<IpPrefix> = (0..40).map(|n| prefix(&format!("10.{n}.0.0/16"))).collect();
let mut peer = peer(&server, 1).await?;
peer.send(&announcement(&prefixes, &[65001])).await?;
wait_for_routes(&server, 40).await?;
peer.announce(announcement(&prefixes, &[65001])).await?;
server.wait_for_routes(40).await?;
let count = |page: Value| page["routes"].as_array().unwrap().len();
let url = format!("{base}/api/routes/received");
crates/integration-tests/tests/blackhole.rs
+141
View File
@@ -0,0 +1,141 @@
//! Blackhole requests (RFC 7999): refused by default, and with
//! `accept_blackholes` taken for one address inside what the peer may
//! announce, whatever their next hop.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_server::policy::{ImportPolicy, PrefixSet};
use eyre::Result;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
const BLACKHOLE: (u16, u16) = (0xFFFF, 0x029A);
async fn server_with(import: ImportPolicy) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(50);
peer.import = import;
})
.await
}
/// Waits until the server has taken `taken` routes of the session's and
/// refused `refused`, and returns the prefixes in its table.
async fn table_after(server: &TestServer, taken: u64, refused: u64) -> Result<BTreeSet<String>> {
let counters = server.registry.list()[0].counters.clone();
wait_until("the routes are dealt with", || {
(counters.routes(), counters.rejected()) == (taken, refused)
})
.await
.map_err(|_| {
eyre::eyre!(
"the server took {} routes and refused {}; expected {taken} and {refused}",
counters.routes(),
counters.rejected()
)
})?;
Ok(server
.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect())
}
/// The policy that lets the peer announce 193.0.0.0/21 and nothing
/// longer than a /24 inside it, as a customer's would.
fn customer_policy(accept_blackholes: bool) -> ImportPolicy {
let mut prefixes = PrefixSet::default();
prefixes.allow(prefix("193.0.0.0/21"), 24);
ImportPolicy {
prefixes: Some(prefixes),
accept_blackholes,
..default_policy()
}
}
#[tokio::test]
async fn test_blackhole_requests_are_refused_unless_accepted() -> Result<()> {
let with_peer = server_with(customer_policy(false)).await?;
let server = &with_peer.server;
let mut peer: TestPeer = server.established(PEER_ASN, PEER_ID).await?;
peer.announce(announcement(&["193.0.0.9/32"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
peer.announce(announcement(&["193.0.0.0/24"], &[PEER_ASN]))
.await?;
assert_eq!(table_after(server, 1, 1).await?, set(&["193.0.0.0/24"]));
let refused = server.rejections.peers();
assert_eq!(refused.len(), 1);
assert_eq!(
refused[0]
.1
.by_reason()
.iter()
.map(|(reason, n)| (reason.as_str(), *n))
.collect::<Vec<_>>(),
[("blackhole request", 1)]
);
Ok(())
}
#[tokio::test]
async fn test_a_blackhole_request_is_taken_for_one_address_in_the_peers_space() -> Result<()> {
let with_peer = server_with(customer_policy(true)).await?;
let server = &with_peer.server;
let mut peer: TestPeer = server.established(PEER_ASN, PEER_ID).await?;
// One address inside the peer's prefix, from a next hop that is no
// neighbor and would not pass as an ordinary route's.
peer.announce(
announcement(&["193.0.0.9/32"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(172, 31, 0, 9))
.with_communities(&[BLACKHOLE]),
)
.await?;
// More than one address: not a blackhole request that is taken.
peer.announce(announcement(&["193.0.0.0/25"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
// One address outside the peer's space.
peer.announce(announcement(&["193.0.8.9/32"], &[PEER_ASN]).with_communities(&[BLACKHOLE]))
.await?;
// And an ordinary route, taken as ever.
peer.announce(announcement(&["193.0.0.0/24"], &[PEER_ASN]))
.await?;
assert_eq!(
table_after(server, 2, 2).await?,
set(&["193.0.0.0/24", "193.0.0.9/32"])
);
let refused = server.rejections.peers();
let mut reasons: Vec<(&str, u64)> = refused[0]
.1
.by_reason()
.iter()
.map(|(reason, n)| (reason.as_str(), *n))
.collect();
reasons.sort();
assert_eq!(
reasons,
[
("blackhole for more than one address", 1),
("prefix not allowed", 1)
]
);
// The route is in the table with its request on it, for the kernel
// and for whoever it goes on to.
let routes = server.received_routes();
let blackhole = routes
.iter()
.find(|route| route.prefix == prefix("193.0.0.9/32"))
.expect("the blackhole route is in the table");
assert!(ImportPolicy::is_blackhole_request(&blackhole.attributes));
Ok(())
}
fn set(prefixes: &[&str]) -> BTreeSet<String> {
prefixes.iter().map(|p| p.to_string()).collect()
}
crates/integration-tests/tests/collision.rs
+45 -109
View File
@@ -3,52 +3,33 @@
//! The test server's BGP Identifier is 127.0.0.1. A test peer picks an
//! identifier above or below that to steer which connection must survive.
use std::net::{Ipv4Addr, SocketAddr};
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, NotificationMessage, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_packet::message::{Message, NotificationMessage};
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::collision::Direction;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
/// Above the server's 127.0.0.1.
const HIGH_ID: Ipv4Addr = Ipv4Addr::new(192, 0, 2, 1);
/// Below the server's 127.0.0.1.
const LOW_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
async fn server_dialing(listener: &TcpListener) -> Result<TestServer> {
let endpoint: SocketAddr = listener.local_addr()?;
TestServer::start_with(LOCAL_ASN, |c| {
/// A server that dials its peer "r1" every 200 ms until it has a session.
async fn server_dialing() -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
}];
peer.import = ImportPolicy::permissive();
})
.await
}
@@ -63,27 +44,13 @@ async fn open_only(peer: &mut TestPeer, router_id: Ipv4Addr) -> Result<()> {
/// Finishes the handshake begun by [`open_only`].
async fn confirm(peer: &mut TestPeer) -> Result<()> {
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
match peer.recv().await? {
Message::KeepAlive => {}
other => eyre::bail!("expected KEEPALIVE, got {other:?}"),
}
peer.send(&Message::KeepAlive).await
}
/// Reads until a NOTIFICATION arrives, skipping KEEPALIVEs.
async fn expect_notification(peer: &mut TestPeer) -> Result<NotificationMessage> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok(n),
Message::KeepAlive => {}
other => eyre::bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
fn is_collision_cease(n: &NotificationMessage) -> bool {
matches!(
n.error,
@@ -91,38 +58,26 @@ fn is_collision_cease(n: &NotificationMessage) -> bool {
)
}
async fn wait_for_sessions(server: &TestServer, n: usize) -> Result<()> {
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().len() != n {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(())
}
/// Peer has the higher identifier, so the connection the *peer* initiated
/// must survive: the server closes its own outbound connection.
#[tokio::test]
async fn test_simultaneous_open_peer_higher_keeps_inbound() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
let ServerWithPeer { server, listener } = server_dialing().await?;
// Server's outbound connection reaches OpenConfirm.
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
let mut dialled = TestPeer::accept(&listener).await?;
open_only(&mut dialled, HIGH_ID).await?;
// The peer now also dials the server.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, HIGH_ID).await?;
let notif = expect_notification(&mut dialled).await?;
let notif = dialled.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(dialled.is_closed().await);
assert!(dialled.closed_within(TIMEOUT).await);
confirm(&mut inbound).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
let session = server.registry.list().remove(0);
assert_eq!(session.direction, Direction::Inbound);
assert_eq!(session.name.as_deref(), Some("r1"));
@@ -131,11 +86,11 @@ async fn test_simultaneous_open_peer_higher_keeps_inbound() -> Result<()> {
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, inbound.recv_update()).await??;
let update = inbound.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
// The dialer notices the surviving session and does not open another.
@@ -151,22 +106,20 @@ async fn test_simultaneous_open_peer_higher_keeps_inbound() -> Result<()> {
/// must survive: the peer's inbound connection is refused.
#[tokio::test]
async fn test_simultaneous_open_peer_lower_keeps_outbound() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
let ServerWithPeer { server, listener } = server_dialing().await?;
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
let mut dialled = TestPeer::accept(&listener).await?;
open_only(&mut dialled, LOW_ID).await?;
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
let notif = expect_notification(&mut inbound).await?;
let notif = inbound.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
assert!(inbound.is_closed().await);
assert!(inbound.closed_within(TIMEOUT).await);
confirm(&mut dialled).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
@@ -175,22 +128,20 @@ async fn test_simultaneous_open_peer_lower_keeps_outbound() -> Result<()> {
/// outcome: here the inbound connection reaches OpenConfirm first.
#[tokio::test]
async fn test_outcome_is_independent_of_arrival_order() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener).await?;
let ServerWithPeer { server, listener } = server_dialing().await?;
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let mut dialled = TestPeer::from_stream(stream);
let mut dialled = TestPeer::accept(&listener).await?;
// Inbound first this time; peer is lower so outbound must still win.
let mut inbound = TestPeer::connect(server.addr).await?;
open_only(&mut inbound, LOW_ID).await?;
open_only(&mut dialled, LOW_ID).await?;
let notif = expect_notification(&mut inbound).await?;
let notif = inbound.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
confirm(&mut dialled).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
assert_eq!(server.registry.list()[0].direction, Direction::Outbound);
Ok(())
}
@@ -203,11 +154,11 @@ async fn test_established_session_wins_by_default() -> Result<()> {
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
let mut second = TestPeer::connect(server.addr).await?;
open_only(&mut second, HIGH_ID).await?;
let notif = expect_notification(&mut second).await?;
let notif = second.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
// The original is untouched.
@@ -215,11 +166,11 @@ async fn test_established_session_wins_by_default() -> Result<()> {
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, first.recv_update()).await??;
let update = first.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
@@ -234,41 +185,26 @@ async fn test_restarted_peer_replaces_stale_session_when_enabled() -> Result<()>
let mut stale = TestPeer::connect(server.addr).await?;
stale.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
stale
.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 9))),
],
announced_routes: vec![IpPrefix::try_from("203.0.113.0/24")?.into()],
}))
.announce(
announcement(&["203.0.113.0/24"], &[PEER_ASN])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 9)),
)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.received_routes().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
server.wait_for_routes(1).await?;
let stale_addr = server.registry.list()[0].peer.addr;
// "Restart": a fresh connection from the same router.
let mut fresh = TestPeer::connect(server.addr).await?;
fresh.handshake(PEER_ASN, HIGH_ID, 90).await?;
let notif = expect_notification(&mut stale).await?;
let notif = stale.expect_notification().await?;
assert!(is_collision_cease(&notif), "{notif:?}");
tokio::time::timeout(TIMEOUT, async {
loop {
let sessions = server.registry.list();
if sessions.len() == 1 && sessions[0].peer.addr != stale_addr {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
wait_until("the fresh session has replaced the stale one", || {
let sessions = server.registry.list();
sessions.len() == 1 && sessions[0].peer.addr != stale_addr
})
.await?;
assert!(
@@ -286,7 +222,7 @@ async fn test_distinct_identifiers_do_not_collide() -> Result<()> {
a.handshake(PEER_ASN, HIGH_ID, 90).await?;
let mut b = TestPeer::connect(server.addr).await?;
b.handshake(PEER_ASN, LOW_ID, 90).await?;
wait_for_sessions(&server, 2).await?;
server.wait_for_sessions(2).await?;
Ok(())
}
@@ -296,13 +232,13 @@ async fn test_identifier_is_released_when_connection_ends() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut first = TestPeer::connect(server.addr).await?;
first.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
drop(first);
wait_for_sessions(&server, 0).await?;
server.wait_for_sessions(0).await?;
let mut second = TestPeer::connect(server.addr).await?;
second.handshake(PEER_ASN, HIGH_ID, 90).await?;
wait_for_sessions(&server, 1).await?;
server.wait_for_sessions(1).await?;
Ok(())
}
@@ -312,7 +248,7 @@ async fn test_peer_with_our_identifier_is_refused() -> Result<()> {
let server = TestServer::start(LOCAL_ASN).await?;
let mut peer = TestPeer::connect(server.addr).await?;
open_only(&mut peer, Ipv4Addr::new(127, 0, 0, 1)).await?;
let notif = expect_notification(&mut peer).await?;
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadBgpId)),
"{notif:?}"
crates/integration-tests/tests/debugging.rs
+29 -79
View File
@@ -2,39 +2,13 @@
//! what was refused.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::default_policy;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, wait_until};
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use serde_json::Value;
use tokio::net::TcpListener;
const TIMEOUT: Duration = Duration::from_secs(5);
fn announcement(prefixes: &[&str], as_path: &[u32]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(as_path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: prefixes
.iter()
.map(|p| IpPrefix::try_from(*p).unwrap().into())
.collect(),
})
}
/// The routes a query returns, as "prefix from AS, best or not".
async fn routes(client: &reqwest::Client, base: &str, query: &str) -> Result<Vec<String>> {
@@ -67,24 +41,21 @@ async fn test_routes_can_be_looked_up() -> Result<()> {
let (base, _http) = server.serve_http().await?;
let client = reqwest::Client::new();
let mut one = TestPeer::connect(server.addr).await?;
one.handshake(65001, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
let mut two = TestPeer::connect(server.addr).await?;
two.handshake(65002, Ipv4Addr::new(10, 0, 0, 2), 90).await?;
one.send(&announcement(
let mut one = server
.established(65001, Ipv4Addr::new(10, 0, 0, 1))
.await?;
let mut two = server
.established(65002, Ipv4Addr::new(10, 0, 0, 2))
.await?;
one.announce(announcement(
&["10.0.0.0/8", "10.1.0.0/16", "10.1.2.0/24", "192.0.2.0/24"],
&[65001],
))
.await?;
// A longer path for one of the same prefixes: it loses.
two.send(&announcement(&["10.1.0.0/16"], &[65002, 65003]))
two.announce(announcement(&["10.1.0.0/16"], &[65002, 65003]))
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != 5 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
server.wait_for_routes(5).await?;
// One prefix: every route for it, the best first.
assert_eq!(
@@ -162,27 +133,15 @@ async fn test_routes_can_be_looked_up() -> Result<()> {
/// something to alert on.
#[tokio::test]
async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(1299, |c| {
c.accept_unconfigured_peers = false;
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "refused-demo".into(),
endpoint: endpoint.into(),
remote_asn: 3333,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy {
max_prefixes: Some(100),
..default_policy()
},
role: None,
require_role: false,
export: Default::default(),
}];
let ServerWithPeer {
server,
listener: _listener,
} = TestServer::with_peer(1299, 3333, |_, peer| {
peer.name = "refused-demo".into();
peer.import = ImportPolicy {
max_prefixes: Some(100),
..default_policy()
};
})
.await?;
let (base, _http) = server.serve_http().await?;
@@ -195,27 +154,23 @@ async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
let refused: Value = get("/api/routes/rejected").await?.json().await?;
assert_eq!(refused, serde_json::json!([]), "nothing refused yet");
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(3333, Ipv4Addr::new(10, 0, 0, 2), 90).await?;
peer.send(&announcement(&["193.0.0.0/21", "10.0.0.0/8"], &[3333]))
let mut peer = server.established(3333, Ipv4Addr::new(10, 0, 0, 2)).await?;
peer.announce(announcement(&["193.0.0.0/21", "10.0.0.0/8"], &[3333]))
.await?;
peer.send(&announcement(
peer.announce(announcement(
&["194.0.1.0/24", "194.0.2.0/24"],
&[3333, 64512],
))
.await?;
peer.send(&announcement(&["193.0.9.0/25"], &[3333, 2121]))
peer.announce(announcement(&["193.0.9.0/25"], &[3333, 2121]))
.await?;
tokio::time::timeout(TIMEOUT, async {
while server
wait_until("four routes are refused", || {
server
.registry
.list()
.first()
.map(|s| s.counters.rejected())
!= Some(4)
{
tokio::time::sleep(Duration::from_millis(5)).await;
}
== Some(4)
})
.await?;
@@ -297,12 +252,7 @@ async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
// The record outlives the session; the count of routes does not.
drop(peer);
tokio::time::timeout(TIMEOUT, async {
while !server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
server.wait_for_sessions(0).await?;
let refused: Value = get("/api/routes/rejected").await?.json().await?;
assert_eq!(refused[0]["total"], 4);
let metrics = get("/metricz").await?.text().await?;
crates/integration-tests/tests/dialer.rs
+109 -165
View File
@@ -5,74 +5,47 @@ use std::net::{Ipv4Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, peer_config, prefix, wait_for, wait_until};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::Message;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::ImportPolicy;
use bgp_server::session::DialState;
use eyre::Result;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// Starts a server configured to dial `listener`'s address as peer "r1".
async fn server_dialing(listener: &TcpListener, remote_asn: u32) -> Result<TestServer> {
let endpoint: SocketAddr = listener.local_addr()?;
TestServer::start_with(LOCAL_ASN, |c| {
/// A server that dials its peer "r1", in `remote_asn`, every 200 ms.
async fn server_dialing(remote_asn: u32) -> Result<ServerWithPeer> {
server_with_peer(remote_asn, false).await
}
/// A server with peer "r1" in `remote_asn`, which it dials every 200 ms
/// unless the peer is `passive`.
async fn server_with_peer(remote_asn: u32, passive: bool) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, remote_asn, |c, peer| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
}];
peer.import = ImportPolicy::permissive();
peer.passive = passive;
})
.await
}
async fn accept(listener: &TcpListener) -> Result<TestPeer> {
let (stream, _) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
Ok(TestPeer::from_stream(stream))
}
#[tokio::test]
async fn test_server_dials_configured_peer_and_establishes() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut peer = accept(&listener).await?;
let mut peer = TestPeer::accept(&listener).await?;
// Both sides send OPEN on connect; the test peer's handshake order works
// whichever side opened TCP.
let open = peer
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let open = peer.handshake(PEER_ASN, PEER_ID, 90).await?;
assert_eq!(open.asn, LOCAL_ASN as u16);
let mut events = server.registry.subscribe();
// Either the Up event is still queued or the session is already listed.
let listed = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(s) = server.registry.list().first().cloned() {
return s;
}
let _ = events.recv().await;
}
})
.await?;
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
assert_eq!(listed.peer.asn, PEER_ASN);
assert_eq!(listed.peer.addr, listener.local_addr()?);
@@ -94,39 +67,26 @@ async fn test_server_dials_configured_peer_and_establishes() -> Result<()> {
server
.local_rib
.announce(LocalRoute::new(
IpPrefix::try_from("10.100.1.0/24")?,
prefix("10.100.1.0/24"),
LocalRouteAttrs::default(),
))
.await?;
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes[0].to_string(), "10.100.1.0/24");
Ok(())
}
#[tokio::test]
async fn test_server_reconnects_after_peer_closes() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut first = accept(&listener).await?;
first
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let mut first = TestPeer::accept(&listener).await?;
first.handshake(PEER_ASN, PEER_ID, 90).await?;
server.wait_for_session().await?;
drop(first);
// Session goes away, then the dialer comes back.
tokio::time::timeout(TIMEOUT, async {
while !server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
server.wait_for_sessions(0).await?;
let (_, state) = server
.registry
.configured_peers()
@@ -135,45 +95,30 @@ async fn test_server_reconnects_after_peer_closes() -> Result<()> {
.unwrap();
assert!(matches!(state, DialState::Backoff { .. }), "{state:?}");
let mut second = accept(&listener).await?;
second
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let mut second = TestPeer::accept(&listener).await?;
second.handshake(PEER_ASN, PEER_ID, 90).await?;
server.wait_for_session().await?;
Ok(())
}
#[tokio::test]
async fn test_wrong_asn_is_refused_with_bad_peer_as() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let _server = server_dialing(&listener, PEER_ASN).await?;
let ServerWithPeer {
server: _server,
listener,
} = server_dialing(PEER_ASN).await?;
let mut peer = accept(&listener).await?;
peer.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let mut peer = TestPeer::accept(&listener).await?;
peer.send_open(65099, PEER_ID, 90).await?;
let _ = peer.recv_open().await?;
let notif = tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::Notification(n) => return Ok::<_, eyre::Error>(n),
Message::KeepAlive => {}
other => eyre::bail!("unexpected {other:?}"),
}
}
})
.await??;
let notif = peer.expect_notification().await?;
assert!(
matches!(notif.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{notif:?}"
);
// The dialer retries rather than giving up.
let _again = accept(&listener).await?;
let _again = TestPeer::accept(&listener).await?;
Ok(())
}
@@ -185,28 +130,19 @@ async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
// so the dialer never succeeds; then connect inbound from loopback.
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_secs(60);
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
..peer_config("r1", at, PEER_ASN)
}];
})
.await?;
// Wrong ASN inbound → refused.
let mut bad = TestPeer::connect(server.addr).await?;
bad.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90).await?;
bad.send_open(65099, PEER_ID, 90).await?;
let _ = bad.recv_open().await?;
let msg = tokio::time::timeout(TIMEOUT, bad.recv()).await??;
let msg = bad.recv().await?;
assert!(
matches!(&msg, Message::Notification(n) if matches!(n.error, BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs))),
"{msg:?}"
@@ -214,17 +150,8 @@ async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
// Right ASN inbound → established, named.
let mut good = TestPeer::connect(server.addr).await?;
good.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let listed = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(s) = server.registry.list().first().cloned() {
return s;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
good.handshake(PEER_ASN, PEER_ID, 90).await?;
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
Ok(())
}
@@ -234,56 +161,33 @@ async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
/// reported as the reason for backing off rather than silently ignored.
#[tokio::test]
async fn test_local_address_is_used_as_the_source() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint: SocketAddr = listener.local_addr()?;
let _server = TestServer::start_with(LOCAL_ASN, |c| {
let ServerWithPeer {
server: _server,
listener,
} = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: Some(Ipv4Addr::LOCALHOST.into()),
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
}];
peer.import = ImportPolicy::permissive();
peer.local_address = Some(Ipv4Addr::LOCALHOST.into());
})
.await?;
let (_stream, from) = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
assert_eq!(from.ip(), std::net::IpAddr::from(Ipv4Addr::LOCALHOST));
// 192.0.2.1 is documentation space; no host has it.
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.connect_retry = Duration::from_millis(200);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: Some(Ipv4Addr::new(192, 0, 2, 1).into()),
families: FamilySet::supported(),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
..peer_config("r1", endpoint, PEER_ASN)
}];
})
.await?;
let reason = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some((_, DialState::Backoff { last_error, .. })) =
server.registry.configured_peers().into_iter().next()
{
return last_error;
}
tokio::time::sleep(Duration::from_millis(20)).await;
let reason = wait_for("the dialer is backing off", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { last_error, .. })) => Some(last_error),
_ => None,
}
})
.await?;
@@ -298,16 +202,14 @@ async fn test_local_address_is_used_as_the_source() -> Result<()> {
/// such a peer was dialled every few seconds for ever.
#[tokio::test]
async fn test_a_peer_that_refuses_the_session_is_dialled_ever_more_slowly() -> Result<()> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let server = server_dialing(&listener, PEER_ASN).await?;
let ServerWithPeer { server, listener } = server_dialing(PEER_ASN).await?;
let mut waits = Vec::new();
for _ in 0..3 {
let mut peer = accept(&listener).await?;
peer.send_open(65099, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let mut peer = TestPeer::accept(&listener).await?;
peer.send_open(65099, PEER_ID, 90).await?;
let _ = peer.recv_open().await?;
let refusal = tokio::time::timeout(TIMEOUT, peer.recv()).await??;
let refusal = peer.recv().await?;
assert!(
matches!(&refusal, Message::Notification(n) if n.error == BgpError::OpenMsg(OpenMsgSubcode::BadPeerAs)),
"{refusal:?}"
@@ -315,17 +217,17 @@ async fn test_a_peer_that_refuses_the_session_is_dialled_ever_more_slowly() -> R
drop(peer);
// How long the dialer says it will wait before trying again.
let wait = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some((_, DialState::Backoff { until, last_error })) =
server.registry.configured_peers().into_iter().next()
{
let wait = wait_for("the dialer is backing off", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { until, last_error })) => {
assert!(last_error.contains("AS65099"), "{last_error}");
return until
.duration_since(std::time::SystemTime::now())
.unwrap_or_default();
Some(
until
.duration_since(std::time::SystemTime::now())
.unwrap_or_default(),
)
}
tokio::time::sleep(Duration::from_millis(5)).await;
_ => None,
}
})
.await?;
@@ -338,3 +240,45 @@ async fn test_a_peer_that_refuses_the_session_is_dialled_ever_more_slowly() -> R
);
Ok(())
}
/// A passive peer is not dialled: its session comes up when it connects,
/// and until then, and after, it is shown as waiting, with how its last
/// session ended.
#[tokio::test]
async fn test_a_passive_peer_is_not_dialled_and_comes_up_when_it_connects() -> Result<()> {
let ServerWithPeer { server, listener } = server_with_peer(PEER_ASN, true).await?;
let state = || server.registry.configured_peers()[0].1.clone();
// Many retry intervals pass, and nothing connects.
assert!(
tokio::time::timeout(Duration::from_secs(1), listener.accept())
.await
.is_err(),
"a passive peer was dialled"
);
assert_eq!(state(), DialState::Passive { last_error: None });
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, PEER_ID, 90).await?;
let listed = server.wait_for_session().await?;
assert_eq!(listed.name.as_deref(), Some("r1"));
drop(peer);
wait_until("the peer is shown as waiting, with how it ended", || {
matches!(
state(),
DialState::Passive {
last_error: Some(_)
}
)
})
.await?;
assert!(server.registry.list().is_empty());
assert!(
tokio::time::timeout(Duration::from_millis(600), listener.accept())
.await
.is_err(),
"a passive peer was dialled after its session ended"
);
Ok(())
}
crates/integration-tests/tests/export_limits.rs
+147
View File
@@ -0,0 +1,147 @@
//! Export limits, end to end: what a peer above or beside us is told is
//! held to the prefixes they list and to as many routes as they allow,
//! whatever else would send it more.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
use bgp_packet::open::{BgpRole, Capability};
use bgp_primitives::constants::AfiSafi;
use bgp_rib::LocalRoute;
use bgp_server::policy::{ExportLimits, PeerRole, PrefixSet};
use eyre::{Result, bail};
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server that originates `originated`, with one configured peer at the
/// address the test connects from, which is a peer of ours held to
/// `limits`.
async fn server_with(originated: &[&str], limits: ExportLimits) -> Result<ServerWithPeer> {
let originate = originated
.iter()
.map(|p| LocalRoute::new(prefix(p), Default::default()))
.collect();
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.originate = originate;
peer.export_limits = Some(Arc::new(limits));
peer.role = Some(PeerRole::Peer);
})
.await
}
/// A session with the server, as its peer.
async fn established(server: &TestServer) -> Result<TestPeer> {
let capabilities = vec![
Capability::FourByteAsn { asn: PEER_ASN },
Capability::multiprotocol(AfiSafi::IPV4_UNICAST),
Capability::multiprotocol(AfiSafi::IPV6_UNICAST),
Capability::BgpRole {
role: BgpRole::Peer,
},
];
server
.established_with(PEER_ASN, PEER_ID, capabilities)
.await
}
/// What the peer has been told once it has been quiet for a while: the
/// prefixes it holds, and the NOTIFICATION that ended the session, if one
/// did.
async fn told(peer: &mut TestPeer) -> (BTreeSet<String>, Option<(BgpError, Vec<u8>)>) {
let mut routes = BTreeSet::new();
while let Ok(message) = peer.recv_within(Duration::from_millis(500)).await {
match message {
Message::Update(update) => {
for route in &update.withdrawn_routes {
routes.remove(&route.prefix.to_string());
}
for route in &update.announced_routes {
routes.insert(route.prefix.to_string());
}
}
Message::Notification(notification) => {
return (routes, Some((notification.error, notification.data)));
}
_ => {}
}
}
(routes, None)
}
fn allowing(prefixes: &[&str]) -> Option<PrefixSet> {
let mut set = PrefixSet::default();
for p in prefixes {
let p = prefix(p);
set.allow(p, p.length());
}
Some(set)
}
/// A route of ours whose prefix is not in the peer's export limits is not
/// announced to it; the one that is, is.
#[tokio::test]
async fn test_a_route_outside_the_export_limits_is_not_announced() -> Result<()> {
let limits = ExportLimits {
prefixes: allowing(&["193.0.0.0/21"]),
max_prefixes: None,
};
let ServerWithPeer { server, .. } =
server_with(&["193.0.0.0/21", "198.51.100.0/24"], limits).await?;
let mut peer = established(&server).await?;
let (routes, ended) = told(&mut peer).await;
assert_eq!(routes, BTreeSet::from(["193.0.0.0/21".to_string()]));
assert_eq!(ended, None);
Ok(())
}
/// One route more than the export limit allows is not announced: the
/// session ends with an Administrative Shutdown that says why (RFC 9003),
/// and the peer is kept down, so that it is told nothing rather than too
/// much.
#[tokio::test]
async fn test_going_over_the_export_limit_ends_the_session_and_keeps_the_peer_down() -> Result<()> {
let limits = ExportLimits {
prefixes: None,
max_prefixes: Some(2),
};
let ServerWithPeer { server, .. } =
server_with(&["193.0.0.0/21", "193.0.8.0/21"], limits).await?;
let mut peer = established(&server).await?;
let (routes, ended) = told(&mut peer).await;
assert_eq!(routes.len(), 2);
assert_eq!(ended, None);
server
.local_rib
.announce(LocalRoute::new(prefix("193.0.16.0/21"), Default::default()))
.await?;
let (more, ended) = told(&mut peer).await;
assert_eq!(more, BTreeSet::new(), "nothing more is announced");
let Some((error, data)) = ended else {
bail!("the session was not ended");
};
assert_eq!(error, BgpError::Cease(CeaseSubcode::AdminShutdown));
let text = String::from_utf8(data[1..].to_vec())?;
assert!(
text.starts_with("export limit of 2 routes reached"),
"{text}"
);
assert!(peer.closed_within(TIMEOUT).await);
// The peer is kept down: a new connection is closed unanswered.
let mut again = TestPeer::connect(server.addr).await?;
assert!(
again.closed_unanswered().await,
"a held-down peer gets no session"
);
Ok(())
}
crates/integration-tests/tests/failure.rs
+6 -34
View File
@@ -6,52 +6,24 @@
//! stops with an error, for whatever supervises it to restart it.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::TIMEOUT;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::Message;
use bgp_rib::RibStopped;
use bgp_server::error::ServerError;
use eyre::{Result, bail};
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
async fn established(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(peer)
}
/// Reads until the server's NOTIFICATION, skipping KEEPALIVEs.
async fn farewell(peer: &mut TestPeer) -> Result<BgpError> {
tokio::time::timeout(TIMEOUT, async {
loop {
match peer.recv().await? {
Message::KeepAlive => {}
Message::Notification(n) => return Ok(n.error),
other => bail!("expected NOTIFICATION, got {other:?}"),
}
}
})
.await?
}
#[tokio::test]
async fn test_losing_the_local_rib_stops_the_server_and_tells_the_peers() -> Result<()> {
let mut server = TestServer::start(LOCAL_ASN).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
let mut peer = server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1))
.await?;
server.local_rib.stop_for_testing();
@@ -61,7 +33,7 @@ async fn test_losing_the_local_rib_stops_the_server_and_tells_the_peers() -> Res
"{outcome:?}"
);
assert_eq!(
farewell(&mut peer).await?,
peer.expect_notification().await?.error,
BgpError::Cease(CeaseSubcode::AdminShutdown)
);
crates/integration-tests/tests/graceful_restart.rs
+15 -42
View File
@@ -4,20 +4,17 @@
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::{Announcement, announcement};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, wait_until as until};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, NotificationMessage, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::message::{Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{Capability, GracefulRestartEntry};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::restart::Leave;
use eyre::{Result, bail};
const TIMEOUT: Duration = Duration::from_secs(5);
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
const PREFIXES: [&str; 3] = ["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"];
@@ -57,37 +54,13 @@ fn capabilities(more: impl IntoIterator<Item = Capability>) -> Vec<Capability> {
capabilities
}
fn announcement(prefixes: &[&str]) -> Message {
Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: prefixes
.iter()
.map(|p| IpPrefix::try_from(*p).unwrap().into())
.collect(),
})
/// The peer's own routes.
fn own(prefixes: &[&str]) -> Announcement {
announcement(prefixes, &[PEER_ASN])
}
fn end_of_rib(family: AfiSafi) -> Message {
Message::Update(UpdateMessage::end_of_rib(family))
}
/// Waits until `done` holds, or fails with `what`.
async fn until(what: &str, mut done: impl FnMut() -> bool) -> Result<()> {
let waited = tokio::time::timeout(TIMEOUT, async {
while !done() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await;
if waited.is_err() {
bail!("timed out waiting until {what}");
}
Ok(())
fn end_of_rib(family: AfiSafi) -> UpdateMessage {
UpdateMessage::end_of_rib(family)
}
/// The prefixes in the server's table, and how many of its routes are
@@ -109,7 +82,7 @@ async fn peer_with_routes(server: &TestServer, offer: Capability) -> Result<Test
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities([offer]))
.await?;
peer.send(&announcement(&PREFIXES)).await?;
peer.announce(own(&PREFIXES)).await?;
until("the routes are in", || server.rib.len() == 3).await?;
Ok(peer)
}
@@ -155,12 +128,12 @@ async fn test_a_restarting_peer_keeps_its_routes_until_it_has_sent_them_again()
server.registry.list().len() == 1 && server.restarts.kept().is_empty()
})
.await?;
peer.send(&announcement(&PREFIXES[..2])).await?;
peer.announce(own(&PREFIXES[..2])).await?;
until("two are fresh", || table(&server).1 == 1).await?;
// The peer has them all, as far as its limit goes.
assert_eq!(server.registry.list()[0].counters.routes(), 3);
peer.send(&end_of_rib(AfiSafi::IPV4_UNICAST)).await?;
peer.announce(end_of_rib(AfiSafi::IPV4_UNICAST)).await?;
until("the one not sent again is gone", || {
table(&server) == (vec!["10.1.0.0/16".into(), "10.2.0.0/16".into()], 0)
})
@@ -213,7 +186,7 @@ async fn test_only_a_restart_keeps_the_routes() -> Result<()> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(offer))
.await?;
peer.send(&announcement(&PREFIXES)).await?;
peer.announce(own(&PREFIXES)).await?;
until("the routes are in", || server.rib.len() == 3).await?;
if let Some(goodbye) = &goodbye {
peer.send(&Message::Notification(goodbye.clone())).await?;
@@ -310,7 +283,7 @@ async fn test_we_offer_graceful_restart_and_say_when_our_routes_are_all_sent() -
// Nothing is originated: the markers are all there is.
let mut markers = Vec::new();
while markers.len() < 2 {
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
match update.end_of_rib_family() {
Some(family) => markers.push(family),
None => bail!("expected End-of-RIB, got {update:?}"),
@@ -346,14 +319,14 @@ async fn test_leaving_tells_each_peer_as_graceful_restart_has_it() -> Result<()>
server.registry.leave(leave, Duration::from_secs(2)).await;
let mut notification = None;
// Whatever came before (End-of-RIB markers) is passed over.
while let Ok(Ok(message)) = tokio::time::timeout(TIMEOUT, peer.recv()).await {
while let Ok(message) = peer.recv().await {
if let Message::Notification(n) = message {
notification = Some(n.error);
break;
}
}
assert_eq!(notification, told, "{leave:?} with {offer:?}");
assert!(peer.is_closed().await);
assert!(peer.closed_within(TIMEOUT).await);
}
Ok(())
}
crates/integration-tests/tests/http_api.rs
+11 -40
View File
@@ -3,46 +3,22 @@
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, LargeCommunity, Message, NextHopPathAttribute,
OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_integration_tests::{TIMEOUT, get_json, prefix};
use bgp_packet::message::LargeCommunity;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
use serde_json::{Value, json};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
async fn established_peer(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
Ok(peer)
}
async fn get_json(client: &reqwest::Client, url: &str) -> Result<Value> {
Ok(client
.get(url)
.send()
.await?
.error_for_status()?
.json()
.await?)
server
.established(PEER_ASN, Ipv4Addr::new(10, 0, 0, 1))
.await
}
#[tokio::test]
@@ -155,16 +131,11 @@ async fn test_received_routes_summary() -> Result<()> {
let client = reqwest::Client::new();
let mut peer = established_peer(&server).await?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN, 65002]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(10, 0, 0, 1))),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(65001, 7)])),
],
announced_routes: vec![prefix("203.0.113.0/24").into()],
}))
peer.announce(
announcement(&["203.0.113.0/24"], &[PEER_ASN, 65002])
.with_next_hop(Ipv4Addr::new(10, 0, 0, 1))
.with_communities(&[(65001, 7)]),
)
.await?;
let received = tokio::time::timeout(TIMEOUT, async {
crates/integration-tests/tests/import_policy.rs
+215 -104
View File
@@ -5,95 +5,51 @@ use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::default_policy;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_for, wait_until};
use bgp_packet::errors::{BgpError, CeaseSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, PathAttribute,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::{ImportPolicy, PrefixSet};
use bgp_server::marks::{Marks, Patterns, Scrub};
use bgp_server::policy::{ImportPolicy, ImportRule, PrefixSet};
use bgp_server::session::DialState;
use eyre::Result;
use tokio::net::TcpListener;
/// AS numbers as they are on the Internet: the checks refuse the private
/// ones the other tests use.
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server with one configured peer, at the address the test connects
/// from, whose routes are judged by `import`. Returns the listener that
/// keeps the server's own dialling occupied.
async fn server_with(import: ImportPolicy) -> Result<(TestServer, TcpListener)> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
/// from, whose routes are judged by `import`. The listener that keeps the
/// server's own dialling occupied comes with it.
async fn server_with(import: ImportPolicy) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.connect_retry = Duration::from_millis(50);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import,
role: None,
require_role: false,
export: Default::default(),
}];
peer.import = import;
})
.await?;
Ok((server, listener))
.await
}
async fn established(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(peer)
server.established(PEER_ASN, PEER_ID).await
}
/// Announces IPv4 `prefixes` with the AS path `path`.
async fn announce(peer: &mut TestPeer, prefixes: &[&str], path: &[u32]) -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
}))
.await
peer.announce(announcement(prefixes, path)).await
}
/// Waits until the server has dealt with `count` announced routes of the
/// session's, taken or refused, and returns the prefixes in its table.
async fn table_after(server: &TestServer, taken: u64, refused: u64) -> Result<BTreeSet<String>> {
let counters = server.registry.list()[0].counters.clone();
tokio::time::timeout(TIMEOUT, async {
while (counters.routes(), counters.rejected()) != (taken, refused) {
tokio::time::sleep(Duration::from_millis(5)).await;
}
wait_until("the routes are dealt with", || {
(counters.routes(), counters.rejected()) == (taken, refused)
})
.await
.map_err(|_| {
@@ -121,7 +77,10 @@ fn set(prefixes: &[&str]) -> BTreeSet<String> {
/// through us. Everything a peer sent used to be taken.
#[tokio::test]
async fn test_routes_that_no_peer_should_announce_are_refused() -> Result<()> {
let (server, _listener) = server_with(default_policy()).await?;
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
// One UPDATE with good and bad prefixes: the good ones are taken.
@@ -159,7 +118,10 @@ async fn test_routes_that_no_peer_should_announce_are_refused() -> Result<()> {
/// and the new one is not acceptable.
#[tokio::test]
async fn test_a_refused_route_replaces_the_one_that_was_taken() -> Result<()> {
let (server, _listener) = server_with(default_policy()).await?;
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.0.0/21", "193.0.16.0/24"], &[PEER_ASN]).await?;
@@ -176,7 +138,10 @@ async fn test_a_refused_route_replaces_the_one_that_was_taken() -> Result<()> {
async fn test_a_peer_is_held_to_its_prefix_list_and_origins() -> Result<()> {
let mut prefixes = PrefixSet::default();
prefixes.allow(prefix("193.0.0.0/21"), 24);
let (server, _listener) = server_with(ImportPolicy {
let ServerWithPeer {
server,
listener: _listener,
} = server_with(ImportPolicy {
prefixes: Some(prefixes),
origin_asns: Some([PEER_ASN, 2121].into()),
..default_policy()
@@ -206,7 +171,7 @@ async fn test_a_peer_is_held_to_its_prefix_list_and_origins() -> Result<()> {
/// again.
#[tokio::test]
async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
let (server, listener) = server_with(ImportPolicy {
let ServerWithPeer { server, listener } = server_with(ImportPolicy {
max_prefixes: Some(3),
hold_down: Duration::from_millis(1500),
..default_policy()
@@ -214,7 +179,7 @@ async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
.await?;
// The server dials the peer as well; that connection is left unanswered
// until the end.
let dialled = tokio::time::timeout(TIMEOUT, listener.accept()).await??;
let dialled = TestPeer::accept(&listener).await?;
let mut peer = established(&server).await?;
// Refused routes do not count towards the limit, and replacing a
@@ -229,58 +194,43 @@ async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
assert_eq!(table_after(&server, 3, 1).await?.len(), 3);
announce(&mut peer, &["193.0.3.0/24"], &[PEER_ASN]).await?;
let notification = tokio::time::timeout(TIMEOUT, async {
loop {
if let Message::Notification(notification) = peer.recv().await? {
return Ok::<_, eyre::Report>(notification);
}
}
})
.await??;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::Cease(CeaseSubcode::MaxPrefixes)
);
drop(peer);
tokio::time::timeout(TIMEOUT, async {
while !server.received_routes().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
wait_until("the peer's routes are gone", || {
server.received_routes().is_empty()
})
.await?;
// Held down: a new connection gets nothing.
let mut early = TestPeer::connect(server.addr).await?;
assert!(
matches!(
tokio::time::timeout(TIMEOUT, early.recv()).await,
Ok(Err(_))
),
early.closed_unanswered().await,
"a peer that is held down is not answered"
);
// Nor is it dialled: once the connection the server had open to it is
// gone, the dialer waits out the hold-down and says so.
drop(dialled);
let reason = tokio::time::timeout(TIMEOUT, async {
loop {
if let Some((_, DialState::Backoff { last_error, .. })) =
server.registry.configured_peers().into_iter().next()
&& last_error.contains("held down")
let reason = wait_for("the dialer says the peer is held down", || {
match server.registry.configured_peers().into_iter().next() {
Some((_, DialState::Backoff { last_error, .. }))
if last_error.contains("held down") =>
{
return last_error;
Some(last_error)
}
tokio::time::sleep(Duration::from_millis(10)).await;
_ => None,
}
})
.await?;
assert!(reason.contains("limit of routes"), "{reason}");
// When the time is up the peer is dialled again, and taken back.
tokio::time::timeout(TIMEOUT, listener.accept()).await??;
TestPeer::accept(&listener).await?;
let mut later = TestPeer::connect(server.addr).await?;
later
.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
later.handshake(PEER_ASN, PEER_ID, 90).await?;
Ok(())
}
@@ -291,7 +241,10 @@ async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()> {
use bgp_packet::message::{CommunitiesPathAttribute, LocalPrefPathAttribute};
use bgp_server::marks::Marks;
let (server, _listener) = server_with(ImportPolicy {
let ServerWithPeer {
server,
listener: _listener,
} = server_with(ImportPolicy {
marks: Marks::parse(
&["1299:*".to_string()],
&["1299:1001".to_string()],
@@ -303,16 +256,10 @@ async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()>
})
.await?;
let mut peer = established(&server).await?;
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::Communities(CommunitiesPathAttribute(vec![(3333, 5), (1299, 2002)])),
],
announced_routes: vec![prefix("193.0.0.0/21").into(), prefix("193.0.8.0/22").into()],
}))
peer.announce(
announcement(&["193.0.0.0/21", "193.0.8.0/22"], &[PEER_ASN])
.with_communities(&[(3333, 5), (1299, 2002)]),
)
.await?;
table_after(&server, 2, 0).await?;
let routes = server.received_routes();
@@ -343,3 +290,167 @@ async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()>
assert_eq!(server.rib.attrs().len(), 1, "they share their attributes");
Ok(())
}
/// The communities of our own AS are taken off what another network
/// sends, before the route is kept, and nothing else is; from a peer that
/// is to steer its routes with them (a customer), they are kept.
#[tokio::test]
async fn test_our_communities_are_not_taken_from_other_networks() -> Result<()> {
let ours_and_theirs = || {
vec![
PathAttribute::Communities(CommunitiesPathAttribute(vec![
(0, 13335),
(3356, 3),
(1299, 5),
])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![large(1299, 0, 0), large(1299, 31, 2), large(3356, 1, 2)],
}),
]
};
let kept = |server: &TestServer| -> Vec<PathAttribute> {
let routes = server.received_routes();
routes[0]
.attributes
.iter()
.filter(|attribute| {
matches!(
attribute,
PathAttribute::Communities(_) | PathAttribute::LargeCommunities(_)
)
})
.cloned()
.collect()
};
let scrubbed = ImportPolicy {
scrub: Some(Scrub {
local_asn: LOCAL_ASN,
zero: true,
}),
..default_policy()
};
let ServerWithPeer {
server,
listener: _listener,
} = server_with(scrubbed).await?;
let mut peer = established(&server).await?;
announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
table_after(&server, 1, 0).await?;
assert_eq!(
kept(&server),
[
PathAttribute::Communities(CommunitiesPathAttribute(vec![(3356, 3)])),
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: vec![large(3356, 1, 2)],
}),
]
);
let ServerWithPeer {
server,
listener: _listener,
} = server_with(default_policy()).await?;
let mut peer = established(&server).await?;
announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
table_after(&server, 1, 0).await?;
assert_eq!(kept(&server), ours_and_theirs());
Ok(())
}
fn large(global_admin: u32, data_1: u32, data_2: u32) -> LargeCommunity {
LargeCommunity {
global_admin,
data_1,
data_2,
}
}
/// Announces IPv4 `prefixes` from the peer itself, with `more` attributes.
async fn announce_with(
peer: &mut TestPeer,
prefixes: &[&str],
more: Vec<PathAttribute>,
) -> Result<()> {
peer.announce(announcement(prefixes, &[PEER_ASN]).with_attributes(more))
.await
}
/// Rules do to some routes what the rest of the policy does to all: by
/// their communities, as the session has marked them, or by their prefix,
/// even when they came in one UPDATE with others. The first rule that
/// holds for a route is the one applied.
#[tokio::test]
async fn test_rules_apply_to_the_routes_they_hold_for() -> Result<()> {
let mut within = PrefixSet::default();
within.allow(prefix("193.0.0.0/21"), 24);
let rule = |communities: &[&str], prefixes: Option<PrefixSet>, local_pref, add: &[&str]| {
let strings = |texts: &[&str]| texts.iter().map(|t| t.to_string()).collect::<Vec<_>>();
ImportRule {
communities: Patterns::parse(&strings(communities)).unwrap(),
prefixes,
local_pref,
marks: Marks::parse(&[], &[], &strings(add)).unwrap(),
}
};
let policy = ImportPolicy {
// The session marks where its routes were learned.
marks: Marks::parse(&[], &[], &["1299:101:1".to_string()]).unwrap(),
rules: vec![
rule(&["3356:3"], None, Some(50), &[]),
rule(&[], Some(within), None, &["1299:31:10"]),
rule(&["1299:101:*"], None, Some(120), &[]),
],
..default_policy()
};
let ServerWithPeer {
server,
listener: _listener,
} = server_with(policy).await?;
let mut peer = established(&server).await?;
announce_with(
&mut peer,
&["194.0.0.0/16"],
vec![PathAttribute::Communities(CommunitiesPathAttribute(vec![
(3356, 3),
]))],
)
.await?;
announce_with(&mut peer, &["193.0.0.0/22", "195.0.0.0/16"], vec![]).await?;
table_after(&server, 3, 0).await?;
let kept = |wanted: &str| {
let routes = server.received_routes();
let route = routes
.iter()
.find(|route| route.prefix.to_string() == wanted)
.unwrap_or_else(|| panic!("{wanted} is in the table"));
let local_pref = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::LocalPref(local_pref) => Some(local_pref.0),
_ => None,
});
let large = route
.attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::LargeCommunities(large) => Some(large.communities.clone()),
_ => None,
})
.unwrap_or_default();
(local_pref, large)
};
let learned = large(1299, 101, 1);
// By its community: the first rule, and only that one.
assert_eq!(kept("194.0.0.0/16"), (Some(50), vec![learned.clone()]));
// By its prefix, apart from the route it came with.
assert_eq!(
kept("193.0.0.0/22"),
(None, vec![learned.clone(), large(1299, 31, 10)])
);
// By the mark the session put on.
assert_eq!(kept("195.0.0.0/16"), (Some(120), vec![learned]));
Ok(())
}
crates/integration-tests/tests/interop.rs
+2 -10
View File
@@ -12,8 +12,6 @@ use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use eyre::Result;
const TIMEOUT: Duration = Duration::from_secs(5);
/// The OPEN a BIRD 2 router sends for an IPv6-only session from AS210036:
/// one optional parameter carrying MP(IPv6 unicast), route refresh,
/// extended message, graceful restart (aware only), 4-octet AS, enhanced
@@ -77,19 +75,13 @@ async fn test_bird_open_and_end_of_rib_are_accepted() -> Result<()> {
peer.send_raw(BIRD_OPEN).await?;
peer.recv_open().await?;
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
match peer.recv().await? {
Message::KeepAlive => {}
other => eyre::bail!("server did not accept BIRD's OPEN: {other:?}"),
}
peer.send(&Message::KeepAlive).await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let session = server.registry.list().remove(0);
let session = server.wait_for_session().await?;
assert_eq!(
session.peer.asn, 210036,
"4-octet ASN must come from the capability, not AS_TRANS"
crates/integration-tests/tests/kernel.rs
+7 -24
View File
@@ -12,16 +12,14 @@ use std::process::Command;
use std::time::{Duration, Instant};
use bgp_fib::FibConfig;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::prefix;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use eyre::Result;
use serde_json::Value;
/// The kernel is given more time than the loopback tests allow.
const TIMEOUT: Duration = Duration::from_secs(10);
const TABLE: u32 = 1100;
const INTERFACE: &str = "ferrix100";
@@ -87,19 +85,8 @@ impl Drop for Interface {
}
async fn announce(peer: &mut TestPeer, prefixes: &[&str], next_hop: Ipv4Addr) -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![65001]),
PathAttribute::NextHop(NextHopPathAttribute(next_hop)),
],
announced_routes: prefixes
.iter()
.map(|p| IpPrefix::try_from(*p).unwrap().into())
.collect(),
}))
.await
peer.announce(announcement(prefixes, &[65001]).with_next_hop(next_hop))
.await
}
/// A peer's routes go from the session to the kernel, and out of it again
@@ -174,12 +161,8 @@ async fn test_routes_from_a_session_reach_the_kernel() -> Result<()> {
);
// One is withdrawn.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![IpPrefix::try_from("198.18.0.0/15").unwrap().into()],
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
peer.announce(withdrawal([prefix("198.18.0.0/15").into()]))
.await?;
kernel_becomes(&["203.0.113.0/24 via 10.100.0.2 dev ferrix100 proto bgp metric 20"]);
// The session ends: packets must not go on following its routes.
crates/integration-tests/tests/negotiation.rs
+33 -81
View File
@@ -4,16 +4,16 @@
use std::net::{Ipv4Addr, Ipv6Addr, SocketAddr};
use std::time::Duration;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ReceivedRoute, TestServer};
use bgp_integration_tests::{peer_config, prefix, wait_for};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NextHopPathAttribute, NlriNextHop,
OriginPathAttribute, PathAttribute, UpdateMessage,
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, PathAttribute, UpdateMessage,
};
use bgp_packet::open::Capability;
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::config::PeerConfig;
use bgp_server::family::{FamilySet, SUPPORTED};
@@ -23,11 +23,6 @@ use eyre::{Result, bail};
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
fn only(family: AfiSafi) -> FamilySet {
[family].into_iter().collect()
@@ -37,35 +32,26 @@ fn four_octet() -> Capability {
Capability::FourByteAsn { asn: PEER_ASN }
}
async fn wait_for_session(server: &TestServer) -> Result<bgp_server::session::SessionHandle> {
Ok(tokio::time::timeout(TIMEOUT, async {
loop {
if let Some(session) = server.registry.list().into_iter().next() {
return session;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
/// Waits until the server has a route, and returns what it has.
async fn some_routes(server: &TestServer) -> Result<Vec<ReceivedRoute>> {
wait_for("a route is in the table", || {
let routes = server.received_routes();
(!routes.is_empty()).then_some(routes)
})
.await?)
.await
}
/// An UPDATE announcing one IPv4 route in the body and one IPv6 route in
/// MP_REACH_NLRI.
fn dual_stack_update() -> UpdateMessage {
UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1)),
prefixes: vec![prefix("2001:db8:1::/48").into()],
}),
],
announced_routes: vec![prefix("192.0.2.0/24").into()],
}
let ipv6 = PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
family: AfiSafi::IPV6_UNICAST,
next_hop: NlriNextHop::Ipv6(Ipv6Addr::new(0x2001, 0xdb8, 0, 0, 0, 0, 0, 1)),
prefixes: vec![prefix("2001:db8:1::/48").into()],
});
announcement(&["192.0.2.0/24"], &[PEER_ASN])
.with_attributes([ipv6])
.into()
}
/// The session carries the families both OPENs offered, and the server's
@@ -93,7 +79,7 @@ async fn test_families_are_the_ones_both_sides_offered() -> Result<()> {
"the server's OPEN offers {family}"
);
}
let session = wait_for_session(&server).await?;
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
Ok(())
}
@@ -114,20 +100,11 @@ async fn test_routes_of_a_family_not_negotiated_are_not_accepted() -> Result<()>
],
)
.await?;
wait_for_session(&server).await?;
server.wait_for_session().await?;
peer.send(&Message::Update(dual_stack_update())).await?;
peer.announce(dual_stack_update()).await?;
let routes = tokio::time::timeout(TIMEOUT, async {
loop {
let routes = server.received_routes();
if !routes.is_empty() {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let routes = some_routes(&server).await?;
// One UPDATE is applied as a whole, so the IPv6 route is not on its way.
assert_eq!(routes.len(), 1);
assert_eq!(routes[0].prefix, prefix("192.0.2.0/24"));
@@ -159,7 +136,7 @@ async fn test_routes_of_a_family_not_negotiated_are_not_sent() -> Result<()> {
],
)
.await?;
let session = wait_for_session(&server).await?;
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV6_UNICAST));
// A second peer that takes IPv4 is sent the route. The advertiser
@@ -168,7 +145,7 @@ async fn test_routes_of_a_family_not_negotiated_are_not_sent() -> Result<()> {
let mut dual = TestPeer::connect(server.addr).await?;
dual.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
let update = tokio::time::timeout(TIMEOUT, dual.recv_update()).await??;
let update = dual.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
let nothing = tokio::time::timeout(Duration::from_millis(200), v6_only.recv_update()).await;
@@ -182,19 +159,11 @@ async fn test_routes_of_a_family_not_negotiated_are_not_sent() -> Result<()> {
async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
let server = TestServer::start_with(LOCAL_ASN, |c| {
// Inbound connections from this address inherit the peer's settings.
let at = SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1);
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "v6-only".into(),
endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: only(AfiSafi::IPV6_UNICAST),
add_path: None,
import: ImportPolicy::permissive(),
role: None,
require_role: false,
export: Default::default(),
..peer_config("v6-only", at, PEER_ASN)
}];
// Keep the dialer out of the way.
c.connect_retry = Duration::from_secs(3600);
@@ -219,7 +188,7 @@ async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
.collect();
assert_eq!(offered, [&Capability::multiprotocol(AfiSafi::IPV6_UNICAST)]);
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(
n.error,
@@ -252,7 +221,7 @@ async fn test_a_peer_offering_an_unsupported_family_is_still_accepted() -> Resul
)
.await?;
let session = wait_for_session(&server).await?;
let session = server.wait_for_session().await?;
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
assert!(
session.four_octet_asn,
@@ -277,13 +246,13 @@ async fn test_a_peer_without_capabilities_gets_two_octet_as_paths() -> Result<()
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, vec![]).await?;
let session = wait_for_session(&server).await?;
let session = server.wait_for_session().await?;
assert!(!session.four_octet_asn);
assert_eq!(session.families, only(AfiSafi::IPV4_UNICAST));
// The test peer now decodes AS_PATH two octets wide, as such a speaker
// would; an UPDATE encoded four octets wide would not parse.
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, vec![prefix("10.100.0.0/16")]);
assert!(
update
@@ -298,26 +267,9 @@ async fn test_a_peer_without_capabilities_gets_two_octet_as_paths() -> Result<()
assert_eq!(as4_path, Some(vec![WIDE_ASN]));
// And its own two-octet paths are read correctly.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN, 64512]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: vec![prefix("192.0.2.0/24").into()],
}))
.await?;
let routes = tokio::time::timeout(TIMEOUT, async {
loop {
let routes = server.received_routes();
if !routes.is_empty() {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
peer.announce(announcement(&["192.0.2.0/24"], &[PEER_ASN, 64512]))
.await?;
let routes = some_routes(&server).await?;
assert!(
routes[0]
.attributes
crates/integration-tests/tests/roles.rs
+44 -116
View File
@@ -3,70 +3,36 @@
//! marked as only to go to customers.
use std::net::{Ipv4Addr, Ipv6Addr};
use std::time::Duration;
use bgp_integration_tests::default_policy;
use bgp_integration_tests::announcement::announcement;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{prefix, wait_until};
use bgp_packet::errors::{BgpError, OpenMsgSubcode};
use bgp_packet::message::{
AsPathAttribute, Message, MpReachNlriPathAttribute, NextHopPathAttribute, NlriNextHop,
OriginPathAttribute, PathAttribute, UpdateMessage,
AsPathAttribute, Message, MpReachNlriPathAttribute, NlriNextHop, OriginPathAttribute,
PathAttribute, UpdateMessage,
};
use bgp_packet::open::{BgpRole, Capability};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::LocalRoute;
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::PeerRole;
use eyre::{Result, bail};
use serde_json::Value;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// What a test holds on to: the server, and the listener its dialling
/// goes to.
struct Setup {
server: TestServer,
_listener: TcpListener,
}
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
/// A server with one configured peer at the address the test connects
/// from, which is `role` to it, and which originates 193.0.0.0/21.
async fn server_with(role: Option<PeerRole>, require_role: bool) -> Result<Setup> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
async fn server_with(role: Option<PeerRole>, require_role: bool) -> Result<ServerWithPeer> {
TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.originate = vec![LocalRoute::new(prefix("193.0.0.0/21"), Default::default())];
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import: default_policy(),
role,
require_role,
export: Default::default(),
}];
})
.await?;
Ok(Setup {
server,
_listener: listener,
peer.role = role;
peer.require_role = require_role;
})
.await
}
/// The capabilities of an ordinary peer, and with `role` what it says it
@@ -84,16 +50,9 @@ fn capabilities(role: Option<BgpRole>) -> Vec<Capability> {
/// A session with the server, on which the peer says it is `says`.
async fn established(server: &TestServer, says: Option<BgpRole>) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90, capabilities(says))
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(peer)
server
.established_with(PEER_ASN, PEER_ID, capabilities(says))
.await
}
/// Announces `prefixes` with the AS path `path` and, with `marked_by`, an
@@ -104,30 +63,17 @@ async fn announce(
path: &[u32],
marked_by: Option<u32>,
) -> Result<()> {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
];
path_attributes.extend(marked_by.map(PathAttribute::OnlyToCustomer));
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
}))
.await
let mut announcement = announcement(prefixes, path);
if let Some(asn) = marked_by {
announcement = announcement.with_only_to_customer(asn);
}
peer.announce(announcement).await
}
/// Waits until the table holds `count` routes, and returns the mark each
/// has, by prefix.
async fn marks(server: &TestServer, count: usize) -> Result<Vec<(String, Option<u32>)>> {
tokio::time::timeout(TIMEOUT, async {
while server.received_routes().len() != count {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.map_err(|_| {
server.wait_for_routes(count).await.map_err(|_| {
eyre::eyre!(
"expected {count} routes, the table has {:?}",
server.received_routes()
@@ -163,26 +109,16 @@ fn announced_role(open: &bgp_packet::open::OpenMessage) -> Option<BgpRole> {
/// shown as one whose peer confirmed its role.
#[tokio::test]
async fn test_roles_that_fit_together_are_confirmed() -> Result<()> {
let Setup { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = TestPeer::connect(server.addr).await?;
let open = peer
.handshake_with(
PEER_ASN,
Ipv4Addr::new(10, 0, 0, 2),
90,
capabilities(Some(BgpRole::Customer)),
)
.handshake_with(PEER_ASN, PEER_ID, 90, capabilities(Some(BgpRole::Customer)))
.await?;
// The peer is our customer: we are its provider.
assert_eq!(announced_role(&open), Some(BgpRole::Provider));
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
server.wait_for_session().await?;
let peers: Value = reqwest::get(format!("{base}/api/peers"))
.await?
.json()
@@ -197,18 +133,13 @@ async fn test_roles_that_fit_together_are_confirmed() -> Result<()> {
/// configured as is refused, with the error RFC 9234 gives for it.
#[tokio::test]
async fn test_a_peer_that_says_it_is_something_else_is_refused() -> Result<()> {
let Setup { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let mut peer = TestPeer::connect(server.addr).await?;
// It thinks it is our peer; we have it down as a customer.
peer.send_open_with(
PEER_ASN,
Ipv4Addr::new(10, 0, 0, 2),
90,
capabilities(Some(BgpRole::Peer)),
)
.await?;
peer.send_open_with(PEER_ASN, PEER_ID, 90, capabilities(Some(BgpRole::Peer)))
.await?;
peer.recv_open().await?;
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(n.error, BgpError::OpenMsg(OpenMsgSubcode::RoleMismatch));
// Subcode 11 on the wire.
@@ -224,7 +155,7 @@ async fn test_a_peer_that_says_it_is_something_else_is_refused() -> Result<()> {
/// our word for what it is, unless the configuration requires it to say.
#[tokio::test]
async fn test_a_peer_that_does_not_say_is_refused_only_where_required() -> Result<()> {
let Setup { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let (base, _http) = server.serve_http().await?;
let _peer = established(&server, None).await?;
let peers: Value = reqwest::get(format!("{base}/api/peers"))
@@ -234,12 +165,12 @@ async fn test_a_peer_that_does_not_say_is_refused_only_where_required() -> Resul
assert_eq!(peers[0]["role"], "customer");
assert_eq!(peers[0]["role_confirmed"], false);
let Setup { server, .. } = server_with(Some(PeerRole::Customer), true).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), true).await?;
let mut peer = TestPeer::connect(server.addr).await?;
peer.send_open_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90, capabilities(None))
peer.send_open_with(PEER_ASN, PEER_ID, 90, capabilities(None))
.await?;
peer.recv_open().await?;
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
match peer.recv().await? {
Message::Notification(n) => {
assert_eq!(n.error, BgpError::OpenMsg(OpenMsgSubcode::RoleMismatch));
}
@@ -253,7 +184,7 @@ async fn test_a_peer_that_does_not_say_is_refused_only_where_required() -> Resul
/// whatever else the peer's policy takes.
#[tokio::test]
async fn test_a_marked_route_from_a_customer_is_a_leak() -> Result<()> {
let Setup { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Customer), false).await?;
let mut peer = established(&server, Some(BgpRole::Customer)).await?;
// The customer's own route, and one its other provider gave it.
@@ -271,13 +202,11 @@ async fn test_a_marked_route_from_a_customer_is_a_leak() -> Result<()> {
marks(&server, 1).await?,
[("194.0.0.0/24".to_string(), None)]
);
tokio::time::timeout(TIMEOUT, async {
while server.refusals.peers().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
wait_until("a route is refused", || {
!server.rejections.peers().is_empty()
})
.await?;
let refused = &server.refusals.peers()[0].1;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 1);
assert_eq!(refused.recent()[0].prefix, prefix("8.8.8.0/24"));
assert_eq!(
@@ -292,7 +221,7 @@ async fn test_a_marked_route_from_a_customer_is_a_leak() -> Result<()> {
/// the route from above or from another peer.
#[tokio::test]
async fn test_a_route_marked_by_someone_else_is_a_leak_from_a_peer() -> Result<()> {
let Setup { server, .. } = server_with(Some(PeerRole::Peer), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Peer), false).await?;
let mut peer = established(&server, Some(BgpRole::Peer)).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN], Some(PEER_ASN)).await?;
@@ -312,7 +241,7 @@ async fn test_a_route_marked_by_someone_else_is_a_leak_from_a_peer() -> Result<(
("195.0.0.0/24".to_string(), Some(PEER_ASN)),
]
);
assert_eq!(server.refusals.peers()[0].1.total(), 1);
assert_eq!(server.rejections.peers()[0].1.total(), 1);
Ok(())
}
@@ -322,14 +251,13 @@ async fn test_a_route_marked_by_someone_else_is_a_leak_from_a_peer() -> Result<(
#[tokio::test]
async fn test_routes_from_a_provider_are_marked_on_the_way_in() -> Result<()> {
for four_octet in [true, false] {
let Setup { server, .. } = server_with(Some(PeerRole::Provider), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(PeerRole::Provider), false).await?;
let mut peer = TestPeer::connect(server.addr).await?;
let mut offered = capabilities(Some(BgpRole::Provider));
if !four_octet {
offered.retain(|capability| !matches!(capability, Capability::FourByteAsn { .. }));
}
peer.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90, offered)
.await?;
peer.handshake_with(PEER_ASN, PEER_ID, 90, offered).await?;
announce(&mut peer, &["194.0.0.0/24"], &[PEER_ASN, 2121], None).await?;
// One that someone further up has marked already keeps that mark.
@@ -363,7 +291,7 @@ async fn test_routes_from_a_provider_are_marked_on_the_way_in() -> Result<()> {
],
"four-octet AS numbers: {four_octet}"
);
assert!(server.refusals.peers().is_empty());
assert!(server.rejections.peers().is_empty());
}
Ok(())
}
@@ -383,9 +311,9 @@ async fn test_our_routes_are_marked_on_the_way_down_or_across() -> Result<()> {
(PeerRole::Provider, BgpRole::Provider, false),
(PeerRole::RouteServer, BgpRole::RouteServer, false),
] {
let Setup { server, .. } = server_with(Some(peer_is), false).await?;
let ServerWithPeer { server, .. } = server_with(Some(peer_is), false).await?;
let mut peer = established(&server, Some(says)).await?;
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert_eq!(update.announced_routes, [prefix("193.0.0.0/21")]);
let mark = update
.path_attributes
@@ -397,9 +325,9 @@ async fn test_our_routes_are_marked_on_the_way_down_or_across() -> Result<()> {
assert_eq!(mark, marked.then_some(LOCAL_ASN), "to our {peer_is:?}");
}
// Nothing is said to a peer whose role is not known.
let Setup { server, .. } = server_with(None, false).await?;
let ServerWithPeer { server, .. } = server_with(None, false).await?;
let mut peer = established(&server, None).await?;
let update = tokio::time::timeout(TIMEOUT, peer.recv_update()).await??;
let update = peer.recv_update().await?;
assert!(
!update
.path_attributes
crates/integration-tests/tests/rpki.rs
+51 -163
View File
@@ -2,34 +2,23 @@
//! cache, a peer that announces routes, and the validation data changing
//! under both.
use std::collections::BTreeSet;
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::default_policy;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::test_cache::TestCache;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::message::{
AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
UpdateMessage,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_integration_tests::test_server::{ServerWithPeer, TestServer};
use bgp_integration_tests::{default_policy, prefix, wait_until};
use bgp_packet::message::{AsPathAttribute, PathAttribute};
use bgp_rpki::{CacheState, RpkiConfig, Vrp};
use bgp_server::config::PeerConfig;
use bgp_server::family::FamilySet;
use bgp_server::policy::ImportPolicy;
use eyre::Result;
use serde_json::Value;
use tokio::net::TcpListener;
const LOCAL_ASN: u32 = 1299;
const PEER_ASN: u32 = 3333;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
const PEER_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
fn vrp(p: &str, max_length: u8, asn: u32) -> Vrp {
Vrp {
@@ -39,13 +28,6 @@ fn vrp(p: &str, max_length: u8, asn: u32) -> Vrp {
}
}
/// What a test holds on to: the server, and the listener its dialling
/// goes to.
struct Setup {
server: TestServer,
_listener: TcpListener,
}
/// A server that validates against `cache`, with one configured peer at
/// the address the test connects from. Returns once the server has the
/// cache's data.
@@ -53,12 +35,9 @@ async fn server_with(
cache: &TestCache,
expire: Option<Duration>,
import: ImportPolicy,
) -> Result<Setup> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
let endpoint = listener.local_addr()?;
) -> Result<ServerWithPeer> {
let cache = cache.address();
let server = TestServer::start_with(LOCAL_ASN, |c| {
c.accept_unconfigured_peers = false;
let setup = TestServer::with_peer(LOCAL_ASN, PEER_ASN, |c, peer| {
c.rpki = Some(RpkiConfig {
caches: vec![cache],
// Data never expires sooner than a refresh and a retry take.
@@ -66,46 +45,19 @@ async fn server_with(
retry: Some(Duration::from_millis(100)),
expire,
});
c.peers = vec![PeerConfig {
graceful_restart: Default::default(),
name: "r1".into(),
endpoint: endpoint.into(),
remote_asn: PEER_ASN,
hold_time_secs: 90,
local_address: None,
families: FamilySet::supported(),
add_path: None,
import,
role: None,
require_role: false,
export: Default::default(),
}];
peer.import = import;
})
.await?;
let rpki = server.rpki.clone().expect("validation is configured");
tokio::time::timeout(TIMEOUT, async {
while rpki.status().state != CacheState::Synchronised {
tokio::time::sleep(Duration::from_millis(5)).await;
}
let rpki = setup.server.rpki.clone().expect("validation is configured");
wait_until("the server has the cache's data", || {
rpki.status().state == CacheState::Synchronised
})
.await?;
Ok(Setup {
server,
_listener: listener,
})
Ok(setup)
}
async fn established(server: &TestServer) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(peer)
server.established(PEER_ASN, PEER_ID).await
}
/// Announces `prefixes` as originated by `origin`, reached through the
@@ -116,53 +68,7 @@ async fn announce(peer: &mut TestPeer, prefixes: &[&str], origin: u32) -> Result
} else {
vec![PEER_ASN, origin]
};
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
],
announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
}))
.await
}
fn set(prefixes: &[&str]) -> BTreeSet<String> {
prefixes.iter().map(|p| p.to_string()).collect()
}
/// Waits until the table holds exactly `expected` and the peer has `held`
/// routes held aside.
async fn table_becomes(server: &TestServer, expected: &[&str], held: u64) -> Result<()> {
let expected = set(expected);
let table = || -> BTreeSet<String> {
server
.received_routes()
.iter()
.map(|route| route.prefix.to_string())
.collect()
};
let held_now = || {
server
.registry
.list()
.first()
.map_or(0, |session| session.counters.held())
};
tokio::time::timeout(TIMEOUT, async {
while table() != expected || held_now() != held {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await
.map_err(|_| {
eyre::eyre!(
"the table has {:?} with {} held; expected {expected:?} with {held} held",
table(),
held_now()
)
})
peer.announce(announcement(prefixes, &path)).await
}
/// A route whose origin is not the one the holder of the addresses
@@ -171,7 +77,7 @@ async fn table_becomes(server: &TestServer, expected: &[&str], held: u64) -> Res
#[tokio::test]
async fn test_an_invalid_route_is_refused_and_the_rest_are_taken() -> Result<()> {
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, PEER_ASN)]).await;
let Setup { server, .. } = server_with(&cache, None, default_policy()).await?;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
@@ -180,7 +86,9 @@ async fn test_an_invalid_route_is_refused_and_the_rest_are_taken() -> Result<()>
// AS, more specific than it may announce.
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
announce(&mut peer, &["193.0.6.0/25"], PEER_ASN).await?;
table_becomes(&server, &["193.0.0.0/21", "194.0.0.0/24"], 1).await?;
server
.table_becomes(&["193.0.0.0/21", "194.0.0.0/24"], 1)
.await?;
let client = reqwest::Client::new();
let get = |path: &str| {
@@ -226,19 +134,19 @@ async fn test_an_invalid_route_is_refused_and_the_rest_are_taken() -> Result<()>
async fn test_routes_follow_the_validation_data() -> Result<()> {
let ripe = vrp("193.0.0.0/21", 24, PEER_ASN);
let cache = TestCache::start(vec![ripe]).await;
let Setup { server, .. } = server_with(&cache, None, default_policy()).await?;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let mut peer = established(&server).await?;
// Nothing is known about 8.8.8.0/24, so it is taken. 193.0.4.0/24 is
// covered, and not for this origin.
announce(&mut peer, &["8.8.8.0/24", "193.0.4.0/24"], 15169).await?;
table_becomes(&server, &["8.8.8.0/24"], 1).await?;
server.table_becomes(&["8.8.8.0/24"], 1).await?;
// The holder of 8.8.8.0/24 authorises another AS: the route in the
// table is now invalid, and goes.
cache.set(vec![ripe, vrp("8.8.8.0/24", 24, 13335)]);
table_becomes(&server, &[], 2).await?;
let refused = &server.refusals.peers()[0].1;
server.table_becomes(&[], 2).await?;
let refused = &server.rejections.peers()[0].1;
assert_eq!(refused.total(), 2);
assert_eq!(refused.recent()[0].prefix, prefix("8.8.8.0/24"));
assert_eq!(&refused.recent()[0].as_path[..], [PEER_ASN, 15169]);
@@ -250,7 +158,9 @@ async fn test_routes_follow_the_validation_data() -> Result<()> {
vrp("8.8.8.0/24", 24, 15169),
vrp("193.0.4.0/24", 24, 15169),
]);
table_becomes(&server, &["8.8.8.0/24", "193.0.4.0/24"], 0).await?;
server
.table_becomes(&["8.8.8.0/24", "193.0.4.0/24"], 0)
.await?;
for route in server.received_routes() {
assert!(
route
@@ -269,27 +179,23 @@ async fn test_routes_follow_the_validation_data() -> Result<()> {
async fn test_a_held_route_the_peer_withdrew_does_not_come_back() -> Result<()> {
let ripe = vrp("193.0.0.0/21", 24, PEER_ASN);
let cache = TestCache::start(vec![ripe]).await;
let Setup { server, .. } = server_with(&cache, None, default_policy()).await?;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24", "193.0.5.0/24"], 15169).await?;
table_becomes(&server, &[], 2).await?;
server.table_becomes(&[], 2).await?;
// One is withdrawn, the other announced again with the right origin.
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![prefix("193.0.4.0/24").into()],
path_attributes: vec![],
announced_routes: vec![],
}))
.await?;
peer.announce(withdrawal([prefix("193.0.4.0/24").into()]))
.await?;
announce(&mut peer, &["193.0.5.0/24"], PEER_ASN).await?;
table_becomes(&server, &["193.0.5.0/24"], 0).await?;
server.table_becomes(&["193.0.5.0/24"], 0).await?;
// Now the old origin would be valid for both. Neither old route is
// there to be brought back.
cache.set(vec![ripe, vrp("193.0.4.0/23", 24, 15169)]);
tokio::time::sleep(Duration::from_millis(500)).await;
table_becomes(&server, &["193.0.5.0/24"], 0).await?;
server.table_becomes(&["193.0.5.0/24"], 0).await?;
assert!(
server.received_routes()[0]
.attributes
@@ -308,18 +214,18 @@ async fn test_a_peer_can_be_exempt_from_validation() -> Result<()> {
accept_rpki_invalid: true,
..default_policy()
};
let Setup { server, .. } = server_with(&cache, None, import).await?;
let ServerWithPeer { server, .. } = server_with(&cache, None, import).await?;
let (base, _http) = server.serve_http().await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
table_becomes(&server, &["193.0.4.0/24"], 0).await?;
server.table_becomes(&["193.0.4.0/24"], 0).await?;
let routes: Value = reqwest::get(format!("{base}/api/routes/received"))
.await?
.json()
.await?;
assert_eq!(routes["routes"][0]["rpki"], "invalid");
assert!(server.refusals.peers().is_empty());
assert!(server.rejections.peers().is_empty());
Ok(())
}
@@ -329,14 +235,14 @@ async fn test_a_peer_can_be_exempt_from_validation() -> Result<()> {
#[tokio::test]
async fn test_held_routes_are_taken_when_the_validation_data_expires() -> Result<()> {
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, PEER_ASN)]).await;
let Setup { server, .. } =
let ServerWithPeer { server, .. } =
server_with(&cache, Some(Duration::from_millis(1500)), default_policy()).await?;
let mut peer = established(&server).await?;
announce(&mut peer, &["193.0.4.0/24"], 15169).await?;
table_becomes(&server, &[], 1).await?;
server.table_becomes(&[], 1).await?;
drop(cache);
table_becomes(&server, &["193.0.4.0/24"], 0).await?;
server.table_becomes(&["193.0.4.0/24"], 0).await?;
assert!(server.rpki.as_ref().unwrap().current().vrps.is_empty());
Ok(())
}
@@ -352,37 +258,21 @@ async fn test_an_unused_as4_path_does_not_make_a_route_valid() -> Result<()> {
use bgp_packet::message::AsPathSegment;
const HOLDER: u32 = 15169;
let cache = TestCache::start(vec![vrp("193.0.0.0/21", 24, HOLDER)]).await;
let Setup { server, .. } = server_with(&cache, None, default_policy()).await?;
let ServerWithPeer { server, .. } = server_with(&cache, None, default_policy()).await?;
// No capabilities: AS numbers are two octets wide on this session.
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake_with(PEER_ASN, Ipv4Addr::new(10, 0, 0, 2), 90, vec![])
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
let mut peer = server.established_with(PEER_ASN, PEER_ID, vec![]).await?;
assert!(!server.registry.list()[0].four_octet_asn);
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![PEER_ASN]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
PathAttribute::As4Path(AsPathAttribute {
segments: vec![AsPathSegment {
ordered: true,
path: vec![3356, HOLDER],
}],
}),
],
announced_routes: vec![prefix("193.0.4.0/24").into()],
}))
.await?;
table_becomes(&server, &[], 1).await?;
let as4_path = PathAttribute::As4Path(AsPathAttribute {
segments: vec![AsPathSegment {
ordered: true,
path: vec![3356, HOLDER],
}],
});
peer.announce(announcement(&["193.0.4.0/24"], &[PEER_ASN]).with_attributes([as4_path]))
.await?;
server.table_becomes(&[], 1).await?;
// The data changes, and every held route is looked at again.
cache.set(vec![
@@ -390,14 +280,12 @@ async fn test_an_unused_as4_path_does_not_make_a_route_valid() -> Result<()> {
vrp("198.51.100.0/24", 24, 64500),
]);
let rpki = server.rpki.clone().unwrap();
tokio::time::timeout(TIMEOUT, async {
while rpki.current().vrps.len() != 2 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
wait_until("the server has the new data", || {
rpki.current().vrps.len() == 2
})
.await?;
// Time enough for the look, and then some.
tokio::time::sleep(Duration::from_millis(600)).await;
table_becomes(&server, &[], 1).await?;
server.table_becomes(&[], 1).await?;
Ok(())
}
crates/integration-tests/tests/transit.rs
+20 -53
View File
@@ -6,31 +6,28 @@ use std::net::Ipv4Addr;
use std::sync::Arc;
use std::time::Duration;
use bgp_integration_tests::announcement::{announcement, withdrawal};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
use bgp_integration_tests::{TIMEOUT, prefix};
use bgp_packet::message::{
AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
PathAttribute, RouteRefreshMessage, UpdateMessage,
CommunitiesPathAttribute, Message, NextHopPathAttribute, PathAttribute, RouteRefreshMessage,
UpdateMessage,
};
use bgp_primitives::constants::AfiSafi;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::{LocalRoute, LocalRouteAttrs};
use bgp_server::marks::{Control, Marks};
use bgp_server::policy::ExportPolicy;
use eyre::Result;
const LOCAL_ASN: u32 = 64601;
const TIMEOUT: Duration = Duration::from_secs(5);
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
/// A server that passes on to every peer what it learned from the others.
async fn transit(customize: impl FnOnce(&mut bgp_server::config::Config)) -> Result<TestServer> {
TestServer::start_with(LOCAL_ASN, |config| {
config.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
});
customize(config);
@@ -40,17 +37,9 @@ async fn transit(customize: impl FnOnce(&mut bgp_server::config::Config)) -> Res
/// The peer with AS `64700 + n`, its session up.
async fn peer(server: &TestServer, n: u8) -> Result<TestPeer> {
let before = server.registry.list().len();
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(64700 + u32::from(n), Ipv4Addr::new(10, 0, 0, n), 90)
.await?;
tokio::time::timeout(TIMEOUT, async {
while server.registry.list().len() != before + 1 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
Ok(peer)
server
.established(64700 + u32::from(n), Ipv4Addr::new(10, 0, 0, n))
.await
}
async fn announce(
@@ -59,31 +48,13 @@ async fn announce(
path: &[u32],
communities: &[(u16, u16)],
) -> Result<()> {
let mut path_attributes = vec![
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path.to_vec()),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
];
if !communities.is_empty() {
path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
communities.to_vec(),
)));
}
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: vec![],
path_attributes,
announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
}))
.await
peer.announce(announcement(prefixes, path).with_communities(communities))
.await
}
async fn withdraw(peer: &mut TestPeer, prefixes: &[&str]) -> Result<()> {
peer.send(&Message::Update(UpdateMessage {
withdrawn_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
path_attributes: vec![],
announced_routes: vec![],
}))
.await
peer.announce(withdrawal(prefixes.iter().map(|p| prefix(p).into())))
.await
}
/// What a peer has been told so far, as the prefixes it would now have
@@ -180,7 +151,7 @@ async fn test_routes_from_one_peer_reach_the_others() -> Result<()> {
// The next hop is ourselves: the address of our end of the session.
announce(&mut one, &["10.4.0.0/16"], &[64701], &[]).await?;
let update = tokio::time::timeout(TIMEOUT, two.recv_update()).await??;
let update = two.recv_update().await?;
assert!(
update
.path_attributes
@@ -220,7 +191,7 @@ async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
// A better route for it comes: that is the one told now.
announce(&mut near, &["10.1.0.0/16"], &[64701, 65001], &[]).await?;
let update = tokio::time::timeout(TIMEOUT, watcher.recv_update()).await??;
let update = watcher.recv_update().await?;
told.take(&update);
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64701, 65001]);
assert!(
@@ -230,7 +201,7 @@ async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
// And goes again: back to the other, with no gap.
withdraw(&mut near, &["10.1.0.0/16"]).await?;
let update = tokio::time::timeout(TIMEOUT, watcher.recv_update()).await??;
let update = watcher.recv_update().await?;
told.take(&update);
assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64702, 65000, 65001]);
@@ -239,7 +210,7 @@ async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
// told of.
let mut told_far = Told::default();
told_becomes(&mut far, &mut told_far, &["10.1.0.0/16"]).await?;
let gone = tokio::time::timeout(TIMEOUT, far.recv_update()).await??;
let gone = far.recv_update().await?;
assert_eq!(gone.withdrawn_routes, vec![prefix("10.1.0.0/16")]);
told_nothing(&mut far).await;
@@ -251,7 +222,7 @@ async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
AfiSafi::IPV4_UNICAST,
)))
.await?;
let again = tokio::time::timeout(TIMEOUT, late.recv_update()).await??;
let again = late.recv_update().await?;
assert_eq!(again.announced_routes, vec![prefix("10.1.0.0/16")]);
Ok(())
}
@@ -294,6 +265,7 @@ async fn test_communities_say_who_is_told() -> Result<()> {
});
config.export = Arc::new(ExportPolicy {
learned_routes: true,
all_paths: false,
marks: Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
prepend: 1,
});
@@ -349,7 +321,7 @@ async fn test_communities_say_who_is_told() -> Result<()> {
&[(0, 64702), (3356, 3)],
)
.await?;
let update = tokio::time::timeout(TIMEOUT, three.recv_update()).await??;
let update = three.recv_update().await?;
assert!(
update
.path_attributes
@@ -378,12 +350,7 @@ async fn test_nothing_learned_is_passed_on_unless_asked_for() -> Result<()> {
let mut told = Told::default();
told_becomes(&mut two, &mut told, &["10.99.0.0/16"]).await?;
announce(&mut one, &["10.1.0.0/16"], &[64701], &[]).await?;
tokio::time::timeout(TIMEOUT, async {
while server.rib.len() != 1 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
})
.await?;
server.wait_for_routes(1).await?;
told_nothing(&mut two).await;
Ok(())
}
crates/integration-tests/tests/update_errors.rs
+19 -40
View File
@@ -2,19 +2,18 @@
//! cost the routes in the UPDATE, not the session.
use std::net::Ipv4Addr;
use std::time::Duration;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::{ReceivedRoute, TestServer};
use bgp_integration_tests::{prefix, wait_for};
use bgp_packet::errors::{BgpError, UpdateMsgSubcode};
use bgp_packet::message::{LocalPrefPathAttribute, Message, PathAttribute};
use bgp_packet::message::{LocalPrefPathAttribute, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_server::metrics::UPDATE_ERRORS;
use eyre::{Result, bail};
use eyre::Result;
const LOCAL_ASN: u32 = 65000;
const PEER_ASN: u32 = 65001;
const TIMEOUT: Duration = Duration::from_secs(5);
const WELL_KNOWN: u8 = 0x40;
const OPTIONAL_TRANSITIVE: u8 = 0xc0;
@@ -58,31 +57,24 @@ fn update(attributes: &[Vec<u8>], nlri: &[u8]) -> Vec<u8> {
const ROUTE_A: [u8; 3] = [16, 10, 1];
const ROUTE_B: [u8; 3] = [16, 10, 2];
fn prefix(s: &str) -> IpPrefix {
IpPrefix::try_from(s).unwrap()
}
async fn established(server: &TestServer, asn: u32) -> Result<TestPeer> {
let mut peer = TestPeer::connect(server.addr).await?;
peer.handshake(asn, Ipv4Addr::new(10, 0, 0, 1), 90).await?;
Ok(peer)
server.established(asn, Ipv4Addr::new(10, 0, 0, 1)).await
}
/// Waits until the server's Adj-RIB-In holds exactly `prefixes`.
async fn wait_for_routes(server: &TestServer, prefixes: &[&str]) -> Result<Vec<ReceivedRoute>> {
let mut expected: Vec<IpPrefix> = prefixes.iter().map(|p| prefix(p)).collect();
expected.sort_by_key(|p| p.to_string());
Ok(tokio::time::timeout(TIMEOUT, async {
loop {
let mut routes = server.received_routes();
routes.sort_by_key(|r| r.prefix.to_string());
if routes.iter().map(|r| &r.prefix).eq(expected.iter()) {
return routes;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
wait_for(&format!("the table holds {prefixes:?}"), || {
let mut routes = server.received_routes();
routes.sort_by_key(|r| r.prefix.to_string());
routes
.iter()
.map(|r| &r.prefix)
.eq(expected.iter())
.then_some(routes)
})
.await?)
.await
}
/// The situation RFC 7606 exists for: a route is in the table, and then an
@@ -205,24 +197,11 @@ async fn test_an_unreadable_prefix_still_resets_the_session() -> Result<()> {
))
.await?;
loop {
match tokio::time::timeout(TIMEOUT, peer.recv()).await?? {
Message::KeepAlive => {}
Message::Notification(n) => {
assert_eq!(
n.error,
BgpError::UpdateMsg(UpdateMsgSubcode::InvalidNetworkField)
);
break;
}
other => bail!("expected NOTIFICATION, got {other:?}"),
}
}
tokio::time::timeout(TIMEOUT, async {
while !server.registry.list().is_empty() {
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await?;
let notification = peer.expect_notification().await?;
assert_eq!(
notification.error,
BgpError::UpdateMsg(UpdateMsgSubcode::InvalidNetworkField)
);
server.wait_for_sessions(0).await?;
Ok(())
}
crates/packet/src/attributes.rs
+122 -133
View File
@@ -9,17 +9,20 @@ use std::net::{Ipv4Addr, Ipv6Addr};
use bitfield::bitfield;
use bytes::{BufMut, BytesMut};
use nom::Err::Failure;
use nom::number::complete::{be_u8, be_u16, be_u32};
use nom::{IResult, Parser};
use nom::Parser;
use nom::number::complete::{be_u8, be_u32};
use serde::Serialize;
use serde_repr::Serialize_repr;
use bgp_primitives::constants::{Afi, AfiSafi, IpVersion, Safi};
use bgp_primitives::constants::{AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::{BgpParserError, EncodeError, ParserContext};
use crate::message::{AS_TRANS, write_nlri};
use crate::message::write_nlri;
use crate::wire::{
Parsed, list, malformed, parse_afi_safi, parse_asn, put_afi_safi, put_asn, put_length, records,
whole,
};
bitfield! {
#[derive(Clone, Debug, Serialize, PartialEq, Eq)]
@@ -74,9 +77,16 @@ pub enum PathAttribute {
/// customer, a peer or a route server client by the AS with this
/// number, and from there on may only go down to customers.
OnlyToCustomer(u32),
/// ORIGINATOR_ID (RFC 4456 §8): the BGP Identifier of the router in our
/// AS that learned the route from outside, put on by the first router
/// that passes the route from one router of the AS to another.
OriginatorId(Ipv4Addr),
/// CLUSTER_LIST (RFC 4456 §8): the clusters the route has been passed
/// on by inside our AS, the last first.
ClusterList(Vec<Ipv4Addr>),
/// An attribute this implementation does not interpret, kept as it
/// came so that it can be passed on.
UnknownPathAttribute {
Unknown {
flags: PathAttributeFlags,
type_code: u8,
payload: Vec<u8>,
@@ -102,18 +112,6 @@ pub(crate) fn defined_flags(type_code: u8) -> Option<(bool, bool)> {
})
}
/// A parsed value, provided the parser used all of it.
fn whole<T>((rest, parsed): (&[u8], T)) -> Result<T, nom::Err<BgpParserError<&[u8]>>> {
if rest.is_empty() {
Ok(parsed)
} else {
Err(Failure(BgpParserError::Malformed(format!(
"{} octets left over",
rest.len()
))))
}
}
/// One attribute as it lies on the wire, split out but not interpreted.
#[derive(Debug, Clone)]
pub struct RawAttribute<'a> {
@@ -198,7 +196,9 @@ impl PathAttribute {
Self::As4Path(_) => AS4_PATH,
Self::LargeCommunities(_) => LARGE_COMMUNITIES,
Self::OnlyToCustomer(_) => ONLY_TO_CUSTOMER,
Self::UnknownPathAttribute { type_code, .. } => *type_code,
Self::OriginatorId(_) => ORIGINATOR_ID,
Self::ClusterList(_) => CLUSTER_LIST,
Self::Unknown { type_code, .. } => *type_code,
}
}
@@ -207,7 +207,7 @@ impl PathAttribute {
fn flags(&self) -> PathAttributeFlags {
match self {
// As they came: the Partial bit has to survive (RFC 4271 §5).
Self::UnknownPathAttribute { flags, .. } => flags.clone(),
Self::Unknown { flags, .. } => flags.clone(),
known => {
let (optional, transitive) = defined_flags(known.type_code())
.expect("every attribute with a variant has defined flags");
@@ -219,12 +219,8 @@ impl PathAttribute {
/// Parses one attribute, header and value, from the front of `buf`.
/// The flags are not checked against the attribute's definition;
/// [`crate::message::UpdateMessage::decode`] does that for attributes off a session.
pub fn from_wire<'a>(
ctx: &ParserContext,
buf: &'a [u8],
) -> IResult<&'a [u8], Self, BgpParserError<&'a [u8]>> {
let (raw, rest) =
split_attribute(buf).map_err(|e| Failure(BgpParserError::Malformed(e.to_string())))?;
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (raw, rest) = split_attribute(buf).map_err(|e| malformed(e.to_string()))?;
let attribute = Self::parse_value(ctx, raw.flags, raw.type_code, raw.value)?;
Ok((rest, attribute))
}
@@ -240,32 +236,50 @@ impl PathAttribute {
) -> Result<Self, nom::Err<BgpParserError<&'a [u8]>>> {
use attr_type::*;
Ok(match type_code {
ORIGIN => Self::Origin(whole(OriginPathAttribute::from_wire(value)?)?),
AS_PATH => Self::AsPath(whole(AsPathAttribute::from_wire(ctx, value)?)?),
NEXT_HOP => Self::NextHop(whole(NextHopPathAttribute::from_wire(value)?)?),
MULTI_EXIT_DISC => {
Self::MultiExitDisc(whole(MultiExitDiscPathAttribute::from_wire(value)?)?)
}
LOCAL_PREF => Self::LocalPref(whole(LocalPrefPathAttribute::from_wire(value)?)?),
ATOMIC_AGGREGATE => {
Self::AtomicAggregate(whole((value, AtomicAggregatePathAttribute {}))?)
}
AGGREGATOR => Self::Aggregator(whole(AggregatorPathAttribute::from_wire(ctx, value)?)?),
COMMUNITIES => Self::Communities(whole(CommunitiesPathAttribute::from_wire(value)?)?),
MP_REACH_NLRI => {
Self::MpReachNlri(whole(MpReachNlriPathAttribute::from_wire(ctx, value)?)?)
}
MP_UNREACH_NLRI => {
Self::MpUnreachNlri(whole(MpUnreachNlriPathAttribute::from_wire(ctx, value)?)?)
}
ORIGIN => Self::Origin(whole("ORIGIN", OriginPathAttribute::from_wire(value)?)?),
AS_PATH => Self::AsPath(whole("AS_PATH", AsPathAttribute::from_wire(ctx, value)?)?),
NEXT_HOP => Self::NextHop(whole("NEXT_HOP", NextHopPathAttribute::from_wire(value)?)?),
MULTI_EXIT_DISC => Self::MultiExitDisc(whole(
"MULTI_EXIT_DISC",
MultiExitDiscPathAttribute::from_wire(value)?,
)?),
LOCAL_PREF => Self::LocalPref(whole(
"LOCAL_PREF",
LocalPrefPathAttribute::from_wire(value)?,
)?),
ATOMIC_AGGREGATE => Self::AtomicAggregate(whole(
"ATOMIC_AGGREGATE",
(value, AtomicAggregatePathAttribute {}),
)?),
AGGREGATOR => Self::Aggregator(whole(
"AGGREGATOR",
AggregatorPathAttribute::from_wire(ctx, value)?,
)?),
COMMUNITIES => Self::Communities(whole(
"COMMUNITIES",
CommunitiesPathAttribute::from_wire(value)?,
)?),
MP_REACH_NLRI => Self::MpReachNlri(whole(
"MP_REACH_NLRI",
MpReachNlriPathAttribute::from_wire(ctx, value)?,
)?),
MP_UNREACH_NLRI => Self::MpUnreachNlri(whole(
"MP_UNREACH_NLRI",
MpUnreachNlriPathAttribute::from_wire(ctx, value)?,
)?),
EXTENDED_COMMUNITIES => Self::ExtendedCommunities(whole(
"EXTENDED_COMMUNITIES",
ExtendedCommunitiesPathAttribute::from_wire(value)?,
)?),
// Always four octets wide, whatever the session's AS_PATH is.
AS4_PATH => Self::As4Path(whole(AsPathAttribute::from_wire_with_width(true, value)?)?),
LARGE_COMMUNITIES => {
Self::LargeCommunities(whole(LargeCommunitiesPathAttribute::from_wire(value)?)?)
}
AS4_PATH => Self::As4Path(whole(
"AS4_PATH",
AsPathAttribute::from_wire_with_width(true, value)?,
)?),
LARGE_COMMUNITIES => Self::LargeCommunities(whole(
"LARGE_COMMUNITIES",
LargeCommunitiesPathAttribute::from_wire(value)?,
)?),
ONLY_TO_CUSTOMER => {
let asn = <[u8; 4]>::try_from(value).map_err(|_| {
malformed(format!(
@@ -275,7 +289,22 @@ impl PathAttribute {
})?;
Self::OnlyToCustomer(u32::from_be_bytes(asn))
}
_ => Self::UnknownPathAttribute {
ORIGINATOR_ID => {
let id = <[u8; 4]>::try_from(value).map_err(|_| {
malformed(format!(
"ORIGINATOR_ID is {} octets long, not four",
value.len()
))
})?;
Self::OriginatorId(Ipv4Addr::from(id))
}
CLUSTER_LIST => Self::ClusterList(
records::<4>(value, "CLUSTER_LIST")?
.iter()
.map(|id| Ipv4Addr::from(*id))
.collect(),
),
_ => Self::Unknown {
flags,
type_code,
payload: value.to_vec(),
@@ -291,26 +320,21 @@ impl PathAttribute {
let mut flags = self.flags();
out.put_u8(0);
out.put_u8(self.type_code());
let length_at = out.len();
out.put_u8(0);
let value_start = out.len();
self.value_to_wire(ctx, out)?;
let length = out.len() - value_start;
let length = out.len() - length_at - 1;
if let Ok(short) = u8::try_from(length)
&& !flags.extended_length()
{
out[start + 2] = short;
out[length_at] = short;
} else {
let long = u16::try_from(length).map_err(|_| EncodeError::TooLong {
what: "path attribute",
len: length,
max: u16::MAX.into(),
})?;
flags.set_extended_length(true);
// Move the value up by one octet to make room.
out.put_u8(0);
out.copy_within(value_start..value_start + length, value_start + 1);
out[start + 2..start + 4].copy_from_slice(&long.to_be_bytes());
out.copy_within(length_at + 1..length_at + 1 + length, length_at + 2);
put_length(out, length_at, 2, length, u16::MAX.into(), "path attribute")?;
}
out[start] = flags.0;
Ok(())
@@ -332,7 +356,13 @@ impl PathAttribute {
Self::As4Path(a) => a.to_wire_with_width(true, out)?,
Self::LargeCommunities(a) => a.to_wire(out),
Self::OnlyToCustomer(asn) => out.put_u32(*asn),
Self::UnknownPathAttribute { payload, .. } => out.put_slice(payload),
Self::OriginatorId(id) => out.put_slice(&id.octets()),
Self::ClusterList(ids) => {
for id in ids {
out.put_slice(&id.octets());
}
}
Self::Unknown { payload, .. } => out.put_slice(payload),
}
Ok(())
}
@@ -354,12 +384,6 @@ pub fn only_to_customer_wire(asn: u32) -> [u8; 7] {
]
}
type Parsed<'a, T> = IResult<&'a [u8], T, BgpParserError<&'a [u8]>>;
fn malformed<'a>(detail: String) -> nom::Err<BgpParserError<&'a [u8]>> {
Failure(BgpParserError::Malformed(detail))
}
/// ORIGIN: how the route came into BGP (RFC 4271 §5.1.1).
#[derive(Debug, PartialEq, Eq, Copy, Clone, Serialize_repr)]
#[repr(u8)]
@@ -478,16 +502,11 @@ impl AsPathAttribute {
)));
}
if len == 0 {
return Err(malformed("AS path segment with no AS numbers".into()));
return Err(malformed("AS path segment with no AS numbers"));
}
let count = usize::from(len);
let (buf, path): (_, Vec<u32>) = if four_octet {
nom::multi::many_m_n(count, count, be_u32).parse(buf)?
} else {
nom::multi::many_m_n(count, count, be_u16)
.parse(buf)
.map(|(buf, asns)| (buf, asns.into_iter().map(u32::from).collect()))?
};
let (buf, path) =
nom::multi::many_m_n(count, count, |buf| parse_asn(four_octet, buf)).parse(buf)?;
Ok((
buf,
AsPathSegment {
@@ -496,16 +515,7 @@ impl AsPathAttribute {
},
))
};
// Not many0: that stops quietly at a segment it cannot read, and a
// truncated segment is an error.
let mut segments = Vec::new();
let mut buf = buf;
while !buf.is_empty() {
let (rest, segment) = parse_segment(buf)?;
segments.push(segment);
buf = rest;
}
let (buf, segments) = list(buf, parse_segment)?;
Ok((buf, Self { segments }))
}
@@ -527,11 +537,7 @@ impl AsPathAttribute {
})?;
out.put_u8(count);
for asn in &segment.path {
if four_octet {
out.put_u32(*asn);
} else {
out.put_u16(u16::try_from(*asn).unwrap_or(AS_TRANS));
}
put_asn(four_octet, *asn, out);
}
}
Ok(())
@@ -605,11 +611,7 @@ impl AggregatorPathAttribute {
/// The AS number is as wide as the session's AS numbers (RFC 6793), so
/// the attribute is six or eight octets.
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, asn) = if ctx.four_octet_asn {
be_u32(buf)?
} else {
be_u16(buf).map(|(buf, asn)| (buf, u32::from(asn)))?
};
let (buf, asn) = parse_asn(ctx.four_octet_asn, buf)?;
let (buf, ip) = be_u32(buf)?;
Ok((
buf,
@@ -623,11 +625,7 @@ impl AggregatorPathAttribute {
/// On a two-octet session an AS number that does not fit is written as
/// AS_TRANS (RFC 6793 §4.2.2).
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
if ctx.four_octet_asn {
out.put_u32(self.asn);
} else {
out.put_u16(u16::try_from(self.asn).unwrap_or(AS_TRANS));
}
put_asn(ctx.four_octet_asn, self.asn, out);
out.put_u32(self.ip.into());
}
}
@@ -636,10 +634,13 @@ impl AggregatorPathAttribute {
pub struct CommunitiesPathAttribute(pub Vec<(u16, u16)>);
impl CommunitiesPathAttribute {
/// Reads all of `buf` as communities, four octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
// At least one: an empty COMMUNITIES is malformed (RFC 7606 §7.8).
let (buf, values) = nom::multi::many1(|i| (be_u16, be_u16).parse(i)).parse(buf)?;
Ok((buf, CommunitiesPathAttribute(values)))
let values = records::<4>(buf, "COMMUNITIES")?
.iter()
.map(|[a, b, c, d]| (u16::from_be_bytes([*a, *b]), u16::from_be_bytes([*c, *d])))
.collect();
Ok((&buf[buf.len()..], CommunitiesPathAttribute(values)))
}
pub fn to_wire(&self, out: &mut BytesMut) {
@@ -657,16 +658,11 @@ pub struct ExtendedCommunitiesPathAttribute {
}
impl ExtendedCommunitiesPathAttribute {
/// Reads all of `buf` as extended communities, eight octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (communities, rest) = buf.as_chunks::<8>();
if communities.is_empty() || !rest.is_empty() {
return Err(malformed(format!(
"{} octets are not a list of extended communities",
buf.len()
)));
}
let communities = records::<8>(buf, "EXTENDED_COMMUNITIES")?;
Ok((
rest,
&buf[buf.len()..],
Self {
extended_communities: communities.iter().copied().map(ExtendedCommunity).collect(),
},
@@ -703,9 +699,17 @@ pub struct LargeCommunitiesPathAttribute {
}
impl LargeCommunitiesPathAttribute {
/// Reads all of `buf` as large communities, twelve octets each.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, communities) = nom::multi::many1(LargeCommunity::from_wire).parse(buf)?;
Ok((buf, Self { communities }))
let communities = records::<12>(buf, "LARGE_COMMUNITIES")?
.iter()
.map(|community| {
let (_, community) =
LargeCommunity::from_wire(community).expect("twelve octets are one");
community
})
.collect();
Ok((&buf[buf.len()..], Self { communities }))
}
pub fn to_wire(&self, out: &mut BytesMut) {
@@ -824,16 +828,9 @@ fn parse_nlri(version: IpVersion, with_path_id: bool, buf: &[u8]) -> Parsed<'_,
pub(crate) fn parse_nlri_list(
version: IpVersion,
with_path_id: bool,
mut buf: &[u8],
buf: &[u8],
) -> Parsed<'_, Vec<Nlri>> {
// Not many0: that stops quietly at a prefix it cannot read.
let mut nlri = Vec::new();
while !buf.is_empty() {
let (rest, parsed) = parse_nlri(version, with_path_id, buf)?;
nlri.push(parsed);
buf = rest;
}
Ok((buf, nlri))
list(buf, |buf| parse_nlri(version, with_path_id, buf))
}
/// The IP version of the prefixes that make up the NLRI of `family`.
@@ -852,12 +849,6 @@ fn ip_version_of_nlri<'a>(
IpVersion::try_from(family.afi).map_err(|_| unsupported())
}
fn parse_family(buf: &[u8]) -> Parsed<'_, AfiSafi> {
let (buf, afi) = be_u16(buf)?;
let (buf, safi) = be_u8(buf)?;
Ok((buf, AfiSafi::new(Afi(afi), Safi(safi))))
}
/// MP_REACH_NLRI: routes of a family other than the one in the message
/// body, with their next hop (RFC 4760 §3).
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
@@ -869,7 +860,7 @@ pub struct MpReachNlriPathAttribute {
impl MpReachNlriPathAttribute {
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, family) = parse_family(buf)?;
let (buf, family) = parse_afi_safi(buf)?;
let version = ip_version_of_nlri(family)?;
let (buf, next_hop_len) = be_u8(buf)?;
@@ -902,8 +893,7 @@ impl MpReachNlriPathAttribute {
}
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
out.put_u16(self.family.afi.0);
out.put_u8(self.family.safi.0);
put_afi_safi(self.family, out);
out.put_u8(self.next_hop.wire_len());
self.next_hop.to_wire(out);
out.put_u8(0);
@@ -921,15 +911,14 @@ pub struct MpUnreachNlriPathAttribute {
impl MpUnreachNlriPathAttribute {
pub fn from_wire<'a>(ctx: &ParserContext, buf: &'a [u8]) -> Parsed<'a, Self> {
let (buf, family) = parse_family(buf)?;
let (buf, family) = parse_afi_safi(buf)?;
let version = ip_version_of_nlri(family)?;
let (buf, prefixes) = parse_nlri_list(version, ctx.has_add_path(family), buf)?;
Ok((buf, Self { family, prefixes }))
}
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) {
out.put_u16(self.family.afi.0);
out.put_u8(self.family.safi.0);
put_afi_safi(self.family, out);
write_nlri(&self.prefixes, ctx.has_add_path(self.family), out);
}
}
crates/packet/src/lib.rs
+1
View File
@@ -4,3 +4,4 @@ pub mod errors;
pub mod message;
pub mod open;
pub mod update_decode;
mod wire;
crates/packet/src/message.rs
+27 -45
View File
@@ -15,6 +15,7 @@ use crate::errors::DecodeError;
use crate::errors::MsgHeaderSubcode;
use crate::errors::OpenMsgSubcode;
use crate::open::OpenMessage;
use crate::wire::{fill_length, put_length};
/// A sentinel AS number used to denote that the actual AS number is provided as
/// a 4 byte value in the capabilities instead.
@@ -162,30 +163,27 @@ impl Message {
pub fn to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
let start = out.len();
out.put_slice(&BGP_MARKER);
// The length, filled in once the body is written.
// The length, filled in once the body is written. It counts the
// header too.
let length_at = out.len();
out.put_u16(0);
out.put_u8(self.type_code());
let written = self.body_to_wire(ctx, out).and_then(|()| {
let length = out.len() - start;
u16::try_from(length)
.ok()
.filter(|length| *length <= BGP_MAX_MSG_LEN)
.ok_or(EncodeError::TooLong {
what: "BGP message",
len: length,
max: BGP_MAX_MSG_LEN.into(),
})
put_length(
out,
length_at,
2,
length,
BGP_MAX_MSG_LEN.into(),
"BGP message",
)
});
match written {
Ok(length) => {
out[start + 16..start + 18].copy_from_slice(&length.to_be_bytes());
Ok(())
}
Err(e) => {
out.truncate(start);
Err(e)
}
if let Err(e) = written {
out.truncate(start);
return Err(e);
}
Ok(())
}
fn body_to_wire(&self, ctx: &ParserContext, out: &mut BytesMut) -> Result<(), EncodeError> {
@@ -337,15 +335,19 @@ impl UpdateMessage {
// The prefixes in the body are IPv4 unicast (RFC 4271 §4.3).
let with_path_id = ctx.has_add_path(AfiSafi::IPV4_UNICAST);
let withdrawn = reserve_length(out);
// Each field's two-octet length is filled in once the field is
// written.
let withdrawn = out.len();
out.put_u16(0);
write_nlri(&self.withdrawn_routes, with_path_id, out);
fill_length(out, withdrawn, "withdrawn routes")?;
fill_length(out, withdrawn, 2, u16::MAX.into(), "withdrawn routes")?;
let attributes = reserve_length(out);
let attributes = out.len();
out.put_u16(0);
for attribute in &self.path_attributes {
attribute.to_wire(ctx, out)?;
}
fill_length(out, attributes, "path attributes")?;
fill_length(out, attributes, 2, u16::MAX.into(), "path attributes")?;
// The NLRI take up the rest of the message.
write_nlri(&self.announced_routes, with_path_id, out);
@@ -353,26 +355,6 @@ impl UpdateMessage {
}
}
/// Leaves room at the end of `out` for a two-octet length that is known
/// only once what it counts has been written, and returns where.
fn reserve_length(out: &mut BytesMut) -> usize {
let at = out.len();
out.put_u16(0);
at
}
/// Fills in the length reserved at `at`: everything written since.
fn fill_length(out: &mut BytesMut, at: usize, what: &'static str) -> Result<(), EncodeError> {
let length = out.len() - at - 2;
let field = u16::try_from(length).map_err(|_| EncodeError::TooLong {
what,
len: length,
max: u16::MAX.into(),
})?;
out[at..at + 2].copy_from_slice(&field.to_be_bytes());
Ok(())
}
/// Writes prefixes as an UPDATE names them: for each, its path identifier
/// if the session uses them for the family (RFC 7911 §3), its length in
/// bits, and the bytes that length covers.
@@ -657,7 +639,7 @@ mod tests {
test_path_attribute_roundtrip!(
test_unknown_path_attribute,
&[0x80, 0x63, 0x04, 0xde, 0xad, 0xbe, 0xef],
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: PathAttributeFlags::new(
/*optional=*/ true, /*transitive=*/ false, /*partial=*/ false,
/*extended_length=*/ false,
@@ -798,7 +780,7 @@ mod tests {
],
UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![PathAttribute::UnknownPathAttribute {
path_attributes: vec![PathAttribute::Unknown {
flags: PathAttributeFlags::new(true, false, false, false),
type_code: 0x63,
payload: vec![0xde, 0xad, 0xbe, 0xef],
@@ -807,7 +789,6 @@ mod tests {
}
);
// Rejects trailing garbage after the NLRI section (a partial prefix).
/// With ADD-PATH in use for a family, each of its prefixes has a
/// four-octet path identifier in front (RFC 7911 §3): in the body for
/// IPv4 unicast, in MP_REACH_NLRI and MP_UNREACH_NLRI for the others.
@@ -1067,6 +1048,7 @@ mod tests {
assert_eq!(narrow.as4_path(), None);
}
/// Rejects trailing garbage after the NLRI section (a partial prefix).
#[test]
fn test_update_rejects_trailing_garbage() {
let ctx = default_v6_context();
crates/packet/src/open.rs
+55 -99
View File
@@ -4,7 +4,7 @@ use bitfield::bitfield;
use bytes::{BufMut, BytesMut};
use nom::{
Err::Failure,
IResult, Parser,
Parser,
number::complete::{be_u8, be_u16, be_u24, be_u32},
};
use serde::Serialize;
@@ -16,26 +16,8 @@ use bgp_primitives::{
parser::{BgpParserError, EncodeError, ProtocolErrorValues},
};
type Parsed<'a, T> = IResult<&'a [u8], T, BgpParserError<&'a [u8]>>;
macro_rules! fail {
( $($arg:expr),* $(,)? ) => {
Failure(BgpParserError::Malformed(format!($($arg),*)))
};
}
/// Parses `buf` to its end as a list of `item`s. Unlike nom's `many0`,
/// which stops quietly at an item it cannot read, an error in an item is an
/// error of the list.
fn list<'a, T>(mut buf: &'a [u8], item: impl Fn(&'a [u8]) -> Parsed<'a, T>) -> Parsed<'a, Vec<T>> {
let mut items = Vec::new();
while !buf.is_empty() {
let (rest, parsed) = item(buf)?;
items.push(parsed);
buf = rest;
}
Ok((buf, items))
}
use crate::message::BGP4_VERSION;
use crate::wire::{Parsed, fill_length, list, malformed, parse_afi_safi, put_afi_safi, whole};
/// The optional parameter type that marks an OPEN whose optional parameters
/// have two-octet lengths (RFC 9072).
@@ -62,7 +44,7 @@ impl OpenMessage {
/// in a well-formed OPEN is nothing.
pub fn from_wire(buf: &[u8]) -> Parsed<'_, Self> {
let (buf, version) = be_u8(buf)?;
if version != 4 {
if version != BGP4_VERSION {
return Err(Failure(BgpParserError::ProtocolError(
ProtocolErrorValues::UnsupportedVersion(version),
)));
@@ -255,7 +237,7 @@ pub enum Capability {
OutboundRouteFilter(Vec<OrfFamily>),
/// Extended Next Hop encoding (RFC 8950).
ExtendedNextHop {
entries: Vec<ExtendedNextHop>,
entries: Vec<ExtendedNextHopEntry>,
},
/// Extended Message (RFC 8654).
ExtendedMessage,
@@ -266,7 +248,7 @@ pub enum Capability {
},
/// Multiple labels (RFC 8277 §2.1): for each address family, how many
/// labels the speaker can handle in one NLRI.
MultiLabelCompat {
MultiLabel {
entries: Vec<MultiLabelEntry>,
},
BgpRole {
@@ -298,10 +280,10 @@ pub enum Capability {
/// Enhanced Route Refresh (RFC 7313).
EnhancedRouteRefresh,
/// Long Lived Graceful Restart (RFC 9494).
LongLivedGracefulRestart(Vec<LongLivedGracefulRestart>),
LongLivedGracefulRestart(Vec<LongLivedGracefulRestartEntry>),
/// A capability this implementation has no definition for, kept as it
/// came (RFC 5492 §3: what is not understood is ignored).
UnknownCapability {
Unknown {
type_code: u8,
value: Vec<u8>,
},
@@ -316,12 +298,6 @@ fn afi_reserved_safi(buf: &[u8]) -> Parsed<'_, (Afi, Safi)> {
Ok((buf, (Afi(afi), Safi(safi))))
}
fn afi_safi(buf: &[u8]) -> Parsed<'_, (Afi, Safi)> {
let (buf, afi) = be_u16(buf)?;
let (buf, safi) = be_u8(buf)?;
Ok((buf, (Afi(afi), Safi(safi))))
}
impl Capability {
fn type_code(&self) -> u8 {
use cap_type::*;
@@ -332,14 +308,14 @@ impl Capability {
Self::ExtendedNextHop { .. } => EXTENDED_NEXT_HOP,
Self::ExtendedMessage => EXTENDED_MESSAGE,
Self::BgpSec { .. } => BGP_SEC,
Self::MultiLabelCompat { .. } => MULTI_LABEL,
Self::MultiLabel { .. } => MULTI_LABEL,
Self::BgpRole { .. } => BGP_ROLE,
Self::GracefulRestart { .. } => GRACEFUL_RESTART,
Self::FourByteAsn { .. } => FOUR_BYTE_ASN,
Self::AddPath { .. } => ADD_PATH,
Self::EnhancedRouteRefresh => ENHANCED_ROUTE_REFRESH,
Self::LongLivedGracefulRestart(_) => LONG_LIVED_GRACEFUL_RESTART,
Self::UnknownCapability { type_code, .. } => *type_code,
Self::Unknown { type_code, .. } => *type_code,
}
}
@@ -387,10 +363,10 @@ impl Capability {
// else: no SAFI is larger than 255.
let safi = u8::try_from(safi)
.map(Safi)
.map_err(|_| fail!("SAFI {} does not fit in one octet", safi))?;
.map_err(|_| malformed(format!("SAFI {safi} does not fit in one octet")))?;
Ok((
buf,
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi(afi),
safi,
nh_afi: Afi(nh_afi),
@@ -398,7 +374,7 @@ impl Capability {
))
})?;
if entries.is_empty() {
return Err(fail!("Extended Next Hop capability with no entries"));
return Err(malformed("Extended Next Hop capability with no entries"));
}
(rest, Self::ExtendedNextHop { entries })
}
@@ -416,14 +392,14 @@ impl Capability {
}
MULTI_LABEL => {
let (rest, entries) = list(value, |buf| {
let (buf, (afi, safi)) = afi_safi(buf)?;
let (buf, AfiSafi { afi, safi }) = parse_afi_safi(buf)?;
let (buf, count) = be_u8(buf)?;
Ok((buf, MultiLabelEntry { afi, safi, count }))
})?;
if entries.is_empty() {
return Err(fail!("Multiple Labels capability with no entries"));
return Err(malformed("Multiple Labels capability with no entries"));
}
(rest, Self::MultiLabelCompat { entries })
(rest, Self::MultiLabel { entries })
}
BGP_ROLE => {
let (rest, role) = be_u8(value)?;
@@ -438,7 +414,7 @@ impl Capability {
// Four bits of flags and twelve of restart time.
let (rest, flags_restart_time) = be_u16(value)?;
let (rest, entries) = list(rest, |buf| {
let (buf, (afi, safi)) = afi_safi(buf)?;
let (buf, AfiSafi { afi, safi }) = parse_afi_safi(buf)?;
let (buf, flags) = be_u8(buf)?;
Ok((
buf,
@@ -476,7 +452,7 @@ impl Capability {
Some(AddPathEntry {
afi: Afi(u16::from_be_bytes([entry[0], entry[1]])),
safi: Safi(entry[2]),
send_recv: AddPathSendRecv::try_from(entry[3]).ok()?,
send_recv: SendReceive::try_from(entry[3]).ok()?,
})
})
.collect();
@@ -485,12 +461,12 @@ impl Capability {
ENHANCED_ROUTE_REFRESH => (value, Self::EnhancedRouteRefresh),
LONG_LIVED_GRACEFUL_RESTART => {
let (rest, entries) = list(value, |buf| {
let (buf, (afi, safi)) = afi_safi(buf)?;
let (buf, AfiSafi { afi, safi }) = parse_afi_safi(buf)?;
let (buf, flags) = be_u8(buf)?;
let (buf, stale_time) = be_u24(buf)?;
Ok((
buf,
LongLivedGracefulRestart {
LongLivedGracefulRestartEntry {
afi,
safi,
flags,
@@ -502,19 +478,13 @@ impl Capability {
}
type_code => (
&[][..],
Self::UnknownCapability {
Self::Unknown {
type_code,
value: value.to_vec(),
},
),
};
if !rest.is_empty() {
return Err(fail!(
"{} octets left over in capability {type_code}",
rest.len()
));
}
Ok(capability)
whole(format_args!("capability {type_code}"), (rest, capability))
}
/// Writes the capability, with its type and length, to the end of
@@ -525,13 +495,7 @@ impl Capability {
let length_at = out.len();
out.put_u8(0);
self.value_to_wire(out)?;
let length = out.len() - length_at - 1;
out[length_at] = u8::try_from(length).map_err(|_| EncodeError::TooLong {
what: "capability",
len: length,
max: u8::MAX.into(),
})?;
Ok(())
fill_length(out, length_at, 1, u8::MAX.into(), "capability")
}
fn value_to_wire(&self, out: &mut BytesMut) -> Result<(), EncodeError> {
@@ -560,10 +524,9 @@ impl Capability {
out.put_u8(header.0);
out.put_u16(afi.0);
}
Capability::MultiLabelCompat { entries } => {
Capability::MultiLabel { entries } => {
for entry in entries {
out.put_u16(entry.afi.0);
out.put_u8(entry.safi.0);
put_afi_safi(AfiSafi::new(entry.afi, entry.safi), out);
out.put_u8(entry.count);
}
}
@@ -585,28 +548,25 @@ impl Capability {
u16::from(*restart_state) << 15 | u16::from(*notification) << 14 | restart_time,
);
for entry in entries {
out.put_u16(entry.afi.0);
out.put_u8(entry.safi.0);
put_afi_safi(AfiSafi::new(entry.afi, entry.safi), out);
out.put_u8(u8::from(entry.preserve_forwarding) << 7);
}
}
Capability::FourByteAsn { asn } => out.put_u32(*asn),
Capability::AddPath { entries } => {
for entry in entries {
out.put_u16(entry.afi.0);
out.put_u8(entry.safi.0);
put_afi_safi(entry.family(), out);
out.put_u8(entry.send_recv as u8);
}
}
Capability::LongLivedGracefulRestart(entries) => {
for entry in entries {
out.put_u16(entry.afi.0);
out.put_u8(entry.safi.0);
put_afi_safi(AfiSafi::new(entry.afi, entry.safi), out);
out.put_u8(entry.flags);
out.put_slice(&entry.stale_time.to_be_bytes()[1..]);
}
}
Capability::UnknownCapability { value, .. } => out.put_slice(value),
Capability::Unknown { value, .. } => out.put_slice(value),
}
Ok(())
}
@@ -629,8 +589,8 @@ impl OrfFamily {
for _ in 0..count {
let (rest, orf_type) = be_u8(buf)?;
let (rest, send_recv) = be_u8(rest)?;
let send_recv = OrfSendRecv::try_from(send_recv)
.map_err(|e| fail!("Failed to parse ORF send/recv: {}", e))?;
let send_recv = SendReceive::try_from(send_recv)
.map_err(|e| malformed(format!("Failed to parse ORF send/recv: {e}")))?;
orfs.push(OutboundRouteFilterEntry {
r#type: OrfType(orf_type),
send_recv,
@@ -662,7 +622,7 @@ impl OrfFamily {
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct OutboundRouteFilterEntry {
pub r#type: OrfType,
pub send_recv: OrfSendRecv,
pub send_recv: SendReceive,
}
/// An ORF type. Any number is one: most are unassigned or vendor-specific.
@@ -687,9 +647,6 @@ pub enum SendReceive {
Both = 3,
}
pub type OrfSendRecv = SendReceive;
pub type AddPathSendRecv = SendReceive;
impl SendReceive {
/// Whether the speaker that advertised this sends.
pub fn sends(self) -> bool {
@@ -723,7 +680,7 @@ impl TryFrom<u8> for SendReceive {
pub struct AddPathEntry {
pub afi: Afi,
pub safi: Safi,
pub send_recv: AddPathSendRecv,
pub send_recv: SendReceive,
}
impl AddPathEntry {
@@ -732,11 +689,11 @@ impl AddPathEntry {
}
}
/// `ExtendedNextHop` allows for the nexthop to be for a different address family
/// `ExtendedNextHopEntry` allows for the nexthop to be for a different address family
/// than the one being used to advertise prefixes for. E.g. it allows for sending
/// IPv4 prefixes with an IPv6 nexthop.
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct ExtendedNextHop {
pub struct ExtendedNextHopEntry {
/// The Address Family to update the nexthop format for.
pub afi: Afi,
/// The Subsequent Address Family to update the nexthop format for.
@@ -783,7 +740,7 @@ impl From<&BgpRole> for u8 {
}
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct LongLivedGracefulRestart {
pub struct LongLivedGracefulRestartEntry {
pub afi: Afi,
pub safi: Safi,
pub flags: u8,
@@ -820,9 +777,8 @@ mod tests {
use eyre::Result;
use crate::open::{
AddPathEntry, AddPathSendRecv, Capability, ExtendedNextHop, GracefulRestartEntry,
MultiLabelEntry, OpenMessage, OpenOption, OrfFamily, OrfSendRecv, OrfType,
OutboundRouteFilterEntry,
AddPathEntry, Capability, ExtendedNextHopEntry, GracefulRestartEntry, MultiLabelEntry,
OpenMessage, OpenOption, OrfFamily, OrfType, OutboundRouteFilterEntry, SendReceive,
};
use bgp_primitives::constants::{Afi, Safi};
@@ -843,7 +799,7 @@ mod tests {
}
test_capability_roundtrip!(
test_multiptotocol,
test_multiprotocol,
&[0x01, 0x04, 0x00, 0x01, 0x00, 0x01],
Capability::MultiProtocol {
afi: Afi::IPV4,
@@ -856,7 +812,7 @@ mod tests {
test_capability_roundtrip!(
test_route_refresh_cisco_unsupported,
&[0x80, 0x00],
Capability::UnknownCapability {
Capability::Unknown {
type_code: 0x80,
value: vec![]
}
@@ -908,17 +864,17 @@ mod tests {
],
Capability::ExtendedNextHop {
entries: vec![
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
},
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::MULTICAST,
nh_afi: Afi::IPV6,
},
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::MPLS_VPN,
nh_afi: Afi::IPV6
@@ -946,7 +902,7 @@ mod tests {
afi: Afi::IPV4,
safi: Safi::UNICAST,
}]),
OpenOption::Capabilities(vec![Capability::UnknownCapability {
OpenOption::Capabilities(vec![Capability::Unknown {
type_code: 0x80,
value: vec![],
}]),
@@ -964,17 +920,17 @@ mod tests {
}]),
OpenOption::Capabilities(vec![Capability::ExtendedNextHop {
entries: vec![
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
},
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::MULTICAST,
nh_afi: Afi::IPV6,
},
ExtendedNextHop {
ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::MPLS_VPN,
nh_afi: Afi::IPV6,
@@ -1002,7 +958,7 @@ mod tests {
assert!(rest.is_empty());
assert_eq!(
cap,
Capability::MultiLabelCompat {
Capability::MultiLabel {
entries: vec![
MultiLabelEntry {
afi: Afi::IPV4,
@@ -1045,7 +1001,7 @@ mod tests {
entries: vec![AddPathEntry {
afi: evpn.0,
safi: evpn.1,
send_recv: AddPathSendRecv::Both,
send_recv: SendReceive::Both,
}],
},
),
@@ -1094,12 +1050,12 @@ mod tests {
AddPathEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
send_recv: AddPathSendRecv::Receive,
send_recv: SendReceive::Receive,
},
AddPathEntry {
afi: Afi::IPV6,
safi: Safi::UNICAST,
send_recv: AddPathSendRecv::Both,
send_recv: SendReceive::Both,
},
],
}
@@ -1159,8 +1115,8 @@ mod tests {
assert_eq!(
cap,
Capability::OutboundRouteFilter(vec![
family(Afi::IPV4, OrfSendRecv::Both),
family(Afi::IPV6, OrfSendRecv::Receive),
family(Afi::IPV4, SendReceive::Both),
family(Afi::IPV6, SendReceive::Receive),
])
);
let mut out = BytesMut::new();
@@ -1176,7 +1132,7 @@ mod tests {
#[test]
fn long_optional_parameters_use_two_octet_lengths() {
let capabilities: Vec<Capability> = (0..60)
.map(|i| Capability::UnknownCapability {
.map(|i| Capability::Unknown {
type_code: 200,
value: vec![i; 4],
})
@@ -1234,7 +1190,7 @@ mod tests {
/// used to be written with the length wrapped around.
#[test]
fn a_capability_too_long_for_its_length_octet_is_refused() {
let entry = ExtendedNextHop {
let entry = ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
crates/packet/src/update_decode.rs
+90 -25
View File
@@ -201,8 +201,9 @@ pub fn merge_as4_path(attributes: &mut Vec<PathAttribute>) {
else {
return;
};
let PathAttribute::As4Path(wide) = attributes.remove(at) else {
return;
let wide = match attributes.remove(at) {
PathAttribute::As4Path(wide) => wide,
other => unreachable!("{other:?} is not the AS4_PATH found at {at}"),
};
for attribute in attributes {
if let PathAttribute::AsPath(path) = attribute {
@@ -406,8 +407,6 @@ impl<'a> AttributeReader<'a> {
// Known by their definition but not modelled: kept as bytes, once
// their length has been checked.
let length_is_valid = match raw.type_code {
ORIGINATOR_ID => Some(value.len() == 4),
CLUSTER_LIST => Some(!value.is_empty() && value.len().is_multiple_of(4)),
AS4_AGGREGATOR => Some(value.len() == 8),
_ => None,
};
@@ -422,7 +421,7 @@ impl<'a> AttributeReader<'a> {
/// The attribute kept as bytes.
fn opaque(raw: &RawAttribute<'_>) -> PathAttribute {
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: raw.flags.clone(),
type_code: raw.type_code,
payload: raw.value.to_vec(),
@@ -503,12 +502,6 @@ mod tests {
OriginPathAttribute,
};
const MED: u8 = 4;
const COMMUNITIES: u8 = 8;
const EXTENDED_COMMUNITIES: u8 = 16;
const LARGE_COMMUNITY: u8 = 32;
const ONLY_TO_CUSTOMER: u8 = 35;
const WELL_KNOWN: u8 = 0x40;
const OPTIONAL: u8 = 0x80;
const OPTIONAL_TRANSITIVE: u8 = 0xc0;
@@ -604,7 +597,7 @@ mod tests {
#[test]
fn a_well_formed_update_is_left_alone() {
let decoded = announce_with(&ebgp(), attr(OPTIONAL, MED, &[0, 0, 0, 50]));
let decoded = announce_with(&ebgp(), attr(OPTIONAL, MULTI_EXIT_DISC, &[0, 0, 0, 50]));
assert!(decoded.errors.is_empty());
assert_eq!(decoded.update.announced_routes, [route()]);
assert_eq!(
@@ -634,7 +627,7 @@ mod tests {
// Nothing when the routes were withdrawn instead: there is no route
// for the attributes to belong to.
let decoded = announce_with(&ebgp(), attr(OPTIONAL, MED, &[1]));
let decoded = announce_with(&ebgp(), attr(OPTIONAL, MULTI_EXIT_DISC, &[1]));
assert_eq!(decoded.attributes_wire, None);
let decoded = UpdateMessage::decode(&ebgp(), &body(&NLRI, &[], &[])).unwrap();
assert_eq!(decoded.attributes_wire, None);
@@ -693,7 +686,11 @@ mod tests {
#[test]
fn a_malformed_attribute_withdraws_the_routes_and_keeps_the_session() {
let cases: Vec<(&str, u8, Vec<u8>)> = vec![
("MED of three octets", MED, attr(OPTIONAL, MED, &[0, 0, 50])),
(
"MED of three octets",
MULTI_EXIT_DISC,
attr(OPTIONAL, MULTI_EXIT_DISC, &[0, 0, 50]),
),
(
"COMMUNITIES with nothing in it",
COMMUNITIES,
@@ -711,13 +708,13 @@ mod tests {
),
(
"LARGE_COMMUNITY of eleven octets",
LARGE_COMMUNITY,
attr(OPTIONAL_TRANSITIVE, LARGE_COMMUNITY, &[0; 11]),
LARGE_COMMUNITIES,
attr(OPTIONAL_TRANSITIVE, LARGE_COMMUNITIES, &[0; 11]),
),
(
"MED marked transitive",
MED,
attr(OPTIONAL_TRANSITIVE, MED, &[0, 0, 0, 50]),
MULTI_EXIT_DISC,
attr(OPTIONAL_TRANSITIVE, MULTI_EXIT_DISC, &[0, 0, 0, 50]),
),
// RFC 9234 §5.
(
@@ -805,13 +802,13 @@ mod tests {
fn an_attribute_that_overruns_the_attributes_withdraws_what_was_found() {
// A MED that claims 200 octets.
let mut attributes = mandatory();
attributes.push(vec![OPTIONAL, MED, 200, 0, 0]);
attributes.push(vec![OPTIONAL, MULTI_EXIT_DISC, 200, 0, 0]);
let decoded = UpdateMessage::decode(&ebgp(), &body(&[], &attributes, &NLRI)).unwrap();
assert_withdrawn(&decoded, Some(MED), "overrun");
assert_withdrawn(&decoded, Some(MULTI_EXIT_DISC), "overrun");
// Two stray octets after the last attribute: not even a header.
let mut attributes = mandatory();
attributes.push(vec![OPTIONAL, MED]);
attributes.push(vec![OPTIONAL, MULTI_EXIT_DISC]);
let decoded = UpdateMessage::decode(&ebgp(), &body(&[], &attributes, &NLRI)).unwrap();
assert_withdrawn(&decoded, None, "stray octets");
}
@@ -862,10 +859,13 @@ mod tests {
#[test]
fn a_repeated_attribute_keeps_its_first_value() {
let mut attributes = mandatory();
attributes.push(attr(OPTIONAL, MED, &[0, 0, 0, 50]));
attributes.push(attr(OPTIONAL, MED, &[0, 0, 0, 99]));
attributes.push(attr(OPTIONAL, MULTI_EXIT_DISC, &[0, 0, 0, 50]));
attributes.push(attr(OPTIONAL, MULTI_EXIT_DISC, &[0, 0, 0, 99]));
let decoded = UpdateMessage::decode(&ebgp(), &body(&[], &attributes, &NLRI)).unwrap();
assert_eq!(handled(&decoded), [(Handling::AttributeDiscard, Some(MED))]);
assert_eq!(
handled(&decoded),
[(Handling::AttributeDiscard, Some(MULTI_EXIT_DISC))]
);
assert!(
decoded
.update
@@ -899,6 +899,21 @@ mod tests {
let reflected = UpdateMessage::decode(&ibgp(), &body(&[], &attributes, &NLRI)).unwrap();
assert!(reflected.errors.is_empty());
assert_eq!(reflected.update.path_attributes.len(), 6);
assert!(
reflected
.update
.path_attributes
.contains(&PathAttribute::OriginatorId(Ipv4Addr::new(10, 0, 0, 1)))
);
assert!(
reflected
.update
.path_attributes
.contains(&PathAttribute::ClusterList(vec![
Ipv4Addr::new(10, 0, 0, 2),
Ipv4Addr::new(10, 0, 0, 3)
]))
);
let mut attributes = mandatory();
attributes.push(local_pref());
@@ -959,6 +974,56 @@ mod tests {
assert_eq!(decoded.update.announced_routes, [route()]);
}
/// AS4_AGGREGATOR (RFC 6793 §4.2.2) is kept as it came, once its
/// length is known to be the four octets of an AS number and the four
/// of an address. One of another length is dropped and the route kept
/// (RFC 6793 §6), like a malformed AGGREGATOR.
#[test]
fn as4_aggregator_is_kept_opaque_and_dropped_if_malformed() {
let ctx = ParserContext {
four_octet_asn: false,
ibgp: false,
add_path: Vec::new(),
};
let attributes = |as4_aggregator: Vec<u8>| {
vec![
origin(),
attr(WELL_KNOWN, AS_PATH, &[2, 1, 0x5b, 0xa0]),
next_hop(),
as4_aggregator,
]
};
// AS 4200000001, 192.0.2.9.
let value = [0xfa, 0x56, 0xea, 0x01, 192, 0, 2, 9];
let good = attr(OPTIONAL_TRANSITIVE, AS4_AGGREGATOR, &value);
let decoded = UpdateMessage::decode(&ctx, &body(&[], &attributes(good), &NLRI)).unwrap();
assert!(decoded.errors.is_empty());
assert_eq!(
decoded.update.path_attributes.last(),
Some(&PathAttribute::Unknown {
flags: PathAttributeFlags(OPTIONAL_TRANSITIVE),
type_code: AS4_AGGREGATOR,
payload: value.to_vec(),
})
);
for (case, bad) in [
("two-octet AS number", &value[2..]),
("no address", &value[..4]),
("nothing", &[][..]),
] {
let bad = attr(OPTIONAL_TRANSITIVE, AS4_AGGREGATOR, bad);
let decoded = UpdateMessage::decode(&ctx, &body(&[], &attributes(bad), &NLRI)).unwrap();
assert_eq!(
handled(&decoded),
[(Handling::AttributeDiscard, Some(AS4_AGGREGATOR))],
"{case}"
);
assert_eq!(decoded.update.announced_routes, [route()], "{case}");
assert_eq!(decoded.update.path_attributes.len(), 3, "{case}");
}
}
/// RFC 6793 §4.2.3. AS4_PATH is the tail of the path; the front is
/// kept from AS_PATH. One longer than the path is not used, and is
/// gone all the same: nothing after the decoder is to see two paths,
@@ -1139,7 +1204,7 @@ mod tests {
assert!(decoded.errors.is_empty());
assert_eq!(
decoded.update.path_attributes[3],
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: PathAttributeFlags(0xe0),
type_code: 99,
payload: vec![1, 2, 3],
crates/packet/src/wire.rs
+189
View File
@@ -0,0 +1,189 @@
//! What the readers and writers of the wire format share: the shape of a
//! parser's result and its errors, the fields that several messages and
//! attributes have in common, and the lengths that are written before what
//! they count.
use bytes::{BufMut, BytesMut};
use nom::Err::Failure;
use nom::IResult;
use nom::number::complete::{be_u8, be_u16, be_u32};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_primitives::parser::{BgpParserError, EncodeError};
use crate::message::AS_TRANS;
/// A parser's result: what is left of the input, and what was read.
pub(crate) type Parsed<'a, T> = IResult<&'a [u8], T, BgpParserError<&'a [u8]>>;
/// The error for input that does not follow the format; `detail` says how.
pub(crate) fn malformed<'a>(detail: impl Into<String>) -> nom::Err<BgpParserError<&'a [u8]>> {
Failure(BgpParserError::Malformed(detail.into()))
}
/// Parses `buf` to its end as a list of `item`s. Unlike nom's `many0`,
/// which stops quietly at an item it cannot read, an error in an item is an
/// error of the list.
pub(crate) fn list<'a, T>(
mut buf: &'a [u8],
item: impl Fn(&'a [u8]) -> Parsed<'a, T>,
) -> Parsed<'a, Vec<T>> {
let mut items = Vec::new();
while !buf.is_empty() {
let (rest, parsed) = item(buf)?;
items.push(parsed);
buf = rest;
}
Ok((buf, items))
}
/// A parsed value, provided the parser used all of the input. `what` names
/// the input in the error.
pub(crate) fn whole<T>(
what: impl std::fmt::Display,
(rest, parsed): (&[u8], T),
) -> Result<T, nom::Err<BgpParserError<&[u8]>>> {
if rest.is_empty() {
Ok(parsed)
} else {
Err(malformed(format!(
"{} octets left over in {what}",
rest.len()
)))
}
}
/// `value` as a list of one or more records of `N` octets each: how the
/// community attributes and CLUSTER_LIST are laid out. An empty list is
/// malformed (RFC 7606 §7.8 for COMMUNITIES, and the others alike). `what`
/// names the attribute in the error.
pub(crate) fn records<'a, const N: usize>(
value: &'a [u8],
what: &str,
) -> Result<&'a [[u8; N]], nom::Err<BgpParserError<&'a [u8]>>> {
let (records, rest) = value.as_chunks::<N>();
if records.is_empty() || !rest.is_empty() {
return Err(malformed(format!(
"{what} is {} octets long, not a non-empty multiple of {N}",
value.len()
)));
}
Ok(records)
}
/// Parses an AS number as wide as the session's: four octets or two
/// (RFC 6793).
pub(crate) fn parse_asn(four_octet: bool, buf: &[u8]) -> Parsed<'_, u32> {
if four_octet {
be_u32(buf)
} else {
be_u16(buf).map(|(buf, asn)| (buf, u32::from(asn)))
}
}
/// Writes an AS number as wide as the session's. On a two-octet session
/// one that does not fit is written as AS_TRANS (RFC 6793 §4.2.2).
pub(crate) fn put_asn(four_octet: bool, asn: u32, out: &mut BytesMut) {
if four_octet {
out.put_u32(asn);
} else {
out.put_u16(u16::try_from(asn).unwrap_or(AS_TRANS));
}
}
/// Parses an address family as the multiprotocol attributes and most
/// capabilities name it: a two-octet AFI and a one-octet SAFI.
pub(crate) fn parse_afi_safi(buf: &[u8]) -> Parsed<'_, AfiSafi> {
let (buf, afi) = be_u16(buf)?;
let (buf, safi) = be_u8(buf)?;
Ok((buf, AfiSafi::new(Afi(afi), Safi(safi))))
}
/// Writes an address family as [`parse_afi_safi`] reads it.
pub(crate) fn put_afi_safi(family: AfiSafi, out: &mut BytesMut) {
out.put_u16(family.afi.0);
out.put_u8(family.safi.0);
}
/// Writes `length` into the field of `width` octets reserved at `at`, if
/// it is no more than `max`. `what` names the field in the error.
pub(crate) fn put_length(
out: &mut BytesMut,
at: usize,
width: usize,
length: usize,
max: usize,
what: &'static str,
) -> Result<(), EncodeError> {
if length > max {
return Err(EncodeError::TooLong {
what,
len: length,
max,
});
}
let bytes = length.to_be_bytes();
out[at..at + width].copy_from_slice(&bytes[bytes.len() - width..]);
Ok(())
}
/// Fills in the field of `width` octets reserved at `at` with how much
/// has been written after it, if that is no more than `max`: for a length
/// that is known only once what it counts has been written. `what` names
/// the field in the error.
pub(crate) fn fill_length(
out: &mut BytesMut,
at: usize,
width: usize,
max: usize,
what: &'static str,
) -> Result<(), EncodeError> {
let length = out.len() - at - width;
put_length(out, at, width, length, max, what)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_length_is_filled_in_after_what_it_counts() {
let mut out = BytesMut::new();
out.put_u8(0xaa);
let at = out.len();
out.put_u16(0);
out.put_slice(&[1, 2, 3]);
fill_length(&mut out, at, 2, u16::MAX.into(), "field").unwrap();
assert_eq!(&out[..], [0xaa, 0, 3, 1, 2, 3]);
let mut out = BytesMut::new();
out.put_u8(0);
out.put_slice(&[0; 300]);
assert_eq!(
fill_length(&mut out, 0, 1, u8::MAX.into(), "field"),
Err(EncodeError::TooLong {
what: "field",
len: 300,
max: 255,
})
);
}
#[test]
fn records_are_a_whole_number_of_them_and_at_least_one() {
assert_eq!(
records::<4>(&[1, 2, 3, 4, 5, 6, 7, 8], "X").unwrap().len(),
2
);
for value in [&[][..], &[1, 2, 3][..], &[1, 2, 3, 4, 5][..]] {
let error = records::<4>(value, "X").unwrap_err();
assert_eq!(
crate::message::describe(&error),
format!(
"X is {} octets long, not a non-empty multiple of 4",
value.len()
)
);
}
}
}
crates/packet/tests/properties.rs
+109 -35
View File
@@ -25,9 +25,9 @@ use bgp_packet::message::{
PathAttributeFlags, RouteRefreshMessage, UpdateMessage,
};
use bgp_packet::open::{
AddPathEntry, AddPathSendRecv, BgpRole, BgpSecHeader, Capability, ExtendedNextHop,
GracefulRestartEntry, LongLivedGracefulRestart, MultiLabelEntry, OpenMessage, OpenOption,
OrfFamily, OrfSendRecv, OrfType, OutboundRouteFilterEntry,
AddPathEntry, BgpRole, BgpSecHeader, Capability, ExtendedNextHopEntry, GracefulRestartEntry,
LongLivedGracefulRestartEntry, MultiLabelEntry, OpenMessage, OpenOption, OrfFamily, OrfType,
OutboundRouteFilterEntry, SendReceive,
};
use bgp_primitives::constants::{Afi, AfiSafi, IpVersion, Safi};
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
@@ -137,19 +137,6 @@ fn extended_community() -> impl Strategy<Value = ExtendedCommunity> {
fn mp_reach(session: Session) -> impl Strategy<Value = MpReachNlriPathAttribute> {
let ip_safi = || prop_oneof![Just(Safi::UNICAST), Just(Safi::MULTICAST)];
let v4 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
any::<[u8; 4]>(),
nlri(session, IpVersion::Ipv4, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV4, safi),
next_hop: NlriNextHop::Ipv4(Ipv4Addr::from(next_hop)),
prefixes,
});
let v6_next_hop = || {
prop_oneof![
any::<[u8; 16]>().prop_map(|a| NlriNextHop::Ipv6(Ipv6Addr::from(a))),
@@ -161,6 +148,27 @@ fn mp_reach(session: Session) -> impl Strategy<Value = MpReachNlriPathAttribute>
}),
]
};
// IPv4 routes have an IPv4 next hop, or an IPv6 one over a link that
// has only IPv6 addresses (RFC 8950).
let v4_next_hop = move || {
prop_oneof![
3 => any::<[u8; 4]>().prop_map(|a| NlriNextHop::Ipv4(Ipv4Addr::from(a))),
1 => v6_next_hop(),
]
};
let v4 = ip_safi()
.prop_flat_map(move |safi| {
(
Just(safi),
v4_next_hop(),
nlri(session, IpVersion::Ipv4, safi),
)
})
.prop_map(|(safi, next_hop, prefixes)| MpReachNlriPathAttribute {
family: AfiSafi::new(Afi::IPV4, safi),
next_hop,
prefixes,
});
let v6 = ip_safi()
.prop_flat_map(move |safi| {
(
@@ -212,7 +220,7 @@ fn unknown_attributes() -> impl Strategy<Value = Vec<PathAttribute>> {
if payload.len() > 255 {
flags |= 0x10;
}
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: PathAttributeFlags(flags),
type_code,
payload,
@@ -292,6 +300,22 @@ fn path_attributes(session: Session) -> impl Strategy<Value = Vec<PathAttribute>
})
})),
proptest::option::of(any::<u32>().prop_map(PathAttribute::OnlyToCustomer)),
// Route reflection's, which only an internal peer sends.
proptest::option::of((any::<[u8; 4]>(), vec(any::<[u8; 4]>(), 1..4))).prop_map(
move |reflected| {
reflected
.filter(|_| session.ibgp)
.map(|(originator, clusters)| {
vec![
PathAttribute::OriginatorId(Ipv4Addr::from(originator)),
PathAttribute::ClusterList(
clusters.into_iter().map(Ipv4Addr::from).collect(),
),
]
})
.unwrap_or_default()
},
),
unknown_attributes(),
);
(mandatory, optional).prop_map(|(mut attributes, optional)| {
@@ -306,6 +330,7 @@ fn path_attributes(session: Session) -> impl Strategy<Value = Vec<PathAttribute>
as4,
large,
otc,
reflected,
unknown,
) = optional;
attributes.extend(
@@ -324,6 +349,7 @@ fn path_attributes(session: Session) -> impl Strategy<Value = Vec<PathAttribute>
.into_iter()
.flatten(),
);
attributes.extend(reflected);
attributes.extend(unknown);
attributes
})
@@ -351,16 +377,16 @@ fn capability() -> impl Strategy<Value = Capability> {
let orf = (
any::<u8>().prop_map(OrfType),
prop_oneof![
Just(OrfSendRecv::Receive),
Just(OrfSendRecv::Send),
Just(OrfSendRecv::Both)
Just(SendReceive::Receive),
Just(SendReceive::Send),
Just(SendReceive::Both)
],
)
.prop_map(|(r#type, send_recv)| OutboundRouteFilterEntry { r#type, send_recv });
let add_path = prop_oneof![
Just(AddPathSendRecv::Receive),
Just(AddPathSendRecv::Send),
Just(AddPathSendRecv::Both)
Just(SendReceive::Receive),
Just(SendReceive::Send),
Just(SendReceive::Both)
];
prop_oneof![
(afi(), safi()).prop_map(|(afi, safi)| Capability::MultiProtocol { afi, safi }),
@@ -376,7 +402,7 @@ fn capability() -> impl Strategy<Value = Capability> {
vec((afi(), safi(), afi()), 1..4).prop_map(|entries| Capability::ExtendedNextHop {
entries: entries
.into_iter()
.map(|(afi, safi, nh_afi)| ExtendedNextHop { afi, safi, nh_afi })
.map(|(afi, safi, nh_afi)| ExtendedNextHopEntry { afi, safi, nh_afi })
.collect(),
}),
Just(Capability::ExtendedMessage),
@@ -385,7 +411,7 @@ fn capability() -> impl Strategy<Value = Capability> {
afi,
}),
vec((afi(), safi(), any::<u8>()), 1..4).prop_map(|entries| {
Capability::MultiLabelCompat {
Capability::MultiLabel {
entries: entries
.into_iter()
.map(|(afi, safi, count)| MultiLabelEntry { afi, safi, count })
@@ -432,12 +458,14 @@ fn capability() -> impl Strategy<Value = Capability> {
Capability::LongLivedGracefulRestart(
entries
.into_iter()
.map(|(afi, safi, flags, stale_time)| LongLivedGracefulRestart {
afi,
safi,
flags,
stale_time,
})
.map(
|(afi, safi, flags, stale_time)| LongLivedGracefulRestartEntry {
afi,
safi,
flags,
stale_time,
},
)
.collect(),
)
}),
@@ -447,7 +475,7 @@ fn capability() -> impl Strategy<Value = Capability> {
}),
vec(any::<u8>(), 0..20)
)
.prop_map(|(type_code, value)| Capability::UnknownCapability { type_code, value }),
.prop_map(|(type_code, value)| Capability::Unknown { type_code, value }),
]
}
@@ -674,7 +702,9 @@ proptest! {
check_decoding(&ctx, &damaged(&wire, &damage))?;
}
/// The parsers below the message level are public too.
/// The parsers below the message level are public too. What they
/// accept must write back as it was read, where the format leaves
/// nothing for the reader to normalise.
#[test]
fn noise_is_never_more_than_an_error_for_the_parts(
session in session(),
@@ -682,8 +712,52 @@ proptest! {
) {
let ctx = session.ctx();
let _ = OpenMessage::from_wire(&bytes);
let _ = Capability::from_wire(&bytes);
let _ = UpdateMessage::decode(&ctx, &bytes);
let _ = PathAttribute::from_wire(&ctx, &bytes);
if let Ok((rest, capability)) = Capability::from_wire(&bytes) {
let consumed = &bytes[..bytes.len() - rest.len()];
let mut rewritten = BytesMut::new();
capability.to_wire(&mut rewritten)
.map_err(|e| TestCaseError::fail(format!("accepted but cannot be written: {e}")))?;
// ADD-PATH families with a Send/Receive value nobody defined
// are left out (RFC 7911 §4), and the reserved octet of
// Multiprotocol and ORF and the reserved bits of Graceful
// Restart are read as zero whatever they were. Everything else
// is written as it was read.
let normalised = matches!(
capability,
Capability::AddPath { .. }
| Capability::MultiProtocol { .. }
| Capability::OutboundRouteFilter(_)
| Capability::GracefulRestart { .. }
);
if !normalised {
prop_assert_eq!(&rewritten[..], consumed, "{:?}", capability);
}
let again = Capability::from_wire(&rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e:?}")))?;
prop_assert_eq!(again, (&[][..], capability));
}
if let Ok((rest, attribute)) = PathAttribute::from_wire(&ctx, &bytes) {
let consumed = &bytes[..bytes.len() - rest.len()];
let mut rewritten = BytesMut::new();
attribute.to_wire(&ctx, &mut rewritten)
.map_err(|e| TestCaseError::fail(format!("accepted but cannot be written: {e}")))?;
// The flags are not checked against the attribute's definition
// here, and are written as defined; and the bits of a prefix
// past its length are cleared. With the flags as defined and
// no prefixes, the attribute is written as it was read.
let names_prefixes = matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
);
if rewritten[0] == consumed[0] && !names_prefixes {
prop_assert_eq!(&rewritten[..], consumed, "{:?}", attribute);
}
let again = PathAttribute::from_wire(&ctx, &rewritten)
.map_err(|e| TestCaseError::fail(format!("written but cannot be read: {e:?}")))?;
prop_assert_eq!(again, (&[][..], attribute));
}
}
}
crates/primitives/src/ip_prefix.rs
+10 -8
View File
@@ -64,7 +64,7 @@ impl IpPrefix {
let mut address = [0u8; 16];
address[..needed].copy_from_slice(&prefix[..needed]);
if let (Some(last), rem @ 1..) = (address[..needed].last_mut(), length % 8) {
*last &= u8::MAX << (8 - rem);
*last &= high_bits_mask(rem);
}
Ok(Self {
address_family,
@@ -98,7 +98,7 @@ impl IpPrefix {
let whole = usize::from(length / 8);
address[..whole].copy_from_slice(&self.address[..whole]);
if !length.is_multiple_of(8) {
address[whole] = self.address[whole] & (u8::MAX << (8 - length % 8));
address[whole] = self.address[whole] & high_bits_mask(length % 8);
}
Self {
address_family: self.address_family,
@@ -122,11 +122,18 @@ impl IpPrefix {
if rem_bits == 0 {
return true;
}
let mask = u8::MAX << (8 - rem_bits);
let mask = high_bits_mask(rem_bits);
(self.address[full_bytes] & mask) == (other.address[full_bytes] & mask)
}
}
/// The mask that keeps the first `bits` bits of an octet and clears the
/// rest, for `bits` from 1 to 7: what the last octet of a prefix whose
/// length is not a whole number of octets is masked with.
fn high_bits_mask(bits: u8) -> u8 {
u8::MAX << (8 - bits)
}
/// A prefix as an UPDATE names it: the prefix and, on a session where
/// ADD-PATH is in use for the address family (RFC 7911), the identifier
/// that tells apart the several routes a peer may announce for it.
@@ -144,11 +151,6 @@ impl Nlri {
pub fn plain(prefixes: impl IntoIterator<Item = IpPrefix>) -> Vec<Self> {
prefixes.into_iter().map(Self::from).collect()
}
/// The bytes this takes on the wire, with or without its identifier.
pub fn wire_len(&self, with_path_id: bool) -> usize {
1 + self.prefix.bytes().len() + if with_path_id { 4 } else { 0 }
}
}
impl From<IpPrefix> for Nlri {
crates/rib/src/table/attrs.rs
+39
View File
@@ -209,6 +209,23 @@ impl AttrSet {
plain
}
/// Whether the route carries the BLACKHOLE community (RFC 7999), which
/// asks for the packets to its prefix to be dropped. Read off the
/// stored bytes without building anything else, for those who ask it
/// of every route.
pub fn is_blackhole(&self) -> bool {
const BLACKHOLE: [u8; 4] = [0xFF, 0xFF, 0x02, 0x9A];
let mut wire = &*self.wire;
while let Ok((attribute, rest)) = split_attribute(wire) {
if attribute.type_code == attr_type::COMMUNITIES {
let (communities, _) = attribute.value.as_chunks::<4>();
return communities.contains(&BLACKHOLE);
}
wire = rest;
}
false
}
/// Puts the AS numbers of the route's path in `out`, in place of what
/// was there: in the order of the path, the neighbor's first and the
/// origin's last, sets and all. Returns whether the path has a set in
@@ -595,6 +612,28 @@ mod tests {
assert!(path.is_empty());
}
/// The BLACKHOLE community is read off the stored bytes.
#[test]
fn a_blackhole_request_is_told_from_the_bytes() {
let store = AttrStore::new();
let with = |communities: Vec<(u16, u16)>| {
let mut attributes = attributes(vec![3333]);
attributes.retain(|attribute| !matches!(attribute, PathAttribute::Communities(_)));
attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
communities,
)));
store.intern(&attributes, None).unwrap()
};
assert!(with(vec![(3333, 100), (0xFFFF, 0x029A)]).is_blackhole());
assert!(!with(vec![(3333, 666), (0xFFFF, 0xFF01)]).is_blackhole());
assert!(
!store
.intern(&attributes(vec![3333]), None)
.unwrap()
.is_blackhole()
);
}
/// Two of the same attribute are not a set: its readers could each
/// take a different one.
#[test]
crates/rib/src/table/journal.rs
+32 -7
View File
@@ -387,9 +387,8 @@ impl std::fmt::Debug for Routes<'_> {
pub enum Event<'a> {
/// Forget what you know. The whole table follows, as [`Event::Best`]
/// or [`Event::Routes`] for every prefix, and then [`Event::Synced`].
/// The first thing a new
/// reader is given, and what a reader that fell too far behind is
/// given in place of what it missed.
/// The first thing a new reader is given, and what a reader that fell
/// too far behind is given in place of what it missed.
Reset,
/// This is now the best route for `prefix`. Given to a follower of the
/// best routes.
@@ -577,8 +576,14 @@ impl Drop for Reader {
enum Drained {
/// Entries were dropped that this reader had not read.
Lagged,
/// `events` were given out; `more` if the budget ran out first.
Read { events: usize, more: bool },
/// `events` were given out; `consumed` if the reader's place in the
/// journal moved at all; `more` if entries were left past the time
/// read up to.
Read {
events: usize,
consumed: bool,
more: bool,
},
}
impl Reader {
@@ -642,6 +647,7 @@ impl Reader {
self.seen = version;
let mut shard = next;
let mut lagged = false;
let mut consumed_any = false;
let mut more_somewhere = false;
while shard < shards {
if events >= budget {
@@ -653,8 +659,13 @@ impl Reader {
lagged = true;
break;
}
Drained::Read { events: read, more } => {
Drained::Read {
events: read,
consumed,
more,
} => {
events += read;
consumed_any |= consumed;
more_somewhere |= more;
}
}
@@ -665,6 +676,14 @@ impl Reader {
continue;
}
self.phase = Phase::Follow { next: 0, upto: 0 };
if next == 0 && !consumed_any && more_somewhere {
// A whole pass got nowhere, though there is more:
// it is all beyond the earliest change still under
// way, which no pass can read past until it ends.
// Asking again at once would spin until then. The
// change wakes the readers when it ends.
return self.progress(events, true);
}
if next != 0 || more_somewhere {
// This pass took up where an earlier call stopped,
// so the shards before that point have not been
@@ -809,6 +828,7 @@ impl Reader {
if cursor == journal.seq {
return Drained::Read {
events: 0,
consumed: false,
more: false,
};
}
@@ -837,8 +857,13 @@ impl Reader {
// Past the last entry there is nothing up to the latest change.
cursor = journal.seq;
}
let consumed = cursor != self.cursors[index];
self.set_cursor(index, &shard, cursor);
Drained::Read { events, more }
Drained::Read {
events,
consumed,
more,
}
}
}
crates/rib/src/table/journal_tests.rs
+39
View File
@@ -517,6 +517,45 @@ async fn a_waiting_reader_is_woken_by_a_change_of_best_route() {
.expect("the change was already there");
}
/// A reader does not read past a change that is still under way, which
/// a change to every shard (a peer's whole table going) is for a while.
/// Changes made by others meanwhile are beyond it. A reader that finds
/// only those must say it is caught up and wait, not ask again at once
/// for as long as the change takes; and when the change ends, it is
/// woken, though the change itself wrote nothing for it.
#[tokio::test]
async fn a_change_under_way_does_not_keep_a_reader_spinning() {
let fixture = Fixture::new(&[external(1), external(2)]);
let mut reader = fixture.table.reader();
let mut mirror = Mirror::default();
mirror.catch_up(&mut reader, 100);
// A long change begins, and another writer announces meanwhile.
let change = fixture.table.begin_change();
fixture.announce(1, &prefix("10.0.0.0/8"), &Route::via(&[2]));
let progress = reader.poll(100, |_| {});
assert_eq!(progress.events, 0, "nothing can be read past the change");
assert!(
progress.caught_up,
"and the reader is to wait, not ask again"
);
let waiting = tokio::time::timeout(std::time::Duration::from_millis(50), reader.changed());
assert!(
waiting.await.is_err(),
"nothing to be had until the change ends"
);
// The change ends without writing anything itself.
drop(change);
tokio::time::timeout(std::time::Duration::from_secs(5), reader.changed())
.await
.expect("woken when the change ended");
mirror.catch_up(&mut reader, 100);
mirror.assert_matches(&fixture.table);
assert_eq!(mirror.best.len(), 1);
}
/// Writers on several threads, a reader on another, reading while they
/// write. When the writers are done and the reader has caught up, it has
/// what the table has.
crates/rib/src/table/mod.rs
+118 -56
View File
@@ -58,6 +58,10 @@ pub struct PeerInfo {
pub interface: u32,
/// Whether the peer is in our own AS.
pub ibgp: bool,
/// Whether routes learned from our other routers are passed on to the
/// peer, and its routes to them: whether it is a route reflector
/// client of ours (RFC 4456).
pub reflector_client: bool,
}
/// A peer's number in a table, given out by [`RibTable::add_peer`]. Small,
@@ -189,13 +193,16 @@ impl Row {
.position(|(index, path)| path.peer() == peer && self.path_id(index) == path_id)
}
fn push(&mut self, path: Path, path_id: u32) {
/// Adds a route. `peers` is how many peers the table has: a prefix
/// that one of them announces, the others mostly announce too, so a
/// row is given room for that many at its first growth.
fn push(&mut self, path: Path, path_id: u32, peers: usize) {
if path_id != 0 && self.path_ids.is_empty() {
self.path_ids = vec![0; self.paths.len()];
}
push_tight(&mut self.paths, path);
push_tight(&mut self.paths, path, peers);
if !self.path_ids.is_empty() {
push_tight(&mut self.path_ids, path_id);
push_tight(&mut self.path_ids, path_id, peers);
}
}
@@ -378,13 +385,6 @@ impl RibTable {
&self.attrs
}
/// Makes room for a peer's routes and returns the number to announce
/// them under.
///
/// The number is the peer's until [`RibTable::remove_peer`] is called
/// with it, and may then be given to another. Whoever was given it must
/// not use it after that: the table cannot tell a late announcement
/// from one of the number's next owner.
/// What is known of the peer with this number, if it is in use. Not to
/// be called from a visitor that is given routes with a shard held.
pub fn peer(&self, index: PeerIndex) -> Option<PeerInfo> {
@@ -392,6 +392,13 @@ impl RibTable {
peers.get(index.as_usize()).copied().flatten()
}
/// Makes room for a peer's routes and returns the number to announce
/// them under.
///
/// The number is the peer's until [`RibTable::remove_peer`] is called
/// with it, and may then be given to another. Whoever was given it must
/// not use it after that: the table cannot tell a late announcement
/// from one of the number's next owner.
pub fn add_peer(&self, info: PeerInfo) -> Result<PeerIndex, TooManyPeers> {
let mut peers = self.peers.lock().unwrap_or_else(PoisonError::into_inner);
let index = match peers.iter().position(Option::is_none) {
@@ -424,29 +431,18 @@ impl RibTable {
/// held while it does, so other peers can come and go meanwhile.
pub fn remove_peer(&self, peer: PeerIndex) -> Outcome {
let mut outcome = Outcome::default();
let mut news = false;
let change = self.begin_change();
for shard in &self.shards {
let mut shard = write(shard);
shard.journals.set_stamp(change.stamp);
let before = shard.journals.mark();
self.change(self.shards.iter().map(|shard| (shard, ())), |shard, ()| {
outcome += shard.remove_peer(peer);
// From here on the shard takes no more routes from the peer.
if let Some(slot) = shard.peers.get_mut(peer.as_usize()) {
*slot = None;
}
shard.tidy_journals();
news |= shard.journals.has_news_since(before);
}
});
// Only now can the number be given out again: no shard knows it.
let mut peers = self.peers.lock().unwrap_or_else(PoisonError::into_inner);
if let Some(slot) = peers.get_mut(peer.as_usize()) {
*slot = None;
}
drop(peers);
if news {
self.wake_readers();
}
outcome
}
@@ -507,30 +503,20 @@ impl RibTable {
/// again, before or during this, is fresh and stays.
pub fn sweep_stale(&self, peer: PeerIndex, versions: &[IpVersion]) -> Outcome {
let mut outcome = Outcome::default();
let mut news = false;
let change = self.begin_change();
for shard in &self.shards {
let mut shard = write(shard);
shard.journals.set_stamp(change.stamp);
let before = shard.journals.mark();
self.change(self.shards.iter().map(|shard| (shard, ())), |shard, ()| {
outcome += shard.remove_where(|row, path| {
path.peer() == peer
&& path.is_stale()
&& versions.contains(&row.prefix.address_family())
});
shard.tidy_journals();
news |= shard.journals.has_news_since(before);
}
drop(change);
if news {
self.wake_readers();
}
});
outcome
}
/// How many routes of `peer` are stale; see [`RibTable::mark_stale`].
/// Looks at every prefix in the table.
pub fn count_stale(&self, peer: PeerIndex) -> usize {
#[cfg(test)]
pub(super) fn count_stale(&self, peer: PeerIndex) -> usize {
self.shards
.iter()
.map(|shard| {
@@ -590,7 +576,7 @@ impl RibTable {
/// Calls `apply` once for each shard that some of `routes` fall in,
/// with that shard locked and those routes. Wakes the readers if a
/// journal one of them follows was written to.
fn by_shard<N>(&self, routes: &[N], mut apply: impl FnMut(&mut Shard, &[(u64, Nlri)]))
fn by_shard<N>(&self, routes: &[N], apply: impl FnMut(&mut Shard, &[(u64, Nlri)]))
where
N: Copy + Into<Nlri>,
{
@@ -616,17 +602,34 @@ impl RibTable {
};
placed.sort_unstable_by_key(|(hash, _)| self.shard_of(*hash));
let mut news = false;
// One time for all of this call's changes, in every shard.
let runs = placed
.chunk_by(|a, b| self.shard_of(a.0) == self.shard_of(b.0))
.map(|run| (&self.shards[self.shard_of(run[0].0)], run));
self.change(runs, apply);
}
/// Makes one change to the table, part by part: `apply` is called with
/// each shard that `parts` names locked for writing, and the part's
/// own item. All the parts are one change, at one time on the clock.
///
/// Readers are woken once the change is complete in every shard, if a
/// journal one of them follows was written to.
fn change<'a, T>(
&'a self,
parts: impl IntoIterator<Item = (&'a RwLock<Shard>, T)>,
mut apply: impl FnMut(&mut Shard, T),
) {
let change = self.begin_change();
for run in placed.chunk_by(|a, b| self.shard_of(a.0) == self.shard_of(b.0)) {
let mut shard = write(&self.shards[self.shard_of(run[0].0)]);
let mut news = false;
for (shard, part) in parts {
let mut shard = write(shard);
shard.journals.set_stamp(change.stamp);
let before = shard.journals.mark();
apply(&mut shard, run);
apply(&mut shard, part);
shard.tidy_journals();
news |= shard.journals.has_news_since(before);
}
drop(change);
if news {
self.wake_readers();
}
@@ -635,12 +638,17 @@ impl RibTable {
/// Takes a time on the clock for a change that is about to be made,
/// and notes that it is under way until the guard drops.
fn begin_change(&self) -> Change<'_> {
let version = self.version.load(std::sync::atomic::Ordering::Acquire);
let stamp = self.clock.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
self.in_progress
.lock()
.unwrap_or_else(PoisonError::into_inner)
.insert(stamp);
Change { table: self, stamp }
Change {
table: self,
stamp,
version,
}
}
/// The time on the clock up to which every change is complete in every
@@ -665,6 +673,23 @@ impl RibTable {
self.len() == 0
}
/// Makes room for `prefixes` prefixes, spread over the shards, so that
/// a table of that size goes in without the shards growing as it
/// does. Growing a shard moves every row it has into a bigger table;
/// over a full table that was a seventh of what a session spent
/// taking it in. Room that is already there is kept.
pub fn expect_prefixes(&self, prefixes: usize) {
let each = prefixes.div_ceil(self.shards.len());
for shard in &self.shards {
let mut shard = write(shard);
let hasher = shard.hasher.clone();
let additional = each.saturating_sub(shard.rows.len());
shard
.rows
.reserve(additional, |row| hasher.hash_one(row.prefix));
}
}
/// How many prefixes have at least one route.
pub fn prefix_count(&self) -> usize {
self.shards.iter().map(|shard| read(shard).rows.len()).sum()
@@ -684,6 +709,23 @@ impl RibTable {
.unwrap_or_default()
}
/// Every route for `prefix`, from the best down: after the best, the
/// route that would be chosen were the ones before it gone, and so on.
pub fn ranked_routes(&self, prefix: &IpPrefix) -> Vec<Route> {
let hash = self.hasher.hash_one(prefix);
let shard = read(&self.shards[self.shard_of(hash)]);
shard
.rows
.find(hash, |row| row.prefix == *prefix)
.map(|row| {
select::ranked(row, &shard.peers)
.into_iter()
.map(|index| row.route(index).to_owned())
.collect()
})
.unwrap_or_default()
}
fn with_row<T>(&self, prefix: &IpPrefix, read_row: impl FnOnce(&Row) -> T) -> Option<T> {
let hash = self.hasher.hash_one(prefix);
let shard = read(&self.shards[self.shard_of(hash)]);
@@ -709,13 +751,6 @@ impl RibTable {
}
}
/// Up to `limit` routes: every route of as many prefixes as fit, in no
/// particular order. For showing some of a table that is too large to
/// show all of.
pub fn some_routes(&self, limit: usize) -> Vec<Route> {
self.routes_where(limit, |_| true)
}
/// Up to `limit` of the routes that `wanted` picks, in no particular
/// order. This looks at routes until it has enough, which for a choosy
/// `wanted` is every route in the table: it is for looking into the
@@ -777,6 +812,9 @@ fn write(shard: &RwLock<Shard>) -> RwLockWriteGuard<'_, Shard> {
struct Change<'a> {
table: &'a RibTable,
stamp: u64,
/// The readers' version when it began, to tell whether they were
/// woken while it was under way.
version: u64,
}
impl Drop for Change<'_> {
@@ -786,6 +824,17 @@ impl Drop for Change<'_> {
.lock()
.unwrap_or_else(PoisonError::into_inner)
.remove(&self.stamp);
// A reader does not read past a change still under way. One that
// another change woke meanwhile found nothing it could read yet,
// and is woken again now that it can.
if self
.table
.version
.load(std::sync::atomic::Ordering::Acquire)
!= self.version
{
self.table.wake_readers();
}
}
}
@@ -809,12 +858,25 @@ fn spread(hash: u64, count: usize) -> usize {
((stirred * count as u64) >> 32) as usize
}
/// The most room a row is given at its first growth, however many peers
/// there are: a route server's clients do not all announce every prefix.
const FIRST_ROOM: usize = 8;
/// Adds to a vector without the doubling that `push` does: a row of a few
/// hundred routes should not hold room for as many again. Grows by a
/// quarter, which keeps adding cheap on average.
fn push_tight<T>(vec: &mut Vec<T>, value: T) {
/// hundred routes should not hold room for as many again. The first
/// growth is to `expected` (at most [`FIRST_ROOM`]), since with several
/// full tables nearly every row gets that many routes, and growing a row
/// one route at a time was a twelfth of what taking in four tables cost.
/// After that it grows by a quarter, which keeps adding cheap on average.
fn push_tight<T>(vec: &mut Vec<T>, value: T, expected: usize) {
if vec.len() == vec.capacity() {
vec.reserve_exact((vec.len() / 4).max(1));
let first = expected.min(FIRST_ROOM);
let more = if vec.capacity() < first {
first - vec.len()
} else {
(vec.len() / 4).max(1)
};
vec.reserve_exact(more);
}
vec.push(value);
}
@@ -856,7 +918,7 @@ impl Shard {
let (outcome, best_changed) = match row.position(peer, path_id) {
None => {
row.push(path, path_id);
row.push(path, path_id, self.peers.len());
self.routes += 1;
let outcome = Outcome {
added: 1,
crates/rib/src/table/select.rs
+35 -1
View File
@@ -27,7 +27,7 @@
//! of all its routes.
use super::attrs::SelectionKey;
use super::{DEFAULT_LOCAL_PREF, PeerInfo, Row};
use super::{DEFAULT_LOCAL_PREF, Path, PeerInfo, Row};
/// Steps 1 to 3 as one number of 48 bits: the larger, the better.
///
@@ -187,6 +187,40 @@ fn choose(row: &mut Row, peers: &[Option<PeerInfo>], scratch: &mut Scratch) -> b
row.best != before
}
/// The routes of `row` from the best down, as indices into its routes:
/// after the best, the one that would be chosen were the best gone, and
/// so on. What an operator asks when looking at a prefix: which route
/// would take over.
pub(super) fn ranked(row: &Row, peers: &[Option<PeerInfo>]) -> Vec<usize> {
// A copy to take the winners out of one by one, and where each of its
// routes is in the row itself.
let mut rest = Row {
prefix: row.prefix,
best: 0,
seqs: row.seqs,
paths: row
.paths
.iter()
.map(|path| Path {
word: path.word,
attrs: std::sync::Arc::clone(&path.attrs),
})
.collect(),
path_ids: row.path_ids.clone(),
};
let mut places: Vec<usize> = (0..row.paths.len()).collect();
let mut scratch = Scratch::default();
let mut order = Vec::with_capacity(row.paths.len());
while !rest.paths.is_empty() {
choose(&mut rest, peers, &mut scratch);
let at = rest.best();
order.push(places[at]);
rest.swap_remove(at);
places.swap_remove(at);
}
order
}
/// For a route just added at `added` that ties with the best in steps 1 to
/// 3: whether it is the new best, if that can be said from the two routes
/// alone. `None` if the whole row has to be looked at.
crates/rib/src/table/tests.rs
+88 -2
View File
@@ -63,6 +63,7 @@ pub(super) fn external(number: u8) -> PeerInfo {
address: IpAddr::from(Ipv4Addr::new(192, 0, 2, number)),
interface: 0,
ibgp: false,
reflector_client: false,
}
}
@@ -70,6 +71,7 @@ pub(super) fn internal(number: u8) -> PeerInfo {
PeerInfo {
asn: 64999,
ibgp: true,
reflector_client: false,
..external(number)
}
}
@@ -259,6 +261,62 @@ fn each_step_of_selection_decides_when_the_ones_before_tie() {
assert_eq!(fixture.best(&p), Some(1));
}
/// The ranking of a prefix's routes is selection run again without the
/// winners: each route in it is what would be chosen were the ones before
/// it gone.
#[test]
fn routes_are_ranked_as_they_would_be_chosen_one_after_another() {
let p = prefix("10.0.0.0/8");
// Peer 0 would win on identifier alone; peer 2 has the highest
// LOCAL_PREF, peer 1 the shortest path of the rest, peer 3 a MED
// that loses to peer 1's within their shared neighbouring AS.
let peers = [external(1), external(2), internal(3), external(4)];
let fixture = Fixture::new(&peers);
fixture.announce(0, &p, &Route::via(&[7, 1, 2]));
fixture.announce(1, &p, &Route::via(&[7, 3]));
fixture.announce(
2,
&p,
&Route {
local_pref: Some(200),
..Route::via(&[8, 1, 2, 3])
},
);
fixture.announce(
3,
&p,
&Route {
med: Some(5),
..Route::via(&[7, 4])
},
);
let ranked: Vec<usize> = fixture
.table
.ranked_routes(&p)
.iter()
.map(|route| {
fixture
.peers
.iter()
.position(|peer| *peer == route.peer)
.unwrap()
})
.collect();
assert_eq!(ranked, vec![2, 1, 3, 0]);
assert_eq!(fixture.best(&p), Some(2));
// Taking the best away makes the second the best, and so on down.
fixture.withdraw(2, &p);
assert_eq!(fixture.best(&p), Some(1));
fixture.withdraw(1, &p);
assert_eq!(fixture.best(&p), Some(3));
assert!(
fixture
.table
.ranked_routes(&prefix("10.1.0.0/16"))
.is_empty()
);
}
/// MEDs are compared only within a neighbouring AS. Taken pair by pair
/// that is not an order: A beats B, B beats C, C beats A. Whatever the
/// order of arrival, the answer must be the same.
@@ -899,13 +957,13 @@ fn a_row_can_hold_more_routes_than_a_sixteen_bit_index_counts() {
let mut scratch = select::Scratch::default();
let count = usize::from(u16::MAX) + 10;
for path_id in 2..=count as u32 {
row.push(path(&worse), path_id);
row.push(path(&worse), path_id, peers.len());
}
// Equal routes from one peer: the lowest identifier is the best.
assert!(!select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), 0);
row.push(path(&better), count as u32 + 1);
row.push(path(&better), count as u32 + 1, peers.len());
assert!(select::after_add(&mut row, &peers, &mut scratch));
assert_eq!(row.best(), count);
assert_eq!(row.route(row.best()).path_id, count as u32 + 1);
@@ -944,3 +1002,31 @@ fn routes_can_be_picked_out_of_the_table() {
);
assert!(table.routes_where(0, |_| true).is_empty());
}
#[test]
fn a_table_told_what_to_expect_has_the_room_before_the_routes_come() {
let table = RibTable::new(8);
table.expect_prefixes(100_000);
let room: usize = table
.shards
.iter()
.map(|shard| read(shard).rows.capacity())
.sum();
assert!(room >= 100_000, "room for {room} prefixes");
// Asking again for less, or for the same, takes nothing away.
table.expect_prefixes(10);
let still: usize = table
.shards
.iter()
.map(|shard| read(shard).rows.capacity())
.sum();
assert_eq!(still, room);
// And it is a table like any other.
let peer = table.add_peer(external(1)).unwrap();
let attrs = table
.attrs()
.intern(&Route::via(&[64512]).attributes(), None)
.unwrap();
table.announce(peer, &attrs, &[Nlri::from(prefix("192.0.2.0/24"))]);
assert_eq!(table.prefix_count(), 1);
}
crates/rpki/src/client.rs
+88 -26
View File
@@ -17,7 +17,6 @@
//! worse: it would go on refusing routes whose holders have since
//! authorised them.
use std::collections::HashSet;
use std::io;
use std::net::SocketAddr;
use std::sync::atomic::{AtomicU64, Ordering};
@@ -37,6 +36,10 @@ use crate::aspa::AspaTable;
use crate::rtr::{Pdu, RtrError, Timers, VERSION_0, VERSION_2, error_code};
use crate::vrp::{Vrp, VrpTable};
// Hashed by the hundred thousand with every answer, and told apart from
// what there was with every reset: a quick hash matters here.
type HashSet<K> = std::collections::HashSet<K, foldhash::fast::RandomState>;
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// How long a cache may take to send the next part of an answer.
@@ -54,6 +57,12 @@ const MAX_RECORDS: usize = 8_000_000;
/// How long to wait before trying the next cache after one has failed.
const NEXT_CACHE_DELAY: Duration = Duration::from_secs(1);
/// How long to wait before asking a cache again that said it has no data
/// yet, rather than the retry interval: it is up and validating, which
/// takes minutes after it starts, and the retry interval is ten unless it
/// has said otherwise, which it has not had the chance to.
const NO_DATA_RETRY: Duration = Duration::from_secs(30);
/// How long a cache that is being left is given to take the reason.
const ERROR_REPORT_TIMEOUT: Duration = Duration::from_secs(1);
@@ -220,8 +229,14 @@ enum SessionError {
Malformed(#[from] RtrError),
#[error("the cache closed the connection")]
Closed,
#[error("the cache did not answer within {RESPONSE_TIMEOUT:?}")]
Timeout,
#[error("the cache did not accept the connection within {0:?}")]
ConnectTimeout(Duration),
#[error("the cache sent nothing for {0:?} in the middle of an answer")]
ResponseTimeout(Duration),
#[error("the cache did not finish its answer within {0:?}")]
AnswerTimeout(Duration),
#[error("the cache confirmed nothing for so long that the data expired")]
Expired,
#[error("the cache reported an error: {text} (code {code})")]
Reported { code: u16, text: String },
#[error("the cache sent {0} where it has no place")]
@@ -243,8 +258,14 @@ impl SessionError {
Self::Malformed(e) => Some(e.code()),
Self::Unexpected(_) => Some(error_code::CORRUPT_DATA),
Self::Inconsistent(problem) => Some(problem.code),
Self::Io(_) | Self::Closed | Self::Timeout | Self::Reported { .. } => None,
Self::Downgrade(_) => None,
Self::Io(_)
| Self::Closed
| Self::ConnectTimeout(_)
| Self::ResponseTimeout(_)
| Self::AnswerTimeout(_)
| Self::Expired
| Self::Reported { .. }
| Self::Downgrade(_) => None,
}
}
}
@@ -419,23 +440,29 @@ impl Client {
self.protocol_version = version;
continue;
}
warn!(%cache, err = %ended, "RPKI cache session ended");
let no_data = matches!(
ended,
SessionError::Reported {
code: error_code::NO_DATA_AVAILABLE,
..
}
);
if no_data {
info!(%cache, "RPKI cache has no data yet; asking again soon");
} else {
warn!(%cache, err = %ended, "RPKI cache session ended");
}
self.status(|status| {
status.state = CacheState::Connecting;
status.last_error = Some(ended.to_string());
});
// The next cache at once; when all have been tried, a rest.
attempt += 1;
// The next cache, or this one when it is back, may speak a
// newer version.
self.protocol_version = VERSION_2;
let wait = if attempt.is_multiple_of(self.config.caches.len()) {
self.retry()
} else {
NEXT_CACHE_DELAY
};
self.wait(wait).await;
let all_tried = attempt.is_multiple_of(self.config.caches.len());
self.wait(rest(all_tried, no_data, self.retry())).await;
}
}
@@ -476,17 +503,14 @@ impl Client {
let stream = match timeout(CONNECT_TIMEOUT, TcpStream::connect(cache)).await {
Ok(Ok(stream)) => stream,
Ok(Err(e)) => return e.into(),
Err(_) => return SessionError::Timeout,
Err(_) => return SessionError::ConnectTimeout(CONNECT_TIMEOUT),
};
let mut connection = Connection {
stream,
buf: BytesMut::with_capacity(64 * 1024),
};
info!(%cache, "Connected to RPKI cache");
self.status(|status| {
status.state = CacheState::Synchronising;
status.protocol_version = self.protocol_version;
});
self.status(|status| status.state = CacheState::Synchronising);
loop {
if let Err(e) = self.synchronise(cache, &mut connection).await {
@@ -515,7 +539,7 @@ impl Client {
() = sleep_until(ask) => {}
() = sleep_until(expires), if expires < ask => {
self.expired();
return SessionError::Timeout;
return SessionError::Expired;
}
}
}
@@ -580,10 +604,13 @@ impl Client {
// one a minute would keep the old data in use for ever.
let deadline = Instant::now() + ANSWER_TIMEOUT;
let mut next = async || {
let wait = RESPONSE_TIMEOUT.min(deadline.saturating_duration_since(Instant::now()));
match timeout(wait, connection.next()).await {
let left = deadline.saturating_duration_since(Instant::now());
match timeout(RESPONSE_TIMEOUT.min(left), connection.next()).await {
Ok(pdu) => pdu,
Err(_) => Err(SessionError::Timeout),
Err(_) if left <= RESPONSE_TIMEOUT => {
Err(SessionError::AnswerTimeout(ANSWER_TIMEOUT))
}
Err(_) => Err(SessionError::ResponseTimeout(RESPONSE_TIMEOUT)),
}
};
@@ -618,6 +645,10 @@ impl Client {
_ => return Err(SessionError::Unexpected("data before a Cache Response")),
}
};
// The cache has answered in the version asked for: that is the
// version of the session.
let protocol_version = self.protocol_version;
self.status(|status| status.protocol_version = protocol_version);
if since.is_some_and(|place| place.session_id != session_id) {
// The cache has started over; its serials mean something else
// now.
@@ -699,7 +730,8 @@ impl Client {
serial,
});
self.timers = timers.unwrap_or_default();
self.expires = Some(Instant::now() + self.expire());
let expires_in = self.expire();
self.expires = Some(Instant::now() + expires_in);
let changes = built.changed.vrps.len() + built.changed.aspa_customers.len();
if first || changes > 0 {
self.publish(built.vrps, (!first).then_some(built.changed));
@@ -712,13 +744,13 @@ impl Client {
changes,
"RPKI data is up to date"
);
let expires_at = SystemTime::now() + self.expire();
let updated_at = SystemTime::now();
self.status(|status| {
status.state = CacheState::Synchronised;
status.session_id = Some(session_id);
status.serial = Some(serial);
status.updated_at = Some(SystemTime::now());
status.expires_at = Some(expires_at);
status.updated_at = Some(updated_at);
status.expires_at = Some(updated_at + expires_in);
status.last_error = None;
});
Ok(Answer::Complete)
@@ -889,3 +921,33 @@ enum Answer {
/// With a request to ask for everything instead.
Reset,
}
/// How long to rest after a session ended: the next cache at once; when
/// all have been tried, the retry interval, or less if the last one is
/// only still validating.
fn rest(all_tried: bool, no_data: bool, retry: Duration) -> Duration {
match (all_tried, no_data) {
(false, _) => NEXT_CACHE_DELAY,
(true, true) => NO_DATA_RETRY.min(retry),
(true, false) => retry,
}
}
#[cfg(test)]
mod tests {
use super::*;
/// A cache that is still validating after it started is asked again
/// in half a minute, not the ten the retry interval is by default.
#[test]
fn a_cache_without_data_yet_is_asked_again_soon() {
let ten_minutes = Duration::from_secs(600);
assert_eq!(rest(true, true, ten_minutes), NO_DATA_RETRY);
assert_eq!(rest(true, false, ten_minutes), ten_minutes);
// Never later than the retry interval says.
let short = Duration::from_secs(5);
assert_eq!(rest(true, true, short), short);
// Another cache first, whatever this one said.
assert_eq!(rest(false, true, ten_minutes), NEXT_CACHE_DELAY);
}
}
crates/rpki/src/lib.rs
-1
View File
@@ -8,7 +8,6 @@
//! tell a route whose origin is authorised from one whose origin is not:
//! the commonest kind of hijack and of mistake.
//!
//!
//! The same way, a network can sign a statement of who its providers are
//! (an ASPA). With those a router can tell a path that could have come
//! about by networks passing routes on as they are paid to from one that
crates/rpki/tests/client.rs
+5 -2
View File
@@ -425,7 +425,10 @@ async fn aspas_are_taken_replaced_and_withdrawn() {
PathValidity::Valid
);
let changed = update.changed.expect("a change to what there was");
assert_eq!(*changed.aspa_customers, [3333, 2121, 1101].into());
assert_eq!(
*changed.aspa_customers,
[3333, 2121, 1101].into_iter().collect()
);
assert!(changed.vrps.is_empty(), "no VRP came or went");
assert_eq!(update.vrps.len(), 1);
@@ -536,7 +539,7 @@ async fn a_cache_that_speaks_version_one_has_no_aspas() {
assert!(update.aspas.is_empty());
assert_eq!(update.vrps.len(), 1);
let changed = update.changed.expect("the difference to what there was");
assert_eq!(*changed.aspa_customers, [3333].into());
assert_eq!(*changed.aspa_customers, [3333].into_iter().collect());
assert!(changed.vrps.is_empty());
let status = rpki.status();
assert_eq!((status.protocol_version, status.aspas), (1, 0));
crates/server/proto/config.proto
+208 -18
View File
@@ -58,31 +58,103 @@ message BgpConfig {
// own.
ExportPolicy export = 12;
// Let routes say, with communities, which peers they are passed on to,
// as the route servers of Internet exchanges do. With L our own AS
// number and P a peer's:
// Let routes say, with communities, which peers they are passed on to
// and how far away we look to them, as the route servers of Internet
// exchanges and transit networks do. With L our own AS number, P a
// peer's, and G a group of sessions (`groups` of a peer):
//
// 0:P or L:0:P not to the peer with AS P
// L:P or L:1:P to the peer with AS P, and then only to peers
// named so
// 0:L or L:0:0 to nobody
// 0:P or L:0:P not to the peer with AS P
// L:P or L:1:P to the peer with AS P, and then only to peers
// named so
// 0:L or L:0:0 to nobody
// L:11:P with our AS once more on the path to the peer
// L:12:P with AS P; twice, three times. P 0 is every
// L:13:P peer in another AS.
// L:20:G not to the sessions of group G
// L:21:G to the sessions of group G, and then only to
// sessions named so, by group or AS
// L:31:G with our AS once more on the path to the
// L:32:G sessions of group G; twice, three times
// L:33:G
//
// A route with none of these goes to everyone it otherwise would. "Not"
// wins over "to". The two-part form is a standard community, with room
// for AS numbers below 65536; the three-part form is a large community
// and works for any. They are taken off a route as it is passed on.
// wins over "to". Of several prepends that apply, the most counts, on
// top of the session's export.as_path_prepend. The two-part form is a
// standard community, with room for AS numbers below 65536; the
// three-part forms are large communities and work for any.
//
// They are honoured on routes we originate too (`communities` of
// `originate`), and on what import.add_communities puts on.
// They are acted on where a route leaves our AS, and taken off there:
// to a peer in our own AS a route goes with them, for that router to act
// on. They are honoured on routes we originate too (`communities` and
// `large_communities` of `originate`), on what import.add_communities
// puts on, and on what a peer in our own AS, or one with
// import.accept_control_communities, sends. From any other peer the
// communities of our own AS are taken off as its routes come in, so that
// nobody outside can say what is done with our routes.
//
// With this on, standard communities L:n are taken for "to the peer
// with AS n": mark routes with something else (large communities
// L:2:n and up, say).
// L:100:n and up, say; functions 2 to 9 are left for marks too).
bool control_communities = 13;
// Graceful restart, for every peer that has no `graceful_restart` of its
// own. Left out, it is on, as the defaults below have it.
GracefulRestart graceful_restart = 14;
// Where this router passes on routes from one router of our AS to
// another (Peer.route_reflector_client), the cluster it does so for, as
// a dotted-quad: what it puts on such routes, and what it does not take
// back (RFC 4456). Give routers that serve the same clients the same
// cluster ID. Defaults to router_id.
string cluster_id = 15;
// Limits on what goes to other networks that are not our customers: to
// every peer whose role is PROVIDER, PEER or ROUTE_SERVER, and to one
// in another AS that has no role. Not to customers, route server
// clients or routers of our own AS. A peer's own `export_limits` takes
// the place of these.
//
// They hold whatever the routes are and wherever they came from, and
// whatever `export` says: a check on everything else, in case it goes
// wrong. With the roles, a route learned from a provider or a peer is
// passed on to customers only; these keep it from going further even
// if a route is not marked as it should be.
ExportLimits export_limits = 16;
// Names for communities, for the status page to show in place of the
// numbers: what the marks on routes mean to a reader. A name for one
// community ("64601:100:2" is "comrack"), or for a kind of them, with
// `*` for the part left open ("64601:102:*" is "interconnection"), which
// is shown with the open part after the name. The daemon itself does
// nothing with them.
repeated CommunityName community_names = 17;
}
// A name for a community, or for a kind of them.
message CommunityName {
// The community, standard ("64601:100") or large ("64601:100:2"), or a
// pattern of them with `*` for a part left open ("64601:102:*").
string community = 1;
// What it is shown as.
string name = 2;
}
// Limits on what is announced to a peer, which hold whatever `export`
// and the routes say.
message ExportLimits {
// Only routes for these prefixes are announced: our own address space
// and our customers', say, with max_length as long as the longest
// prefix announced from it. Routes for other prefixes, learned or
// originated, are not. Empty allows any.
repeated PrefixRange prefixes = 1;
// The most routes that are announced to the peer at a time. If more
// were to be, none of them is: the session is ended with a Cease /
// Administrative Shutdown NOTIFICATION that says why (RFC 9003), and the
// peer is kept down for its import.hold_down_seconds, as when it sends
// us too many. Set it somewhat above what is expected. Zero is no
// limit.
uint32 max_prefixes = 2;
}
// Graceful restart (RFC 4724): BGP restarting without the routes going.
@@ -148,6 +220,22 @@ message Kernel {
// was not announced again is removed.
// Zero removes them at once.
optional uint32 stale_seconds = 4;
// The kernel's table whose routes say how a next hop that is not a
// neighbour is reached: its connected and static routes, and an IGP's
// if there is one. Defaults to 254, the main table. A route whose next
// hop is covered by such a route is installed with that route's gateway
// and interface, the kernel not looking gateways up in turn. Routes put
// in the table by BGP, this daemon's or another's, never count: a next
// hop reached only by way of a BGP route rests on what may itself rest
// on a next hop.
uint32 resolution_table = 5;
// Whether the default route in that table counts as the way to a next
// hop. Off by default: on a router the default route points at a
// transit provider, and a next hop "reached" only by sending the
// packets to the Internet at large is not reached.
bool resolve_via_default_route = 6;
}
// Where to get the validated ROA payloads that say which AS may originate
@@ -178,7 +266,8 @@ message Rpki {
}
// A route originated by this router. It is announced with an AS_PATH of the
// local AS and, on each session, that session's local address as next hop.
// local AS (empty, to a peer in our own AS) and, on each session, that
// session's local address as next hop.
message OriginatedRoute {
// The prefix in CIDR notation, e.g. "192.0.2.0/24" or "2001:db8::/32".
string prefix = 1;
@@ -280,6 +369,48 @@ message Peer {
// Graceful restart with this peer. What is set here takes the place of
// what the top-level `graceful_restart` says.
GracefulRestart graceful_restart = 12;
// The groups the session is in, by number, for control communities to
// name sessions by (see control_communities): those at one Internet
// exchange, say, or those over one link, whatever AS the peers are. Use
// the same numbers on all routers of the AS: a route marked on one is
// acted on by another. From 1.
repeated uint32 groups = 13;
// Pass on to this peer, a router of our own AS, the routes learned from
// our other routers, and pass on the routes it sends to all of them: we
// are its route reflector (RFC 4456). Without it, as BGP has it, a route
// learned from one router of our AS goes to no other, and every router
// needs a session with every other. Routes go with this router as next
// hop, so that the peer can reach what lies behind another router
// through this one. Only for a peer in our own AS: other networks are
// told of learned routes by `export` and the roles alone. The peer's
// `export` still has to pass on learned routes.
bool route_reflector_client = 14;
// Limits on what is announced to the peer, in the place of the
// top-level export_limits. They apply to the same peers as those do: to
// one whose role is PROVIDER, PEER or ROUTE_SERVER, or that is in
// another AS and has no role.
ExportLimits export_limits = 15;
// Do not open connections to the peer: wait for it to connect. The
// session is configured in full, and comes up when the peer connects
// from `endpoint`'s address; the port of `endpoint` is not used. For a
// new peer whose side is not ready yet, or one that is to open the
// connections itself.
bool passive = 16;
// About how many routes the peer sends when all is well: a full table,
// a few hundred from an exchange peer, one from a customer. Nothing is
// decided by it. The status page compares the routes the peer has in
// the table with it, and shows an established session with fewer than
// half as needing attention: a peer that is up but has sent next to
// nothing looks, without this, exactly like one that is fine. Zero, the
// default, compares nothing. The routing table is also made with room
// for the largest of these from the start, so that it need not grow,
// and so use twice its memory for a moment, while the tables come in.
uint32 expected_routes = 17;
}
// What is announced to a peer.
@@ -290,8 +421,9 @@ message Peer {
//
// - Never back to the peer a route came from, nor to a peer whose AS is
// on its path already.
// - A route from a peer in our own AS is not passed to another such peer
// (there is no route reflection).
// - A route from a peer in our own AS is not passed to another such peer,
// unless one of them is a route reflector client of ours
// (Peer.route_reflector_client).
// - Where peers have roles (RFC 9234): a route learned from a provider, a
// peer or a route server goes on to customers only. So with roles on
// all peers, turning this on makes us transit for our customers and for
@@ -310,6 +442,14 @@ message ExportPolicy {
// Pass on the best routes learned from other peers.
bool learned_routes = 1;
// Pass on every route learned from other peers, not only the best of
// each prefix, each with a path identifier (ADD-PATH, RFC 7911). For a
// route collector, which wants to see every path. Needs learned_routes,
// and the peer's add_path to offer sending; for a family the peer did
// not agree to receive several routes for, the best goes as usual. The
// export policy and the limits apply to every route alike.
bool all_paths = 6;
// Communities put on every route announced to the peer: which router
// it went through, say. "64601:100", or a well-known name.
repeated string add_communities = 2;
@@ -367,7 +507,7 @@ message ImportPolicy {
// one, as from an upstream that is asked to send it.
bool accept_default_route = 4;
// The longest prefixes taken. Default to 24 and 48, the longest that are
// The longest prefixes taken. Defaults to 24 and 48, the longest that are
// routed between networks.
uint32 max_ipv4_prefix_length = 5;
uint32 max_ipv6_prefix_length = 6;
@@ -426,6 +566,55 @@ message ImportPolicy {
// The LOCAL_PREF of the peer's routes: of two routes for a prefix the
// one with the higher is chosen, whatever else they have. Left out, 100.
optional uint32 local_pref = 18;
// By default the communities of our own AS (L:n and L:n:m; and 0:n,
// with control_communities on) are taken off the routes of a peer in
// another AS as they come in, before add_communities puts ours on: what
// they ask of us is not for others to ask. True keeps them, for a
// customer that is to steer its own routes with control communities, or
// a route server's client. Not for a peer in our own AS, whose routes
// keep them in any case.
bool accept_control_communities = 19;
// What is done to some of the routes, beyond what is done to all of
// them above. Each route that is taken is looked at by the rules in
// turn, and the first that holds for it is applied; the rest are not
// looked at. A rule sees the route as the statements above leave it:
// with our communities taken off and ours put on.
repeated ImportRule rules = 20;
// By default a route with the BLACKHOLE community (65535:666, RFC
// 7999), which asks for the traffic to its prefix to be dropped, is
// refused ("blackhole request"). True takes such a route if it is for
// one address (a /32 or /128), whatever the longest prefix taken
// otherwise, and inside what the peer may announce; its next hop is
// not looked at. The traffic to the address is then dropped by this
// router (a blackhole route in the kernel), and the route goes on to
// the peers the export policy sends it to, community and all, for them
// to do the same. For a customer under attack, and for a route server
// whose blackhole routes are to be honoured.
bool accept_blackholes = 21;
}
// What is done to the routes with some communities, or for some prefixes:
// those learned at one exchange, say, by the mark another router of ours
// put on them, to be preferred less; or one prefix of a customer's, to be
// marked for a prepend.
message ImportRule {
// The rule holds for routes with one of these communities: one
// ("64601:100"), all of an AS ("64601:*"), or large ones with three
// parts ("64601:101:*"). Left out, for routes with any.
repeated string match_communities = 1;
// And for routes for a prefix in these. Left out, for any prefix.
repeated PrefixRange match_prefixes = 2;
// The LOCAL_PREF of the routes, in place of import.local_pref.
optional uint32 local_pref = 3;
// Communities put on the routes.
repeated string add_communities = 4;
repeated string add_large_communities = 5;
}
// A prefix, and optionally the more specific prefixes inside it.
@@ -442,7 +631,8 @@ enum AddPath {
ADD_PATH_OFF = 0;
// Accept several routes for a prefix from the peer.
ADD_PATH_RECEIVE = 1;
// Send the peer several routes for a prefix.
// Offer to send the peer several routes for a prefix. As yet only the
// best is sent all the same.
ADD_PATH_SEND = 2;
ADD_PATH_BOTH = 3;
}
crates/server/src/advertise.rs
+820 -115
View File
File diff suppressed because it is too large. Load diff
crates/server/src/collision.rs
+5 -4
View File
@@ -105,10 +105,10 @@ impl CollisionTable {
/// Call when a connection receives the peer's OPEN. `one_identifier`
/// says that the address is that of one router, which holds for a
/// configured peer. `None` means this
/// connection lost and must be closed with Cease / Connection Collision
/// Resolution. Otherwise any rival has been told to close and this
/// connection holds the slot until the guard drops.
/// configured peer. `None` means this connection lost and must be
/// closed with Cease / Connection Collision Resolution. Otherwise any
/// rival has been told to close and this connection holds the slot
/// until the guard drops.
pub fn admit(
&self,
local_id: Ipv4Addr,
@@ -143,6 +143,7 @@ impl CollisionTable {
}
if let Some(kill) = existing.kill.take() {
COLLISIONS.with_label_values(&["existing"]).inc();
// Fails only if the rival has gone already.
let _ = kill.send(());
}
}
crates/server/src/config.rs
+685 -135
View File
File diff suppressed because it is too large. Load diff
crates/server/src/connection.rs
+317 -204
View File
@@ -13,13 +13,13 @@ use std::pin::Pin;
use std::sync::Arc;
use std::time::Duration;
use bgp_fib::InterfacesWatch;
use bgp_fib::TopologyWatch;
use bgp_packet::errors::{BgpError, DecodeError, MsgHeaderSubcode};
use bgp_packet::message::RouteRefreshMessage;
use bgp_packet::message::{
BGP_HEADER_LEN, BGP_MAX_MSG_LEN, Decoded, Message, PathAttribute, UpdateMessage,
};
use bgp_packet::open::AddPathSendRecv;
use bgp_packet::open::SendReceive;
use bgp_packet::update_decode::{UpdateError, merge_as4_path};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_primitives::ip_prefix::Nlri;
@@ -41,14 +41,15 @@ use crate::config::PeerConfig;
use crate::error::SessionError;
use crate::family::FamilySet;
use crate::fsm::{Action, Event, Fsm, FsmConfig, Negotiated, Received};
use crate::holddown::{HeldDown, HoldDowns};
use crate::metrics::{
MESSAGES_RECEIVED, MESSAGES_SENT, PEER_ROUTES, PREFIX_LIMIT_EXCEEDED,
SESSIONS_ESTABLISHED_TOTAL, UPDATE_ERRORS, message_type_name,
EXPORT_LIMIT_EXCEEDED, MESSAGES_RECEIVED, MESSAGES_SENT, PEER_ADVERTISED, PEER_ROUTES,
PEER_TCP, PREFIX_LIMIT_EXCEEDED, SESSIONS_ESTABLISHED_TOTAL, UPDATE_ERRORS, message_type_name,
};
use crate::panic::contained;
use crate::policy::{self, ExportPolicy, HeldDown, HoldDowns, ImportPolicy, PeerRole};
use crate::policy::{self, ExportLimits, ExportPolicy, ImportPolicy, PeerRole};
use crate::reach::{Reach, neighbor_at};
use crate::refusals::{PeerRefusals, RefusalLog};
use crate::rejections::{PeerRejections, RejectionLog};
use crate::restart::{self, Agreed, GracefulRestartConfig, Leave, Restarts};
use crate::routes::{self, Held};
use crate::rpki::{Checks, Subject, Validator};
@@ -83,11 +84,12 @@ pub struct ServerContext {
/// Peers that are being kept down for going over their limits.
pub hold_downs: HoldDowns,
/// What peers announced that was not taken.
pub refusals: RefusalLog,
pub rejections: RejectionLog,
/// The data for route origin validation, where that is configured.
pub rpki: Option<Rpki>,
/// The machine's interfaces, where routes are installed in the kernel.
pub interfaces: Option<InterfacesWatch>,
/// The kernel's view of the machine, where routes are installed in
/// the kernel: what says whether a next hop can be reached.
pub topology: Option<TopologyWatch>,
/// The routes kept of peers that are restarting, and how far the peers
/// are with sending theirs since the start (graceful restart).
pub restarts: Arc<Restarts>,
@@ -105,7 +107,7 @@ pub struct PeerSettings {
/// The address families to offer the peer.
pub families: FamilySet,
/// The directions to offer the peer ADD-PATH for (RFC 7911).
pub add_path: Option<AddPathSendRecv>,
pub add_path: Option<SendReceive>,
/// Which of the peer's routes are taken.
pub import: Arc<ImportPolicy>,
/// What the peer is to us, if that is known.
@@ -114,8 +116,16 @@ pub struct PeerSettings {
pub require_role: bool,
/// What the peer is told, and how.
pub export: Arc<ExportPolicy>,
/// Limits on what the peer is told, if it is one they are for.
pub export_limits: Option<Arc<ExportLimits>>,
/// The groups the session is in, for control communities.
pub groups: Arc<[u32]>,
/// Whether the peer is a route reflector client of ours (RFC 4456).
pub reflector_client: bool,
/// Graceful restart with the peer.
pub graceful_restart: GracefulRestartConfig,
/// Whether the peer is left to connect to us, and not dialled.
pub passive: bool,
}
impl PeerSettings {
@@ -125,6 +135,7 @@ impl PeerSettings {
pub fn unconfigured(
hold_time_secs: u16,
export: Arc<ExportPolicy>,
export_limits: Option<Arc<ExportLimits>>,
graceful_restart: GracefulRestartConfig,
) -> Self {
Self {
@@ -137,7 +148,11 @@ impl PeerSettings {
role: None,
require_role: false,
export,
export_limits,
groups: Arc::default(),
reflector_client: false,
graceful_restart,
passive: false,
}
}
}
@@ -154,7 +169,11 @@ impl From<&PeerConfig> for PeerSettings {
role: peer.role,
require_role: peer.require_role,
export: Arc::clone(&peer.export),
export_limits: peer.export_limits.clone(),
groups: peer.groups.as_slice().into(),
reflector_client: peer.reflector_client,
graceful_restart: peer.graceful_restart,
passive: peer.passive,
}
}
}
@@ -169,6 +188,19 @@ pub struct SessionEnd {
pub result: Result<(), SessionError>,
}
/// Why a session ended when nothing went wrong.
pub const CLOSED_AT_OUR_REQUEST: &str = "session closed at our request";
impl SessionEnd {
/// Why the session ended, in a sentence.
pub fn reason(&self) -> String {
match &self.result {
Ok(()) => CLOSED_AT_OUR_REQUEST.to_string(),
Err(e) => e.to_string(),
}
}
}
/// Runs the BGP session on `stream` until it ends. `direction` is which
/// side opened the connection.
pub async fn run_session(
@@ -194,91 +226,13 @@ pub async fn run_session(
Err(panic) => Err(session.close_after_panic(panic).await),
};
let established = session.established.is_some();
let hold_downs = &session.server.hold_downs;
if matches!(result, Err(SessionError::PrefixLimitExceeded { .. })) {
// A peer that sends far more than it should will do so again on
// the next session; it is left alone for a while.
hold_downs.hold(
peer_addr.into(),
session.import.hold_down,
HeldDown::OverLimit,
);
PREFIX_LIMIT_EXCEEDED
.with_label_values(&[session.refusals.label()])
.inc();
}
// A configured peer whose sessions keep coming up and failing is
// slowed down. Not one that is nobody in particular: such peers are
// only taken on a test network, where many share an address.
if let (Some(up), Some(_)) = (&session.established, &session.peer_name)
&& let Some(hold) =
hold_downs.session_ended(peer_addr.into(), up.since.elapsed(), result.is_err())
{
warn!(
peer_addr = %socket_text(peer_addr),
hold_down = ?hold,
"The peer's sessions keep failing soon after they come up; keeping it down for a while"
);
}
hold_down_after(&session, &result);
// Whatever ended the session, routes learned over it are gone
// (RFC 4271 §8.2.2), unless the peer is restarting (RFC 4724 §4.2).
// Do this before the registration drops so that the routes are
// Done before the registration drops, so that the routes are
// withdrawn, or marked stale, before the session is seen to be down.
if let Some(established) = session.established.take() {
let leaving = *session.leave.borrow();
let keep_for = established
.graceful_restart
.as_ref()
.filter(|agreed| leaving.is_none() && restart::ends_in_restart(&result, agreed))
.map(|agreed| {
let theirs = Duration::from_secs(agreed.restart_time.into());
(
restart::versions(&agreed.families),
theirs.min(session.graceful_restart.stale_routes),
)
})
.filter(|(versions, time)| !versions.is_empty() && !time.is_zero());
let Established {
peer_id,
rib_peer,
routes,
_registration,
..
} = established;
// The peer has no session to count its routes for.
let _ = PEER_ROUTES.remove_label_values(&[session.refusals.label()]);
if leaving == Some(Leave::Restart) {
// The daemon is about to go, and its routes with it; the
// kernel keeps its own. Taking them out first would only hold
// the restart up.
routes.forget();
} else if let Some((versions, time)) = keep_for {
let kept = routes.keep(versions.clone()).await;
session.server.restarts.keep(
(peer_addr.into(), peer_id.asn),
peer_addr,
session.peer_name.clone(),
rib_peer,
versions,
kept,
time,
);
info!(
peer = %peer_id,
kept_routes = kept,
kept_for = ?time,
"Session ended; the peer is restarting, and its routes are kept meanwhile"
);
} else {
let outcome = routes.remove().await;
info!(
peer = %peer_id,
removed_routes = outcome.removed,
best_changed = outcome.best_changed,
"Session ended"
);
}
release_routes(&session, established, &result).await;
}
session.close().await;
SessionEnd {
@@ -287,6 +241,115 @@ pub async fn run_session(
}
}
/// Keeps the peer down for a while if the session that just ended says
/// to: a peer that sent far more than it should, or that we were to send
/// far more than we should, will be so again on the next session, and is
/// left alone until it or we are put right. A configured peer whose
/// sessions keep coming up and failing is slowed down; not one that is
/// nobody in particular, such as are only taken on a test network, where
/// many share an address.
fn hold_down_after(session: &Session, result: &Result<(), SessionError>) {
let address = PeerAddress::from(session.peer_addr);
let hold_downs = &session.server.hold_downs;
let over_limit = match result {
Err(SessionError::PrefixLimitExceeded { .. }) => {
Some((HeldDown::OverLimit, &*PREFIX_LIMIT_EXCEEDED))
}
Err(SessionError::ExportLimitExceeded { .. }) => {
Some((HeldDown::OverExportLimit, &*EXPORT_LIMIT_EXCEEDED))
}
_ => None,
};
if let Some((why, exceeded)) = over_limit {
hold_downs.hold(address, session.peer.import.hold_down, why);
exceeded
.with_label_values(&[session.rejections.label()])
.inc();
}
if let (Some(up), Some(_)) = (&session.established, &session.peer.name)
&& let Some(hold) = hold_downs.session_ended(address, up.since.elapsed(), result.is_err())
{
warn!(
peer_addr = %socket_text(session.peer_addr),
hold_down = ?hold,
"The peer's sessions keep failing soon after they come up; keeping it down for a while"
);
}
}
/// Whether, and for how long, the routes of a session that ended as
/// `result` says are to be kept for the peer's next session (RFC 4724
/// §4.2): the IP versions whose routes are, and the time. `None` if they
/// are to go now.
fn keep_for(
session: &Session,
established: &Established,
result: &Result<(), SessionError>,
) -> Option<(Vec<IpVersion>, Duration)> {
let agreed = established.graceful_restart.as_ref()?;
if session.leave.borrow().is_some() || !restart::ends_in_restart(result, agreed) {
return None;
}
let versions = restart::versions(&agreed.families);
let theirs = Duration::from_secs(agreed.restart_time.into());
let time = theirs.min(session.peer.graceful_restart.stale_routes);
(!versions.is_empty() && !time.is_zero()).then_some((versions, time))
}
/// What becomes of the routes of a session that has ended: kept for the
/// peer's next session if it is restarting, left where they are if the
/// daemon is, and removed otherwise.
async fn release_routes(
session: &Session,
established: Established,
result: &Result<(), SessionError>,
) {
let keep = keep_for(session, &established, result);
let Established {
peer_id,
rib_peer,
routes,
_registration,
..
} = established;
// The peer has no session to count its routes for.
// Fails only if the session never set it, which is nothing to do.
let _ = PEER_ROUTES.remove_label_values(&[session.rejections.label()]);
let _ = PEER_ADVERTISED.remove_label_values(&[session.rejections.label()]);
PEER_TCP.remove(session.rejections.label());
if *session.leave.borrow() == Some(Leave::Restart) {
// The daemon is about to go, and its routes with it; the kernel
// keeps its own. Taking them out first would only hold the
// restart up.
routes.forget();
} else if let Some((versions, time)) = keep {
let kept = routes.keep(versions.clone()).await;
session.server.restarts.keep(
(session.peer_addr.into(), peer_id.asn),
session.peer_addr,
session.peer.name.clone(),
rib_peer,
versions,
kept,
time,
);
info!(
peer = %peer_id,
kept_routes = kept,
kept_for = ?time,
"Session ended; the peer is restarting, and its routes are kept meanwhile"
);
} else {
let outcome = routes.remove().await;
info!(
peer = %peer_id,
removed_routes = outcome.removed,
best_changed = outcome.best_changed,
"Session ended"
);
}
}
/// What exists only once a session is Established.
struct Established {
/// When it was.
@@ -296,6 +359,8 @@ struct Established {
rib_peer: PeerIndex,
/// Messages the rest of the server queued for the peer.
outbound: mpsc::Receiver<Message>,
/// Where the rest of the server asks for the session to end, and why.
ending: mpsc::UnboundedReceiver<SessionError>,
/// The routes learned from the peer, which go when the session does.
routes: PeerRoutes,
/// Whose routes the import policy is judging.
@@ -312,7 +377,7 @@ struct Established {
held: Held,
/// The address families the peer can be asked to send again, if it can
/// be asked at all.
refreshable: Option<FamilySet>,
route_refresh: bool,
/// The address families of the session.
families: FamilySet,
/// Graceful restart (RFC 4724), if both sides offered it.
@@ -323,14 +388,25 @@ struct Established {
/// again yet, if there are any.
stale: Option<Stale>,
/// Where the sweeps of stale routes say what they did.
swept: (
mpsc::UnboundedSender<Outcome>,
mpsc::UnboundedReceiver<Outcome>,
),
swept: Sweeps,
/// Keeps the session listed in the registry.
_registration: SessionGuard,
}
/// Where the sweeps of a session's stale routes, which run on their own
/// threads, say what they did.
struct Sweeps {
report: mpsc::UnboundedSender<Outcome>,
reported: mpsc::UnboundedReceiver<Outcome>,
}
impl Default for Sweeps {
fn default() -> Self {
let (report, reported) = mpsc::unbounded_channel();
Self { report, reported }
}
}
/// The routes kept from a restarting peer's last session, while its new
/// one sends them again: those of the IP versions it has not sent an
/// End-of-RIB marker for yet, which are kept until `until` at most.
@@ -345,7 +421,7 @@ impl Established {
fn import<'a>(
&'a mut self,
policy: &'a ImportPolicy,
refusals: &'a PeerRefusals,
rejections: &'a PeerRejections,
) -> (routes::Import<'a>, &'a mut Held) {
let import = routes::Import {
peer: self.rib_peer,
@@ -355,7 +431,7 @@ impl Established {
.validation
.as_mut()
.map_or(Checks::NONE, |validation| validation.validator.checks()),
refusals,
rejections,
};
(import, &mut self.held)
}
@@ -406,10 +482,8 @@ impl PeerRoutes {
/// Removes the routes and says what that did.
async fn remove(mut self) -> Outcome {
let (table, peer) = (Arc::clone(&self.table), self.peer);
// Looks at every prefix, which for a full table is a tenth of a
// second: too long to hold up the other sessions on this thread.
match tokio::task::spawn_blocking(move || Self::remove_now(&table, peer)).await {
let peer = self.peer;
match off_thread(&self.table, move |table| table.remove_peer(peer)).await {
Ok(outcome) => {
self.removed = true;
outcome
@@ -436,7 +510,7 @@ impl PeerRoutes {
.filter(|version| !versions.contains(version))
.collect();
let removed = table.sweep_stale(peer, &others).removed;
marked - removed
marked.saturating_sub(removed)
})
.await;
match kept {
@@ -459,12 +533,31 @@ impl PeerRoutes {
fn remove_now(table: &RibTable, peer: PeerIndex) -> Outcome {
let outcome = table.remove_peer(peer);
// Most of the peer's attribute sets were its alone.
table.attrs().sweep();
outcome
}
}
/// Runs `work` on the table on a thread that can take its time, and then
/// sweeps the attribute sets that nothing refers to any more. For what
/// looks at every prefix: removing a peer's routes, which for a full
/// table takes a tenth of a second, too long to hold up the sessions that
/// share this thread. Fails only if `work` panicked.
pub(crate) async fn off_thread<T: Send + 'static>(
table: &Arc<RibTable>,
work: impl FnOnce(&RibTable) -> T + Send + 'static,
) -> Result<T, tokio::task::JoinError> {
let table = Arc::clone(table);
tokio::task::spawn_blocking(move || {
let outcome = work(&table);
// Most of the attribute sets the work let go of were its routes'
// alone.
table.attrs().sweep();
outcome
})
.await
}
impl Drop for PeerRoutes {
fn drop(&mut self) {
if self.removed {
@@ -559,18 +652,10 @@ struct Session {
local_addr: SocketAddr,
direction: Direction,
server: Arc<ServerContext>,
/// Configured name of the peer, if it is a configured one.
peer_name: Option<String>,
/// Which of the peer's routes are taken.
import: Arc<ImportPolicy>,
/// What the peer is to us, if that is known.
role: Option<PeerRole>,
/// What the peer is told, and how.
export: Arc<ExportPolicy>,
/// What is configured for the peer.
peer: PeerSettings,
/// Where what is refused is noted.
refusals: Arc<PeerRefusals>,
/// Graceful restart with the peer, as configured.
graceful_restart: GracefulRestartConfig,
rejections: Arc<PeerRejections>,
/// Says when the daemon is leaving, and how.
leave: watch::Receiver<Option<Leave>>,
}
@@ -592,7 +677,7 @@ impl Session {
local_asn: server.local_asn,
hold_time_secs: peer.hold_time_secs,
expected_asn: peer.expected_asn,
families: peer.families,
families: peer.families.clone(),
add_path: peer.add_path,
role: peer.role,
require_role: peer.require_role,
@@ -608,7 +693,7 @@ impl Session {
.unwrap_or_else(|| PeerAddress::from(peer_addr).to_string());
let leave = server.registry.leaving();
let session = Session {
refusals: server.refusals.of_peer(peer_addr.into(), &label),
rejections: server.rejections.of_peer(peer_addr.into(), &label),
fsm,
frames: FrameReader::new(reader),
writer,
@@ -625,12 +710,8 @@ impl Session {
local_addr,
direction,
server,
peer_name: peer.name,
import: peer.import,
role: peer.role,
export: peer.export,
peer,
said: Said::new(),
graceful_restart: peer.graceful_restart,
leave,
};
Ok((session, first_actions))
@@ -816,7 +897,7 @@ impl Session {
SessionError::PrefixLimitExceeded { .. } => warn!(
peer_addr = %socket_text(self.peer_addr),
err = %e,
hold_down = ?self.import.hold_down,
hold_down = ?self.peer.import.hold_down,
"Ending the session and keeping the peer down"
),
_ => error!(
@@ -867,7 +948,7 @@ impl Session {
self.peer_addr.into(),
negotiated.peer_identifier,
self.direction,
self.peer_name.is_some(),
self.peer.name.is_some(),
);
if self.collision.is_none() {
info!(peer_addr = %socket_text(self.peer_addr), "Closing connection: lost collision resolution");
@@ -900,6 +981,8 @@ impl Session {
address: self.peer_addr.ip(),
interface: PeerAddress::from(self.peer_addr).interface,
ibgp: negotiated.ibgp,
// The configuration allows it only inside our AS.
reflector_client: self.peer.reflector_client && negotiated.ibgp,
};
// A peer back from a restart takes over the routes kept of it.
let kept = self
@@ -918,70 +1001,27 @@ impl Session {
collision.set_established();
}
let (outbound_tx, outbound) = mpsc::channel(OUTBOUND_QUEUE_CAPACITY);
let mut handle = SessionHandle::new(
peer_id,
self.peer_name.clone(),
self.direction,
self.local_addr,
negotiated,
rib_peer,
outbound_tx,
);
handle.role = self.role;
handle.role_confirmed = negotiated.role_confirmed;
handle.export = Arc::clone(&self.export);
// Routes from other networks are validated, where there is data
// to validate against and the peer's policy does not say to take
// invalid ones anyway.
let subject = Subject::of(
&self.import,
self.role,
negotiated.peer_asn,
negotiated.ibgp,
);
// And where routes go into the kernel, a route has to have a next
// hop the kernel can use.
let reach = self
.server
.interfaces
.clone()
.map(|interfaces| Reach::new(interfaces, neighbor_at(self.peer_addr)));
let subject = subject.filter(|_| self.server.rpki.is_some());
let validation =
Validator::new(self.server.rpki.clone().zip(subject), reach).map(|validator| {
Validation {
validator,
changes: handle.validate(subject),
}
});
let mut handle = self.session_handle(negotiated, peer_id, rib_peer, outbound_tx);
let ending = handle.listen_for_end();
let validation = self.validation(&mut handle, negotiated);
self.established = Some(Established {
since: Instant::now(),
validation,
held: Held::default(),
refreshable: negotiated
.route_refresh
.then(|| negotiated.families.clone()),
route_refresh: negotiated.route_refresh,
families: negotiated.families.clone(),
graceful_restart: negotiated.graceful_restart.clone(),
end_of_rib: FamilySet::default(),
stale: None,
swept: mpsc::unbounded_channel(),
swept: Sweeps::default(),
peer_id,
rib_peer,
outbound,
ending,
routes: PeerRoutes::new(Arc::clone(&self.server.rib), rib_peer),
from: policy::RouteSource {
peer_asn: negotiated.peer_asn,
local_asn: self.server.local_asn,
ibgp: negotiated.ibgp,
four_octet_asn: negotiated.four_octet_asn,
role: self.role,
peer_address: self.peer_addr.ip(),
local_address: self.local_addr.ip(),
extended_next_hop: negotiated.extended_next_hop,
},
from: self.route_source(negotiated),
counters: Arc::clone(&handle.counters),
routes_metric: PEER_ROUTES.with_label_values(&[self.refusals.label()]),
routes_metric: PEER_ROUTES.with_label_values(&[self.rejections.label()]),
_registration: self.server.registry.register(handle),
});
if let Some(kept) = kept {
@@ -990,6 +1030,80 @@ impl Session {
Ok(())
}
/// The session as the rest of the server is to see it: what was
/// negotiated, and what is configured for the peer.
fn session_handle(
&self,
negotiated: &Negotiated,
peer_id: PeerId,
rib_peer: PeerIndex,
outbound: mpsc::Sender<Message>,
) -> SessionHandle {
let peer = &self.peer;
let mut handle = SessionHandle::new(
peer_id,
peer.name.clone(),
self.direction,
self.local_addr,
negotiated,
rib_peer,
outbound,
);
handle.role = peer.role;
handle.role_confirmed = negotiated.role_confirmed;
handle.export = Arc::clone(&peer.export);
// To a network above or beside us, or one we know nothing of.
handle.export_limits = peer
.export_limits
.clone()
.filter(|_| ExportLimits::apply_to(peer.role, negotiated.ibgp));
handle.groups = Arc::clone(&peer.groups);
// The configuration allows it only inside our AS.
handle.reflector_client = peer.reflector_client && negotiated.ibgp;
handle
}
/// How the peer's routes are checked as the world changes, if they
/// are: against the RPKI, where there is data to validate against and
/// the peer's policy does not say to take invalid routes anyway; and,
/// where routes go into the kernel, for a next hop the kernel can use.
fn validation(
&self,
handle: &mut SessionHandle,
negotiated: &Negotiated,
) -> Option<Validation> {
let subject = Subject::of(
&self.peer.import,
self.peer.role,
negotiated.peer_asn,
negotiated.ibgp,
)
.filter(|_| self.server.rpki.is_some());
let reach = self
.server
.topology
.clone()
.map(|topology| Reach::new(topology, neighbor_at(self.peer_addr)));
Validator::new(self.server.rpki.clone().zip(subject), reach).map(|validator| Validation {
validator,
changes: handle.validate(subject),
})
}
/// Whose routes the import policy is judging.
fn route_source(&self, negotiated: &Negotiated) -> policy::RouteSource {
policy::RouteSource {
peer_asn: negotiated.peer_asn,
local_asn: self.server.local_asn,
ibgp: negotiated.ibgp,
four_octet_asn: negotiated.four_octet_asn,
role: self.peer.role,
peer_address: self.peer_addr.ip(),
local_address: self.local_addr.ip(),
extended_next_hop: negotiated.extended_next_hop,
}
}
/// Takes over the routes kept from the peer's last session, which are
/// stale until it sends them again (RFC 4724 §4.2). Those of the
/// families whose forwarding it did not keep over its restart, or all
@@ -1023,13 +1137,13 @@ impl Session {
&self.server.rib,
established.rib_peer,
go,
&established.swept.0,
&established.swept.report,
);
}
if !stay.is_empty() {
established.stale = Some(Stale {
versions: stay,
until: Box::pin(sleep(self.graceful_restart.stale_routes)),
until: Box::pin(sleep(self.peer.graceful_restart.stale_routes)),
});
}
}
@@ -1054,10 +1168,10 @@ impl Session {
&self.server.rib,
established.rib_peer,
vec![version],
&established.swept.0,
&established.swept.report,
);
}
if let Some(name) = &self.peer_name
if let Some(name) = &self.peer.name
&& established
.families
.iter()
@@ -1082,7 +1196,7 @@ impl Session {
&self.server.rib,
established.rib_peer,
stale.versions,
&established.swept.0,
&established.swept.report,
);
}
}
@@ -1102,7 +1216,7 @@ impl Session {
}
// Every route swept away was stale.
established.counters.clear_stale(outcome.removed);
match established.account(outcome, 0, &self.import) {
match established.account(outcome, 0, &self.peer.import) {
Ok(()) => vec![],
Err(e) => self.fsm.handle(Event::LocalFailure(e)),
}
@@ -1117,7 +1231,7 @@ impl Session {
let Some(established) = &mut self.established else {
return Ok(());
};
let (import, held) = established.import(&self.import, &self.refusals);
let (import, held) = established.import(&self.peer.import, &self.rejections);
let routes::Applied { outcome, rejected } =
routes::apply(&self.server.rib, &import, held, update, attributes_wire)?;
// One line for every UPDATE is too much to log by default: a full
@@ -1131,7 +1245,7 @@ impl Session {
best_changed = outcome.best_changed,
"Received UPDATE"
);
established.account(&outcome, rejected, &self.import)
established.account(&outcome, rejected, &self.peer.import)
}
/// Looks at the peer's routes again after the validation data has
@@ -1141,7 +1255,7 @@ impl Session {
let Some(established) = &mut self.established else {
return vec![];
};
let (import, held) = established.import(&self.import, &self.refusals);
let (import, held) = established.import(&self.peer.import, &self.rejections);
let routes::Applied { outcome, rejected } =
routes::revalidate(&self.server.rib, &import, held, suspects);
if outcome != Outcome::default() {
@@ -1153,16 +1267,15 @@ impl Session {
"Revalidated the peer's routes"
);
}
if let Err(e) = established.account(&outcome, rejected, &self.import) {
if let Err(e) = established.account(&outcome, rejected, &self.peer.import) {
return self.fsm.handle(Event::LocalFailure(e));
}
// Invalid routes that there was no room to hold may be acceptable
// now, and only the peer has them.
if established.held.take_overflowed()
&& let Some(families) = &established.refreshable
{
if established.held.take_overflowed() && established.route_refresh {
info!(peer = %established.peer_id, "Asking the peer to send its routes again");
return families
return established
.families
.iter()
.map(|family| {
Action::Send(Message::RouteRefresh(RouteRefreshMessage::request(family)))
@@ -1225,12 +1338,7 @@ fn sweep(
) {
let (rib, done) = (Arc::clone(rib), done.clone());
tokio::spawn(async move {
let swept = tokio::task::spawn_blocking(move || {
let outcome = rib.sweep_stale(peer, &versions);
rib.attrs().sweep();
outcome
})
.await;
let swept = off_thread(&rib, move |table| table.sweep_stale(peer, &versions)).await;
match swept {
// The session may have ended meanwhile.
Ok(outcome) => {
@@ -1242,10 +1350,13 @@ fn sweep(
}
/// What the rest of the server has for an established session: a message
/// to send to the peer, or word that the validation data has changed.
/// to send to the peer, a request to end the session, word that the
/// validation data has changed, the outcome of a sweep of stale routes,
/// or the time being up for the routes kept from the peer's last session.
async fn from_server(established: &mut Option<Established>) -> Wake {
let Some(Established {
outbound,
ending,
validation,
stale,
swept,
@@ -1267,9 +1378,10 @@ async fn from_server(established: &mut Option<Established>) -> Wake {
}
};
tokio::select! {
Some(error) = ending.recv() => Wake::Event(Event::LocalFailure(error)),
Some(msg) = outbound.recv() => Wake::Event(Event::Outbound(msg)),
Some(suspects) = invalidated => Wake::Revalidate(suspects),
Some(outcome) = swept.1.recv() => Wake::Swept(outcome),
Some(outcome) = swept.reported.recv() => Wake::Swept(outcome),
() = stale_timeout => Wake::StaleTimeout,
// The registry holds the sending halves for as long as the session
// is registered, so neither closes under us; if they somehow did,
@@ -1496,8 +1608,8 @@ mod tests {
registry: SessionRegistry::new(),
collisions: CollisionTable::new(false),
hold_downs: HoldDowns::default(),
interfaces: None,
refusals: RefusalLog::default(),
topology: None,
rejections: RejectionLog::default(),
rpki: None,
restarts: Restarts::new(rib, true, [], 120),
});
@@ -1506,7 +1618,7 @@ mod tests {
peer_addr,
Direction::Inbound,
server,
PeerSettings::unconfigured(90, Arc::default(), GracefulRestartConfig::default()),
PeerSettings::unconfigured(90, Arc::default(), None, GracefulRestartConfig::default()),
)
.unwrap();
@@ -1547,8 +1659,8 @@ mod tests {
registry: SessionRegistry::new(),
collisions: CollisionTable::new(false),
hold_downs: HoldDowns::default(),
interfaces: None,
refusals: RefusalLog::default(),
topology: None,
rejections: RejectionLog::default(),
rpki: None,
restarts: Restarts::new(rib, true, [], 120),
});
@@ -1557,7 +1669,7 @@ mod tests {
peer_addr,
Direction::Inbound,
server,
PeerSettings::unconfigured(90, Arc::default(), GracefulRestartConfig::default()),
PeerSettings::unconfigured(90, Arc::default(), None, GracefulRestartConfig::default()),
)
.unwrap();
session.send_hold_time = Duration::from_millis(100);
@@ -1595,6 +1707,7 @@ mod tests {
address: Ipv4Addr::new(192, 0, 2, 1).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap();
let update = UpdateMessage {
@@ -1622,7 +1735,7 @@ mod tests {
},
policy: &ImportPolicy::permissive(),
checks: Checks::NONE,
refusals: &RefusalLog::default().of_peer(
rejections: &RejectionLog::default().of_peer(
std::net::IpAddr::from(Ipv4Addr::new(192, 0, 2, 1)).into(),
"test",
),
crates/server/src/dialer.rs
+30 -3
View File
@@ -18,7 +18,7 @@ use tracing::{info, warn};
use crate::collision::Direction;
use crate::config::PeerConfig;
use crate::connection::{PeerSettings, ServerContext, run_session};
use crate::connection::{CLOSED_AT_OUR_REQUEST, PeerSettings, ServerContext, run_session};
use crate::session::DialState;
/// Upper bound for exponential backoff between connection attempts.
@@ -32,6 +32,8 @@ pub fn spawn_dialers(
) -> Vec<JoinHandle<()>> {
peers
.iter()
// A passive peer is left to connect to us.
.filter(|peer| !peer.passive)
.map(|peer| tokio::spawn(dial_loop(peer.clone(), Arc::clone(server), connect_retry)))
.collect()
}
@@ -119,7 +121,11 @@ async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry:
if end.established {
backoff = connect_retry;
}
end.result.map_err(|e| e.to_string())
if end.result.is_ok() {
Ok(())
} else {
Err(end.reason())
}
}
Err(e) => Err(format!("connection lost at once: {e}")),
},
@@ -128,7 +134,7 @@ async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry:
};
let reason = match ended {
Ok(()) => "session closed at our request".to_string(),
Ok(()) => CLOSED_AT_OUR_REQUEST.to_string(),
Err(reason) => {
warn!(peer = %peer.name, endpoint = %peer.endpoint, err = %reason, "Session attempt ended");
reason
@@ -148,3 +154,24 @@ async fn dial_loop(peer: PeerConfig, server: Arc<ServerContext>, connect_retry:
backoff = (backoff * 2).min(MAX_BACKOFF);
}
}
#[cfg(test)]
mod tests {
use super::*;
/// The wait before dialling again is shortened by up to a quarter, by
/// chance, so that peers that went away together are not all dialled
/// at the same moment (RFC 4271 §10).
#[test]
fn the_wait_is_jittered_by_up_to_a_quarter() {
let wait = Duration::from_secs(100);
let mut shortest = wait;
for _ in 0..200 {
let jittered = jittered(wait);
assert!(jittered <= wait, "{jittered:?}");
assert!(jittered >= wait.mul_f64(0.75), "{jittered:?}");
shortest = shortest.min(jittered);
}
assert!(shortest < wait, "the wait varies");
}
}
crates/server/src/error.rs
+99 -5
View File
@@ -82,6 +82,13 @@ pub enum SessionError {
/// down for a while after.
#[error("the peer went over its limit of {limit} routes")]
PrefixLimitExceeded { limit: u32 },
/// We were to announce the peer more routes than its export limit
/// allows, and did not. It is kept down for a while after.
#[error(
"{routes} routes were to be advertised to the peer, over its export limit of {limit}; \
none of them was"
)]
ExportLimitExceeded { limit: u32, routes: usize },
/// The peer took nothing we sent for this long (RFC 9687): it has
/// stopped reading, and there is no telling it so.
#[error("the peer accepted no data for {0:?}")]
@@ -117,6 +124,17 @@ impl SessionError {
Self::Collision => BgpError::Cease(CeaseSubcode::ConnCollisionResolution),
Self::TableFull(_) => BgpError::Cease(CeaseSubcode::OutOfResources),
Self::PrefixLimitExceeded { .. } => BgpError::Cease(CeaseSubcode::MaxPrefixes),
// Nothing is wrong at the peer's end, as far as we know; its
// operators are told what is at ours.
Self::ExportLimitExceeded { limit, .. } => {
return Some(NotificationMessage {
error: BgpError::Cease(CeaseSubcode::AdminShutdown),
data: shutdown_communication(&format!(
"export limit of {limit} routes reached; no more routes were sent, \
and the session is held down for a while"
)),
});
}
// The session will be set up again; nothing is wrong at the
// peer's end.
Self::Panicked(_) => BgpError::Cease(CeaseSubcode::AdminReset),
@@ -136,6 +154,19 @@ impl SessionError {
}
}
/// A Shutdown Communication (RFC 9003 §2): text for the peer's operators,
/// at most 255 octets of UTF-8, after its length.
fn shutdown_communication(text: &str) -> Vec<u8> {
let mut end = text.len().min(255);
while !text.is_char_boundary(end) {
end -= 1;
}
let mut data = Vec::with_capacity(end + 1);
data.push(end as u8);
data.extend_from_slice(&text.as_bytes()[..end]);
data
}
/// Why the server could not start, or could not go on.
#[derive(Debug, Error)]
pub enum ServerError {
@@ -164,6 +195,8 @@ pub enum ServerError {
pub enum ConfigError {
#[error("invalid router_id {0:?}: must be a dotted-quad IPv4 address other than 0.0.0.0")]
RouterId(String),
#[error("invalid cluster_id {0:?}: must be a dotted-quad IPv4 address other than 0.0.0.0")]
ClusterId(String),
#[error("local_asn must be non-zero")]
LocalAsn,
#[error("invalid IP address {0:?}")]
@@ -178,6 +211,21 @@ pub enum ConfigError {
RpkiNoCaches,
#[error("export: {0}")]
Export(#[from] ExportProblem),
#[error("export_limits: {0}")]
ExportLimits(ImportProblem),
#[error("community_names {community:?}: {problem}")]
CommunityName {
community: String,
problem: crate::marks::MarksProblem,
},
#[error("community_names {community:?}: the name is empty")]
CommunityNameEmpty { community: String },
#[error(
"accept_unconfigured_peers is set and export.learned_routes passes learned routes on: \
a peer that is not configured has no role, so the routes it sends would go to every \
peer, and every route to it"
)]
UnconfiguredPeersWithoutRole,
#[error("rpki: the port of cache {0:?} is required")]
RpkiCachePort(String),
#[error("graceful_restart: {0}")]
@@ -220,6 +268,18 @@ pub enum PeerProblem {
RequireRoleWithoutRole,
#[error("role has a value that is not one of its settings")]
Role,
#[error("groups has a 0 in it; groups are numbered from 1")]
Group,
#[error("route_reflector_client is set for a peer in another AS; it is for routers of our own")]
ReflectorClientOutsideAs,
#[error(
"role is not set, and learned routes are passed on (export.learned_routes): without a \
role, the routes the peer sends would go to every other peer, and it would be told of \
every route. Set role to PROVIDER, CUSTOMER, PEER, ROUTE_SERVER or ROUTE_SERVER_CLIENT"
)]
NoRole,
#[error("export_limits: {0}")]
ExportLimits(ImportProblem),
#[error("import: {0}")]
Import(#[from] ImportProblem),
#[error("export: {0}")]
@@ -231,7 +291,7 @@ pub enum PeerProblem {
/// What is wrong with a `graceful_restart`.
#[derive(Debug, Clone, PartialEq, Eq, Error)]
pub enum GracefulRestartProblem {
#[error("restart_time_seconds is {0}; it is at most 4095")]
#[error("restart_time_seconds is {0}; it is at most {max}", max = crate::restart::MAX_RESTART_TIME)]
RestartTime(u32),
}
@@ -261,6 +321,12 @@ pub enum ImportProblem {
PathLength(u32),
#[error(transparent)]
Marks(#[from] crate::marks::MarksProblem),
#[error("rule {number}: {problem}")]
Rule {
/// From 1, as an operator counts.
number: usize,
problem: Box<ImportProblem>,
},
}
/// What is wrong with an `export` policy.
@@ -268,8 +334,10 @@ pub enum ImportProblem {
pub enum ExportProblem {
#[error(transparent)]
Marks(#[from] crate::marks::MarksProblem),
#[error("as_path_prepend of {0} is more than 16")]
Prepend(u32),
#[error("as_path_prepend of {value} is more than {max}")]
Prepend { value: u32, max: u32 },
#[error("all_paths needs learned_routes")]
AllPathsWithoutLearnedRoutes,
}
/// What is wrong with one `originate` entry.
@@ -279,8 +347,8 @@ pub enum OriginateProblem {
Prefix(#[from] bgp_primitives::ip_prefix::PrefixError),
#[error("origin {0:?} must be IGP, EGP or INCOMPLETE")]
Origin(String),
#[error("as_path_prepend must be at most {max}")]
Prepend { max: u32 },
#[error("as_path_prepend of {value} is more than {max}")]
Prepend { value: u32, max: u32 },
#[error("{0}")]
Community(#[from] bgp_packet::community::CommunityError),
#[error("listed more than once")]
@@ -344,5 +412,31 @@ mod tests {
notification(SessionError::Io(io::ErrorKind::BrokenPipe.into())),
None
);
// Going over an export limit is ours to explain: an Administrative
// Shutdown, with the reason for the peer's operators (RFC 9003).
let (error, data) = notification(SessionError::ExportLimitExceeded {
limit: 50,
routes: 900_000,
})
.unwrap();
assert_eq!(error, BgpError::Cease(CeaseSubcode::AdminShutdown));
let text = std::str::from_utf8(&data[1..]).unwrap();
assert_eq!(usize::from(data[0]), text.len());
assert!(
text.starts_with("export limit of 50 routes reached"),
"{text}"
);
}
/// A Shutdown Communication is at most 255 octets, and whole
/// characters (RFC 9003 §2).
#[test]
fn a_shutdown_communication_is_cut_to_fit() {
let long = "é".repeat(200);
let data = shutdown_communication(&long);
assert_eq!(usize::from(data[0]), data.len() - 1);
assert_eq!(data[0], 254);
assert!(std::str::from_utf8(&data[1..]).is_ok());
assert_eq!(shutdown_communication(""), [0]);
}
}
crates/server/src/export.rs
+420 -50
View File
@@ -10,19 +10,27 @@
//!
//! - not back to the peer it came from, and not to a peer whose AS is on
//! its path already (it would refuse it as a loop);
//! - not from one peer in our own AS to another (RFC 4271 §9.2.1: there is
//! no route reflection here);
//! - not from one peer in our own AS to another (RFC 4271 §9.2.1), unless
//! one of the two is a route reflector client of ours (RFC 4456), and
//! then not back to the router the route started at;
//! - not up or across if it came down or across: a route marked "only to
//! customer" goes to customers only (RFC 9234 §5, egress, 2);
//! - not where its communities say it is not to go: NO_EXPORT and
//! NO_ADVERTISE (RFC 1997), and those of [`Control`].
//! NO_ADVERTISE (RFC 1997), and, where it leaves our AS, those of
//! [`Control`], which also say how many more times our AS goes on its
//! path.
//!
//! What goes is changed as BGP has it: our AS on the path and ourselves as
//! next hop towards another network, the MED it came with dropped; the
//! path as it was and a LOCAL_PREF towards a peer in our own AS. To a
//! route server's client the route goes as it came (RFC 7947 §2.2).
//! path as it was and a LOCAL_PREF towards a peer in our own AS. A route
//! passed from one router of our AS to another is marked with where it
//! started and with our cluster (ORIGINATOR_ID, CLUSTER_LIST), and goes,
//! like every route, with ourselves as next hop: there is no resolving a
//! next hop that is not a neighbor, so the peer reaches what lies behind
//! the other router through us. To a route server's client the route goes
//! as it came (RFC 7947 §2.2).
use std::net::IpAddr;
use std::net::{IpAddr, Ipv4Addr};
use bgp_packet::message::{
AsPathAttribute, AsPathSegment, LocalPrefPathAttribute, MpReachNlriPathAttribute,
@@ -33,7 +41,7 @@ use bgp_primitives::ip_prefix::Nlri;
use bgp_rib::table::{DEFAULT_LOCAL_PREF, PeerIndex};
use crate::advertise::{Towards, ipv6_next_hop};
use crate::marks::{self, Control, communities_of, well_known_allow};
use crate::marks::{self, Control, Named, communities_of, well_known_allow};
use crate::policy::{ExportPolicy, PeerRole};
/// The session a route may go out on, as far as that decides whether it
@@ -46,9 +54,26 @@ pub struct Egress<'a> {
/// are marked with.
pub rib_peer: PeerIndex,
pub policy: &'a ExportPolicy,
/// The groups the session is in, for control communities to name it
/// by.
pub groups: &'a [u32],
/// The communities by which routes say who is told of them, if they
/// are honoured.
pub control: Option<Control>,
/// What route reflection needs to know of the session.
pub reflection: Reflection,
}
/// Route reflection (RFC 4456), as far as one session goes.
#[derive(Debug, Clone, Copy)]
pub struct Reflection {
/// The cluster this router reflects routes for.
pub cluster_id: Ipv4Addr,
/// Whether the peer is a route reflector client of ours.
pub client: bool,
/// The peer's BGP Identifier: a route that started at the peer is not
/// passed back to it.
pub peer_router_id: Ipv4Addr,
}
/// Where a learned route came from.
@@ -57,6 +82,11 @@ pub struct Source {
pub peer: PeerIndex,
/// Whether that peer is in our own AS.
pub ibgp: bool,
/// Whether that peer is a route reflector client of ours.
pub reflector_client: bool,
/// Its BGP Identifier: where the route started inside our AS, if it
/// says nothing else.
pub router_id: Ipv4Addr,
}
/// A route ready to be announced on a session: its attributes, but for
@@ -78,16 +108,25 @@ impl Egress<'_> {
/// whether it goes.
pub fn dress(&self, attributes: &mut Vec<PathAttribute>) -> bool {
let towards = &self.towards;
let (communities, large) = communities_of(attributes);
if let Some(control) = self.control {
if !control.allows(communities, large, self.peer_asn) {
let mut prepends = self.policy.prepend;
// Control communities are acted on where a route leaves our AS. To
// a peer in it the route goes with them, for that router to act
// on where it passes the route on.
if let Some(control) = self.control.filter(|_| !towards.ibgp) {
let (communities, large) = communities_of(attributes);
let session = Named {
asn: self.peer_asn,
groups: self.groups,
};
if !control.allows(communities, large, session) {
return false;
}
prepends = prepends.saturating_add(control.prepends(large, session));
control.strip(attributes);
}
self.policy.marks.apply(attributes);
if self.policy.prepend > 0 && !towards.ibgp {
let ours = std::iter::repeat_n(towards.local_asn, usize::from(self.policy.prepend));
if prepends > 0 && !towards.ibgp {
let ours = std::iter::repeat_n(towards.local_asn, usize::from(prepends));
for attribute in attributes.iter_mut() {
if let PathAttribute::AsPath(path) = attribute {
prepend(path, ours);
@@ -122,21 +161,46 @@ impl Egress<'_> {
source: Source,
routes: IpVersion,
) -> Option<Outgoing> {
let reflected = source.ibgp && self.towards.ibgp;
if !self.may_go(&attributes, source, reflected) {
return None;
}
let came_with = next_hop.or_else(|| {
attributes.iter().find_map(|attribute| match attribute {
PathAttribute::NextHop(next_hop) => Some(NlriNextHop::Ipv4(next_hop.0)),
_ => None,
})
});
if !self.rewrite(&mut attributes, source, reflected) {
return None;
}
let next_hop = self.next_hop_for(routes, came_with)?;
Some(Outgoing {
attributes,
next_hop,
})
}
/// Whether a route with these attributes, from `source`, is for this
/// session at all, as far as who may be told of what goes: the export
/// policy, route reflection, the AS path, the well-known communities
/// and the Only to Customer mark.
fn may_go(&self, attributes: &[PathAttribute], source: Source, reflected: bool) -> bool {
let towards = &self.towards;
if !self.policy.learned_routes || source.peer == self.rib_peer {
return None;
return false;
}
if source.ibgp && towards.ibgp {
return None;
// From one router of our AS to another only by way of a route
// reflector, which we are only for our clients: what a client sends
// goes to all our routers, and what the others send goes to the
// clients (RFC 4456 §6).
if reflected && !source.reflector_client && !self.reflection.client {
return false;
}
let mut came_with = next_hop;
let mut marked = false;
let mut on_path = false;
for attribute in &attributes {
for attribute in attributes {
match attribute {
PathAttribute::NextHop(next_hop) => {
came_with = came_with.or(Some(NlriNextHop::Ipv4(next_hop.0)));
}
PathAttribute::OnlyToCustomer(_) => marked = true,
PathAttribute::AsPath(path) => {
on_path = path
@@ -149,24 +213,36 @@ impl Egress<'_> {
}
// The peer would refuse it as having been there before.
if on_path && !towards.ibgp {
return None;
return false;
}
// Whoever sent it said how far it may go (RFC 1997). On a route
// of our own the same communities are for those who get it from
// us, and it goes out with them.
if !well_known_allow(communities_of(&attributes).0, towards.ibgp) {
return None;
if !well_known_allow(communities_of(attributes).0, towards.ibgp) {
return false;
}
// It came down or across once: it goes on down only.
if marked
let only_down = marked
&& !towards.ibgp
&& towards
.role
.is_some_and(|role| !role.may_be_sent_only_to_customer_routes())
{
return None;
}
.is_some_and(|role| !role.may_be_sent_only_to_customer_routes());
!only_down
}
/// Makes the attributes those of the route as this session announces
/// it: what is ours to say is taken off, what is for routers of the
/// AS it was learned in stays only inside ours, a reflected route gets
/// its reflection attributes, and the path and LOCAL_PREF are as the
/// peer is to see them. Then the session's own marks ([`Egress::dress`]).
/// `false` if the route is not to go after all.
fn rewrite(
&self,
attributes: &mut Vec<PathAttribute>,
source: Source,
reflected: bool,
) -> bool {
let towards = &self.towards;
let transparent = self.transparent();
attributes.retain_mut(|attribute| match attribute {
// Ours to say, each way.
@@ -178,10 +254,13 @@ impl Egress<'_> {
// (RFC 4271 §5.1.4, §5.1.5).
PathAttribute::MultiExitDisc(_) => towards.ibgp || transparent,
PathAttribute::LocalPref(_) => towards.ibgp,
// Only for routers of our AS, and only for a route that is
// reflected from one to another; put on afresh below.
PathAttribute::OriginatorId(_) | PathAttribute::ClusterList(_) => reflected,
// What we do not understand goes on if it says it may, marked
// as having passed a router that did not look at it; and no
// further if not (RFC 4271 §5).
PathAttribute::UnknownPathAttribute { flags, .. } => {
PathAttribute::Unknown { flags, .. } => {
let passes = flags.optional() && flags.transitive();
if passes {
flags.set_partial(true);
@@ -190,13 +269,16 @@ impl Egress<'_> {
}
_ => true,
});
if reflected && !self.reflect(attributes, source) {
return false;
}
if towards.ibgp {
if !attributes
.iter()
.any(|attribute| matches!(attribute, PathAttribute::LocalPref(_)))
{
marks::put(
&mut attributes,
attributes,
PathAttribute::LocalPref(LocalPrefPathAttribute(DEFAULT_LOCAL_PREF)),
);
}
@@ -208,17 +290,24 @@ impl Egress<'_> {
}) {
Some(path) => prepend(path, ours),
None => marks::put(
&mut attributes,
attributes,
AsPathAttribute::from_asns(vec![towards.local_asn]),
),
}
}
if !self.dress(&mut attributes) {
return None;
}
self.dress(attributes)
}
// Ourselves, unless we are a route server and the client can use
// the next hop the route came with.
/// The next hop of a route for prefixes of IP version `routes`:
/// ourselves, unless we are a route server and the client can use the
/// next hop the route came with. `None` where our address on the
/// session is of the wrong kind for the routes.
fn next_hop_for(
&self,
routes: IpVersion,
came_with: Option<NlriNextHop>,
) -> Option<NlriNextHop> {
let towards = &self.towards;
let ours = match (routes, towards.next_hop) {
(IpVersion::Ipv4, IpAddr::V4(ours)) => NlriNextHop::Ipv4(ours),
(IpVersion::Ipv6, IpAddr::V6(ours)) => ipv6_next_hop(ours),
@@ -228,13 +317,44 @@ impl Egress<'_> {
let same_kind = |theirs: &NlriNextHop| {
matches!(theirs, NlriNextHop::Ipv4(_)) == matches!(ours, NlriNextHop::Ipv4(_))
};
let next_hop = came_with
.filter(|theirs| transparent && same_kind(theirs))
.unwrap_or(ours);
Some(Outgoing {
attributes,
next_hop,
})
Some(
came_with
.filter(|theirs| self.transparent() && same_kind(theirs))
.unwrap_or(ours),
)
}
}
impl Egress<'_> {
/// Marks a route reflected from one router of our AS to another with
/// where it started and with our cluster (RFC 4456 §8). Returns whether
/// it goes: not back to the router it started at, nor round a circle
/// of reflectors that has been through our cluster already.
fn reflect(&self, attributes: &mut Vec<PathAttribute>, source: Source) -> bool {
let originator = attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::OriginatorId(originator) => Some(*originator),
_ => None,
})
.unwrap_or(source.router_id);
if originator == self.reflection.peer_router_id {
return false;
}
let mut clusters = attributes
.iter()
.find_map(|attribute| match attribute {
PathAttribute::ClusterList(clusters) => Some(clusters.clone()),
_ => None,
})
.unwrap_or_default();
if clusters.contains(&self.reflection.cluster_id) {
return false;
}
clusters.insert(0, self.reflection.cluster_id);
marks::put(attributes, PathAttribute::OriginatorId(originator));
marks::put(attributes, PathAttribute::ClusterList(clusters));
true
}
}
@@ -327,7 +447,10 @@ pub(crate) fn withdrawals(
routes: &[Nlri],
towards: &Towards,
) -> Vec<UpdateMessage> {
let at_once = (UPDATE_ROOM - 16) / nlri_octets(version);
// What the message needs besides the prefixes: its two length fields,
// or the header of MP_UNREACH_NLRI; 16 octets is more than either.
const WITHDRAWAL_OVERHEAD: usize = 16;
let at_once = (UPDATE_ROOM - WITHDRAWAL_OVERHEAD) / nlri_octets(version);
// IPv4 routes that went in MP_REACH_NLRI are withdrawn the same way.
let in_body =
version == IpVersion::Ipv4 && !(towards.extended_next_hop && towards.next_hop.is_ipv6());
@@ -361,14 +484,14 @@ mod tests {
use std::net::Ipv4Addr;
use bgp_packet::message::{
CommunitiesPathAttribute, MultiExitDiscPathAttribute, OriginPathAttribute,
PathAttributeFlags,
CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
MultiExitDiscPathAttribute, OriginPathAttribute, PathAttributeFlags,
};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerInfo, RibTable};
use super::*;
use crate::marks::Marks;
use crate::marks::{Marks, large};
const LOCAL: u32 = 64601;
const THEM: u32 = 64602;
@@ -385,6 +508,7 @@ mod tests {
address: Ipv4Addr::new(192, 0, 2, n).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap()
};
@@ -405,6 +529,7 @@ mod tests {
fn all_routes() -> ExportPolicy {
ExportPolicy {
learned_routes: true,
all_paths: false,
..ExportPolicy::default()
}
}
@@ -429,7 +554,24 @@ mod tests {
role: Option<PeerRole>,
to_ibgp: bool,
from_ibgp: bool,
groups: &'a [u32],
control: Option<Control>,
to_client: bool,
from_client: bool,
}
/// Our cluster, and the BGP Identifiers of the peers a route comes
/// from and goes to.
const CLUSTER: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 9);
const FROM_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 1);
const TO_ID: Ipv4Addr = Ipv4Addr::new(10, 0, 0, 2);
fn reflection(client: bool) -> Reflection {
Reflection {
cluster_id: CLUSTER,
client,
peer_router_id: TO_ID,
}
}
impl Case<'_> {
@@ -440,11 +582,15 @@ mod tests {
peer_asn: if self.to_ibgp { LOCAL } else { THEM },
rib_peer: to,
policy: self.policy,
groups: self.groups,
control: self.control,
reflection: reflection(self.to_client),
};
let source = Source {
peer: from,
ibgp: self.from_ibgp,
reflector_client: self.from_client,
router_id: FROM_ID,
};
egress.outgoing(attributes, None, source, IpVersion::Ipv4)
}
@@ -456,7 +602,10 @@ mod tests {
role: None,
to_ibgp: false,
from_ibgp: false,
groups: &[],
control: None,
to_client: false,
from_client: false,
}
}
@@ -471,12 +620,12 @@ mod tests {
communities(&[(3356, 3)]),
// Understood by nobody here: one that may go on, one that may
// not.
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: PathAttributeFlags::new(true, true, false, false),
type_code: 200,
payload: vec![1],
},
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
flags: PathAttributeFlags::new(true, false, false, false),
type_code: 201,
payload: vec![2],
@@ -493,7 +642,7 @@ mod tests {
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![LOCAL, 3356, 64700]),
communities(&[(3356, 3)]),
PathAttribute::UnknownPathAttribute {
PathAttribute::Unknown {
// Marked as having passed a router that did not know it.
flags: PathAttributeFlags::new(true, true, true, false),
type_code: 200,
@@ -518,11 +667,15 @@ mod tests {
peer_asn: THEM,
rib_peer: from,
policy: &policy,
groups: &[],
control: None,
reflection: reflection(false),
};
let source = Source {
peer: from,
ibgp: false,
reflector_client: false,
router_id: FROM_ID,
};
assert!(
egress
@@ -592,6 +745,105 @@ mod tests {
assert!(from_internal.out(route(vec![])).is_some());
}
/// Route reflection (RFC 4456): between routers of our AS only by way
/// of a client of ours, marked with where the route started and with
/// our cluster, and never back to where it started or round a circle.
#[test]
fn a_route_is_reflected_to_and_from_clients_only() {
let policy = all_routes();
let between = |from_client, to_client| Case {
to_ibgp: true,
from_ibgp: true,
from_client,
to_client,
..plain(&policy)
};
let preferred = || route(vec![PathAttribute::LocalPref(LocalPrefPathAttribute(200))]);
// Neither a client: as BGP has it, nothing.
assert!(between(false, false).out(preferred()).is_none());
// To a client, and from one: marked, the preference as it was, and
// ourselves as next hop.
for (from_client, to_client) in [(false, true), (true, false), (true, true)] {
let outgoing = between(from_client, to_client)
.out(preferred())
.unwrap_or_else(|| panic!("from {from_client} to {to_client}"));
assert_eq!(
outgoing.attributes,
[
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![3356, 64700]),
PathAttribute::LocalPref(LocalPrefPathAttribute(200)),
PathAttribute::OriginatorId(FROM_ID),
PathAttribute::ClusterList(vec![CLUSTER]),
]
);
assert_eq!(
outgoing.next_hop,
NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 1))
);
}
// Reflected before: where it started stays, our cluster goes first.
let elsewhere = Ipv4Addr::new(10, 0, 0, 5);
let other_cluster = Ipv4Addr::new(10, 0, 0, 7);
let reflected_before = |originator, clusters: Vec<Ipv4Addr>| {
route(vec![
PathAttribute::LocalPref(LocalPrefPathAttribute(200)),
PathAttribute::OriginatorId(originator),
PathAttribute::ClusterList(clusters),
])
};
let outgoing = between(false, true)
.out(reflected_before(elsewhere, vec![other_cluster]))
.unwrap();
assert!(
outgoing
.attributes
.contains(&PathAttribute::OriginatorId(elsewhere))
);
assert!(
outgoing
.attributes
.contains(&PathAttribute::ClusterList(vec![CLUSTER, other_cluster]))
);
// Not back to the router it started at, nor round to our cluster.
assert!(
between(false, true)
.out(reflected_before(TO_ID, vec![other_cluster]))
.is_none()
);
assert!(
between(false, true)
.out(reflected_before(elsewhere, vec![other_cluster, CLUSTER]))
.is_none()
);
// To another network they are not sent; a route from another
// network to a client of ours gets none.
let from_inside = Case {
from_ibgp: true,
..plain(&policy)
};
let outgoing = from_inside
.out(reflected_before(elsewhere, vec![other_cluster]))
.unwrap();
assert!(!outgoing.attributes.iter().any(|attribute| matches!(
attribute,
PathAttribute::OriginatorId(_) | PathAttribute::ClusterList(_)
)));
let to_client = Case {
to_ibgp: true,
to_client: true,
..plain(&policy)
};
let outgoing = to_client.out(route(vec![])).unwrap();
assert!(!outgoing.attributes.iter().any(|attribute| matches!(
attribute,
PathAttribute::OriginatorId(_) | PathAttribute::ClusterList(_)
)));
}
/// RFC 9234 §5, egress: what came down or across goes on down only,
/// and what goes down or across is marked.
#[test]
@@ -671,6 +923,7 @@ mod tests {
|texts: &[&str]| -> Vec<String> { texts.iter().map(|text| text.to_string()).collect() };
let policy = ExportPolicy {
learned_routes: true,
all_paths: false,
marks: Marks::parse(&strings(&["64601:900"]), &strings(&["64601:5"]), &[]).unwrap(),
prepend: 2,
};
@@ -711,6 +964,98 @@ mod tests {
);
}
fn large_communities(communities: &[LargeCommunity]) -> PathAttribute {
PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
communities: communities.to_vec(),
})
}
/// Control communities are for where a route leaves our AS. To a
/// router of our own the route goes with them, whatever they say, for
/// it to act on; to another network they decide, by group as well as
/// by AS, and are taken off, while our marks go on.
#[test]
fn control_communities_are_acted_on_where_routes_leave_our_as() {
let policy = all_routes();
let control = Some(Control { local_asn: LOCAL });
let mark = large(LOCAL, 100, 7);
let tagged = |actions: &[LargeCommunity]| {
route(vec![large_communities(
&[actions, std::slice::from_ref(&mark)].concat(),
)])
};
// Inside: as it came, though it says "to nobody".
let inside = Case {
to_ibgp: true,
control,
..plain(&policy)
};
let nobody = [large(LOCAL, 0, 0), large(LOCAL, 31, 2)];
let kept = inside
.out(tagged(&nobody))
.expect("our own router hears of it");
assert!(
kept.attributes.contains(&large_communities(
&[&nobody[..], std::slice::from_ref(&mark)].concat()
)),
"{:?}",
kept.attributes
);
// Outside, it is done as it says.
let at_two = Case {
groups: &[2],
control,
..plain(&policy)
};
assert!(at_two.out(tagged(&nobody)).is_none());
// Prepended where the group is named, and not elsewhere; the
// community that asked is gone, the mark is not.
let prepend_at_two = [large(LOCAL, 31, 2)];
let elsewhere = Case {
groups: &[1],
control,
..plain(&policy)
};
for (case, path) in [
(&at_two, vec![LOCAL, LOCAL, 3356, 64700]),
(&elsewhere, vec![LOCAL, 3356, 64700]),
] {
let outgoing = case.out(tagged(&prepend_at_two)).unwrap();
assert_eq!(
outgoing.attributes,
[
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(path),
large_communities(std::slice::from_ref(&mark)),
]
);
}
// A route without one, as an anycast prefix would be, is not.
let bare = at_two.out(route(vec![])).unwrap();
assert_eq!(
bare.attributes[1],
AsPathAttribute::from_asns(vec![LOCAL, 3356, 64700])
);
// The session's own prepend and the route's add up.
let prepending = ExportPolicy {
prepend: 1,
..all_routes()
};
let both = Case {
groups: &[2],
control,
..plain(&prepending)
};
let outgoing = both.out(tagged(&prepend_at_two)).unwrap();
assert_eq!(
outgoing.attributes[1],
AsPathAttribute::from_asns(vec![LOCAL, LOCAL, LOCAL, 3356, 64700])
);
}
/// As many prefixes to a message as fit, and the path as AS4_PATH too
/// where the session cannot carry it.
#[test]
@@ -773,5 +1118,30 @@ mod tests {
routes.len()
);
assert!(gone.len() < 8);
let in_mp_unreach = |update: &UpdateMessage, count: usize| {
update.withdrawn_routes.is_empty()
&& update.path_attributes.iter().any(|attribute| {
matches!(attribute, PathAttribute::MpUnreachNlri(mp) if mp.prefixes.len() == count)
})
};
let v6_routes: Vec<Nlri> = (0..3)
.map(|n| {
IpPrefix::try_from(format!("2001:db8:{n}::/48").as_str())
.unwrap()
.into()
})
.collect();
let gone = withdrawals(IpVersion::Ipv6, &v6_routes, &narrow);
assert_eq!(gone.len(), 1);
assert!(in_mp_unreach(&gone[0], 3), "{:?}", gone[0]);
// IPv4 routes that went with IPv6 next hops (RFC 8950) too.
let over_v6 = Towards {
extended_next_hop: true,
next_hop: "2001:db8::2".parse::<std::net::Ipv6Addr>().unwrap().into(),
..narrow
};
let gone = withdrawals(IpVersion::Ipv4, &routes[..2], &over_v6);
assert_eq!(gone.len(), 1);
assert!(in_mp_unreach(&gone[0], 2), "{:?}", gone[0]);
}
}
crates/server/src/fsm.rs
+82 -66
View File
@@ -17,7 +17,7 @@ use std::time::Duration;
use bgp_packet::errors::{BgpError, CeaseSubcode, DecodeError, OpenMsgSubcode};
use bgp_packet::message::{AS_TRANS, BGP4_VERSION, Message, NotificationMessage, UpdateMessage};
use bgp_packet::open::{
AddPathEntry, AddPathSendRecv, Capability, ExtendedNextHop, OpenMessage, OpenOption,
AddPathEntry, Capability, ExtendedNextHopEntry, OpenMessage, OpenOption, SendReceive,
};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
@@ -69,7 +69,7 @@ pub struct FsmConfig {
pub families: FamilySet,
/// The directions to offer ADD-PATH (RFC 7911) for, on each of
/// `families`; `None` for neither.
pub add_path: Option<AddPathSendRecv>,
pub add_path: Option<SendReceive>,
/// What the peer is to us, if that is configured. Our OPEN then says
/// what we are to it, and the peer's must not say otherwise (RFC 9234).
/// Not for a peer in our own AS.
@@ -352,34 +352,9 @@ impl Fsm {
});
}
// RFC 9234 §4.2: the two ends have to agree on what they are to
// each other.
let mut their_roles = open
.capabilities()
.filter_map(|capability| match capability {
Capability::BgpRole { role } => Some(*role),
_ => None,
});
let their_role = their_roles.next();
let role_mismatch = |theirs| SessionError::RoleMismatch {
configured: self.config.role,
theirs,
};
// Saying it twice is saying it once; saying two things is saying
// nothing that can be relied on.
if their_roles.any(|role| Some(role) != their_role) {
return self.fail(role_mismatch(None));
}
let role_confirmed = match (self.config.role, their_role) {
(Some(configured), Some(theirs)) if theirs == configured.theirs() => true,
(Some(_), Some(theirs)) => return self.fail(role_mismatch(Some(theirs))),
(Some(_), None) if self.config.require_role => {
return self.fail(role_mismatch(None));
}
// A peer that does not say is taken at our word for it; and
// what a peer says of itself means nothing where nothing is
// configured to hold it against.
(Some(_), None) | (None, _) => false,
let role_confirmed = match self.negotiate_role(&open) {
Ok(confirmed) => confirmed,
Err(e) => return self.fail(e),
};
let ours = self.config.families.clone();
@@ -388,26 +363,7 @@ impl Fsm {
if families.is_empty() {
return self.fail(SessionError::NoCommonAddressFamily { ours, theirs });
}
// ADD-PATH is in use in a direction, for a family the session
// carries, when one side offered to send and the other to receive
// (RFC 7911 §4).
let (mut add_path_receive, mut add_path_send) =
(FamilySet::default(), FamilySet::default());
if let Some(ours) = self.config.add_path {
let theirs = open.capabilities().flat_map(|capability| match capability {
Capability::AddPath { entries } => entries.as_slice(),
_ => &[],
});
for entry in theirs.filter(|entry| families.contains(entry.family())) {
if ours.receives() && entry.send_recv.sends() {
add_path_receive.insert(entry.family());
}
if ours.sends() && entry.send_recv.receives() {
add_path_send.insert(entry.family());
}
}
}
let (add_path_receive, add_path_send) = self.negotiate_add_path(&open, &families);
// RFC 4724 §3: in use if both offered it.
let graceful_restart = self
@@ -477,13 +433,9 @@ impl Fsm {
error,
SessionError::HoldTimerExpired | SessionError::Collision
);
let notification = error.notification().map(|notification| {
if for_good && self.notifications_restart() {
restart::hard_reset(notification)
} else {
notification
}
});
let notification = error
.notification()
.map(|notification| self.as_sent(notification, for_good));
let mut actions: Vec<Action> = notification
.map(|notification| Action::Send(Message::Notification(notification)))
.into_iter()
@@ -520,16 +472,81 @@ impl Fsm {
error: BgpError::Cease(CeaseSubcode::AdminShutdown),
data: vec![],
};
let notification = if self.notifications_restart() {
restart::hard_reset(shutdown)
} else {
shutdown
};
let notification = self.as_sent(shutdown, true);
let mut actions = vec![Action::Send(Message::Notification(notification))];
actions.extend(self.close(Ok(())));
actions
}
/// What the peer says it is to us, held against what is configured
/// (RFC 9234 §4.2): whether the two agree, or an error if they do not.
/// A peer that does not say is taken at our word for it, unless it is
/// required to; what a peer says of itself means nothing where nothing
/// is configured to hold it against.
fn negotiate_role(&self, open: &OpenMessage) -> Result<bool, SessionError> {
let mut their_roles = open
.capabilities()
.filter_map(|capability| match capability {
Capability::BgpRole { role } => Some(*role),
_ => None,
});
let their_role = their_roles.next();
let mismatch = |theirs| SessionError::RoleMismatch {
configured: self.config.role,
theirs,
};
// Saying it twice is saying it once; saying two things is saying
// nothing that can be relied on.
if their_roles.any(|role| Some(role) != their_role) {
return Err(mismatch(None));
}
match (self.config.role, their_role) {
(Some(configured), Some(theirs)) if theirs == configured.theirs() => Ok(true),
(Some(_), Some(theirs)) => Err(mismatch(Some(theirs))),
(Some(_), None) if self.config.require_role => Err(mismatch(None)),
(Some(_), None) | (None, _) => Ok(false),
}
}
/// The families the peer sends us several routes for a prefix in, and
/// those we send it several in: ADD-PATH is in use in a direction, for
/// a family the session carries, when one side offered to send and the
/// other to receive (RFC 7911 §4).
fn negotiate_add_path(
&self,
open: &OpenMessage,
families: &FamilySet,
) -> (FamilySet, FamilySet) {
let (mut receive, mut send) = (FamilySet::default(), FamilySet::default());
let Some(ours) = self.config.add_path else {
return (receive, send);
};
let theirs = open.capabilities().flat_map(|capability| match capability {
Capability::AddPath { entries } => entries.as_slice(),
_ => &[],
});
for entry in theirs.filter(|entry| families.contains(entry.family())) {
if ours.receives() && entry.send_recv.sends() {
receive.insert(entry.family());
}
if ours.sends() && entry.send_recv.receives() {
send.insert(entry.family());
}
}
(receive, send)
}
/// `notification` as it goes to the peer: where NOTIFICATIONs count as
/// restarts (RFC 8538) and this one is `for_good`, as a Hard Reset, so
/// that the peer drops our routes rather than keeping them.
fn as_sent(&self, notification: NotificationMessage, for_good: bool) -> NotificationMessage {
if for_good && self.notifications_restart() {
restart::hard_reset(notification)
} else {
notification
}
}
fn close(&mut self, result: Result<(), SessionError>) -> Vec<Action> {
self.phase = Phase::Closed;
vec![Action::Close(result)]
@@ -538,7 +555,7 @@ impl Fsm {
/// IPv4 unicast routes with next hops that are IPv6 addresses, as the
/// Extended Next Hop capability names it (RFC 8950 §3).
const IPV4_WITH_IPV6_NEXT_HOP: ExtendedNextHop = ExtendedNextHop {
const IPV4_WITH_IPV6_NEXT_HOP: ExtendedNextHopEntry = ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
@@ -883,8 +900,7 @@ mod tests {
#[test]
fn add_path_is_in_use_where_one_side_sends_and_the_other_receives() {
use bgp_packet::open::SendReceive::{Both, Receive, Send};
let negotiated = |ours: Option<AddPathSendRecv>,
theirs: Vec<(AfiSafi, AddPathSendRecv)>| {
let negotiated = |ours: Option<SendReceive>, theirs: Vec<(AfiSafi, SendReceive)>| {
let (mut fsm, first) = Fsm::new(FsmConfig {
add_path: ours,
..config()
@@ -1049,7 +1065,7 @@ mod tests {
// Offered for something else: IPv6 routes with IPv4 next hops,
// say.
let other = Capability::ExtendedNextHop {
entries: vec![ExtendedNextHop {
entries: vec![ExtendedNextHopEntry {
afi: Afi::IPV6,
safi: Safi::UNICAST,
nh_afi: Afi::IPV4,
crates/server/src/holddown.rs
+174
View File
@@ -0,0 +1,174 @@
//! Peers that are being kept down for a while: those that went over a
//! limit, ours or theirs, and those whose sessions keep failing.
use std::collections::HashMap;
use std::sync::Mutex;
use std::time::{Duration, Instant};
use crate::address::PeerAddress;
use crate::sync::IgnorePoison;
/// Why a peer is being kept down.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HeldDown {
/// It went over its limit of routes.
OverLimit,
/// We were to announce it more routes than its export limit allows.
OverExportLimit,
/// Its sessions keep coming up and failing.
Flapping,
}
impl HeldDown {
pub fn as_str(self) -> &'static str {
match self {
Self::OverLimit => "held down after going over its limit of routes",
Self::OverExportLimit => {
"held down after more routes were to be advertised to it than its export limit allows"
}
Self::Flapping => "held down because its sessions keep failing",
}
}
}
/// A session that fails sooner than this after coming up counts as a
/// flap.
const FLAP_WITHIN: Duration = Duration::from_secs(60);
/// This many flaps in a row are let pass: a router being restarted, a
/// cable put back.
const FLAPS_ALLOWED: u32 = 2;
/// How long a peer is kept down after the first flap too many; twice as
/// long after each one more, up to `FLAP_HOLD_MAX`.
const FLAP_HOLD: Duration = Duration::from_secs(1);
const FLAP_HOLD_MAX: Duration = Duration::from_secs(300);
/// Peers that are being kept down, and until when: those that went over
/// their limit, and those whose sessions keep failing. Shared by the
/// sessions, which put peers here, and by the dialer and the listeners,
/// which leave them alone meanwhile.
///
/// Every session that comes up and goes costs a look at every route in
/// the table, and tells every other peer of the changes. A peer that
/// does so over and over, by fault or by design, is slowed down.
#[derive(Debug, Default)]
pub struct HoldDowns {
held: Mutex<HashMap<PeerAddress, (Instant, HeldDown)>>,
/// How many times in a row each peer's session has failed soon after
/// coming up.
flaps: Mutex<HashMap<PeerAddress, u32>>,
}
impl HoldDowns {
/// Keeps the peer at `address` down for `duration` from now.
pub fn hold(&self, address: PeerAddress, duration: Duration, why: HeldDown) {
let until = Instant::now() + duration;
let mut held = self.held.locked();
// Not for less long than it already is.
if held
.get(&address)
.is_none_or(|(already, _)| *already < until)
{
held.insert(address, (until, why));
}
}
/// How much longer the peer at `address` is kept down, if it is, and
/// why.
pub fn remaining(&self, address: PeerAddress) -> Option<(Duration, HeldDown)> {
let mut held = self.held.locked();
let (until, why) = *held.get(&address)?;
let remaining = until.saturating_duration_since(Instant::now());
if remaining.is_zero() {
held.remove(&address);
return None;
}
Some((remaining, why))
}
/// A session with the peer at `address` that had come up has ended,
/// `lasted` after it did; `failed` if not at our own wish. Returns for
/// how long the peer is kept down for it, if it is.
pub fn session_ended(
&self,
address: PeerAddress,
lasted: Duration,
failed: bool,
) -> Option<Duration> {
let mut flaps = self.flaps.locked();
if !failed || lasted >= FLAP_WITHIN {
flaps.remove(&address);
return None;
}
let count = flaps.entry(address).or_insert(0);
*count += 1;
let over = count.checked_sub(FLAPS_ALLOWED + 1)?;
let hold = FLAP_HOLD
.checked_mul(1u32.checked_shl(over).unwrap_or(u32::MAX))
.map_or(FLAP_HOLD_MAX, |hold| hold.min(FLAP_HOLD_MAX));
drop(flaps);
self.hold(address, hold, HeldDown::Flapping);
Some(hold)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_peer_is_held_down_for_as_long_as_was_said() {
let held = HoldDowns::default();
let address = PeerAddress::from("192.0.2.1".parse::<std::net::IpAddr>().unwrap());
let other = PeerAddress::from("192.0.2.2".parse::<std::net::IpAddr>().unwrap());
// Two failures soon after coming up are let pass; from the third
// the peer is kept down, twice as long each time, up to a limit.
let flaps = HoldDowns::default();
let soon = Duration::from_secs(5);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, true), Some(FLAP_HOLD));
assert_eq!(
flaps.session_ended(address, soon, true),
Some(2 * FLAP_HOLD)
);
assert_eq!(
flaps.remaining(address).map(|(_, why)| why),
Some(HeldDown::Flapping)
);
assert_eq!(flaps.remaining(other), None, "each peer for itself");
for _ in 0..40 {
flaps.session_ended(address, soon, true);
}
assert_eq!(
flaps.session_ended(address, soon, true),
Some(FLAP_HOLD_MAX)
);
// A session that lasted, or that we ended ourselves, starts the
// count over.
assert_eq!(flaps.session_ended(address, FLAP_WITHIN, true), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, false), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(held.remaining(address), None);
held.hold(address, Duration::from_secs(600), HeldDown::OverLimit);
let (remaining, why) = held.remaining(address).expect("held");
assert!(remaining > Duration::from_secs(590), "{remaining:?}");
assert_eq!(why, HeldDown::OverLimit);
assert_eq!(held.remaining(other), None);
// Not made shorter by a lesser reason.
held.hold(address, Duration::from_secs(1), HeldDown::Flapping);
let (remaining, why) = held.remaining(address).expect("held");
assert!(remaining > Duration::from_secs(590), "{remaining:?}");
assert_eq!(why, HeldDown::OverLimit);
held.hold(other, Duration::ZERO, HeldDown::OverLimit);
assert_eq!(held.remaining(other), None);
assert_eq!(
held.held.locked().len(),
1,
"a hold that is over is forgotten"
);
}
}
crates/server/src/lib.rs
+2 -1
View File
@@ -8,12 +8,13 @@ pub mod error;
pub mod export;
pub mod family;
mod fsm;
pub(crate) mod holddown;
pub mod marks;
pub mod metrics;
mod panic;
pub mod policy;
mod reach;
pub mod refusals;
pub mod rejections;
pub mod restart;
mod routes;
pub mod rpki;
crates/server/src/marks.rs
+329 -71
View File
@@ -9,7 +9,9 @@
//! - [`Marks`]: communities taken off and put on every route from a peer
//! (`import`) or to one (`export`).
//! - [`Control`]: communities on a route that say which peers it is, and
//! is not, passed on to.
//! is not, passed on to, and how far away we look to them.
//! - [`Scrub`]: the communities of our own AS taken off what other
//! networks send, so that only we and those we let can ask for that.
use bgp_packet::community::{CommunityError, parse_large, parse_standard};
use bgp_packet::message::{
@@ -66,12 +68,64 @@ fn part<T: std::str::FromStr>(text: &str, whole: &str) -> Result<Option<T>, Mark
}
}
/// Communities, standard and large, by patterns of them: `64601:100`,
/// `64601:*`, `64601:2:*`.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct Patterns {
standard: Vec<Pattern>,
large: Vec<LargePattern>,
}
impl Patterns {
/// From the configuration: patterns of standard communities (two
/// parts) and of large ones (three).
pub fn parse(texts: &[String]) -> Result<Self, MarksProblem> {
let mut patterns = Self::default();
for text in texts {
let parts: Vec<&str> = text.split(':').collect();
match parts.as_slice() {
[high, low] => patterns
.standard
.push(Pattern(part(high, text)?, part(low, text)?)),
[admin, first, second] => patterns.large.push(LargePattern(
part(admin, text)?,
part(first, text)?,
part(second, text)?,
)),
_ => return Err(MarksProblem::Pattern(text.clone())),
}
}
Ok(patterns)
}
pub fn is_empty(&self) -> bool {
self.standard.is_empty() && self.large.is_empty()
}
fn matches(&self, community: (u16, u16)) -> bool {
self.standard
.iter()
.any(|pattern| pattern.matches(community))
}
fn matches_large(&self, community: &LargeCommunity) -> bool {
self.large.iter().any(|pattern| pattern.matches(community))
}
/// Whether `attributes` have a community that one of the patterns
/// matches.
pub fn match_any(&self, attributes: &[PathAttribute]) -> bool {
let (standard, large) = communities_of(attributes);
standard.iter().any(|community| self.matches(*community))
|| large.iter().any(|community| self.matches_large(community))
}
}
/// What is done to the communities of every route that goes one way on
/// one session: some are taken off, then some are put on.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct Marks {
remove: Vec<Pattern>,
remove_large: Vec<LargePattern>,
remove: Patterns,
add: Vec<(u16, u16)>,
add_large: Vec<LargeCommunity>,
}
@@ -84,21 +138,10 @@ impl Marks {
add: &[String],
add_large: &[String],
) -> Result<Self, MarksProblem> {
let mut marks = Self::default();
for text in remove {
let parts: Vec<&str> = text.split(':').collect();
match parts.as_slice() {
[high, low] => marks
.remove
.push(Pattern(part(high, text)?, part(low, text)?)),
[admin, first, second] => marks.remove_large.push(LargePattern(
part(admin, text)?,
part(first, text)?,
part(second, text)?,
)),
_ => return Err(MarksProblem::Pattern(text.clone())),
}
}
let mut marks = Self {
remove: Patterns::parse(remove)?,
..Self::default()
};
for text in add {
let community = parse_standard(text)?;
if !marks.add.contains(&community) {
@@ -120,9 +163,9 @@ impl Marks {
/// Does it to `attributes`.
pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
if !self.remove.is_empty() || !self.add.is_empty() {
if !self.remove.standard.is_empty() || !self.add.is_empty() {
let mut communities = take_communities(attributes);
communities.retain(|community| !self.remove.iter().any(|p| p.matches(*community)));
communities.retain(|community| !self.remove.matches(*community));
for community in &self.add {
if !communities.contains(community) {
communities.push(*community);
@@ -130,9 +173,9 @@ impl Marks {
}
put_communities(attributes, communities);
}
if !self.remove_large.is_empty() || !self.add_large.is_empty() {
if !self.remove.large.is_empty() || !self.add_large.is_empty() {
let mut large = take_large_communities(attributes);
large.retain(|community| !self.remove_large.iter().any(|p| p.matches(community)));
large.retain(|community| !self.remove.matches_large(community));
for community in &self.add_large {
if !large.contains(community) {
large.push(community.clone());
@@ -196,26 +239,52 @@ pub(crate) fn put(attributes: &mut Vec<PathAttribute>, attribute: PathAttribute)
}
}
/// The communities by which a route says who is told of it, as Internet
/// exchanges' route servers have them. With `L` our own AS number and `P`
/// a peer's:
/// The communities by which a route says who is told of it, and with how
/// many prepends, as Internet exchanges' route servers and transit
/// networks have them. With `L` our own AS number, `P` a peer's and `G` a
/// group of sessions (`groups` of a peer):
///
/// | Standard | Large | |
/// |---|---|---|
/// | `0:P` | `L:0:P` | not to the peer with AS `P` |
/// | `L:P` | `L:1:P` | to the peer with AS `P`, and with that only to peers named so |
/// | `0:L` | `L:0:0` | to nobody |
/// | | `L:11:P`, `L:12:P`, `L:13:P` | our AS once, twice, three times more on the path to AS `P`; `P` 0 for everyone |
/// | | `L:20:G` | not to the sessions of group `G` |
/// | | `L:21:G` | to the sessions of group `G`, and with that only to sessions named so |
/// | | `L:31:G`, `L:32:G`, `L:33:G` | our AS once, twice, three times more on the path to group `G` |
///
/// A route with none of them goes to everyone. What says "not" wins over
/// what says "to". A standard community has room for AS numbers below
/// 65536 only; the large ones are for the others, and work for all.
/// what says "to"; of several prepends, the most counts. A standard
/// community has room for AS numbers below 65536 only; the large ones
/// are for the others, and work for all.
///
/// These are taken off a route as it is passed on: they were for us.
/// They are acted on where a route leaves our AS, and taken off there:
/// they were for us. Other large communities of ours are marks, which go
/// on.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Control {
pub local_asn: u32,
}
/// What a large community of ours asks for, by its second part.
const NOT_TO_AS: u32 = 0;
const ONLY_TO_AS: u32 = 1;
const PREPEND_TO_AS: [u32; 3] = [11, 12, 13];
const NOT_TO_GROUP: u32 = 20;
const ONLY_TO_GROUP: u32 = 21;
const PREPEND_TO_GROUP: [u32; 3] = [31, 32, 33];
/// Whether a large community of ours with `function` as its second part
/// asks for something, and is not a mark.
fn is_action(function: u32) -> bool {
matches!(
function,
NOT_TO_AS | ONLY_TO_AS | NOT_TO_GROUP | ONLY_TO_GROUP
) || PREPEND_TO_AS.contains(&function)
|| PREPEND_TO_GROUP.contains(&function)
}
/// The well-known communities that limit how far a route goes
/// (RFC 1997).
const NO_EXPORT: (u16, u16) = (0xFFFF, 0xFF01);
@@ -226,29 +295,45 @@ const NO_EXPORT_SUBCONFED: (u16, u16) = (0xFFFF, 0xFF03);
/// all, as far as the well-known ones say: NO_ADVERTISE keeps it from
/// everyone, NO_EXPORT from other networks. (There are no confederations
/// here, so NO_EXPORT_SUBCONFED is the same as NO_EXPORT.)
pub fn well_known_allow(communities: &[(u16, u16)], peer_is_internal: bool) -> bool {
pub fn well_known_allow(communities: &[(u16, u16)], ibgp: bool) -> bool {
!communities.iter().any(|community| {
*community == NO_ADVERTISE
|| (!peer_is_internal && matches!(*community, NO_EXPORT | NO_EXPORT_SUBCONFED))
|| (!ibgp && matches!(*community, NO_EXPORT | NO_EXPORT_SUBCONFED))
})
}
/// A session a route may go out on, as control communities name it.
#[derive(Debug, Clone, Copy)]
pub struct Named<'a> {
pub asn: u32,
pub groups: &'a [u32],
}
impl Named<'_> {
fn is(&self, asn: u32) -> bool {
asn == self.asn
}
fn in_group(&self, group: u32) -> bool {
self.groups.contains(&group)
}
}
impl Control {
/// Our AS number as half a standard community, if it fits.
fn local_16(&self) -> Option<u16> {
u16::try_from(self.local_asn).ok()
}
/// Whether a route with these communities is passed on to the peer
/// with AS number `peer_asn`.
/// Whether a route with these communities is passed on to `session`.
pub fn allows(
&self,
communities: &[(u16, u16)],
large: &[LargeCommunity],
peer_asn: u32,
session: Named<'_>,
) -> bool {
let local_16 = self.local_16();
let peer_16 = u16::try_from(peer_asn).ok();
let peer_16 = u16::try_from(session.asn).ok();
let (mut not_to_peer, mut to_peer, mut to_nobody, mut to_some) =
(false, false, false, false);
for (high, low) in communities {
@@ -267,10 +352,15 @@ impl Control {
continue;
}
match (community.data_1, community.data_2) {
(0, 0) => to_nobody = true,
(0, asn) => not_to_peer |= asn == peer_asn,
(1, asn) => {
to_peer |= asn == peer_asn;
(NOT_TO_AS, 0) => to_nobody = true,
(NOT_TO_AS, asn) => not_to_peer |= session.is(asn),
(ONLY_TO_AS, asn) => {
to_peer |= session.is(asn);
to_some = true;
}
(NOT_TO_GROUP, group) => not_to_peer |= session.in_group(group),
(ONLY_TO_GROUP, group) => {
to_peer |= session.in_group(group);
to_some = true;
}
_ => {}
@@ -279,31 +369,103 @@ impl Control {
!not_to_peer && (to_peer || !(to_nobody || to_some))
}
/// How many more times our AS goes on the path of a route with these
/// large communities to `session`: the most that one of them asks.
pub fn prepends(&self, large: &[LargeCommunity], session: Named<'_>) -> u8 {
let mut most = 0;
for community in large {
if community.global_admin != self.local_asn {
continue;
}
let (function, target) = (community.data_1, community.data_2);
let times = if let Some(at) = PREPEND_TO_AS.iter().position(|f| *f == function) {
(target == 0 || session.is(target)).then_some(at + 1)
} else if let Some(at) = PREPEND_TO_GROUP.iter().position(|f| *f == function) {
session.in_group(target).then_some(at + 1)
} else {
None
};
// At most three: the position in a list of three, plus one.
most = most.max(times.map_or(0, |times| times as u8));
}
most
}
/// Takes off `attributes` the communities that were for us to act on.
pub fn strip(&self, attributes: &mut Vec<PathAttribute>) {
let local_16 = self.local_16();
let ours = |community: &(u16, u16)| community.0 == 0 || Some(community.0) == local_16;
let has = attributes.iter().any(|attribute| match attribute {
PathAttribute::Communities(communities) => communities.0.iter().any(ours),
_ => false,
});
if has {
let mut communities = take_communities(attributes);
communities.retain(|community| !ours(community));
put_communities(attributes, communities);
}
let ours_large = |community: &LargeCommunity| {
community.global_admin == self.local_asn && community.data_1 <= 1
};
let has_large = attributes.iter().any(|attribute| match attribute {
PathAttribute::LargeCommunities(large) => large.communities.iter().any(ours_large),
_ => false,
});
if has_large {
let mut large = take_large_communities(attributes);
large.retain(|community| !ours_large(community));
put_large_communities(attributes, large);
}
remove_where(
attributes,
|community| community.0 == 0 || Some(community.0) == local_16,
|community| community.global_admin == self.local_asn && is_action(community.data_1),
);
}
}
/// The communities of our own AS, taken off the routes of a peer in
/// another AS as they come in. What they ask of us (see [`Control`]), and
/// what our marks say of a route, is not for others to say: a peer at an
/// exchange that put `L:0:0` on its routes would decide which of our
/// customers hear of them. The standard form of "not to" (`0:P`) is only
/// ours to act on where control communities are, and so taken off only
/// then.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Scrub {
pub local_asn: u32,
/// Take off `0:n` too.
pub zero: bool,
}
impl Scrub {
fn ours(&self, community: &(u16, u16)) -> bool {
(self.zero && community.0 == 0) || u32::from(community.0) == self.local_asn
}
fn ours_large(&self, community: &LargeCommunity) -> bool {
community.global_admin == self.local_asn
}
/// Whether `attributes` have a community to take off: asked of every
/// UPDATE, so that those without one are taken as they came.
pub fn finds(&self, attributes: &[PathAttribute]) -> bool {
let (standard, large) = communities_of(attributes);
standard.iter().any(|community| self.ours(community))
|| large.iter().any(|community| self.ours_large(community))
}
pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
remove_where(
attributes,
|community| self.ours(community),
|community| self.ours_large(community),
);
}
}
/// Takes off `attributes` the standard communities for which `standard`
/// holds and the large ones for which `large` does, leaving the
/// attributes as they were if there are none.
fn remove_where(
attributes: &mut Vec<PathAttribute>,
standard: impl Fn(&(u16, u16)) -> bool,
large: impl Fn(&LargeCommunity) -> bool,
) {
let (has, has_large) = {
let (communities, large_communities) = communities_of(attributes);
(
communities.iter().any(&standard),
large_communities.iter().any(&large),
)
};
if has {
let mut communities = take_communities(attributes);
communities.retain(|community| !standard(community));
put_communities(attributes, communities);
}
if has_large {
let mut large_communities = take_large_communities(attributes);
large_communities.retain(|community| !large(community));
put_large_communities(attributes, large_communities);
}
}
@@ -413,7 +575,7 @@ mod tests {
fn communities_say_which_peers_a_route_goes_to() {
let control = Control { local_asn: 64601 };
let to = |communities: &[(u16, u16)], large: &[LargeCommunity], peer: u32| {
control.allows(communities, large, peer)
control.allows(communities, large, named(peer, &[]))
};
// Nothing said: everyone.
assert!(to(&[(3356, 3)], &[], 64602));
@@ -451,6 +613,7 @@ mod tests {
&[large(3356, 0, 64602), large(64601, 2, 64602)],
64602
));
assert!(to(&[], &[large(64601, 100, 64602)], 64602));
// A peer whose AS number does not fit a standard community is not
// meant by one.
assert!(to(&[(0, 1)], &[], 65537));
@@ -458,27 +621,122 @@ mod tests {
// Our own AS number may not fit either: then only the large ones
// can name us.
let control = Control { local_asn: wide };
assert!(control.allows(&[(0, 64602)], &[], 64603));
assert!(!control.allows(&[(0, 64602)], &[], 64602));
assert!(!control.allows(&[], &[large(wide, 0, 0)], 64602));
assert!(control.allows(&[(0, 64602)], &[], named(64603, &[])));
assert!(!control.allows(&[(0, 64602)], &[], named(64602, &[])));
assert!(!control.allows(&[], &[large(wide, 0, 0)], named(64602, &[])));
}
fn named(asn: u32, groups: &[u32]) -> Named<'_> {
Named { asn, groups }
}
/// Sessions are named by group as well as by AS: the same network at
/// two exchanges is two sessions in two groups.
#[test]
fn communities_name_sessions_by_group() {
let control = Control { local_asn: 64601 };
let at_one = named(13335, &[1]);
let at_two = named(13335, &[2, 10]);
let allows = |large: &[LargeCommunity], session| control.allows(&[], large, session);
// Not to group 2: the same peer elsewhere still hears of it.
assert!(!allows(&[large(64601, 20, 2)], at_two));
assert!(allows(&[large(64601, 20, 2)], at_one));
// Only to group 1, or to group 1 and AS 13335 anywhere.
assert!(allows(&[large(64601, 21, 1)], at_one));
assert!(!allows(&[large(64601, 21, 1)], at_two));
assert!(!allows(&[large(64601, 21, 1)], named(64602, &[])));
assert!(allows(
&[large(64601, 21, 1), large(64601, 1, 13335)],
at_two
));
// A session in several groups is in each of them.
assert!(!allows(&[large(64601, 20, 10)], at_two));
// "Not" wins, whichever way the session is named.
assert!(!allows(
&[large(64601, 1, 13335), large(64601, 20, 2)],
at_two
));
assert!(!allows(
&[large(64601, 21, 2), large(64601, 0, 13335)],
at_two
));
}
#[test]
fn communities_say_how_far_away_we_look() {
let control = Control { local_asn: 64601 };
let at_two = named(13335, &[2]);
let prepends = |large: &[LargeCommunity], session| control.prepends(large, session);
assert_eq!(prepends(&[], at_two), 0);
// To a group, to an AS, to everyone.
assert_eq!(prepends(&[large(64601, 31, 2)], at_two), 1);
assert_eq!(prepends(&[large(64601, 33, 2)], at_two), 3);
assert_eq!(prepends(&[large(64601, 31, 1)], at_two), 0);
assert_eq!(prepends(&[large(64601, 12, 13335)], at_two), 2);
assert_eq!(prepends(&[large(64601, 12, 64602)], at_two), 0);
assert_eq!(prepends(&[large(64601, 11, 0)], at_two), 1);
// The most counts, not the sum.
assert_eq!(
prepends(
&[
large(64601, 31, 2),
large(64601, 13, 13335),
large(64601, 12, 0)
],
at_two
),
3
);
// Group 0 is no group; someone else's say nothing.
assert_eq!(prepends(&[large(64601, 31, 0)], named(13335, &[])), 0);
assert_eq!(prepends(&[large(3356, 13, 0)], at_two), 0);
}
/// From another network, ours are taken off, and nothing else is.
#[test]
fn our_communities_are_not_taken_from_others() {
let scrub = Scrub {
local_asn: 64601,
zero: true,
};
let mut attributes = route(
&[(0, 64602), (3356, 3), (64601, 900)],
&[large(64601, 0, 0), large(64601, 100, 1), large(3356, 0, 5)],
);
assert!(scrub.finds(&attributes));
scrub.apply(&mut attributes);
assert_eq!(attributes, route(&[(3356, 3)], &[large(3356, 0, 5)]));
assert!(!scrub.finds(&attributes));
// `0:n` only where it means something to us.
let scrub = Scrub {
zero: false,
..scrub
};
assert!(!scrub.finds(&route(&[(0, 64602)], &[])));
// An AS number too wide for a standard community has none of them.
let wide = Scrub {
local_asn: 4_200_000_001,
zero: false,
};
assert!(!wide.finds(&route(&[(64601, 1)], &[large(64601, 0, 0)])));
assert!(wide.finds(&route(&[], &[large(4_200_000_001, 100, 1)])));
}
#[test]
fn what_was_for_us_is_taken_off_as_a_route_is_passed_on() {
let control = Control { local_asn: 64601 };
let actions = [0, 1, 11, 12, 13, 20, 21, 31, 32, 33].map(|f| large(64601, f, 5));
let marks = [2, 3, 9, 10, 14, 30, 100].map(|f| large(64601, f, 7));
let mut attributes = route(
&[(0, 64602), (3356, 3), (64601, 64603)],
&[
large(64601, 0, 5),
large(64601, 1, 6),
large(64601, 2, 7),
large(3356, 0, 5),
],
&[&actions[..], &marks[..], &[large(3356, 0, 5)]].concat(),
);
control.strip(&mut attributes);
// Our marks go on; what was for us to do does not.
assert_eq!(
attributes,
route(&[(3356, 3)], &[large(64601, 2, 7), large(3356, 0, 5)])
route(&[(3356, 3)], &[&marks[..], &[large(3356, 0, 5)]].concat())
);
let mut all = route(&[(0, 1)], &[large(64601, 0, 0)]);
control.strip(&mut all);
crates/server/src/metrics.rs
+156 -2
View File
@@ -4,8 +4,8 @@ use std::sync::LazyLock;
use bgp_packet::message::Message;
use prometheus::{
IntCounter, IntCounterVec, IntGauge, IntGaugeVec, register_int_counter,
register_int_counter_vec, register_int_gauge, register_int_gauge_vec,
GaugeVec, IntCounter, IntCounterVec, IntGauge, IntGaugeVec, register_gauge_vec,
register_int_counter, register_int_counter_vec, register_int_gauge, register_int_gauge_vec,
};
pub static MESSAGES_SENT: LazyLock<IntCounterVec> = LazyLock::new(|| {
@@ -69,6 +69,148 @@ pub static PEER_ROUTES: LazyLock<IntGaugeVec> = LazyLock::new(|| {
.expect("metric registration")
});
pub static PEER_ADVERTISED: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_peer_advertised_routes",
"Routes announced to an established peer and not withdrawn since",
&["peer"]
)
.expect("metric registration")
});
/// A session's TCP connection as the kernel reports it (`TCP_INFO`),
/// sampled when the metrics are scraped. Seconds, octets, segments, and
/// octets per second; the busy, limited and byte figures are cumulative
/// over the connection's life.
pub struct PeerTcp {
pub rtt_seconds: GaugeVec,
pub min_rtt_seconds: GaugeVec,
pub rtt_variance_seconds: GaugeVec,
pub cwnd_segments: IntGaugeVec,
pub mss_bytes: IntGaugeVec,
pub send_window_bytes: IntGaugeVec,
pub receive_window_bytes: IntGaugeVec,
pub send_queue_bytes: IntGaugeVec,
pub not_sent_bytes: IntGaugeVec,
pub receive_queue_bytes: IntGaugeVec,
pub unacked_segments: IntGaugeVec,
pub retransmitted_segments: IntGaugeVec,
pub lost_segments: IntGaugeVec,
pub busy_seconds: GaugeVec,
pub receive_window_limited_seconds: GaugeVec,
pub send_buffer_limited_seconds: GaugeVec,
pub sent_bytes: IntGaugeVec,
pub received_bytes: IntGaugeVec,
pub delivery_rate_bytes_per_second: IntGaugeVec,
pub pacing_rate_bytes_per_second: IntGaugeVec,
}
impl PeerTcp {
/// Forgets the peer's series, when its session has ended.
pub fn remove(&self, label: &str) {
for gauge in [
&self.rtt_seconds,
&self.min_rtt_seconds,
&self.rtt_variance_seconds,
&self.busy_seconds,
&self.receive_window_limited_seconds,
&self.send_buffer_limited_seconds,
] {
let _ = gauge.remove_label_values(&[label]);
}
for gauge in [
&self.cwnd_segments,
&self.mss_bytes,
&self.send_window_bytes,
&self.receive_window_bytes,
&self.send_queue_bytes,
&self.not_sent_bytes,
&self.receive_queue_bytes,
&self.unacked_segments,
&self.retransmitted_segments,
&self.lost_segments,
&self.sent_bytes,
&self.received_bytes,
&self.delivery_rate_bytes_per_second,
&self.pacing_rate_bytes_per_second,
] {
let _ = gauge.remove_label_values(&[label]);
}
}
}
pub static PEER_TCP: LazyLock<PeerTcp> = LazyLock::new(|| {
let seconds = |name: &str, help: &str| {
register_gauge_vec!(format!("bgp_peer_tcp_{name}"), help, &["peer"])
.expect("metric registration")
};
let count = |name: &str, help: &str| {
register_int_gauge_vec!(format!("bgp_peer_tcp_{name}"), help, &["peer"])
.expect("metric registration")
};
PeerTcp {
rtt_seconds: seconds(
"rtt_seconds",
"Smoothed round-trip time of the session's TCP connection",
),
min_rtt_seconds: seconds(
"min_rtt_seconds",
"Least round-trip time seen on the connection",
),
rtt_variance_seconds: seconds(
"rtt_variance_seconds",
"Round-trip time variance of the connection",
),
cwnd_segments: count("cwnd_segments", "Congestion window, in segments"),
mss_bytes: count("mss_bytes", "Maximum segment size"),
send_window_bytes: count(
"send_window_bytes",
"The peer's receive window: what it lets us send",
),
receive_window_bytes: count(
"receive_window_bytes",
"Our receive window: what we let the peer send",
),
send_queue_bytes: count(
"send_queue_bytes",
"Octets written and not yet acknowledged by the peer",
),
not_sent_bytes: count("not_sent_bytes", "Octets written and not yet sent"),
receive_queue_bytes: count("receive_queue_bytes", "Octets received and not yet read"),
unacked_segments: count("unacked_segments", "Segments sent and not yet acknowledged"),
retransmitted_segments: count(
"retransmitted_segments",
"Segments retransmitted over the connection's life",
),
lost_segments: count("lost_segments", "Segments thought lost now"),
busy_seconds: seconds(
"busy_seconds",
"Time the connection had something to send, over its life",
),
receive_window_limited_seconds: seconds(
"receive_window_limited_seconds",
"Of the busy time, time held up by the peer's receive window",
),
send_buffer_limited_seconds: seconds(
"send_buffer_limited_seconds",
"Of the busy time, time held up by our send buffer being full",
),
sent_bytes: count("sent_bytes", "Octets sent over the connection's life"),
received_bytes: count(
"received_bytes",
"Octets received over the connection's life",
),
delivery_rate_bytes_per_second: count(
"delivery_rate_bytes_per_second",
"Delivery rate the kernel measured most recently",
),
pacing_rate_bytes_per_second: count(
"pacing_rate_bytes_per_second",
"Rate the kernel paces the connection at",
),
}
});
pub static PEER_MAX_PREFIXES: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_peer_max_prefixes",
@@ -87,6 +229,15 @@ pub static PREFIX_LIMIT_EXCEEDED: LazyLock<IntCounterVec> = LazyLock::new(|| {
.expect("metric registration")
});
pub static EXPORT_LIMIT_EXCEEDED: LazyLock<IntCounterVec> = LazyLock::new(|| {
register_int_counter_vec!(
"bgp_export_limit_exceeded_total",
"Sessions ended because more routes were to be announced to the peer than its export limit allows",
&["peer"]
)
.expect("metric registration")
});
pub static RPKI_VRPS: LazyLock<IntGaugeVec> = LazyLock::new(|| {
register_int_gauge_vec!(
"bgp_rpki_vrps",
@@ -206,6 +357,7 @@ pub fn init() {
LazyLock::force(&PEER_ROUTES);
LazyLock::force(&PEER_MAX_PREFIXES);
LazyLock::force(&PREFIX_LIMIT_EXCEEDED);
LazyLock::force(&EXPORT_LIMIT_EXCEEDED);
LazyLock::force(&RPKI_VRPS);
LazyLock::force(&RPKI_ASPAS);
LazyLock::force(&RPKI_UPDATES);
@@ -218,6 +370,8 @@ pub fn init() {
LazyLock::force(&RIB_JOURNAL_ENTRIES);
LazyLock::force(&ROUTES_EXPORTED);
LazyLock::force(&PEER_UP);
LazyLock::force(&PEER_ADVERTISED);
LazyLock::force(&PEER_TCP);
BUILD_INFO
.with_label_values(&[env!("CARGO_PKG_VERSION")])
.set(1);
crates/server/src/policy.rs
+347 -183
View File
@@ -26,8 +26,8 @@
use std::collections::{HashMap, HashSet};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::sync::{Arc, LazyLock, Mutex};
use std::time::{Duration, Instant};
use std::sync::{Arc, LazyLock};
use std::time::Duration;
use bgp_packet::message::{
AS_TRANS, AsPathAttribute, LocalPrefPathAttribute, NlriNextHop, PathAttribute,
@@ -36,9 +36,7 @@ use bgp_packet::open::BgpRole;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use crate::address::PeerAddress;
use crate::marks::{self, Marks};
use crate::sync::IgnorePoison;
use crate::marks::{self, Marks, Patterns, Scrub};
/// The longest prefixes routed between networks: anything more specific is
/// filtered nearly everywhere, and is usually a leak of internal routes.
@@ -93,17 +91,93 @@ pub struct ImportPolicy {
/// who never asked for it. A route server's routes always have its
/// clients as next hops, and are taken either way.
pub accept_third_party_next_hop: bool,
/// Take routes with the BLACKHOLE community for single addresses, and
/// have the kernel drop the traffic to them (RFC 7999).
pub accept_blackholes: bool,
/// The prefixes this router originates. Those, and what is inside
/// them, are ours and not taken from another network, unless
/// `prefixes` says in so many words that this peer may announce them:
/// a customer with a part of our space. Empty for a peer in our own
/// AS.
pub own_prefixes: Arc<PrefixSet>,
/// The prefixes this router originates, each by itself. A peer in our
/// own AS that announces one of them originates it too, as with an
/// anycast prefix that both announce: its route is of no use here,
/// where the prefix is local, and is ignored, neither taken nor
/// counted as rejected. Empty for a peer in another AS, whose routes
/// for them `own_prefixes` refuses.
pub originated: Arc<PrefixSet>,
/// Keep the communities of our own AS on the peer's routes, for it to
/// steer them with control communities: a customer. Without it, those
/// of a peer in another AS are taken off by `scrub`.
pub accept_control_communities: bool,
/// The communities of our own AS taken off the peer's routes as they
/// come in, first; `None` for a peer in our own AS and for one with
/// `accept_control_communities`.
pub scrub: Option<Scrub>,
/// Communities taken off and put on every route that is taken.
pub marks: Marks,
/// The LOCAL_PREF every route that is taken is given; `None` leaves
/// it at the default.
pub local_pref: Option<u32>,
/// What is done to some of the routes that are taken, beyond the
/// above: each route by the first rule that holds for it.
pub rules: Vec<ImportRule>,
/// For a peer in our own AS, what a route passed on inside the AS
/// must not say it has already been through (RFC 4456 §8); `None` for
/// a peer in another AS, whose routes carry no such thing.
pub reflection: Option<ReflectorIds>,
}
/// Who we are to routes passed on from one router of our AS to another:
/// a route whose ORIGINATOR_ID is our BGP Identifier started here, and one
/// whose CLUSTER_LIST holds our cluster ID has been passed on by us
/// already. Either has gone round in a circle, and is not taken.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct ReflectorIds {
pub router_id: Ipv4Addr,
pub cluster_id: Ipv4Addr,
}
/// What is done to the routes with some communities, or for some
/// prefixes, as they are taken.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ImportRule {
/// The rule is for routes with one of these communities, as the
/// route has them once the import policy has marked it; for any if
/// empty.
pub communities: Patterns,
/// And for routes whose prefix is in these; for any if `None`.
pub prefixes: Option<PrefixSet>,
/// The LOCAL_PREF it gives them, in place of the policy's.
pub local_pref: Option<u32>,
/// Communities it puts on them.
pub marks: Marks,
}
impl ImportRule {
/// Whether the rule is for routes with `attributes`, as far as their
/// communities go: the same for all of an UPDATE's routes.
pub fn holds_for(&self, attributes: &[PathAttribute]) -> bool {
self.communities.is_empty() || self.communities.match_any(attributes)
}
/// Whether it is for routes for `prefix`, as far as that goes.
pub fn holds_for_prefix(&self, prefix: &IpPrefix) -> bool {
self.prefixes
.as_ref()
.is_none_or(|prefixes| prefixes.allows(prefix))
}
pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
self.marks.apply(attributes);
if let Some(local_pref) = self.local_pref {
marks::put(
attributes,
PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)),
);
}
}
}
impl Default for ImportPolicy {
@@ -125,9 +199,15 @@ impl Default for ImportPolicy {
accept_aspa_invalid: false,
accept_as_sets: false,
accept_third_party_next_hop: false,
accept_blackholes: false,
own_prefixes: Arc::default(),
originated: Arc::default(),
accept_control_communities: false,
scrub: None,
marks: Marks::default(),
local_pref: None,
rules: Vec::new(),
reflection: None,
}
}
}
@@ -148,20 +228,28 @@ impl ImportPolicy {
accept_aspa_invalid: true,
accept_as_sets: true,
accept_third_party_next_hop: true,
accept_blackholes: false,
..Self::default()
}
}
}
impl ImportPolicy {
/// Whether routes are changed on their way in.
pub fn changes_routes(&self) -> bool {
!self.marks.is_empty() || self.local_pref.is_some()
/// Whether routes with `attributes` are changed on their way in.
/// Asked of every UPDATE: those that come as they are to be kept are
/// taken as they came, without being written out again.
pub fn changes(&self, attributes: &[PathAttribute]) -> bool {
!self.marks.is_empty()
|| self.local_pref.is_some()
|| self.scrub.is_some_and(|scrub| scrub.finds(attributes))
}
/// Makes of `attributes` what the route is kept with: marked, and
/// with its preference.
/// Makes of `attributes` what the route is kept with: without what
/// others may not say for us, marked, and with its preference.
pub fn change(&self, attributes: &mut Vec<PathAttribute>) {
if let Some(scrub) = self.scrub {
scrub.apply(attributes);
}
self.marks.apply(attributes);
if let Some(local_pref) = self.local_pref {
marks::put(
@@ -178,12 +266,51 @@ pub struct ExportPolicy {
/// Pass on the best routes learned from other peers, and not only
/// what we originate.
pub learned_routes: bool,
/// Pass on every route learned from other peers, not only the best
/// of each prefix, told apart by path identifiers (ADD-PATH, RFC
/// 7911), for the families the peer agreed to receive them for; a
/// route collector asks for this. For other families the best goes.
pub all_paths: bool,
/// Communities taken off and put on every route announced.
pub marks: Marks,
/// How many more times our AS goes on the path.
pub prepend: u8,
}
/// Limits on what is announced to a network that is not our customer,
/// which hold whatever the export policy and the routes say: a check on
/// both, in case either goes wrong.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ExportLimits {
/// The prefixes that may be announced; `None` for any.
pub prefixes: Option<PrefixSet>,
/// The most routes announced at a time.
pub max_prefixes: Option<u32>,
}
impl ExportLimits {
/// Whether they are for a session to a peer that is `role` to us, in
/// another AS or not (`ibgp`): one above or beside us, or one we know
/// nothing of. Customers, route server clients and our own routers
/// are told of whatever they are to be.
pub fn apply_to(role: Option<PeerRole>, ibgp: bool) -> bool {
!ibgp && role.is_none_or(PeerRole::is_above_or_beside)
}
/// Whether a route for `prefix` may be announced.
pub fn allows(&self, prefix: &IpPrefix) -> bool {
self.prefixes
.as_ref()
.is_none_or(|prefixes| prefixes.allows(prefix))
}
/// The limit that `routes` routes at a time would go over, if they
/// would.
pub fn exceeded_by(&self, routes: usize) -> Option<u32> {
self.max_prefixes.filter(|limit| routes > *limit as usize)
}
}
/// Why a route was not taken.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Rejection {
@@ -219,10 +346,14 @@ pub enum Rejection {
/// The path has an AS_SET in it (RFC 9774).
AsSet,
/// The route asks for the traffic to its prefix to be dropped: it has
/// the BLACKHOLE community (RFC 7999). Such requests are not acted on,
/// and the route is not taken as an ordinary one either, which would
/// send the traffic to wherever its next hop drops it.
/// the BLACKHOLE community (RFC 7999), and the policy does not take
/// such requests (`accept_blackholes`). The route is not taken as an
/// ordinary one either, which would send the traffic to wherever its
/// next hop drops it.
Blackhole,
/// A blackhole request for more than one address: only a /32 or a
/// /128 is taken as one.
BlackholeNotHost,
/// The next hop is no address packets could be sent to, or is our
/// own, or is an IPv6 address for an IPv4 route on a session that did
/// not agree to that (RFC 8950).
@@ -234,6 +365,13 @@ pub enum Rejection {
/// The next hop cannot be reached from this machine as its interfaces
/// are now, so the kernel could not be given the route.
NextHopUnreachable,
/// The prefix is a network of one of this machine's interfaces, or
/// inside one: the machine reaches it directly, and a route by way of
/// a peer would compete with that.
OwnNetwork,
/// The route was passed on inside our AS and has come back: it says it
/// started at this router, or was passed on by it (RFC 4456 §8).
ReflectionLoop,
}
impl Rejection {
@@ -243,11 +381,11 @@ impl Rejection {
pub(crate) fn may_pass(self) -> bool {
matches!(
self,
Self::RpkiInvalid | Self::AspaInvalid | Self::NextHopUnreachable
Self::RpkiInvalid | Self::AspaInvalid | Self::NextHopUnreachable | Self::OwnNetwork
)
}
pub const ALL: [Self; 18] = [
pub const ALL: [Self; 21] = [
Self::BogonPrefix,
Self::TooShort,
Self::TooSpecific,
@@ -262,10 +400,13 @@ impl Rejection {
Self::Leak,
Self::AsSet,
Self::Blackhole,
Self::BlackholeNotHost,
Self::NextHopUnusable,
Self::NextHopNotPeer,
Self::OwnPrefix,
Self::NextHopUnreachable,
Self::OwnNetwork,
Self::ReflectionLoop,
];
/// Where the reason is in [`Rejection::ALL`].
@@ -289,10 +430,13 @@ impl Rejection {
Self::Leak => "route leak (Only to Customer)",
Self::AsSet => "AS_SET in path",
Self::Blackhole => "blackhole request",
Self::BlackholeNotHost => "blackhole for more than one address",
Self::NextHopUnusable => "next hop unusable",
Self::NextHopNotPeer => "next hop is not the peer",
Self::OwnPrefix => "prefix is our own",
Self::NextHopUnreachable => "next hop unreachable",
Self::OwnNetwork => "prefix is a network of this router",
Self::ReflectionLoop => "route reflected back to us",
}
}
}
@@ -424,17 +568,23 @@ impl ImportPolicy {
.is_some_and(|segment| segment.ordered);
let origin = facts.last.filter(|_| ends_in_sequence);
if from.ibgp {
if let Some(us) = self.reflection {
let came_back = attributes.iter().any(|attribute| match attribute {
PathAttribute::OriginatorId(originator) => *originator == us.router_id,
PathAttribute::ClusterList(clusters) => clusters.contains(&us.cluster_id),
_ => false,
});
if came_back {
return Err(Rejection::ReflectionLoop);
}
}
return Ok(origin);
}
if facts.has_local_asn {
return Err(Rejection::OwnAs);
}
let blackhole = attributes.iter().any(|attribute| {
matches!(attribute, PathAttribute::Communities(communities)
if communities.0.contains(&BLACKHOLE))
});
if blackhole {
if !self.accept_blackholes && Self::is_blackhole_request(attributes) {
return Err(Rejection::Blackhole);
}
// RFC 9234 §5, ingress, 1 and 2. A route that is only to go to
@@ -513,6 +663,35 @@ impl ImportPolicy {
}
}
/// Whether the routes with these attributes ask for the traffic to
/// them to be dropped: the BLACKHOLE community (RFC 7999).
pub fn is_blackhole_request(attributes: &[PathAttribute]) -> bool {
attributes.iter().any(|attribute| {
matches!(attribute, PathAttribute::Communities(communities)
if communities.0.contains(&BLACKHOLE))
})
}
/// Judges the prefix of a blackhole request that the policy takes
/// (`accept_blackholes`): one address, and inside what the peer may
/// announce. The longest prefix taken otherwise does not apply, nor
/// does the longest allowed inside a listed prefix: a request is for
/// one address by its nature.
pub fn check_blackhole_prefix(&self, prefix: &IpPrefix) -> Result<(), Rejection> {
if prefix.length() != prefix.address_family().bits() {
return Err(Rejection::BlackholeNotHost);
}
if !self.accept_bogons && is_bogon_prefix(prefix) {
return Err(Rejection::BogonPrefix);
}
match &self.prefixes {
Some(allowed) if !allowed.covers(prefix) => Err(Rejection::PrefixNotAllowed),
Some(_) => Ok(()),
None if self.own_prefixes.covers(prefix) => Err(Rejection::OwnPrefix),
None => Ok(()),
}
}
/// Judges the next hop of routes of IP version `routes`.
///
/// It has to be an address packets can be sent to and not our own
@@ -715,9 +894,9 @@ fn is_listed_bogon(prefix: &IpPrefix) -> bool {
BOGONS.iter().any(|bogon| bogon.contains(prefix))
}
/// A list of prefixes a peer may announce. Each entry allows the prefix
/// itself and, if it says so, the more specific prefixes inside it up to a
/// length.
/// A set of prefixes: what a peer may announce, what is our own, what may
/// be announced to a peer. Each entry allows the prefix itself and, if it
/// says so, the more specific prefixes inside it up to a length.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct PrefixSet {
/// For each prefix listed, the longest prefix inside it that is
@@ -740,14 +919,19 @@ impl PrefixSet {
}
}
pub fn len(&self) -> usize {
self.entries.len()
}
pub fn is_empty(&self) -> bool {
self.entries.is_empty()
}
/// Whether `prefix` is inside some entry, whatever length the entry
/// allows inside it.
pub fn covers(&self, prefix: &IpPrefix) -> bool {
self.lengths[Self::version_index(prefix)]
.iter()
.take_while(|length| **length <= prefix.length())
.any(|length| self.entries.contains_key(&prefix.shortened(*length)))
}
/// Whether `prefix` is allowed by some entry.
pub fn allows(&self, prefix: &IpPrefix) -> bool {
// An entry that covers it is the prefix cut to the entry's length,
@@ -770,104 +954,6 @@ impl PrefixSet {
}
}
/// Why a peer is being kept down.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HeldDown {
/// It went over its limit of routes.
OverLimit,
/// Its sessions keep coming up and failing.
Flapping,
}
impl HeldDown {
pub fn as_str(self) -> &'static str {
match self {
Self::OverLimit => "held down after going over its limit of routes",
Self::Flapping => "held down because its sessions keep failing",
}
}
}
/// A session that fails sooner than this after coming up counts as a
/// flap.
const FLAP_WITHIN: Duration = Duration::from_secs(60);
/// This many flaps in a row are let pass: a router being restarted, a
/// cable put back.
const FLAPS_ALLOWED: u32 = 2;
/// How long a peer is kept down after the first flap too many; twice as
/// long after each one more, up to `FLAP_HOLD_MAX`.
const FLAP_HOLD: Duration = Duration::from_secs(1);
const FLAP_HOLD_MAX: Duration = Duration::from_secs(300);
/// Peers that are being kept down, and until when: those that went over
/// their limit, and those whose sessions keep failing. Shared by the
/// sessions, which put peers here, and by the dialer and the listeners,
/// which leave them alone meanwhile.
///
/// Every session that comes up and goes costs a look at every route in
/// the table, and tells every other peer of the changes. A peer that
/// does so over and over, by fault or by design, is slowed down.
#[derive(Debug, Default)]
pub struct HoldDowns {
held: Mutex<HashMap<PeerAddress, (Instant, HeldDown)>>,
/// How many times in a row each peer's session has failed soon after
/// coming up.
flaps: Mutex<HashMap<PeerAddress, u32>>,
}
impl HoldDowns {
/// Keeps the peer at `address` down for `duration` from now.
pub fn hold(&self, address: PeerAddress, duration: Duration, why: HeldDown) {
let until = Instant::now() + duration;
let mut held = self.held.locked();
// Not for less long than it already is.
if held
.get(&address)
.is_none_or(|(already, _)| *already < until)
{
held.insert(address, (until, why));
}
}
/// How much longer the peer at `address` is kept down, if it is, and
/// why.
pub fn remaining(&self, address: PeerAddress) -> Option<(Duration, HeldDown)> {
let mut held = self.held.locked();
let (until, why) = *held.get(&address)?;
let remaining = until.saturating_duration_since(Instant::now());
if remaining.is_zero() {
held.remove(&address);
return None;
}
Some((remaining, why))
}
/// A session with the peer at `address` that had come up has ended,
/// `lasted` after it did; `failed` if not at our own wish. Returns for
/// how long the peer is kept down for it, if it is.
pub fn session_ended(
&self,
address: PeerAddress,
lasted: Duration,
failed: bool,
) -> Option<Duration> {
let mut flaps = self.flaps.locked();
if !failed || lasted >= FLAP_WITHIN {
flaps.remove(&address);
return None;
}
let count = flaps.entry(address).or_insert(0);
*count += 1;
let over = count.checked_sub(FLAPS_ALLOWED + 1)?;
let hold = FLAP_HOLD
.checked_mul(1u32.checked_shl(over).unwrap_or(u32::MAX))
.map_or(FLAP_HOLD_MAX, |hold| hold.min(FLAP_HOLD_MAX));
drop(flaps);
self.hold(address, hold, HeldDown::Flapping);
Some(hold)
}
}
#[cfg(test)]
mod tests {
use bgp_packet::message::{AsPathSegment, CommunitiesPathAttribute};
@@ -1024,6 +1110,49 @@ mod tests {
);
}
/// A route passed on inside our AS that says it started here, or that
/// it was passed on by our cluster already, has gone round in a circle
/// (RFC 4456 §8).
#[test]
fn a_reflected_route_that_comes_back_is_not_taken() {
let us = Ipv4Addr::new(10, 0, 0, 1);
let cluster = Ipv4Addr::new(10, 0, 0, 9);
let policy = ImportPolicy {
reflection: Some(ReflectorIds {
router_id: us,
cluster_id: cluster,
}),
..ImportPolicy::default()
};
let internal = RouteSource {
peer_asn: 1299,
ibgp: true,
..EXTERNAL
};
let reflected = |originator, clusters: &[Ipv4Addr]| {
let mut attributes = path(&[]);
attributes.push(PathAttribute::OriginatorId(originator));
attributes.push(PathAttribute::ClusterList(clusters.to_vec()));
attributes
};
let elsewhere = Ipv4Addr::new(10, 0, 0, 5);
let other_cluster = Ipv4Addr::new(10, 0, 0, 7);
assert_eq!(
policy
.check_path(&reflected(elsewhere, &[other_cluster]), &internal)
.map(drop),
Ok(())
);
assert_eq!(
policy.check_path(&reflected(us, &[other_cluster]), &internal),
Err(Rejection::ReflectionLoop)
);
assert_eq!(
policy.check_path(&reflected(elsewhere, &[other_cluster, cluster]), &internal),
Err(Rejection::ReflectionLoop)
);
}
#[test]
fn the_path_must_begin_with_the_peers_as_unless_it_is_a_route_server() {
let policy = ImportPolicy::default();
@@ -1184,6 +1313,42 @@ mod tests {
assert!(!Peer.may_be_sent_only_to_customer_routes());
}
/// AS_TRANS (23456) is a bogon where real numbers fit, and stands in
/// for one that does not on a session with two-octet AS numbers.
#[test]
fn as_trans_is_a_bogon_only_where_wide_numbers_fit() {
assert!(is_bogon_asn(u32::from(AS_TRANS), true));
assert!(!is_bogon_asn(u32::from(AS_TRANS), false));
assert!(is_bogon_asn(0, false));
assert!(is_bogon_asn(64512, false));
assert!(!is_bogon_asn(3333, false));
}
/// Export limits are for networks above or beside us, and for one we
/// know nothing of; not for customers, route server clients or our own
/// routers, which are told of what they are to be.
#[test]
fn export_limits_are_for_those_above_or_beside_us() {
use PeerRole::*;
for role in [Provider, Peer, RouteServer] {
assert!(ExportLimits::apply_to(Some(role), false), "{role:?}");
}
assert!(ExportLimits::apply_to(None, false));
for role in [Customer, RouteServerClient] {
assert!(!ExportLimits::apply_to(Some(role), false), "{role:?}");
}
assert!(!ExportLimits::apply_to(None, true));
let limits = ExportLimits {
prefixes: None,
max_prefixes: Some(2),
};
assert_eq!(limits.exceeded_by(2), None);
assert_eq!(limits.exceeded_by(3), Some(2));
assert!(limits.allows(&IpPrefix::try_from("203.0.113.0/24").unwrap()));
assert_eq!(ExportLimits::default().exceeded_by(usize::MAX), None);
}
#[test]
fn the_path_is_read_with_its_sets() {
let mut out = vec![99];
@@ -1210,7 +1375,7 @@ mod tests {
assert!(out.is_empty());
}
/// The counts of refusals are kept by a reason's place in the list.
/// The counts of rejections are kept by a reason's place in the list.
#[test]
fn every_reason_is_at_its_own_place_in_the_list() {
for (at, rejection) in Rejection::ALL.iter().enumerate() {
@@ -1247,6 +1412,62 @@ mod tests {
assert_eq!(lenient.check_path(&with_set, &EXTERNAL), Ok(None));
}
/// With accept_blackholes, a request to drop traffic is taken for one
/// address inside what the peer may announce, however long a prefix
/// the policy takes otherwise; and for nothing else.
#[test]
fn a_blackhole_request_is_taken_for_one_address_where_allowed() {
let mut attributes = path(&[3333, 2121]);
attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(vec![
(0xFFFF, 0x029A),
])));
assert!(ImportPolicy::is_blackhole_request(&attributes));
assert!(!ImportPolicy::is_blackhole_request(&path(&[3333])));
let mut policy = ImportPolicy {
accept_blackholes: true,
..ImportPolicy::default()
};
// The path passes as any other's would.
assert_eq!(policy.check_path(&attributes, &EXTERNAL), Ok(Some(2121)));
// Without a prefix list: any one routable address but ours.
let check = |policy: &ImportPolicy, p: &str| policy.check_blackhole_prefix(&prefix(p));
assert_eq!(check(&policy, "193.0.0.9/32"), Ok(()));
assert_eq!(check(&policy, "2001:67c:2e8::9/128"), Ok(()));
assert_eq!(
check(&policy, "193.0.0.0/24"),
Err(Rejection::BlackholeNotHost)
);
assert_eq!(
check(&policy, "2001:67c:2e8::/64"),
Err(Rejection::BlackholeNotHost)
);
assert_eq!(
check(&policy, "10.0.0.1/32"),
Err(Rejection::BogonPrefix),
"a bogon is a bogon, one address or not"
);
let mut ours = PrefixSet::default();
ours.allow(prefix("193.0.0.0/21"), 24);
policy.own_prefixes = Arc::new(ours);
assert_eq!(check(&policy, "193.0.0.9/32"), Err(Rejection::OwnPrefix));
// With one: inside a listed prefix, whatever length the list
// allows inside it, and ours or not.
let mut allowed = PrefixSet::default();
allowed.allow(prefix("193.0.0.0/21"), 24);
policy.prefixes = Some(allowed);
assert_eq!(check(&policy, "193.0.0.9/32"), Ok(()));
assert_eq!(
check(&policy, "193.0.8.9/32"),
Err(Rejection::PrefixNotAllowed)
);
assert_eq!(
Rejection::BlackholeNotHost.as_str(),
"blackhole for more than one address"
);
}
/// A request to drop traffic is neither acted on nor taken as an
/// ordinary route, whose next hop would drop the traffic all the same.
/// It is told apart from a prefix that is merely too long, which is
@@ -1524,63 +1745,6 @@ mod tests {
);
}
#[test]
fn a_peer_is_held_down_for_as_long_as_was_said() {
let held = HoldDowns::default();
let address = PeerAddress::from("192.0.2.1".parse::<std::net::IpAddr>().unwrap());
let other = PeerAddress::from("192.0.2.2".parse::<std::net::IpAddr>().unwrap());
// Two failures soon after coming up are let pass; from the third
// the peer is kept down, twice as long each time, up to a limit.
let flaps = HoldDowns::default();
let soon = Duration::from_secs(5);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, true), Some(FLAP_HOLD));
assert_eq!(
flaps.session_ended(address, soon, true),
Some(2 * FLAP_HOLD)
);
assert_eq!(
flaps.remaining(address).map(|(_, why)| why),
Some(HeldDown::Flapping)
);
assert_eq!(flaps.remaining(other), None, "each peer for itself");
for _ in 0..40 {
flaps.session_ended(address, soon, true);
}
assert_eq!(
flaps.session_ended(address, soon, true),
Some(FLAP_HOLD_MAX)
);
// A session that lasted, or that we ended ourselves, starts the
// count over.
assert_eq!(flaps.session_ended(address, FLAP_WITHIN, true), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(flaps.session_ended(address, soon, false), None);
assert_eq!(flaps.session_ended(address, soon, true), None);
assert_eq!(held.remaining(address), None);
held.hold(address, Duration::from_secs(600), HeldDown::OverLimit);
let (remaining, why) = held.remaining(address).expect("held");
assert!(remaining > Duration::from_secs(590), "{remaining:?}");
assert_eq!(why, HeldDown::OverLimit);
assert_eq!(held.remaining(other), None);
// Not made shorter by a lesser reason.
held.hold(address, Duration::from_secs(1), HeldDown::Flapping);
let (remaining, why) = held.remaining(address).expect("held");
assert!(remaining > Duration::from_secs(590), "{remaining:?}");
assert_eq!(why, HeldDown::OverLimit);
held.hold(other, Duration::ZERO, HeldDown::OverLimit);
assert_eq!(held.remaining(other), None);
assert_eq!(
held.held.locked().len(),
1,
"a hold that is over is forgotten"
);
}
/// The quick look at a prefix's first octets never lets through what
/// the list has.
fn assert_quick_check_agrees(prefix: IpPrefix) {
crates/server/src/reach.rs
+114 -53
View File
@@ -9,10 +9,11 @@
//! aside like a route the RPKI finds invalid, and looked at again when
//! the interfaces change (see `crate::routes::Held`).
//!
//! Only next hops that are neighbors on one of the machine's interfaces
//! count as reachable: the way to any other is not looked up. Routes from
//! a peer in our own AS that did not set itself as next hop are therefore
//! held, where they used to be chosen and then not installed.
//! A next hop that is a neighbor on one of the machine's interfaces can
//! be reached, and so can one that the kernel's own routes know the way
//! to; see `bgp_fib::routes` for which routes count. Routes from a peer
//! in our own AS that did not set itself as next hop are taken if a static
//! route or an IGP's says how to reach it, and held otherwise.
use std::cell::RefCell;
use std::collections::HashMap;
@@ -20,9 +21,9 @@ use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use bgp_fib::InterfacesWatch;
use bgp_fib::interfaces::Interfaces;
use bgp_fib::TopologyWatch;
use bgp_fib::plan::{Neighbor, resolve, resolve_next_hop};
use bgp_fib::topology::Topology;
use bgp_packet::message::NlriNextHop;
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
@@ -38,19 +39,32 @@ use crate::session::SessionRegistry;
/// judged it by the interfaces as they were.
const SETTLE: Duration = Duration::from_millis(500);
/// The interfaces to judge one peer's next hops by.
/// The kernel's view of the machine to judge one peer's next hops by.
#[derive(Clone, Copy)]
pub(crate) struct NextHops<'a> {
pub(crate) interfaces: &'a Interfaces,
pub(crate) topology: &'a Topology,
/// The peer: where a link-local next hop is.
pub(crate) neighbor: Neighbor,
}
impl NextHops<'_> {
/// Whether the kernel can be given a route for `prefix` by way of
/// `next_hop`.
pub(crate) fn reaches(&self, prefix: &IpPrefix, next_hop: Option<&NlriNextHop>) -> bool {
resolve(self.interfaces, prefix, next_hop, Some(self.neighbor)).is_ok()
/// `next_hop`; or, for a route that asks for its packets to be
/// dropped (`blackhole`), as a route to nowhere.
pub(crate) fn reaches(
&self,
prefix: &IpPrefix,
next_hop: Option<&NlriNextHop>,
blackhole: bool,
) -> bool {
resolve(
self.topology,
prefix,
next_hop,
Some(self.neighbor),
blackhole,
)
.is_ok()
}
/// The two halves of [`NextHops::reaches`], for an UPDATE's worth of
@@ -58,52 +72,52 @@ impl NextHops<'_> {
/// is asked once, and whether a prefix is one the machine reaches by
/// itself, which is asked of each.
pub(crate) fn next_hop_is_near(&self, routes: IpVersion, next_hop: &NlriNextHop) -> bool {
resolve_next_hop(self.interfaces, routes, Some(next_hop), Some(self.neighbor)).is_ok()
resolve_next_hop(self.topology, routes, Some(next_hop), Some(self.neighbor)).is_ok()
}
pub(crate) fn is_own_network(&self, prefix: &IpPrefix) -> bool {
self.interfaces.has_network_around(prefix)
self.topology.interfaces.has_network_around(prefix)
}
}
/// A session's view of the interfaces: those in use, replaced when the
/// kernel says they have changed. Asking whether they have costs the load
/// of one number, which is done for every UPDATE.
/// A session's view of the topology: the one in use, replaced when the
/// kernel says it has changed. Asking whether it has costs the load of
/// one number, which is done for every UPDATE.
pub(crate) struct Reach {
watch: InterfacesWatch,
watch: TopologyWatch,
neighbor: Neighbor,
interfaces: Option<Arc<Interfaces>>,
topology: Option<Arc<Topology>>,
}
impl Reach {
pub(crate) fn new(watch: InterfacesWatch, neighbor: Neighbor) -> Self {
let interfaces = watch.borrow().clone();
pub(crate) fn new(watch: TopologyWatch, neighbor: Neighbor) -> Self {
let topology = watch.borrow().clone();
Self {
watch,
neighbor,
interfaces,
topology,
}
}
/// What to judge next hops by now; nothing until the kernel has said
/// what interfaces there are.
/// what interfaces and routes there are.
pub(crate) fn next_hops(&mut self) -> Option<NextHops<'_>> {
if self.watch.has_changed().unwrap_or(false) {
self.interfaces = self.watch.borrow_and_update().clone();
self.topology = self.watch.borrow_and_update().clone();
}
self.interfaces.as_deref().map(|interfaces| NextHops {
interfaces,
self.topology.as_deref().map(|topology| NextHops {
topology,
neighbor: self.neighbor,
})
}
}
/// Starts the task that, whenever the machine's interfaces change, finds
/// the routes in the table whose next hop can no longer be reached and has
/// their sessions take them out, and has every session look again at the
/// routes it holds.
/// Starts the task that, whenever the machine's interfaces or routes
/// change, finds the routes in the table whose next hop can no longer be
/// reached and has their sessions take them out, and has every session
/// look again at the routes it holds.
pub(crate) fn spawn_revalidator(
mut watch: InterfacesWatch,
mut watch: TopologyWatch,
rib: Arc<RibTable>,
registry: SessionRegistry,
) -> JoinHandle<()> {
@@ -112,7 +126,7 @@ pub(crate) fn spawn_revalidator(
// which the server notices for itself.
while watch.changed().await.is_ok() {
tokio::time::sleep(SETTLE).await;
let Some(interfaces) = watch.borrow_and_update().clone() else {
let Some(topology) = watch.borrow_and_update().clone() else {
continue;
};
let sessions = registry.list();
@@ -127,10 +141,9 @@ pub(crate) fn spawn_revalidator(
}
let table = Arc::clone(&rib);
// Looks at every route: not for a thread that sessions run on.
let found = tokio::task::spawn_blocking(move || {
now_unreachable(&table, &interfaces, &neighbors)
})
.await;
let found =
tokio::task::spawn_blocking(move || now_unreachable(&table, &topology, &neighbors))
.await;
let mut suspects = match found {
Ok(suspects) => suspects,
Err(e) => {
@@ -141,7 +154,7 @@ pub(crate) fn spawn_revalidator(
info!(
now_unreachable = suspects.values().map(Vec::len).sum::<usize>(),
sessions = sessions.len(),
"The machine's interfaces changed; looking at next hops again"
"The machine's interfaces or routes changed; looking at next hops again"
);
// Every session, also those with nothing found: each has its
// held routes to look at again.
@@ -162,11 +175,11 @@ pub(crate) fn neighbor_at(address: SocketAddr) -> Neighbor {
}
/// The routes in the table that the kernel could not be given, as the
/// interfaces are now, by peer. `neighbors` says, for each peer by its
/// machine is now, by peer. `neighbors` says, for each peer by its
/// number in the table, where it is.
fn now_unreachable(
rib: &RibTable,
interfaces: &Interfaces,
topology: &Topology,
neighbors: &[Option<Neighbor>],
) -> HashMap<PeerIndex, Vec<Nlri>> {
// There are as many next hops as neighbors, and a table's worth of
@@ -177,17 +190,23 @@ fn now_unreachable(
let Some(Some(neighbor)) = neighbors.get(route.peer.as_usize()) else {
return false;
};
if interfaces.has_network_around(&route.prefix) {
if topology.interfaces.has_network_around(&route.prefix) {
return true;
}
// A blackhole resolves whatever its next hop: nothing of it can
// have become unreachable.
if route.attrs.is_blackhole() {
return false;
}
let next_hop = route.attrs.next_hop();
let key = (next_hop, route.prefix.address_family(), route.peer);
!*resolved.borrow_mut().entry(key).or_insert_with(|| {
resolve(
interfaces,
topology,
&route.prefix,
next_hop.as_ref(),
Some(*neighbor),
false,
)
.is_ok()
})
@@ -206,15 +225,15 @@ fn now_unreachable(
mod tests {
use std::net::{IpAddr, Ipv4Addr};
use bgp_fib::netlink::{Address, Link, Message};
use bgp_fib::netlink::{Address, Link, Message, TABLE_MAIN};
use bgp_packet::message::{
AsPathAttribute, NextHopPathAttribute, OriginPathAttribute, PathAttribute, UpdateMessage,
};
use bgp_rib::table::PeerInfo;
use super::*;
use crate::policy::{ImportPolicy, Rejection, RouteSource};
use crate::refusals::RefusalLog;
use crate::policy::{ImportPolicy, PrefixSet, Rejection, RouteSource};
use crate::rejections::RejectionLog;
use crate::routes::{self, Held, Import};
use crate::rpki::Checks;
@@ -224,8 +243,8 @@ mod tests {
/// A machine with eth2 (192.0.2.1/24), and eth3 (198.51.100.1/24) if
/// `both`.
fn interfaces(both: bool) -> Interfaces {
let mut interfaces = Interfaces::default();
fn interfaces(both: bool) -> Topology {
let mut interfaces = Topology::new(TABLE_MAIN, false);
let all = [
(2, "192.0.2.1", "192.0.2.0/24"),
(3, "198.51.100.1", "198.51.100.0/24"),
@@ -261,7 +280,7 @@ mod tests {
struct Bed {
table: RibTable,
policy: ImportPolicy,
refusals: RefusalLog,
rejections: RejectionLog,
}
impl Bed {
@@ -269,7 +288,7 @@ mod tests {
Self {
table: RibTable::new(4),
policy: ImportPolicy::permissive(),
refusals: RefusalLog::default(),
rejections: RejectionLog::default(),
}
}
@@ -284,6 +303,7 @@ mod tests {
address: address.into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap(),
address,
@@ -297,7 +317,7 @@ mod tests {
fn with_import<T>(
&self,
peer: &mut Peer,
interfaces: &Interfaces,
interfaces: &Topology,
work: impl FnOnce(&RibTable, &Import<'_>, &mut Held) -> T,
) -> T {
let address = IpAddr::V4(peer.address);
@@ -316,19 +336,19 @@ mod tests {
policy: &self.policy,
checks: Checks {
next_hops: Some(NextHops {
interfaces,
topology: interfaces,
neighbor: address.into(),
}),
..Checks::NONE
},
refusals: &self.refusals.of_peer(address.into(), "test"),
rejections: &self.rejections.of_peer(address.into(), "test"),
};
work(&self.table, &import, &mut peer.held)
}
/// The peer announces `route` with a path of `hops` AS numbers and
/// itself as next hop.
fn announce(&self, peer: &mut Peer, interfaces: &Interfaces, route: &str, hops: usize) {
fn announce(&self, peer: &mut Peer, interfaces: &Topology, route: &str, hops: usize) {
let update = UpdateMessage {
withdrawn_routes: vec![],
path_attributes: vec![
@@ -343,7 +363,7 @@ mod tests {
});
}
fn revalidate(&self, peer: &mut Peer, interfaces: &Interfaces, suspects: &[Nlri]) {
fn revalidate(&self, peer: &mut Peer, interfaces: &Topology, suspects: &[Nlri]) {
self.with_import(peer, interfaces, |table, import, held| {
routes::revalidate(table, import, held, suspects)
});
@@ -354,6 +374,47 @@ mod tests {
}
}
/// A route for a network of this machine, or for part of one, is held
/// under a reason of its own, not as one whose next hop is out of
/// reach: the next hop is fine, the prefix is the trouble.
#[test]
fn a_route_for_a_network_of_this_machine_is_held_as_such() {
let bed = Bed::new();
let mut peer = bed.peer("192.0.2.2", 65001);
let interfaces = interfaces(false);
bed.announce(&mut peer, &interfaces, "192.0.2.0/24", 1);
bed.announce(&mut peer, &interfaces, "192.0.2.128/25", 1);
assert_eq!((bed.table.len(), peer.held.len()), (0, 2));
let refused = bed.rejections.peers();
assert_eq!(refused[0].1.by_reason(), [(Rejection::OwnNetwork, 2)]);
}
/// A peer in our own AS that originates a prefix we originate too (an
/// anycast prefix) has its route for it ignored: not taken, not held,
/// not counted as refused. What it had announced before goes.
#[test]
fn a_prefix_originated_here_too_is_ignored() {
let mut bed = Bed::new();
let mut peer = bed.peer("192.0.2.2", 65001);
let interfaces = interfaces(false);
bed.announce(&mut peer, &interfaces, "203.0.113.0/24", 1);
assert_eq!(bed.table.len(), 1);
let mut originated = PrefixSet::default();
originated.allow(prefix("203.0.113.0/24"), 24);
bed.policy.originated = Arc::new(originated);
bed.announce(&mut peer, &interfaces, "203.0.113.0/24", 1);
assert_eq!((bed.table.len(), peer.held.len()), (0, 0));
assert!(
bed.rejections.peers().is_empty(),
"nothing counted as refused"
);
// Only the prefix itself: what is inside it is judged as usual.
bed.announce(&mut peer, &interfaces, "203.0.113.0/25", 1);
assert_eq!(bed.table.len(), 1);
}
/// RFC 4271 §9.1.2.1: a route whose next hop cannot be reached is not
/// chosen, however good it looks; and it is when it can be.
#[test]
@@ -370,7 +431,7 @@ mod tests {
assert_eq!(bed.best(route), Some(near.index));
assert_eq!(bed.table.len(), 1, "the other is not in the table at all");
assert_eq!((near.held.len(), far.held.len()), (0, 1));
let refused = bed.refusals.peers();
let refused = bed.rejections.peers();
assert_eq!(refused.len(), 1);
assert_eq!(
refused[0].1.by_reason(),
crates/server/src/refusals.rs → crates/server/src/rejections.rs
+22 -22
View File
@@ -2,7 +2,7 @@
//!
//! A refused route is not stored, so without this there would be nothing
//! to show for it but a number. For each peer the reasons are counted and
//! the latest refusals are kept: enough to see what a peer is doing wrong,
//! the latest rejections are kept: enough to see what a peer is doing wrong,
//! or what in our policy is. They are kept by the peer's address and
//! outlive the session, since the session a peer was cut off on is the one
//! worth looking at.
@@ -25,7 +25,7 @@ use crate::sync::IgnorePoison;
/// How many of a peer's latest refused routes are kept.
pub const RECENT_REFUSALS: usize = 200;
/// How many peers refusals are kept for. More than any configuration has;
/// How many peers rejections are kept for. More than any configuration has;
/// it bounds what hosts that are let in without being configured (on a
/// test network) can make us keep.
const MAX_PEERS: usize = 8192;
@@ -36,7 +36,7 @@ const LOG_INTERVAL: Duration = Duration::from_secs(60);
/// A route a peer announced that its import policy refused.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RefusedRoute {
pub struct RejectedRoute {
pub prefix: IpPrefix,
/// The peer's identifier for the route; zero without ADD-PATH.
pub path_id: u32,
@@ -55,15 +55,15 @@ pub struct RefusedRoute {
pub at: SystemTime,
}
/// What the routes of one UPDATE came with, for [`PeerRefusals::note`].
/// What the routes of one UPDATE came with, for [`PeerRejections::note`].
pub(crate) struct Came<'a> {
pub(crate) as_path: &'a Arc<[u32]>,
pub(crate) attributes: &'a Arc<[PathAttribute]>,
pub(crate) next_hop: Option<&'a NlriNextHop>,
}
/// The refusals of one peer.
pub struct PeerRefusals {
/// The rejections of one peer.
pub struct PeerRejections {
/// What the peer is called in the log and in metrics: its configured
/// name, or its address.
label: String,
@@ -74,13 +74,13 @@ pub struct PeerRefusals {
struct State {
/// The latest refused routes, oldest first.
recent: VecDeque<RefusedRoute>,
recent: VecDeque<RejectedRoute>,
logged: Option<Instant>,
/// How many were refused since the log last said so.
unlogged: u64,
}
impl PeerRefusals {
impl PeerRejections {
fn new(label: &str) -> Self {
Self {
label: label.to_string(),
@@ -113,7 +113,7 @@ impl PeerRefusals {
if state.recent.len() == RECENT_REFUSALS {
state.recent.pop_front();
}
state.recent.push_back(RefusedRoute {
state.recent.push_back(RejectedRoute {
prefix: route.prefix,
path_id: route.path_id,
reason,
@@ -139,7 +139,7 @@ impl PeerRefusals {
reason = reason.as_str(),
as_path = ?&came.as_path[..],
others_since_last_report = state.unlogged + count - 1,
"Refused a route"
"Rejected a route"
);
state.logged = Some(Instant::now());
state.unlogged = 0;
@@ -172,25 +172,25 @@ impl PeerRefusals {
}
/// The latest refused routes, newest first.
pub fn recent(&self) -> Vec<RefusedRoute> {
pub fn recent(&self) -> Vec<RejectedRoute> {
self.state.locked().recent.iter().rev().cloned().collect()
}
}
/// The refusals of every peer, by address. Cheap to clone; clones share
/// The rejections of every peer, by address. Cheap to clone; clones share
/// one record.
#[derive(Clone, Default)]
pub struct RefusalLog(Arc<Mutex<HashMap<PeerAddress, Arc<PeerRefusals>>>>);
pub struct RejectionLog(Arc<Mutex<HashMap<PeerAddress, Arc<PeerRejections>>>>);
impl RefusalLog {
impl RejectionLog {
/// The record for the peer at `address`, made if there was none.
/// `label` is what to call the peer; see [`PeerRefusals::label`].
pub(crate) fn of_peer(&self, address: PeerAddress, label: &str) -> Arc<PeerRefusals> {
/// `label` is what to call the peer; see [`PeerRejections::label`].
pub(crate) fn of_peer(&self, address: PeerAddress, label: &str) -> Arc<PeerRejections> {
let mut peers = self.0.locked();
if let Some(existing) = peers.get(&address) {
return Arc::clone(existing);
}
let record = Arc::new(PeerRefusals::new(label));
let record = Arc::new(PeerRejections::new(label));
// Past the bound the record is still made, for the session to
// count in, but it is not kept after.
if peers.len() < MAX_PEERS {
@@ -200,7 +200,7 @@ impl RefusalLog {
}
/// Every peer that has had a route refused, with its record.
pub fn peers(&self) -> Vec<(PeerAddress, Arc<PeerRefusals>)> {
pub fn peers(&self) -> Vec<(PeerAddress, Arc<PeerRejections>)> {
self.0
.locked()
.iter()
@@ -225,9 +225,9 @@ mod tests {
#[test]
fn refusals_are_counted_by_reason_and_the_latest_are_kept() {
let log = RefusalLog::default();
let log = RejectionLog::default();
let address = PeerAddress::from("192.0.2.1".parse::<std::net::IpAddr>().unwrap());
let record = log.of_peer(address, "refusals-test-1");
let record = log.of_peer(address, "rejections-test-1");
assert!(log.peers().is_empty(), "nothing refused yet");
let path: Arc<[u32]> = vec![3333, 64512].into();
@@ -269,7 +269,7 @@ mod tests {
// The peer's next session finds the same record.
let again = log.of_peer(address, "another-name");
assert_eq!(again.total(), 3);
assert_eq!(again.label(), "refusals-test-1");
assert_eq!(again.label(), "rejections-test-1");
assert_eq!(log.peers().len(), 1);
}
@@ -277,7 +277,7 @@ mod tests {
/// amount of memory.
#[test]
fn only_so_many_refused_routes_are_kept() {
let record = PeerRefusals::new("refusals-test-2");
let record = PeerRejections::new("rejections-test-2");
let path: Arc<[u32]> = vec![3333].into();
let attributes: Arc<[PathAttribute]> = Vec::new().into();
let came = Came {
crates/server/src/restart.rs
+117 -15
View File
@@ -28,8 +28,10 @@ use bgp_packet::message::NotificationMessage;
use bgp_packet::open::{Capability, GracefulRestartEntry, OpenMessage};
use bgp_primitives::constants::{AfiSafi, IpVersion};
use bgp_rib::table::{PeerIndex, RibTable};
use tracing::{error, info};
use crate::connection::off_thread;
use tokio::sync::watch;
use tracing::info;
use crate::address::PeerAddress;
use crate::error::SessionError;
@@ -343,20 +345,21 @@ impl Restarts {
let Some(kept) = kept else {
return;
};
let rib = Arc::clone(&self.rib);
let outcome = tokio::task::spawn_blocking(move || {
let outcome = rib.remove_peer(kept.rib_peer);
rib.attrs().sweep();
outcome
})
.await
.unwrap_or_default();
info!(
peer = %peer.0,
peer_asn = peer.1,
removed_routes = outcome.removed,
"The peer did not come back in time; removed the routes kept while it restarted"
);
let rib_peer = kept.rib_peer;
match off_thread(&self.rib, move |table| table.remove_peer(rib_peer)).await {
Ok(outcome) => info!(
peer = %peer.0,
peer_asn = peer.1,
removed_routes = outcome.removed,
"The peer did not come back in time; removed the routes kept while it restarted"
),
Err(e) => error!(
peer = %peer.0,
peer_asn = peer.1,
err = %e,
"The peer did not come back in time, and removing the routes kept for it failed"
),
}
}
/// The routes kept of `peer`, for its new session to take over.
@@ -445,3 +448,102 @@ pub(crate) fn ends_in_restart(result: &Result<(), SessionError>, agreed: &Agreed
_ => false,
}
}
#[cfg(test)]
mod tests {
use std::io;
use super::*;
fn agreed(notification: bool) -> Agreed {
Agreed {
restart_time: 120,
restarted: false,
notification,
families: FamilySet::supported(),
forwarding: FamilySet::default(),
}
}
/// Which ends of a session count as the peer restarting (RFC 4724
/// §4.2, RFC 8538): a connection that drops always; a NOTIFICATION,
/// or an expired hold timer, only where both sides agreed that they
/// do, and never a Hard Reset.
#[test]
fn a_session_ends_in_a_restart_as_agreed() {
let dropped: Result<(), SessionError> =
Err(io::Error::from(io::ErrorKind::BrokenPipe).into());
let hold_timer = Err(SessionError::HoldTimerExpired);
let notified = Err(SessionError::NotifiedByPeer(BgpError::Cease(
CeaseSubcode::AdminReset,
)));
let hard_reset = Err(SessionError::NotifiedByPeer(BgpError::Cease(
CeaseSubcode::HardReset,
)));
let ours = Err(SessionError::PrefixLimitExceeded { limit: 1 });
for (result, plain, with_notifications) in [
(&dropped, true, true),
(&hold_timer, false, true),
(&notified, false, true),
(&hard_reset, false, false),
(&ours, false, false),
(&Ok(()), false, false),
] {
assert_eq!(ends_in_restart(result, &agreed(false)), plain, "{result:?}");
assert_eq!(
ends_in_restart(result, &agreed(true)),
with_notifications,
"{result:?} with notifications"
);
}
}
/// A Hard Reset carries the NOTIFICATION it stands in for in its data,
/// and one that is a Hard Reset already is left alone.
#[test]
fn a_hard_reset_wraps_the_notification() {
let shutdown = NotificationMessage {
error: BgpError::Cease(CeaseSubcode::AdminShutdown),
data: vec![7],
};
let reset = hard_reset(shutdown);
assert_eq!(reset.error, BgpError::Cease(CeaseSubcode::HardReset));
assert_eq!(reset.data, [6, 2, 7]);
assert_eq!(hard_reset(reset.clone()), reset);
}
/// The configuration's graceful restart settings: what is given
/// overrides the default, a restart time has to fit its twelve bits,
/// and `disabled` is the opposite of `enabled`.
#[test]
fn graceful_restart_settings_override_the_defaults() {
let proto = |text: &str| {
protobuf::text_format::parse_from_str::<crate::proto::config::GracefulRestart>(text)
.unwrap()
};
let defaults = GracefulRestartConfig::default();
let same = defaults.overridden_by(&proto("")).unwrap();
assert_eq!(same, defaults);
let changed = defaults
.overridden_by(&proto(
"disabled: true restart_time_seconds: 30 stale_routes_seconds: 10",
))
.unwrap();
assert!(!changed.enabled);
assert_eq!(changed.restart_time, 30);
assert_eq!(changed.stale_routes, Duration::from_secs(10));
assert_eq!(
defaults
.overridden_by(&proto("restart_time_seconds: 4095"))
.unwrap()
.restart_time,
MAX_RESTART_TIME
);
assert_eq!(
defaults
.overridden_by(&proto("restart_time_seconds: 4096"))
.unwrap_err(),
crate::error::GracefulRestartProblem::RestartTime(4096)
);
}
}
crates/server/src/routes.rs
+147 -44
View File
@@ -8,13 +8,13 @@ use std::sync::Arc;
use bgp_packet::message::{NlriNextHop, PathAttribute, UpdateMessage, only_to_customer_wire};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::Nlri;
use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
use bgp_primitives::parser::EncodeError;
use bgp_rib::table::{AttrSet, Outcome, PeerIndex, RibTable};
use bgp_rpki::{PathValidity, Validity};
use crate::policy::{self, ImportPolicy, Rejection, RouteSource};
use crate::refusals::{Came, PeerRefusals};
use crate::rejections::{Came, PeerRejections};
use crate::rpki::Checks;
/// The most routes of one peer that are held aside as invalid. Far more
@@ -40,7 +40,7 @@ pub(crate) struct Import<'a> {
/// RPKI.
pub(crate) checks: Checks<'a>,
/// Where what is refused is noted.
pub(crate) refusals: &'a PeerRefusals,
pub(crate) rejections: &'a PeerRejections,
}
/// A peer's routes that are not in the routing table because the RPKI's
@@ -129,7 +129,10 @@ pub(crate) fn apply(
as_path: OnceCell::new(),
refused_attributes: OnceCell::new(),
mark: only_to_customer_mark(import, update),
blackhole: import.policy.accept_blackholes
&& ImportPolicy::is_blackhole_request(&update.path_attributes),
changed: OnceCell::new(),
rules_holding: OnceCell::new(),
};
applying.withdraw();
// Routes in the body have their next hop in the NEXT_HOP attribute,
@@ -175,6 +178,10 @@ struct Applying<'a> {
/// What the policy makes of the path, and the AS the routes originate
/// from if it passes.
path_verdict: Result<Option<u32>, Rejection>,
/// Whether the routes ask for their traffic to be dropped, and the
/// policy takes such requests: then the next hop is nothing to them,
/// and the prefix is judged as a request's.
blackhole: bool,
/// Whether the ASPAs find the path invalid: asked only if a route
/// gets as far as that mattering.
path_invalid: OnceCell<bool>,
@@ -187,6 +194,10 @@ struct Applying<'a> {
/// The UPDATE's attributes as the policy changes them, if it does:
/// made when the first route is taken.
changed: OnceCell<Vec<PathAttribute>>,
/// For each of the policy's rules, whether it holds for the UPDATE's
/// routes as far as their communities go: asked once, if there are
/// rules.
rules_holding: OnceCell<Vec<bool>>,
}
impl Applying<'_> {
@@ -230,16 +241,25 @@ impl Applying<'_> {
return Ok(());
}
};
// Asked once for all of them: they have the one next hop.
let near = self
.import
.checks
.next_hops
.is_none_or(|next_hops| next_hops.next_hop_is_near(family, &used_next_hop));
// Asked once for all of them: they have the one next hop. For a
// blackhole it is not asked at all: the packets go nowhere.
let near = self.blackhole
|| self
.import
.checks
.next_hops
.is_none_or(|next_hops| next_hops.next_hop_is_near(family, &used_next_hop));
// Nearly always every route is taken, and then nothing is copied.
let mut accepted = Cow::Borrowed(nlri);
let mut invalid = Vec::new();
for (at, route) in nlri.iter().enumerate() {
if self.import.policy.originated.allows(&route.prefix) {
if let Cow::Borrowed(_) = accepted {
accepted = Cow::Owned(nlri[..at].to_vec());
}
self.ignore(route);
continue;
}
match self.judge(route, origin, near) {
Ok(()) => {
if let Cow::Owned(accepted) = &mut accepted {
@@ -260,16 +280,85 @@ impl Applying<'_> {
if accepted.is_empty() && invalid.is_empty() {
return Ok(());
}
let attrs = self.attributes(next_hop)?;
for route in invalid {
self.held.hold(route, &attrs);
if self.import.policy.rules.is_empty() {
let attrs = self.attributes(next_hop, None)?;
for route in invalid {
self.held.hold(route, &attrs);
}
if !accepted.is_empty() {
self.applied.outcome += self.table.announce(self.import.peer, &attrs, &accepted);
}
return Ok(());
}
if !accepted.is_empty() {
self.applied.outcome += self.table.announce(self.import.peer, &attrs, &accepted);
// A rule may hold for some of the routes and not for others: they
// go in with an attribute set for each rule that holds, and one
// for those for which none does.
let mut by_rule: Vec<(Option<usize>, Vec<Nlri>, Vec<Nlri>)> = Vec::new();
let routes = accepted.iter().map(|route| (route, true));
for (route, taken) in routes.chain(invalid.iter().map(|route| (route, false))) {
let rule = self.rule_for(&route.prefix);
let at = match by_rule.iter().position(|(of, ..)| *of == rule) {
Some(at) => at,
None => {
by_rule.push((rule, Vec::new(), Vec::new()));
by_rule.len() - 1
}
};
if taken {
by_rule[at].1.push(*route);
} else {
by_rule[at].2.push(*route);
}
}
for (rule, taken, held) in by_rule {
let attrs = self.attributes(next_hop, rule)?;
for route in held {
self.held.hold(route, &attrs);
}
if !taken.is_empty() {
self.applied.outcome += self.table.announce(self.import.peer, &attrs, &taken);
}
}
Ok(())
}
/// The first of the policy's rules that holds for the route for
/// `prefix`, if one does.
fn rule_for(&self, prefix: &IpPrefix) -> Option<usize> {
let rules = &self.import.policy.rules;
let holding = self.rules_holding.get_or_init(|| {
// Rules look at the route as the policy keeps it: with the
// communities the session puts on.
let kept = self.changed();
rules.iter().map(|rule| rule.holds_for(kept)).collect()
});
rules
.iter()
.zip(holding)
.position(|(rule, holds)| *holds && rule.holds_for_prefix(prefix))
}
/// The UPDATE's attributes as its routes are kept, but for what a
/// rule does: as the policy changes them, without the prefixes, and
/// marked "only to customer" if they are to be.
fn changed(&self) -> &Vec<PathAttribute> {
self.changed.get_or_init(|| {
let mut attributes = self.update.path_attributes.clone();
// Not the route's own: they hold the prefixes.
attributes.retain(|attribute| {
!matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
)
});
self.import.policy.change(&mut attributes);
if let Some(marked_by) = self.mark {
attributes.push(PathAttribute::OnlyToCustomer(marked_by));
}
attributes
})
}
/// Judges what routes announced together have in common: the path,
/// and the next hop. `routes` is their IP version. Returns the AS they
/// originate from, if the path says, and their next hop.
@@ -293,9 +382,13 @@ impl Applying<'_> {
.copied()
.or_else(in_attributes)
.ok_or(Rejection::NextHopUnusable)?;
self.import
.policy
.check_next_hop(&next_hop, routes, &self.import.from)?;
// The traffic to a blackholed prefix goes nowhere: where the next
// hop would have taken it does not matter.
if !self.blackhole {
self.import
.policy
.check_next_hop(&next_hop, routes, &self.import.from)?;
}
Ok((origin, next_hop))
}
@@ -304,7 +397,11 @@ impl Applying<'_> {
/// comes before what may pass.
fn judge(&self, route: &Nlri, origin: Option<u32>, near: bool) -> Result<(), Rejection> {
let import = self.import;
import.policy.check_prefix(&route.prefix)?;
if self.blackhole {
import.policy.check_blackhole_prefix(&route.prefix)?;
} else {
import.policy.check_prefix(&route.prefix)?;
}
let origin_invalid = import
.checks
.vrps
@@ -318,17 +415,29 @@ impl Applying<'_> {
{
return Err(Rejection::AspaInvalid);
}
let reachable = near
&& !import
.checks
.next_hops
.is_some_and(|next_hops| next_hops.is_own_network(&route.prefix));
if !reachable {
if import
.checks
.next_hops
.is_some_and(|next_hops| next_hops.is_own_network(&route.prefix))
{
return Err(Rejection::OwnNetwork);
}
if !near {
return Err(Rejection::NextHopUnreachable);
}
Ok(())
}
/// Takes no notice of `route`, a prefix this router originates itself,
/// from a peer in our own AS (see `ImportPolicy::originated`): it is
/// not taken, and not counted as refused. What the peer had announced
/// for it before is taken out.
fn ignore(&mut self, route: &Nlri) {
self.applied.outcome += self
.table
.withdraw(self.import.peer, std::slice::from_ref(route));
}
/// Notes that `refused`, whose next hop is `next_hop` if they came in
/// MP_REACH_NLRI, were not taken, and takes out what the peer had
/// announced for them before.
@@ -340,34 +449,28 @@ impl Applying<'_> {
.get_or_init(|| attributes_of(self.update)),
next_hop,
};
self.import.refusals.note(refused, rejection, &came);
self.import.rejections.note(refused, rejection, &came);
self.applied.rejected += refused.len();
self.applied.outcome += self.table.withdraw(self.import.peer, refused);
}
/// The attribute set the UPDATE's routes go into the table with.
fn attributes(&self, next_hop: Option<&NlriNextHop>) -> Result<Arc<AttrSet>, EncodeError> {
fn attributes(
&self,
next_hop: Option<&NlriNextHop>,
rule: Option<usize>,
) -> Result<Arc<AttrSet>, EncodeError> {
let store = self.table.attrs();
if let Some(rule) = rule {
let mut attributes = self.changed().clone();
self.import.policy.rules[rule].apply(&mut attributes);
return store.intern(&attributes, next_hop);
}
// What the policy puts on the peer's routes (communities that say
// where they were learned, a preference) is the same for all of
// an UPDATE's routes, and worked out once for them.
if self.import.policy.changes_routes() {
let changed = self.changed.get_or_init(|| {
let mut attributes = self.update.path_attributes.clone();
// Not the route's own: they hold the prefixes.
attributes.retain(|attribute| {
!matches!(
attribute,
PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
)
});
self.import.policy.change(&mut attributes);
if let Some(marked_by) = self.mark {
attributes.push(PathAttribute::OnlyToCustomer(marked_by));
}
attributes
});
return store.intern(changed, next_hop);
if self.import.policy.changes(&self.update.path_attributes) {
return store.intern(self.changed(), next_hop);
}
let Some(marked_by) = self.mark else {
if let Some(set) = self
@@ -470,7 +573,7 @@ pub(crate) fn revalidate(
applied.rejected += 1;
route.attrs.as_path(&mut path);
let (attributes, next_hop) = route.attrs.attributes().unwrap_or_default();
import.refusals.note(
import.rejections.note(
std::slice::from_ref(suspect),
rejection,
&Came {
crates/server/src/rpki.rs
+7 -4
View File
@@ -158,10 +158,13 @@ impl<'a> Checks<'a> {
return Some(Rejection::AspaInvalid);
}
}
if let Some(next_hops) = self.next_hops
&& !next_hops.reaches(prefix, attrs.next_hop().as_ref())
{
return Some(Rejection::NextHopUnreachable);
if let Some(next_hops) = self.next_hops {
if next_hops.is_own_network(prefix) {
return Some(Rejection::OwnNetwork);
}
if !next_hops.reaches(prefix, attrs.next_hop().as_ref(), attrs.is_blackhole()) {
return Some(Rejection::NextHopUnreachable);
}
}
None
}
crates/server/src/server.rs
+164 -88
View File
@@ -21,18 +21,19 @@ use bgp_rpki::Rpki;
use crate::address::{PeerAddress, socket_text};
use crate::advertise::spawn_advertiser;
use crate::collision::{CollisionTable, Direction};
use crate::config::Config;
use crate::connection::{PeerSettings, ServerContext, run_session};
use crate::config::{Config, PeerConfig};
use crate::connection::{PeerSettings, ServerContext, off_thread, run_session};
use crate::dialer::spawn_dialers;
use crate::error::ServerError;
use crate::holddown::HoldDowns;
use crate::metrics::{CONNECTIONS_REFUSED, PEER_MAX_PREFIXES};
use crate::panic;
use crate::policy::HoldDowns;
use crate::policy::ExportLimits;
use crate::reach;
use crate::refusals::RefusalLog;
use crate::rejections::RejectionLog;
use crate::restart::{Leave, Restarts};
use crate::rpki::spawn_revalidator;
use crate::session::{ConfiguredPeer, SessionRegistry};
use crate::session::{ConfiguredPeer, DialState, SessionRegistry};
use crate::sync::IgnorePoison;
use crate::throttle::Throttle;
@@ -86,7 +87,7 @@ pub struct BgpServer {
/// Live view of established sessions.
pub registry: SessionRegistry,
/// What peers announced that was not taken.
pub refusals: RefusalLog,
pub rejections: RejectionLog,
/// The data for route origin validation, where that is configured.
pub rpki: Option<Rpki>,
/// The task that keeps that data up to date.
@@ -117,7 +118,7 @@ impl BgpServer {
for route in &config.originate {
local_rib.announce(route.clone()).await?;
}
let configured = config
let configured: Vec<ConfiguredPeer> = config
.peers
.iter()
.map(|p| ConfiguredPeer {
@@ -125,12 +126,19 @@ impl BgpServer {
endpoint: p.endpoint.clone(),
remote_asn: p.remote_asn,
role: p.role,
passive: p.passive,
expected_routes: p.expected_routes,
})
.collect();
// The client starts now: with luck the data is there before the
// first routes are.
let (rpki, rpki_client) = config.rpki.clone().map(Rpki::spawn).unzip();
let rib = Arc::new(RibTable::new(table_shards()));
// Sized for the largest table a peer is expected to send: the
// prefixes of the others are mostly the same ones.
if let Some(expected) = configured.iter().filter_map(|p| p.expected_routes).max() {
rib.expect_prefixes(expected as usize);
}
// Started now, so that a machine on which it cannot be done stops
// the daemon from starting. There is nothing to install yet.
let (kernel, kernel_task) = match config.kernel {
@@ -161,7 +169,7 @@ impl BgpServer {
rib,
local_rib,
registry: SessionRegistry::with_configured(configured),
refusals: RefusalLog::default(),
rejections: RejectionLog::default(),
config,
})
}
@@ -177,43 +185,9 @@ impl BgpServer {
/// Runs the server. Returns only if something it cannot do without has
/// failed, after closing its sessions.
pub async fn run(self) -> Result<(), ServerError> {
pub async fn run(mut self) -> Result<(), ServerError> {
crate::metrics::init();
info!(
router_id = %self.config.router_id,
local_asn = self.config.local_asn,
peers = self.config.peers.len(),
"BGP server started"
);
for peer in &self.config.peers {
if let Some(limit) = peer.import.max_prefixes {
PEER_MAX_PREFIXES
.with_label_values(&[&peer.name])
.set(limit.into());
} else {
warn!(
peer = %peer.name,
"No limit on the routes taken from this peer (import.max_prefixes)"
);
}
// Without a role there is no rule about which learned routes
// go to the peer: all of them do.
if peer.export.learned_routes
&& peer.role.is_none()
&& peer.remote_asn != self.config.local_asn
{
warn!(
peer = %peer.name,
"Every route learned from any peer is passed on to this one: it has no role \
(export.learned_routes without role)"
);
}
}
if self.config.accept_unconfigured_peers {
warn!("Taking sessions and routes from any host (accept_unconfigured_peers)");
}
log_startup(&self.config);
let context = Arc::new(ServerContext {
router_id: self.config.router_id,
local_asn: self.config.local_asn,
@@ -221,9 +195,9 @@ impl BgpServer {
registry: self.registry.clone(),
collisions: CollisionTable::new(self.config.collision_detect_established_state),
hold_downs: HoldDowns::default(),
refusals: self.refusals.clone(),
rejections: self.rejections.clone(),
rpki: self.rpki.clone(),
interfaces: self.kernel.as_ref().map(Fib::interfaces),
topology: self.kernel.as_ref().map(Fib::topology),
restarts: Arc::clone(&self.restarts),
});
// What an earlier run left in the kernel and no peer has announced
@@ -236,39 +210,7 @@ impl BgpServer {
}
});
}
// The tasks the server cannot do without. None of them ends while
// the server runs. The advertiser comes first: it has to be
// listening before a dialer brings up the first session.
let mut tasks: FuturesUnordered<NamedTask> = FuturesUnordered::new();
tasks.push(named(
"advertiser",
spawn_advertiser(
self.local_rib.clone(),
Arc::clone(&self.rib),
self.registry.clone(),
self.config.local_asn,
self.config.control,
self.restarts.convergence().watch(),
),
));
tasks.push(named("sweeper", spawn_sweeper(Arc::clone(&self.rib))));
if let (Some(rpki), Some(client)) = (self.rpki.clone(), self.rpki_client) {
tasks.push(named("RPKI client", client));
tasks.push(named(
"revalidator",
spawn_revalidator(rpki, Arc::clone(&self.rib), self.registry.clone()),
));
}
if let Some(interfaces) = context.interfaces.clone() {
tasks.push(named(
"next hop revalidator",
reach::spawn_revalidator(interfaces, Arc::clone(&self.rib), self.registry.clone()),
));
}
for dialer in spawn_dialers(&self.config.peers, &context, self.config.connect_retry) {
tasks.push(named("dialer", dialer));
}
let mut tasks = self.spawn_tasks(&context);
let mut kernel_task = self.kernel_task;
let connections = ConnectionCounts::default();
@@ -314,6 +256,46 @@ impl BgpServer {
accepting.push(accept_one(listen_addr, listener));
}
}
/// Starts the tasks the server cannot do without. None of them ends
/// while the server runs. The advertiser comes first: it has to be
/// listening before a dialer brings up the first session.
fn spawn_tasks(&mut self, context: &Arc<ServerContext>) -> FuturesUnordered<NamedTask> {
let tasks = FuturesUnordered::new();
tasks.push(named(
"advertiser",
spawn_advertiser(
self.local_rib.clone(),
Arc::clone(&self.rib),
self.registry.clone(),
self.config.local_asn,
self.config.control,
self.config.cluster_id,
self.restarts.convergence().watch(),
),
));
tasks.push(named("sweeper", spawn_sweeper(Arc::clone(&self.rib))));
if let (Some(rpki), Some(client)) = (self.rpki.clone(), self.rpki_client.take()) {
tasks.push(named("RPKI client", client));
tasks.push(named(
"revalidator",
spawn_revalidator(rpki, Arc::clone(&self.rib), self.registry.clone()),
));
}
if let Some(topology) = context.topology.clone() {
tasks.push(named(
"next hop revalidator",
reach::spawn_revalidator(topology, Arc::clone(&self.rib), self.registry.clone()),
));
}
for peer in self.config.peers.iter().filter(|peer| peer.passive) {
info!(peer = %peer.name, endpoint = %peer.endpoint, "Waiting for the peer to connect (passive)");
}
for dialer in spawn_dialers(&self.config.peers, context, self.config.connect_retry) {
tasks.push(named("dialer", dialer));
}
tasks
}
}
/// Resolves when the task that keeps the kernel's routes has ended, which
@@ -367,22 +349,99 @@ pub async fn leave_for_restart(registry: &SessionRegistry, kernel: Option<&Fib>)
{
warn!("Letting go of the kernel's routes took too long");
}
let left = registry.leave(Leave::Restart, SHUTDOWN_GRACE).await;
if left > 0 {
warn!(sessions = left, "Some sessions did not close in time");
}
leave_sessions(registry, Leave::Restart).await;
}
/// Closes every session politely and waits a moment for them to go: each
/// peer is told we are going (Cease / Administrative Shutdown), so that it
/// stops using our routes now and not when its hold timer expires.
pub async fn close_sessions(registry: &SessionRegistry) {
let left = registry.shutdown(SHUTDOWN_GRACE).await;
leave_sessions(registry, Leave::Stop).await;
}
/// Closes every session as `leave` says, and waits a moment for them to
/// go.
async fn leave_sessions(registry: &SessionRegistry, leave: Leave) {
let left = registry.leave(leave, SHUTDOWN_GRACE).await;
if left > 0 {
warn!(sessions = left, "Some sessions did not close in time");
}
}
/// Says what the configuration is, and what it leaves undone.
fn log_startup(config: &Config) {
info!(
router_id = %config.router_id,
local_asn = config.local_asn,
peers = config.peers.len(),
"BGP server started"
);
for peer in &config.peers {
if let Some(limit) = peer.import.max_prefixes {
PEER_MAX_PREFIXES
.with_label_values(&[&peer.name])
.set(limit.into());
} else {
warn!(
peer = %peer.name,
"No limit on the routes taken from this peer (import.max_prefixes)"
);
}
}
warn_about_export_limits(config);
if config.accept_unconfigured_peers {
warn!("Taking sessions and routes from any host (accept_unconfigured_peers)");
}
}
/// Says what the export limits leave undone: the peers they are for that
/// have none, and the routes of our own that they leave out, which is
/// likely a mistake in one or the other.
fn warn_about_export_limits(config: &Config) {
let limited: Vec<&PeerConfig> = config
.peers
.iter()
.filter(|peer| ExportLimits::apply_to(peer.role, peer.remote_asn == config.local_asn))
.collect();
let unlimited: Vec<&str> = limited
.iter()
.filter(|peer| peer.export_limits.is_none())
.map(|peer| peer.name.as_str())
.collect();
if !unlimited.is_empty() {
warn!(
peers = unlimited.join(", "),
"Nothing limits what these peers are told (export_limits)"
);
}
// Each distinct set of limits once, with the peers that have it.
let mut distinct: Vec<(&Arc<ExportLimits>, Vec<&str>)> = Vec::new();
for peer in &limited {
let Some(limits) = &peer.export_limits else {
continue;
};
match distinct
.iter_mut()
.find(|(seen, _)| Arc::ptr_eq(seen, limits))
{
Some((_, peers)) => peers.push(&peer.name),
None => distinct.push((limits, vec![&peer.name])),
}
}
for route in &config.originate {
for (limits, peers) in &distinct {
if !limits.allows(&route.prefix) {
warn!(
prefix = %route.prefix,
peers = peers.join(", "),
"An originated route is not announced to these peers: export_limits.prefixes \
leaves it out"
);
}
}
}
}
/// Opens a listening socket on `addr`.
fn listen(addr: SocketAddr) -> io::Result<TcpListener> {
let socket = match addr {
@@ -420,6 +479,7 @@ fn admit(
None if config.accept_unconfigured_peers => PeerSettings::unconfigured(
config.hold_time_secs,
Arc::clone(&config.export),
config.export_limits.clone(),
config.graceful_restart,
),
None => return Err(Refusal::NotAPeer),
@@ -437,10 +497,23 @@ async fn serve(
peer: PeerSettings,
_slot: ConnectionSlot,
) {
// Nobody dials a passive peer, so how its session ended is noted here.
let passive = if peer.passive {
peer.name.clone()
} else {
None
};
let registry = context.registry.clone();
let end = run_session(stream, peer_addr, Direction::Inbound, context, peer).await;
if let Err(e) = end.result {
if let Err(e) = &end.result {
warn!(peer_addr = %socket_text(peer_addr), err = %e, "BGP session terminated with error");
}
if let Some(name) = passive {
let state = DialState::Passive {
last_error: Some(end.reason()),
};
registry.set_dial_state(&name, state);
}
}
/// Why an incoming connection was closed without a session.
@@ -450,7 +523,9 @@ enum Refusal {
NotAPeer,
/// Its address has as many connections open as one may.
TooMany,
/// The peer went over its limit of routes and is being kept down.
/// The peer is being kept down: it went over its limit of routes, we
/// were to send it more than its export limit allows, or its sessions
/// keep failing.
HeldDown,
}
@@ -547,10 +622,11 @@ fn spawn_sweeper(rib: Arc<RibTable>) -> JoinHandle<()> {
if !piled_up && swept_at.elapsed() < SWEEP_INTERVAL {
continue;
}
let table = Arc::clone(&rib);
// Takes each of the store's locks in turn and looks at every
// set: not for a thread that sessions run on.
let _ = tokio::task::spawn_blocking(move || table.attrs().sweep()).await;
if let Err(e) = off_thread(&rib, |_| ()).await {
error!(err = %e, "Sweeping the attribute sets failed");
}
swept_at = Instant::now();
kept = rib.attrs().len();
}
crates/server/src/session.rs
+80 -9
View File
@@ -1,4 +1,7 @@
//! Registry of established BGP sessions.
//! The sessions as the rest of the server sees them: the registry of
//! those that are established, the configured peers and where the dialer
//! is with each, the events a session's coming and going raise, and the
//! word that the daemon is leaving.
//!
//! A session registers itself when it reaches Established and is removed when
//! its [`SessionGuard`] drops, so a session that errors out is always
@@ -6,7 +9,7 @@
//! the registry to enumerate live sessions and queue outbound messages.
use std::collections::HashMap;
use std::net::SocketAddr;
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, RwLock};
use std::time::{Duration, SystemTime};
@@ -18,10 +21,11 @@ use tokio::sync::{broadcast, mpsc, watch};
use crate::address::{Endpoint, PeerAddress, socket_text};
use crate::collision::Direction;
use crate::error::SessionError;
use crate::family::FamilySet;
use crate::fsm::Negotiated;
use crate::metrics::{PEER_UP, SESSIONS_ESTABLISHED};
use crate::policy::{ExportPolicy, PeerRole};
use crate::policy::{ExportLimits, ExportPolicy, PeerRole};
use crate::restart::{Agreed, Leave};
use crate::rpki::Subject;
use crate::sync::IgnorePoisonRw;
@@ -53,6 +57,12 @@ pub enum DialState {
/// How the last attempt or session ended.
last_error: String,
},
/// The peer is passive: not dialled, waiting for it to connect.
Passive {
/// How its last session, or the last connection from it, ended,
/// if one did.
last_error: Option<String>,
},
}
/// A peer from the configuration file, whether or not it is currently up.
@@ -63,6 +73,10 @@ pub struct ConfiguredPeer {
pub remote_asn: u32,
/// What the peer is to us, if that was said.
pub role: Option<PeerRole>,
/// Whether it is left to the peer to connect.
pub passive: bool,
/// About how many routes it sends when all is well, if that was said.
pub expected_routes: Option<u32>,
}
/// Counts kept by a session as it goes, for whoever wants to look.
@@ -79,6 +93,10 @@ pub struct SessionCounters {
/// Routes of the peer's kept from its last session, which it has not
/// sent again yet (graceful restart).
stale: AtomicU64,
/// Routes announced to the peer and not withdrawn since: what it
/// holds from us, every route counted where it is sent several for
/// a prefix.
advertised: AtomicU64,
}
impl SessionCounters {
@@ -98,6 +116,14 @@ impl SessionCounters {
self.held.store(held as u64, Ordering::Relaxed);
}
pub fn advertised(&self) -> u64 {
self.advertised.load(Ordering::Relaxed)
}
pub(crate) fn set_advertised(&self, advertised: usize) {
self.advertised.store(advertised as u64, Ordering::Relaxed);
}
pub fn stale(&self) -> u64 {
self.stale.load(Ordering::Relaxed)
}
@@ -165,7 +191,17 @@ pub struct SessionHandle {
pub counters: Arc<SessionCounters>,
/// What the peer is told, and how.
pub export: Arc<ExportPolicy>,
/// Limits on what the peer is told, where there are any for it.
pub export_limits: Option<Arc<ExportLimits>>,
/// The groups the session is in, for control communities.
pub groups: Arc<[u32]>,
/// Whether the peer is a route reflector client of ours (RFC 4456).
pub reflector_client: bool,
/// The peer's BGP Identifier.
pub peer_router_id: Ipv4Addr,
outbound: mpsc::Sender<Message>,
/// Where the session is asked to end, if it listens for that.
ending: Option<mpsc::UnboundedSender<SessionError>>,
/// Which of the RPKI's checks the peer's routes are subject to, if
/// any, and where the session is told to look at its routes again.
validation: Option<(Option<Subject>, mpsc::UnboundedSender<Vec<Nlri>>)>,
@@ -203,11 +239,34 @@ impl SessionHandle {
graceful_restart: negotiated.graceful_restart.clone(),
counters: Arc::default(),
export: Arc::default(),
export_limits: None,
groups: Arc::default(),
reflector_client: false,
peer_router_id: negotiated.peer_identifier,
outbound,
ending: None,
validation: None,
}
}
/// Has the session listen for being asked to end: what is asked with
/// [`SessionHandle::end`] arrives at the receiver returned.
pub(crate) fn listen_for_end(&mut self) -> mpsc::UnboundedReceiver<SessionError> {
let (tx, rx) = mpsc::unbounded_channel();
self.ending = Some(tx);
rx
}
/// Asks the session to end because of `error`: it tells the peer with
/// the NOTIFICATION for it, and closes. What was queued before may go
/// out first; the caller queues nothing more.
pub(crate) fn end(&self, error: SessionError) {
if let Some(ending) = &self.ending {
// Fails only if the session has ended already.
let _ = ending.send(error);
}
}
/// Marks the session's routes as subject to validation with the
/// RPKI's data. What is sent with [`SessionHandle::revalidate`]
/// arrives at the receiver returned.
@@ -237,6 +296,14 @@ impl SessionHandle {
/// Whether `other` is a handle to this same session, and not to another
/// session with the same peer.
/// What the peer goes by in the metrics: its configured name, or its
/// address.
pub fn metric_label(&self) -> String {
self.name
.clone()
.unwrap_or_else(|| crate::address::PeerAddress::from(self.peer.addr).to_string())
}
pub fn is_same_session(&self, other: &SessionHandle) -> bool {
self.outbound.same_channel(&other.outbound)
}
@@ -287,7 +354,14 @@ impl SessionRegistry {
let (events, _) = broadcast::channel(EVENT_CHANNEL_CAPACITY);
let dial_states = configured
.iter()
.map(|p| (p.name.clone(), DialState::Idle))
.map(|p| {
let state = if p.passive {
DialState::Passive { last_error: None }
} else {
DialState::Idle
};
(p.name.clone(), state)
})
.collect();
for peer in &configured {
PEER_UP.with_label_values(&[&peer.name]).set(0);
@@ -340,6 +414,7 @@ impl SessionRegistry {
if let Some(name) = &handle.name {
PEER_UP.with_label_values(&[name]).set(1);
}
// Fails only when nobody is listening, which is fine.
let _ = self.events.send(SessionEvent::Up(handle));
SessionGuard {
registry: self.clone(),
@@ -360,7 +435,7 @@ impl SessionRegistry {
}
}
/// True once [`shutdown`](Self::shutdown) has begun; dialers stop
/// True once [`leave`](Self::leave) has been called; dialers stop
/// reconnecting.
pub fn is_shutting_down(&self) -> bool {
self.shutting_down.load(Ordering::Relaxed)
@@ -370,10 +445,6 @@ impl SessionRegistry {
/// Administrative Shutdown (RFC 4486) so it can drop the session at once
/// instead of discovering a dead TCP connection. Waits up to `grace` for
/// the sessions to finish; returns how many were still open.
pub async fn shutdown(&self, grace: Duration) -> usize {
self.leave(Leave::Stop, grace).await
}
/// Closes every connection because the daemon is going, as `leave`
/// says: for good, or to restart (see [`Leave`]). Waits up to `grace`
/// for the sessions to finish; returns how many were still open.
crates/web/Cargo.toml
+1
View File
@@ -22,6 +22,7 @@ serde.workspace = true
tokio.workspace = true
tokio-stream.workspace = true
tracing.workspace = true
nix = { workspace = true, features = ["hostname"] }
[lints]
workspace = true
crates/web/src/api.rs
+549 -79
View File
@@ -1,20 +1,21 @@
//! JSON handlers and their view types. Everything here is read-only: the
//! daemon is configured through its configuration file.
use std::collections::{BTreeMap, BinaryHeap, HashMap};
use std::net::IpAddr;
use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet};
use std::net::{IpAddr, Ipv4Addr};
use std::time::{SystemTime, UNIX_EPOCH};
use axum::Json;
use axum::extract::{Query, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use bgp_packet::community;
use bgp_packet::message::{NlriNextHop, OriginPathAttribute, PathAttribute};
use bgp_packet::message::{
AsPathAttribute, ExtendedCommunity, NlriNextHop, OriginPathAttribute, PathAttribute,
};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, RibTable, Route, RouteRef};
use bgp_rib::table::{AttrSet, PeerIndex, RibTable, Route, RouteRef, SelectionKey};
use bgp_rib::{LocalRoute, RibStopped};
use bgp_rpki::{CacheState, Update};
use bgp_server::address::{PeerAddress, socket_text};
@@ -24,7 +25,7 @@ use bgp_server::rpki::PathCheck;
use bgp_server::session::{ConfiguredPeer, DialState, PeerId, SessionHandle, SessionRegistry};
use serde::{Deserialize, Serialize};
use crate::AppState;
use crate::{AppState, CommunityName};
// ---------------------------------------------------------------------------
// Views
@@ -50,6 +51,12 @@ pub struct StatusView {
pub received_routes: usize,
/// Prefixes with at least one such route, and so with a best route.
pub received_prefixes: usize,
/// Whether large communities of our AS ask for things here
/// (`control_communities`).
pub control_communities: bool,
/// Names for communities, as configured, for showing in place of the
/// numbers.
pub community_names: Vec<CommunityName>,
}
#[derive(Debug, Clone, Serialize)]
@@ -58,7 +65,8 @@ pub struct PeerView {
pub name: Option<String>,
pub peer_addr: String,
pub peer_asn: u32,
/// `Established`, `Connecting`, `Backoff` or `Idle`.
/// `Established`, `Connecting`, `Backoff`, `Idle`, or `Passive` for a
/// peer that is not dialled and has not connected.
pub state: String,
/// `inbound` or `outbound`: which side opened the live TCP connection.
pub direction: Option<String>,
@@ -81,16 +89,22 @@ pub struct PeerView {
/// How many routes the peer announced on this session that its import
/// policy refused.
pub rejected_routes: Option<u64>,
/// Routes announced to the peer and not withdrawn since.
pub advertised_routes: Option<u64>,
/// How many of the peer's routes are held out of the table as invalid
/// by the RPKI's data, to be taken if that changes.
pub held_routes: Option<u64>,
/// About how many routes the peer sends when all is well, from the
/// configuration (`expected_routes`), for `routes` to be compared with.
pub expected_routes: Option<u32>,
/// What the peer is to us, if configured: `provider`, `customer`,
/// `peer`, `route server` or `route server client`.
pub role: Option<&'static str>,
/// Whether the peer's OPEN said the same (RFC 9234). `false` for a
/// peer that does not say; one that says otherwise gets no session.
pub role_confirmed: Option<bool>,
/// Why the last connection attempt or session ended, while in Backoff.
/// Why the last connection attempt or session ended, while in Backoff;
/// for a passive peer, why its last session ended, if one did.
pub last_error: Option<String>,
pub retry_at_unix: Option<u64>,
/// Graceful restart (RFC 4724), if both sides offered it.
@@ -147,7 +161,9 @@ impl From<&SessionHandle> for PeerView {
add_path_send: Some(s.add_path_send.iter().map(|f| f.to_string()).collect()),
routes: Some(s.counters.routes()),
rejected_routes: Some(s.counters.rejected()),
advertised_routes: Some(s.counters.advertised()),
held_routes: Some(s.counters.held()),
expected_routes: None,
role: s.role.map(PeerRole::as_str),
role_confirmed: s.role.map(|_| s.role_confirmed),
last_error: None,
@@ -167,6 +183,7 @@ impl PeerView {
DialState::Backoff { until, last_error } => {
("Backoff", Some(last_error.clone()), Some(unix_secs(*until)))
}
DialState::Passive { last_error } => ("Passive", last_error.clone(), None),
};
Self {
name: Some(peer.name.clone()),
@@ -184,7 +201,9 @@ impl PeerView {
add_path_send: None,
routes: None,
rejected_routes: None,
advertised_routes: None,
held_routes: None,
expected_routes: peer.expected_routes,
role: peer.role.map(PeerRole::as_str),
role_confirmed: None,
last_error,
@@ -202,7 +221,7 @@ impl PeerView {
pub fn peer_views(registry: &SessionRegistry, restarts: &Restarts) -> Vec<PeerView> {
let kept = restarts.kept();
let sessions = registry.list();
let mut claimed = std::collections::HashSet::new();
let mut claimed = HashSet::new();
let mut views: Vec<PeerView> = registry
.configured_peers()
.iter()
@@ -215,6 +234,7 @@ pub fn peer_views(registry: &SessionRegistry, restarts: &Restarts) -> Vec<PeerVi
claimed.insert(session.peer);
PeerView {
name: Some(peer.name.clone()),
expected_routes: peer.expected_routes,
..PeerView::from(session)
}
}
@@ -289,6 +309,124 @@ pub struct AttrSummary {
pub local_pref: Option<u32>,
pub communities: Vec<String>,
pub large_communities: Vec<String>,
/// The attributes not shown above, in the order the route has them,
/// as text: `aggregator`, `AS64500 192.0.2.1`.
pub other: Vec<OtherAttribute>,
}
/// An attribute of a route that the summary has no field for.
#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct OtherAttribute {
pub name: String,
pub value: String,
}
/// An AS path as it is usually written: AS_SETs in braces.
fn as_path_text(path: &AsPathAttribute) -> String {
let segments = path.segments.iter().map(|segment| {
let asns = segment
.path
.iter()
.map(u32::to_string)
.collect::<Vec<_>>()
.join(" ");
if segment.ordered {
asns
} else {
format!("{{{asns}}}")
}
});
segments.collect::<Vec<_>>().join(" ")
}
/// Octets as hexadecimal digits: `ab01`.
fn hex(octets: &[u8]) -> String {
octets.iter().map(|o| format!("{o:02x}")).collect()
}
/// An extended community (RFC 4360) as text. Route targets and route
/// origins, of the three kinds of global administrator, by name; the rest
/// as their eight octets.
fn extended_community_text(community: &ExtendedCommunity) -> String {
let b = community.0;
let kind = match b[1] {
0x02 => "rt",
0x03 => "ro",
_ => return format!("0x{}", hex(&b)),
};
let u16_at = |at: usize| u16::from_be_bytes([b[at], b[at + 1]]);
let u32_at = |at: usize| u32::from_be_bytes([b[at], b[at + 1], b[at + 2], b[at + 3]]);
match b[0] {
0x00 => format!("{kind} {}:{}", u16_at(2), u32_at(4)),
0x01 => format!("{kind} {}:{}", Ipv4Addr::from(u32_at(2)), u16_at(6)),
0x02 => format!("{kind} {}:{}", u32_at(2), u16_at(6)),
_ => format!("0x{}", hex(&b)),
}
}
/// The name and value of an attribute the summary has no field for, or
/// `None` for one it has.
fn other_attribute(attribute: &PathAttribute) -> Option<OtherAttribute> {
let other = |name: &str, value: String| {
Some(OtherAttribute {
name: name.to_string(),
value,
})
};
let list = |items: Vec<String>, separator| items.join(separator);
match attribute {
PathAttribute::Origin(_)
| PathAttribute::AsPath(_)
| PathAttribute::NextHop(_)
| PathAttribute::MultiExitDisc(_)
| PathAttribute::LocalPref(_)
| PathAttribute::Communities(_)
| PathAttribute::LargeCommunities(_)
| PathAttribute::MpReachNlri(_)
| PathAttribute::MpUnreachNlri(_) => None,
PathAttribute::AtomicAggregate(_) => other("atomic aggregate", "set".to_string()),
PathAttribute::Aggregator(a) => other("aggregator", format!("AS{} {}", a.asn, a.ip)),
PathAttribute::ExtendedCommunities(e) => other(
"extended communities",
list(
e.extended_communities
.iter()
.map(extended_community_text)
.collect(),
", ",
),
),
PathAttribute::As4Path(p) => other("AS4_PATH", as_path_text(p)),
PathAttribute::OnlyToCustomer(asn) => other("only to customer", format!("AS{asn}")),
PathAttribute::OriginatorId(id) => other("originator ID", id.to_string()),
PathAttribute::ClusterList(ids) => other(
"cluster list",
list(ids.iter().map(Ipv4Addr::to_string).collect(), " "),
),
PathAttribute::Unknown {
flags,
type_code,
payload,
} => {
// "optional transitive, 2 octets: ab01"
let flags: Vec<&str> = [
(flags.optional(), "optional"),
(flags.transitive(), "transitive"),
(flags.partial(), "partial"),
]
.into_iter()
.filter_map(|(set, word)| set.then_some(word))
.collect();
let mut value = format!("{} octets", payload.len());
if !flags.is_empty() {
value = format!("{}, {value}", flags.join(" "));
}
if !payload.is_empty() {
value = format!("{value}: {}", hex(payload));
}
other(&format!("attribute {type_code}"), value)
}
}
}
pub fn summarize(attributes: &[PathAttribute], next_hop: Option<&NlriNextHop>) -> AttrSummary {
@@ -321,7 +459,7 @@ pub fn summarize(attributes: &[PathAttribute], next_hop: Option<&NlriNextHop>) -
PathAttribute::LargeCommunities(l) => {
s.large_communities = l.communities.iter().map(community::format_large).collect()
}
_ => {}
other => s.other.extend(other_attribute(other)),
}
}
s
@@ -339,6 +477,9 @@ pub struct ReceivedRouteView {
pub path_id: u32,
/// Whether this is the route chosen for its prefix.
pub best: bool,
/// The route's place among the prefix's routes: 1 for the best, 2 for
/// the route that would be chosen were the best gone, and so on.
pub rank: u32,
/// Whether it is kept from a session of the peer's that ended, and not
/// sent again since (graceful restart).
pub stale: bool,
@@ -406,7 +547,7 @@ impl PeerNames {
/// A view of one route. `None` if the route's session has just ended:
/// the route is on its way out of the table.
pub(crate) fn view(&self, route: &Route) -> Option<ReceivedRouteView> {
pub(crate) fn view(&self, route: &Route, rank: u32) -> Option<ReceivedRouteView> {
let (peer, name, check) = self.peers.get(&route.peer)?;
let origin = route.attrs.key().origin();
// What a session stored was accepted by the decoder, and parses.
@@ -418,6 +559,7 @@ impl PeerNames {
prefix: route.prefix.to_string(),
path_id: route.path_id,
best: route.best,
rank,
stale: route.stale,
rpki: self
.rpki
@@ -474,6 +616,8 @@ pub async fn status(State(state): State<AppState>) -> Result<Json<StatusView>, U
local_routes: state.local_rib.len().await?,
received_routes: state.rib.len(),
received_prefixes: state.rib.prefix_count(),
control_communities: state.info.control_communities,
community_names: state.info.community_names.clone(),
}))
}
@@ -518,7 +662,8 @@ const DEFAULT_ROUTE_LIMIT: usize = 1000;
const MAX_ROUTE_LIMIT: usize = 100_000;
/// Which routes to return. At most one of `prefix`, `address` and `within`
/// says where to look; `peer` and `best` narrow what is found there.
/// says where to look; `peer`, `origin` and `best` narrow what is found
/// there.
#[derive(Debug, Default, Deserialize)]
pub struct RouteQuery {
/// About this many routes to a page. A page holds whole prefixes: it
@@ -540,11 +685,68 @@ pub struct RouteQuery {
within: Option<String>,
/// Only routes from this peer: its configured name, or its address.
peer: Option<String>,
/// Only routes originated by this AS: the last of the AS path, as a
/// number, with or without `AS` in front. Routes with an empty path
/// are this router's own AS's.
origin: Option<String>,
/// Only the best route of each prefix.
#[serde(default)]
best: bool,
}
/// What narrows the routes found where a query looks.
struct Narrowing {
/// The peers whose routes are wanted, if not every peer's.
peers: Option<Vec<PeerIndex>>,
/// The AS whose routes are wanted, if not every AS's.
origin: Option<u32>,
/// This router's AS: what an empty path originates from.
local_asn: u32,
best: bool,
}
impl Narrowing {
fn of(query: &RouteQuery, names: &PeerNames, local_asn: u32) -> Result<Self, QueryError> {
let origin = match &query.origin {
None => None,
Some(text) => Some(parse_asn(text)?),
};
Ok(Self {
peers: query.peer.as_deref().map(|peer| names.matching(peer)),
origin,
local_asn,
best: query.best,
})
}
fn wants(&self, peer: PeerIndex, best: bool, key: &SelectionKey) -> bool {
(!self.best || best)
&& self
.peers
.as_ref()
.is_none_or(|peers| peers.contains(&peer))
&& self.origin.is_none_or(|origin| {
let originated_by = if key.as_path_len == 0 {
Some(self.local_asn)
} else {
key.origin()
};
originated_by == Some(origin)
})
}
}
/// An AS number given in a query: `64496` or `AS64496`.
fn parse_asn(text: &str) -> Result<u32, QueryError> {
let digits = text
.strip_prefix("AS")
.or_else(|| text.strip_prefix("as"))
.unwrap_or(text);
digits
.parse()
.map_err(|_| QueryError::Bad(format!("{text:?} is not an AS number")))
}
/// Why a query was not answered.
pub enum QueryError {
/// The query makes no sense; the text says how.
@@ -571,15 +773,16 @@ enum Scope {
Within(IpPrefix),
}
/// A prefix given in a query.
fn parse_prefix(text: &str) -> Result<IpPrefix, QueryError> {
IpPrefix::try_from(text).map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
}
impl Scope {
fn of(query: &RouteQuery) -> Result<Self, QueryError> {
let prefix = |text: &str| {
IpPrefix::try_from(text)
.map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
};
match (&query.prefix, &query.address, &query.within) {
(None, None, None) => Ok(Self::All),
(Some(text), None, None) => Ok(Self::Exactly(prefix(text)?)),
(Some(text), None, None) => Ok(Self::Exactly(parse_prefix(text)?)),
(None, Some(text), None) => {
let address: IpAddr = text
.parse()
@@ -592,7 +795,7 @@ impl Scope {
host.map(Self::Covering)
.map_err(|e| QueryError::Bad(e.to_string()))
}
(None, None, Some(text)) => Ok(Self::Within(prefix(text)?)),
(None, None, Some(text)) => Ok(Self::Within(parse_prefix(text)?)),
_ => Err(QueryError::Bad(
"give at most one of prefix, address and within".into(),
)),
@@ -605,7 +808,7 @@ impl Scope {
pub struct RoutesPage {
/// The routes, prefix by prefix in address order (IPv4 first, a
/// shorter prefix before the longer ones at the same address), the
/// best route of each prefix first. A lookup by address has the most
/// routes of each prefix from the best down. A lookup by address has the most
/// specific prefix first instead.
pub routes: Vec<ReceivedRouteView>,
/// If there are routes after these: what to pass as `after` to get
@@ -626,14 +829,10 @@ enum Place {
impl Place {
fn of(query: &RouteQuery) -> Result<Self, QueryError> {
let prefix = |text: &str| {
IpPrefix::try_from(text)
.map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
};
match (&query.after, &query.before) {
(None, None) => Ok(Self::Start),
(Some(after), None) => Ok(Self::After(prefix(after)?)),
(None, Some(before)) => Ok(Self::Before(prefix(before)?)),
(Some(after), None) => Ok(Self::After(parse_prefix(after)?)),
(None, Some(before)) => Ok(Self::Before(parse_prefix(before)?)),
(Some(_), Some(_)) => Err(QueryError::Bad(
"give at most one of after and before".into(),
)),
@@ -754,37 +953,34 @@ fn find_routes(
names: &PeerNames,
scope: &Scope,
place: Place,
query: &RouteQuery,
narrowing: &Narrowing,
limit: usize,
) -> RoutesPage {
let peers = query.peer.as_deref().map(|peer| names.matching(peer));
let wanted = |peer: PeerIndex, best: bool| {
(!query.best || best) && peers.as_ref().is_none_or(|peers| peers.contains(&peer))
let wanted =
|peer: PeerIndex, best: bool, attrs: &AttrSet| narrowing.wants(peer, best, attrs.key());
// The routes of one prefix from the best down, each with its rank
// among all of them: that stays what it is whatever the query keeps.
let ranked = |prefix: &IpPrefix| -> Vec<(u32, Route)> {
(1..).zip(table.ranked_routes(prefix)).collect()
};
let views = |mut routes: Vec<Route>| -> Vec<ReceivedRouteView> {
routes.retain(|route| wanted(route.peer, route.best));
let views = |routes: Vec<(u32, Route)>| -> Vec<ReceivedRouteView> {
routes
.iter()
.filter_map(|route| names.view(route))
.filter(|(_, route)| wanted(route.peer, route.best, &route.attrs))
.filter_map(|(rank, route)| names.view(route, *rank))
.collect()
};
// The routes of each prefix come with the best one first.
let best_first = |routes: &mut Vec<Route>| routes.sort_by_key(|route| !route.best);
// A listing: a page of it.
let listed = |inside: Option<&IpPrefix>| {
let page = page_of_prefixes(table, place, limit, |route| {
inside.is_none_or(|outer| outer.contains(&route.prefix))
&& wanted(route.peer, route.best)
&& wanted(route.peer, route.best, route.attrs)
});
let routes = page
.prefixes
.iter()
.flat_map(|prefix| {
let mut routes = table.routes(prefix);
best_first(&mut routes);
views(routes)
})
.flat_map(|prefix| views(ranked(prefix)))
.collect();
let edge = |more: bool, prefix: Option<&IpPrefix>| {
prefix.filter(|_| more).map(IpPrefix::to_string)
@@ -796,29 +992,21 @@ fn find_routes(
}
};
// A lookup: all there is, which is little.
let looked_up = |routes: Vec<Route>| RoutesPage {
routes: views(routes),
let looked_up = |routes: Vec<ReceivedRouteView>| RoutesPage {
routes,
next_after: None,
prev_before: None,
};
match scope {
Scope::All => listed(None),
Scope::Within(outer) => listed(Some(outer)),
Scope::Exactly(prefix) => {
let mut routes = table.routes(prefix);
best_first(&mut routes);
looked_up(routes)
}
Scope::Exactly(prefix) => looked_up(views(ranked(prefix))),
// Place the longest prefix that could cover the address to the
// shortest: one lookup for each length.
Scope::Covering(host) => looked_up(
(0..=host.length())
.rev()
.flat_map(|length| {
let mut routes = table.routes(&host.shortened(length));
best_first(&mut routes);
routes
})
.flat_map(|length| views(ranked(&host.shortened(length))))
.collect(),
),
}
@@ -849,10 +1037,11 @@ pub async fn received_routes(
let scope = Scope::of(&query)?;
let place = Place::of(&query)?;
let names = PeerNames::of(&state);
let narrowing = Narrowing::of(&query, &names, state.info.local_asn)?;
// Looking through the table, copying, parsing and sorting many routes
// takes long enough to keep off the threads that sessions run on.
let page = tokio::task::spawn_blocking(move || {
find_routes(&state.rib, &names, &scope, place, &query, limit)
find_routes(&state.rib, &names, &scope, place, &narrowing, limit)
})
.await
// Only if making the views panicked.
@@ -861,8 +1050,8 @@ pub async fn received_routes(
}
/// Validation with the RPKI: where the data comes from and how fresh it
/// is.
#[derive(Debug, Clone, Serialize)]
/// is. The default is the view of a router that does not validate.
#[derive(Debug, Clone, Default, Serialize)]
pub struct RpkiView {
/// Whether validation is configured at all.
pub enabled: bool,
@@ -885,21 +1074,9 @@ pub struct RpkiView {
}
pub async fn rpki(State(state): State<AppState>) -> Json<RpkiView> {
// Validation not configured: nothing to say, and `enabled` false.
let Some(status) = state.rpki.as_ref().map(|rpki| rpki.status()) else {
return Json(RpkiView {
enabled: false,
cache: None,
state: None,
session_id: None,
serial: None,
protocol_version: None,
vrps_ipv4: 0,
vrps_ipv6: 0,
aspas: 0,
updated_at_unix: None,
expires_at_unix: None,
last_error: None,
});
return Json(RpkiView::default());
};
Json(RpkiView {
enabled: true,
@@ -938,6 +1115,9 @@ pub struct KernelView {
/// How many of them are kept from before the daemon started though no
/// peer has announced them again; they go when `stale_seconds` is up.
pub kept: usize,
/// How many of them are blackhole routes, which drop the packets for
/// their prefix (RFC 7999).
pub blackholes: usize,
/// Prefixes whose best route is not in the kernel because its next hop
/// cannot be used, by why.
pub unresolved: BTreeMap<&'static str, usize>,
@@ -971,6 +1151,7 @@ pub async fn kernel(State(state): State<AppState>) -> Json<KernelView> {
synced: status.synced,
installed: status.installed,
kept: status.kept,
blackholes: status.blackholes,
unresolved: status.unresolved.into_iter().collect(),
refused: status.refused,
errors: status.errors,
@@ -994,7 +1175,7 @@ pub async fn kernel(State(state): State<AppState>) -> Json<KernelView> {
/// What one peer announced that was refused.
#[derive(Debug, Clone, Serialize)]
pub struct RefusalsView {
pub struct RejectionsView {
/// The peer's configured name, or its address if it has none.
pub peer: String,
pub peer_addr: String,
@@ -1003,11 +1184,11 @@ pub struct RefusalsView {
/// The same, by reason.
pub by_reason: BTreeMap<&'static str, u64>,
/// The latest of them, newest first.
pub recent: Vec<RefusedRouteView>,
pub recent: Vec<RejectedRouteView>,
}
#[derive(Debug, Clone, Serialize)]
pub struct RefusedRouteView {
pub struct RejectedRouteView {
pub prefix: String,
pub path_id: u32,
pub reason: &'static str,
@@ -1022,7 +1203,7 @@ pub struct RefusedRouteView {
}
#[derive(Debug, Default, Deserialize)]
pub struct RefusalQuery {
pub struct RejectionQuery {
/// Only this peer: its configured name, or its address.
peer: Option<String>,
}
@@ -1031,10 +1212,10 @@ pub struct RefusalQuery {
/// each peer that had any, how many and why, and the latest of them.
pub async fn rejected_routes(
State(state): State<AppState>,
Query(query): Query<RefusalQuery>,
) -> Json<Vec<RefusalsView>> {
let mut peers: Vec<RefusalsView> = state
.refusals
Query(query): Query<RejectionQuery>,
) -> Json<Vec<RejectionsView>> {
let mut peers: Vec<RejectionsView> = state
.rejections
.peers()
.into_iter()
.filter(|(address, record)| {
@@ -1043,7 +1224,7 @@ pub async fn rejected_routes(
.as_deref()
.is_none_or(|wanted| record.label() == wanted || address.is_called(wanted))
})
.map(|(address, record)| RefusalsView {
.map(|(address, record)| RejectionsView {
peer: record.label().to_string(),
peer_addr: address.to_string(),
total: record.total(),
@@ -1055,7 +1236,7 @@ pub async fn rejected_routes(
recent: record
.recent()
.into_iter()
.map(|route| RefusedRouteView {
.map(|route| RejectedRouteView {
prefix: route.prefix.to_string(),
path_id: route.path_id,
reason: route.reason.as_str(),
@@ -1096,6 +1277,7 @@ mod tests {
address: Ipv4Addr::new(192, 0, 2, n).into(),
interface: 0,
ibgp: false,
reflector_client: false,
})
.unwrap()
});
@@ -1235,6 +1417,73 @@ mod tests {
/// The place a page begins is a prefix, not a count: what comes and
/// goes elsewhere in the table between two pages moves nothing.
#[test]
fn a_lookup_by_origin_finds_the_routes_whose_path_ends_in_that_as() {
let prefixes = some_prefixes();
let (table, _) = table(&prefixes);
let narrowed = |origin: &str| Narrowing {
peers: None,
origin: Some(parse_asn(origin).ok().unwrap()),
local_asn: 65000,
best: false,
};
let found = |narrowing: &Narrowing| {
walk(&table, 1000, false, |route| {
narrowing.wants(route.peer, route.best, route.attrs.key())
})
.concat()
.len()
};
// The first peer's paths end in 65010, the second's in 65002.
assert_eq!(found(&narrowed("AS65010")), prefixes.len());
assert_eq!(found(&narrowed("65002")), prefixes.len().div_ceil(3));
// 65001 is on every path, but originates nothing.
assert_eq!(found(&narrowed("65001")), 0);
assert!(parse_asn("AS-65001").is_err());
}
#[test]
fn a_route_with_an_empty_path_is_originated_by_this_router_s_as() {
let table = RibTable::new(1);
let peer = table
.add_peer(PeerInfo {
asn: 65000,
router_id: Ipv4Addr::new(10, 0, 0, 9),
address: Ipv4Addr::new(192, 0, 2, 9).into(),
interface: 0,
ibgp: true,
reflector_client: false,
})
.unwrap();
let attrs = table
.attrs()
.intern(
&[
PathAttribute::Origin(OriginPathAttribute::IGP),
AsPathAttribute::from_asns(vec![]),
PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 9))),
],
None,
)
.unwrap();
table.announce(peer, &attrs, &[prefix("193.0.0.0/21")]);
let ours = |origin: u32| Narrowing {
peers: None,
origin: Some(origin),
local_asn: 65000,
best: false,
};
let found = |narrowing: &Narrowing| {
walk(&table, 10, false, |route| {
narrowing.wants(route.peer, route.best, route.attrs.key())
})
.concat()
.len()
};
assert_eq!(found(&ours(65000)), 1);
assert_eq!(found(&ours(65001)), 0);
}
#[test]
fn a_table_that_changes_between_pages_does_not_shift_them() {
let prefixes = some_prefixes();
@@ -1258,4 +1507,225 @@ mod tests {
assert_eq!(second.prefixes[0], in_order[at + 1]);
assert!(second.more_before);
}
/// The attributes without a field in the summary are listed as text,
/// in the order the route has them.
#[test]
fn other_attributes_as_text() {
use bgp_packet::message::{
AggregatorPathAttribute, AsPathSegment, AtomicAggregatePathAttribute,
ExtendedCommunitiesPathAttribute, PathAttributeFlags,
};
let attributes = [
PathAttribute::AsPath(AsPathAttribute { segments: vec![] }),
PathAttribute::AtomicAggregate(AtomicAggregatePathAttribute {}),
PathAttribute::Aggregator(AggregatorPathAttribute {
asn: 64500,
ip: Ipv4Addr::new(192, 0, 2, 1),
}),
PathAttribute::ExtendedCommunities(ExtendedCommunitiesPathAttribute {
extended_communities: vec![
ExtendedCommunity([0x00, 0x02, 0xfd, 0xe8, 0, 0, 0, 100]),
ExtendedCommunity([0x01, 0x03, 192, 0, 2, 1, 0, 7]),
ExtendedCommunity([0x02, 0x02, 0, 3, 0x0d, 0x44, 0, 5]),
ExtendedCommunity([0x03, 0x0c, 1, 2, 3, 4, 5, 6]),
],
}),
PathAttribute::As4Path(AsPathAttribute {
segments: vec![
AsPathSegment {
ordered: true,
path: vec![64500, 64501],
},
AsPathSegment {
ordered: false,
path: vec![64502, 64503],
},
],
}),
PathAttribute::OnlyToCustomer(64500),
PathAttribute::OriginatorId(Ipv4Addr::new(192, 0, 2, 9)),
PathAttribute::ClusterList(vec![
Ipv4Addr::new(10, 0, 0, 1),
Ipv4Addr::new(10, 0, 0, 2),
]),
PathAttribute::Unknown {
flags: PathAttributeFlags::new(true, true, false, false),
type_code: 99,
payload: vec![0xab, 0x01],
},
];
let other: Vec<(String, String)> = summarize(&attributes, None)
.other
.into_iter()
.map(|a| (a.name, a.value))
.collect();
let expected = [
("atomic aggregate", "set"),
("aggregator", "AS64500 192.0.2.1"),
(
"extended communities",
"rt 65000:100, ro 192.0.2.1:7, rt 200004:5, 0x030c010203040506",
),
("AS4_PATH", "64500 64501 {64502 64503}"),
("only to customer", "AS64500"),
("originator ID", "192.0.2.9"),
("cluster list", "10.0.0.1 10.0.0.2"),
("attribute 99", "optional transitive, 2 octets: ab01"),
]
.map(|(name, value)| (name.to_string(), value.to_string()));
assert_eq!(other, expected);
}
}
/// `GET /api/tcp?peer=ADDRESS:PORT`: what the kernel knows of the TCP
/// connection of the session with that peer, as `/api/peers` names it.
#[derive(Debug, Deserialize)]
pub struct TcpQuery {
pub peer: String,
}
/// A session's TCP connection, as the kernel reports it (Linux only).
/// Times in milliseconds, sizes in octets, rates in octets per second.
#[derive(Debug, Serialize)]
pub struct TcpView {
pub state: &'static str,
pub congestion: Option<String>,
pub rtt_ms: f64,
pub rtt_variance_ms: f64,
pub min_rtt_ms: f64,
pub rto_ms: f64,
pub mss: u32,
pub pmtu: u32,
pub cwnd_segments: u32,
pub ssthresh_segments: Option<u32>,
/// The peer's receive window: what it lets us send.
pub send_window: u32,
/// Our receive window: what we let the peer send.
pub receive_window: u32,
pub receive_space: u32,
pub send_window_scale: u8,
pub receive_window_scale: u8,
/// Octets written by us and not yet acknowledged, and of those the
/// octets not even sent yet.
pub send_queue: u32,
pub not_sent: u32,
pub recv_queue: u32,
pub unacked_segments: u32,
pub retrans_segments: u32,
pub lost_segments: u32,
pub total_retrans: u32,
pub bytes_retrans: u64,
pub pacing_rate: u64,
pub delivery_rate: u64,
pub bytes_sent: u64,
pub bytes_acked: u64,
pub bytes_received: u64,
pub segments_out: u32,
pub segments_in: u32,
/// How long the connection had something to send, and of that how
/// long it was held up by the peer's window and by our send buffer.
pub busy_ms: f64,
pub limited_by_receive_window_ms: f64,
pub limited_by_send_buffer_ms: f64,
/// The same two, as shares of the busy time, 0 to 1.
pub limited_by_receive_window_share: f64,
pub limited_by_send_buffer_share: f64,
pub last_data_sent_ms: u32,
pub last_data_received_ms: u32,
pub last_ack_received_ms: u32,
}
/// The answer to `/api/tcp`.
#[derive(Debug, Serialize)]
pub struct TcpAnswer {
/// Whether the kernel could be asked and knew the connection.
pub available: bool,
/// Why not, if not.
pub reason: Option<String>,
pub connection: Option<TcpView>,
}
pub async fn tcp(
State(state): State<AppState>,
Query(query): Query<TcpQuery>,
) -> Result<Json<TcpAnswer>, QueryError> {
let session = state
.registry
.list()
.into_iter()
.find(|s| socket_text(s.peer.addr) == query.peer)
.ok_or_else(|| QueryError::Bad(format!("no session with {}", query.peer)))?;
let answer =
match bgp_fib::sockdiag::tcp_connection(session.local_addr, session.peer.addr).await {
Ok(Some(c)) => {
let ms = |us: u64| us as f64 / 1000.0;
let share = |part: u64| {
if c.busy == 0 {
0.0
} else {
part as f64 / c.busy as f64
}
};
TcpAnswer {
available: true,
reason: None,
connection: Some(TcpView {
state: c.state_name(),
congestion: c.congestion.clone(),
rtt_ms: ms(c.rtt.into()),
rtt_variance_ms: ms(c.rtt_variance.into()),
min_rtt_ms: ms(c.min_rtt.into()),
rto_ms: ms(c.rto.into()),
mss: c.mss,
pmtu: c.pmtu,
cwnd_segments: c.cwnd,
ssthresh_segments: c.ssthresh,
send_window: c.send_window,
receive_window: c.receive_window,
receive_space: c.receive_space,
send_window_scale: c.send_window_scale,
receive_window_scale: c.receive_window_scale,
send_queue: c.send_queue,
not_sent: c.not_sent,
recv_queue: c.recv_queue,
unacked_segments: c.unacked,
retrans_segments: c.retrans,
lost_segments: c.lost,
total_retrans: c.total_retrans,
bytes_retrans: c.bytes_retrans,
pacing_rate: c.pacing_rate,
delivery_rate: c.delivery_rate,
bytes_sent: c.bytes_sent,
bytes_acked: c.bytes_acked,
bytes_received: c.bytes_received,
segments_out: c.segments_out,
segments_in: c.segments_in,
busy_ms: ms(c.busy),
limited_by_receive_window_ms: ms(c.limited_by_receive_window),
limited_by_send_buffer_ms: ms(c.limited_by_send_buffer),
limited_by_receive_window_share: share(c.limited_by_receive_window),
limited_by_send_buffer_share: share(c.limited_by_send_buffer),
last_data_sent_ms: c.last_data_sent_ms,
last_data_received_ms: c.last_data_received_ms,
last_ack_received_ms: c.last_ack_received_ms,
}),
}
}
Ok(None) => TcpAnswer {
available: false,
reason: Some(if cfg!(target_os = "linux") {
"the kernel has no such connection".to_string()
} else {
"only on Linux".to_string()
}),
connection: None,
},
Err(e) => TcpAnswer {
available: false,
reason: Some(format!("the kernel could not be asked: {e}")),
connection: None,
},
};
Ok(Json(answer))
}
crates/web/src/events.rs
+8 -5
View File
@@ -52,7 +52,8 @@ pub enum WireEvent {
/// table.
Lagged {
stream: &'static str,
missed: u64,
#[serde(skip_serializing_if = "Option::is_none")]
missed: Option<u64>,
},
}
@@ -71,7 +72,7 @@ impl WireEvent {
},
Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
stream: "sessions",
missed: n,
missed: Some(n),
},
}
}
@@ -86,7 +87,7 @@ impl WireEvent {
},
Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
stream: "local_routes",
missed: n,
missed: Some(n),
},
}
}
@@ -146,16 +147,18 @@ fn best_routes(state: &AppState) -> impl Stream<Item = WireEvent> + use<> {
let names = PeerNames::of(&state_for_views);
for change in changes {
let event = match change {
Change::Best(route) => match names.view(&route) {
Change::Best(route) => match names.view(&route, 1) {
Some(route) => WireEvent::BestChanged { route },
None => continue,
},
Change::Withdrawn(prefix) => WireEvent::BestWithdrawn {
prefix: prefix.to_string(),
},
// The journal does not say how many changes went
// by before the reader was given the table again.
Change::Lagged => WireEvent::Lagged {
stream: "best_routes",
missed: 0,
missed: None,
},
};
if events.send(event).await.is_err() {
crates/web/src/lib.rs
+41 -17
View File
@@ -14,22 +14,22 @@ use std::sync::Arc;
use std::time::{Instant, SystemTime};
use axum::Router;
use axum::http::header;
use axum::response::{Html, IntoResponse, Redirect};
use axum::response::{Html, Redirect};
use axum::routing::get;
use bgp_fib::Fib;
use bgp_rib::LocalRib;
use bgp_rib::table::RibTable;
use bgp_rpki::Rpki;
use bgp_server::refusals::RefusalLog;
use bgp_server::rejections::RejectionLog;
use bgp_server::restart::Restarts;
use bgp_server::session::SessionRegistry;
use serde::Serialize;
use tokio::net::TcpListener;
use tracing::info;
pub use api::{
AttrSummary, ConfigView, KernelView, LocalRouteView, PeerView, ReceivedRouteView, RefusalsView,
RefusedRouteView, RoutesPage, RpkiView, StatusView,
AttrSummary, ConfigView, KernelView, LocalRouteView, PeerView, ReceivedRouteView,
RejectedRouteView, RejectionsView, RoutesPage, RpkiView, StatusView,
};
/// Static facts about the router shown in status output.
@@ -46,6 +46,21 @@ pub struct RouterInfo {
/// The configuration file as read at startup (path and raw text), shown
/// on the status page. `None` when the server was configured in code.
pub config: Option<(String, String)>,
/// Whether large communities of our AS ask for things here
/// (`control_communities`), so that the status page can say what.
pub control_communities: bool,
/// Names for communities, as configured (`community_names`), for the
/// status page to show in place of the numbers.
pub community_names: Vec<CommunityName>,
}
/// A name for a community, or for a kind of them.
#[derive(Debug, Clone, Serialize)]
pub struct CommunityName {
/// The community, or a pattern of them with `*` for a part left open:
/// `64601:100:2`, `64601:102:*`.
pub community: String,
pub name: String,
}
impl RouterInfo {
@@ -58,9 +73,23 @@ impl RouterInfo {
started_at: SystemTime::now(),
hostname: hostname(),
config: None,
control_communities: false,
community_names: Vec::new(),
}
}
/// Records the names configured for communities.
pub fn with_community_names(mut self, names: Vec<CommunityName>) -> Self {
self.community_names = names;
self
}
/// Records whether control communities are acted on.
pub fn with_control_communities(mut self, on: bool) -> Self {
self.control_communities = on;
self
}
/// Records the configuration file so the status page can show it.
pub fn with_config(mut self, path: impl Into<String>, text: impl Into<String>) -> Self {
self.config = Some((path.into(), text.into()));
@@ -71,13 +100,10 @@ impl RouterInfo {
/// Best-effort host name without pulling in a dependency: ask the `hostname`
/// utility, which exists on every platform we run on.
fn hostname() -> String {
std::process::Command::new("hostname")
.output()
nix::unistd::gethostname()
.ok()
.filter(|o| o.status.success())
.and_then(|o| String::from_utf8(o.stdout).ok())
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.and_then(|name| name.into_string().ok())
.filter(|name| !name.is_empty())
.unwrap_or_else(|| "unknown host".into())
}
@@ -89,7 +115,7 @@ pub struct AppState {
/// The routing table: routes learned from peers and the best of them.
pub rib: Arc<RibTable>,
/// What peers announced that was not taken.
pub refusals: RefusalLog,
pub rejections: RejectionLog,
/// The data for route origin validation, where that is configured.
pub rpki: Option<Rpki>,
/// What keeps the best routes in the kernel, where that is configured.
@@ -101,11 +127,8 @@ pub struct AppState {
/// The status page, embedded so the binary is self-contained.
const STATUSZ_HTML: &str = include_str!("../static/statusz.html");
async fn statusz() -> impl IntoResponse {
(
[(header::CONTENT_TYPE, "text/html; charset=utf-8")],
Html(STATUSZ_HTML),
)
async fn statusz() -> Html<&'static str> {
Html(STATUSZ_HTML)
}
/// Builds the router. Exposed so tests can drive it without a socket.
@@ -121,6 +144,7 @@ pub fn router(state: AppState) -> Router {
.route("/api/routes/rejected", get(api::rejected_routes))
.route("/api/rpki", get(api::rpki))
.route("/api/kernel", get(api::kernel))
.route("/api/tcp", get(api::tcp))
.route("/api/events", get(events::stream))
.route("/metricz", get(metrics::metricz))
.with_state(state)
crates/web/src/metrics.rs
+83 -3
View File
@@ -5,7 +5,7 @@ use axum::extract::State;
use axum::http::{StatusCode, header};
use axum::response::IntoResponse;
use bgp_server::metrics::{
LOCAL_ROUTES, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES, RIB_ROUTES,
LOCAL_ROUTES, PEER_TCP, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES, RIB_ROUTES,
};
use prometheus::{Encoder, TextEncoder};
@@ -24,15 +24,95 @@ pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
RIB_PREFIXES.set(gauge(state.rib.prefix_count()));
RIB_ATTRIBUTE_SETS.set(gauge(state.rib.attrs().len()));
RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len()));
sample_tcp(&state).await;
let mut buf = Vec::new();
match TextEncoder::new().encode(&prometheus::gather(), &mut buf) {
let encoder = TextEncoder::new();
match encoder.encode(&prometheus::gather(), &mut buf) {
Ok(()) => (
StatusCode::OK,
[(header::CONTENT_TYPE, "text/plain; version=0.0.4")],
[(header::CONTENT_TYPE, encoder.format_type())],
buf,
)
.into_response(),
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
}
}
/// Asks the kernel about every established session's TCP connection and
/// sets the peer's gauges: one netlink round trip a session, at scrape
/// time, so that there is history to look back on. A session the kernel
/// has no connection for any more loses its series.
async fn sample_tcp(state: &AppState) {
let tcp = &*PEER_TCP;
for session in state.registry.list() {
let label = session.metric_label();
let connection =
bgp_fib::sockdiag::tcp_connection(session.local_addr, session.peer.addr).await;
let Ok(Some(c)) = connection else {
tcp.remove(&label);
continue;
};
let seconds = |us: u64| us as f64 / 1e6;
let int = |n: u64| i64::try_from(n).unwrap_or(i64::MAX);
let label = [label.as_str()];
tcp.rtt_seconds
.with_label_values(&label)
.set(seconds(c.rtt.into()));
tcp.min_rtt_seconds
.with_label_values(&label)
.set(seconds(c.min_rtt.into()));
tcp.rtt_variance_seconds
.with_label_values(&label)
.set(seconds(c.rtt_variance.into()));
tcp.cwnd_segments
.with_label_values(&label)
.set(c.cwnd.into());
tcp.mss_bytes.with_label_values(&label).set(c.mss.into());
tcp.send_window_bytes
.with_label_values(&label)
.set(c.send_window.into());
tcp.receive_window_bytes
.with_label_values(&label)
.set(c.receive_window.into());
tcp.send_queue_bytes
.with_label_values(&label)
.set(c.send_queue.into());
tcp.not_sent_bytes
.with_label_values(&label)
.set(c.not_sent.into());
tcp.receive_queue_bytes
.with_label_values(&label)
.set(c.recv_queue.into());
tcp.unacked_segments
.with_label_values(&label)
.set(c.unacked.into());
tcp.retransmitted_segments
.with_label_values(&label)
.set(c.total_retrans.into());
tcp.lost_segments
.with_label_values(&label)
.set(c.lost.into());
tcp.busy_seconds
.with_label_values(&label)
.set(seconds(c.busy));
tcp.receive_window_limited_seconds
.with_label_values(&label)
.set(seconds(c.limited_by_receive_window));
tcp.send_buffer_limited_seconds
.with_label_values(&label)
.set(seconds(c.limited_by_send_buffer));
tcp.sent_bytes
.with_label_values(&label)
.set(int(c.bytes_sent));
tcp.received_bytes
.with_label_values(&label)
.set(int(c.bytes_received));
tcp.delivery_rate_bytes_per_second
.with_label_values(&label)
.set(int(c.delivery_rate));
tcp.pacing_rate_bytes_per_second
.with_label_values(&label)
.set(int(c.pacing_rate));
}
}
crates/web/static/statusz.html
+793 -152
View File
File diff suppressed because it is too large. Load diff
docs/README.md
+28
View File
@@ -0,0 +1,28 @@
# Documentation sources
This directory holds the sources of the Ferrix documentation site, built
with [Zola](https://www.getzola.org):
```bash
zola --root docs serve # http://127.0.0.1:1111, rebuilt on every change
zola --root docs build # into docs/public
```
The pages under `content/` are Markdown with Zola additions, which a code
forge does not render:
- The block between `+++` lines at the top of each page is its front
matter: the title, description and position in the navigation.
- `{{ example(name="...") }}` and `{{ file(name="...") }}` insert a file
from [`examples/`](examples/), whole, at that point. Open the file to
read it.
- Links of the form `@/docs/...` are internal links that Zola resolves and
checks at build time. On the forge they do not work; the target is the
path after `@/`, under `content/`.
To read the documentation as intended, build the site, or read it where
it is published (`base_url` in [`config.toml`](config.toml)).
`STYLE.md` is the writing guide. CI builds the site on every push and fails
on a link to a page or an anchor that does not exist; `zola --root docs
check` also checks external links.
docs/content/docs/development/architecture.md
+6 -3
View File
@@ -163,9 +163,12 @@ UPDATE message, and waits when the session's output queue is full.
### Kernel Route Installation
The `fib` crate reads best path changes and maintains the kernel routing
table over rtnetlink, with one route for each prefix through the next hop
of its best path. It also receives the router's interfaces and addresses
from the kernel, which determine whether a next hop is resolvable.
table over rtnetlink, with one route for each prefix through the resolved
next hop of its best path. It also receives the router's interfaces,
addresses and the routes of the resolution table from the kernel, which
determine whether and how a next hop is resolvable; the import policy
consults the same view, so that a route with an unresolvable next hop is
held rather than selected.
### HTTP Interface
docs/content/docs/development/testing.md
+80 -5
View File
@@ -99,6 +99,7 @@ test phases.
| `hijacks` | FRR advertises a local prefix, a route with BIRD as third-party next hop, and the exchange LAN prefix; all are rejected |
| `nexthops` | A preferred route from FRR with an unresolvable next hop is held, and the route from BIRD is selected and installed; the selection changes when the next hop becomes resolvable, and reverts when it becomes unresolvable |
| `transit` | FRR as customer and BIRD as provider: FRR's routes are advertised to BIRD with the local AS prepended; BIRD's routes are advertised to FRR with the OTC attribute; a control community prevents the advertisement of one FRR route to BIRD; traffic is forwarded between their networks |
| `collector` | FRR as a route collector with ADD-PATH and `all_paths`: BIRD peers twice, FRR receives both of BIRD's routes for a prefix with distinct path identifiers, and one is withdrawn by its identifier when its session is disabled |
| `v6only` | Exchange LAN without IPv4 addresses: one IPv6 session with each router for both address families; IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel; IPv4 forwarding verified with `ping` |
| `linklocal` | Exchange LAN with link-local addresses only, with the same addresses on a second interface: sessions with FRR (by interface, with radvd) and BIRD at `fe80::…%ix0`; both address families in BGP and in the kernel; forwarding verified with `ping`; interface disabled and enabled |
@@ -172,11 +173,85 @@ performance: approximately 1 million IPv4 prefixes, mostly /24, and
in UPDATE messages; AS paths of several hops; and communities on most
routes.
Table 5 lists results on an Apple M-series computer with a release build.
Table 5 lists `full_table` results on an Apple M-series computer with a
release build: 1.23 million prefixes from each of four neighbors, 4.92
million routes.
| Measurement | Result |
|---|---|
| 1.23 million routes from one neighbor, accepted into the routing table | 0.47 seconds |
| The same routes received by a second neighbor (`transit`) | 2.1 seconds, approximately 4 routes per UPDATE message |
| 1 million IPv4 routes installed in the kernel (Linux virtual machine) | Slightly more than 1 second |
| The same routes removed from the kernel | 0.5 seconds |
| Decoding the UPDATE messages, one thread | 0.43 seconds, 11 million routes per second |
| Decoding, interning attributes, inserting and selecting, one thread | 1.9 seconds, 2.5 million routes per second |
| The same, 12 threads | 0.7 seconds, 6.9 million routes per second |
| Four neighbors send the table over TCP to the in-process server | 0.9 seconds, 5.2 million routes per second |
| Memory per route, attribute sets included (`--features count-memory`) | 77 bytes |
## Profiling
The `scripts/profile/full-table.sh` script measures the daemon itself, as
built for production, while one or more neighbors send it a full routing
table, and records where its CPU time goes. It requires Linux with `perf`,
`inferno` (`cargo install inferno`), `curl`, `jq`, and `sudo`. On another
operating system, run it in the Linux virtual machine:
```bash
scripts/linux-vm.sh scripts/profile/full-table.sh
```
The script builds the daemon with the `profiling` Cargo profile, which is
the release profile with line tables, and with frame pointers, so that
`perf` can attribute samples to functions. It starts the daemon in a
network namespace of its own, connected by a virtual Ethernet pair to a
second namespace in which the load generator
(`crates/bench/examples/feed.rs`) connects as the neighbors and sends the
synthetic table. Routes that the daemon installs go into the first
namespace's routing table and do not affect the host. Table 6 lists the
environment variables that control the run.
| Variable | Effect | Default |
|---|---|---|
| `PEERS` | The number of neighbors that send the table | 1 |
| `KERNEL` | `1` installs the best routes in the kernel; `0` keeps them in the routing table only | 1 |
| `SCALE_DOWN` | Divides the size of the table | 1 |
| `HOLD_SECS` | How long the sessions stay open after the table is sent | 30 |
| `FREQ` | The sampling frequency in hertz | 1999 |
| `OUT` | The directory for the results | `~/profile` |
| `JEMALLOC_CONF` | jemalloc options for the daemon (its `MALLOC_CONF`), for example `thp:always` | none |
When run through `scripts/linux-vm.sh`, pass the variables with `env`,
because the virtual machine does not receive the host's environment:
```bash
scripts/linux-vm.sh env PEERS=4 KERNEL=0 scripts/profile/full-table.sh
```
The script prints how long the routes take to arrive in the routing table
and in the kernel, and the daemon's resident memory. It writes the
following files to the results directory:
| File | Contents |
|---|---|
| `flame.svg` | An interactive flame graph of the daemon's threads |
| `top-self.txt` | Functions by the time spent in their own code |
| `top-children.txt` | Functions by the time spent in them and everything they call |
| `stacks.folded` | The collapsed stacks the flame graph is drawn from |
| `perf.data`, `daemon.log`, `feed.log` | The raw samples and the logs |
Table 7 lists results in the Linux virtual machine (8 virtual CPUs on an
Apple M-series computer) with the synthetic table of 1.23 million
prefixes.
| Measurement | Result |
|---|---|
| One neighbor; routes in the routing table | 0.8 seconds, 1.5 million routes per second |
| One neighbor; the best routes installed in the kernel | 2.0 seconds after the first route |
| One neighbor, without the kernel; routes in the routing table | 0.6 seconds, 2.0 million routes per second |
| Four neighbors; 4.92 million routes in the routing table | 1.1 seconds, 4.4 million routes per second |
| Four neighbors; the best routes installed in the kernel | 2.2 seconds after the first route |
| Four neighbors; routes the kernel was asked to install | 1.25 million, for 1.23 million prefixes |
| Resident memory with one neighbor's table installed | 440 MB |
| Resident memory with four neighbors' tables installed | 630 MB |
A session is processed on one thread, so one neighbor's table arrives at
the rate of one CPU core, and several neighbors' tables arrive in
parallel. Installing routes in the kernel takes longer than receiving
them, and continues after the last route has arrived.
docs/content/docs/getting-started/installation.md
+1 -1
View File
@@ -55,7 +55,7 @@ To build and install the daemon:
bgp --version
```
The command displays the installed version, for example `bgp 0.2.0`.
The command displays the installed version, for example `bgp 0.2.5`.
## Building a Static Executable
docs/content/docs/getting-started/introduction.md
+4 -5
View File
@@ -55,19 +55,18 @@ Ferrix provides the following capabilities:
## Release Status
> [!IMPORTANT]
> Ferrix 0.2.0 is an early release, in production use on one small
> Ferrix 0.2.5 is an early release, in production use on one small
> network. Ferrix is tested against FRR, BIRD, and Routinator in an
> automated interoperability lab and against real Linux kernels.
Ferrix 0.2.0 does not support the following features:
Ferrix 0.2.5 does not support the following features:
- Route reflection, per-client best path selection for route servers, and
ADD-PATH for advertised routes.
- Per-client best path selection for route servers, and ADD-PATH for
advertised routes.
- TCP Authentication Option (TCP-AO), TCP MD5 signatures, and the
Generalized TTL Security Mechanism (GTSM).
- Long-Lived Graceful Restart, Bidirectional Forwarding Detection (BFD),
and configuration changes without a restart.
- Recursive resolution of next hops that are not directly connected.
For the implementation status of each RFC, see
[Supported Standards](@/docs/reference/rfcs.md).
docs/content/docs/getting-started/quick-start.md
+1 -1
View File
@@ -112,6 +112,6 @@ this metric, see [Monitoring Ferrix](@/docs/operations/monitoring.md#alerting-ru
## Next Steps
- [Configuring BGP Neighbors](@/docs/guides/peers.md)
- [BGP Neighbors](@/docs/guides/peers.md)
- [Import Policy](@/docs/guides/import-policy.md)
- [Configuration Examples](@/docs/guides/examples.md)
docs/content/docs/guides/graceful-restart.md
+1 -1
View File
@@ -77,7 +77,7 @@ When the daemon stops in response to SIGTERM, it ends each session for
good. On a session on which the N bit is in use, it sends a Cease
NOTIFICATION message with the subcode Hard Reset.
## When a Neighbor Restarts
## Neighbor Restart
Ferrix retains the routes of a neighbor whose session ends in one of the
following ways, if graceful restart is in use on the session:
docs/content/docs/guides/import-policy.md
+186 -21
View File
@@ -48,6 +48,12 @@ For IBGP neighbors (where `remote_asn` equals `local_asn`), Ferrix applies
only the checks on the prefix. The AS path checks do not apply, and the
next hop can be any usable address.
An IBGP neighbor can advertise a prefix that the local router also
originates, for example an anycast prefix that both routers originate.
Ferrix ignores such a route: it does not accept the route, and does not
count it as rejected. This applies only to the originated prefix itself,
not to more specific prefixes within it.
Ferrix always rejects the routes in Table 2. These checks cannot be
disabled.
@@ -56,23 +62,71 @@ disabled.
| Route from an EBGP neighbor whose AS path contains the local AS number | `own AS in path` |
| Route whose next hop is unusable: unspecified, loopback, multicast, or an address of the local router | `next hop unusable` |
| Route with the Only to Customer attribute received from the wrong direction (see [BGP Roles](@/docs/guides/roles.md)) | `route leak (Only to Customer)` |
| Route from an EBGP neighbor with the BLACKHOLE community (RFC 7999) | `blackhole request` |
| Route from an EBGP neighbor with the BLACKHOLE community (RFC 7999), unless `accept_blackholes` is set | `blackhole request` |
| Route with the BLACKHOLE community that is not a host route (/32 or /128), with `accept_blackholes` set | `blackhole for more than one address` |
### Blackhole Requests
### Blackhole Routes
A network under a denial-of-service attack can advertise a route for the
attacked address with the BLACKHOLE community (65535:666), to request that
other networks discard traffic to the address. At an Internet exchange, the
route server distributes such routes to its members with the exchange's
blackhole address as the next hop. These routes are usually host routes
(/32 or /128).
other networks discard traffic to the address (RFC 7999). At an Internet
exchange, the route server distributes such routes to its members with
the exchange's blackhole address as the next hop. These routes are host
routes (/32 or /128).
Ferrix does not support blackhole routing. It rejects every route with the
BLACKHOLE community from an EBGP neighbor, with the reason
`blackhole request`, regardless of the prefix length. A route with the
BLACKHOLE community that is accepted as an ordinary route directs traffic
to the blackhole next hop, which discards it, so Ferrix does not accept
these routes as ordinary routes either.
By default, Ferrix rejects every route with the BLACKHOLE community from
an EBGP neighbor, with the reason `blackhole request`, regardless of the
prefix length. Ferrix does not accept such a route as an ordinary route
either, because that would direct traffic to the blackhole next hop,
which discards it.
To honor blackhole requests from a neighbor, set `accept_blackholes` in
its import policy:
```textproto
peers {
name: "customer-a"
endpoint { address: "192.0.2.10" }
remote_asn: 64501
role: CUSTOMER
import {
prefixes { prefix: "203.0.113.0/24" }
accept_blackholes: true
}
}
```
With `accept_blackholes` set, Ferrix accepts a route with the BLACKHOLE
community if all of the following conditions are met:
- The prefix is a single address: a /32 or a /128. The
`max_ipv4_prefix_length` and `max_ipv6_prefix_length` limits do not
apply to blackhole routes. A blackhole route for a longer prefix is
rejected with the reason `blackhole for more than one address`.
- The address is within a prefix that the neighbor may announce, if
`prefixes` is configured. The `max_length` of the entry does not apply.
- The route passes the AS path, origin, and RPKI checks that apply to
every route.
The next hop of a blackhole route is not checked, and need not be
reachable: Ferrix never forwards traffic to it. If the route is the active
route for its prefix, which a host route is unless a longer-matching
route exists, Ferrix installs a blackhole route in the kernel, and the
router discards traffic to the address. See [Blackhole
Routes](@/docs/guides/kernel.md#blackhole-routes) in the kernel guide.
Ferrix advertises an accepted blackhole route like any other route, with
the BLACKHOLE community, to the neighbors that the export policy and the
export limits permit. A provider that honors the community then discards
the traffic before it reaches the local network. To keep a blackhole
route from being advertised, the originator adds the NO_EXPORT or
NO_ADVERTISE community, as RFC 7999 recommends.
> [!NOTE]
> Ferrix does not install a blackhole route for a prefix that it
> originates with the BLACKHOLE community. To discard traffic locally for
> such a prefix, add a kernel blackhole route, for example
> `ip route add blackhole 203.0.113.9/32`.
### Next Hop Check
@@ -229,8 +283,7 @@ set their local preference:
peers {
name: "ixp-a-rs"
import {
remove_communities: "64500:2:*"
remove_communities: "64500:3:*"
remove_communities: "64496:*"
add_large_communities: "64500:2:1001"
local_pref: 150
}
@@ -240,7 +293,9 @@ peers {
The statements have the following effect:
- `remove_communities` removes matching communities. Ferrix applies it
before adding communities. A pattern has two fields for a standard
after it removes the communities of the local AS (see
[Communities of the Local AS](#communities-of-the-local-as)), and before
adding communities. A pattern has two fields for a standard
community and three for a large community; each field is a number or `*`.
Examples: `64500:100`, `64500:*`, `64500:2:*`, `*:*`.
- `add_communities` and `add_large_communities` add communities to every
@@ -251,17 +306,127 @@ The statements have the following effect:
value is preferred. Default: 100.
> [!TIP]
> Remove the local AS's informational communities from received routes, so
> that a neighbor cannot set them. A common local preference scheme assigns
> 200 to customer routes, 150 to peer and route server routes, and 100 to
> transit routes.
> A common local preference scheme assigns 200 to customer routes, 150 to
> peer and route server routes, and 100 to transit routes.
The examples in this documentation use large communities `64500:2:x` to
record where a route was learned and `64500:3:r` to record the router that
advertised it. These values do not overlap the
[control communities](@/docs/guides/passing-routes-on.md#control-communities),
which use `64500:0:*` and `64500:1:*`, or the standard communities
`64500:n`, which control communities interpret as "advertise to AS n".
or the standard communities `64500:n`, which control communities interpret
as "advertise to AS n".
## Recording Where Routes Are Learned
Informational communities record, on each route, where the local AS
learned it. Other routers in the local AS, neighbors, and looking glasses
can then see it, and import rules and control communities can act on it.
A consistent scheme uses large communities with a function number of 100
or higher, for example:
| Community | Meaning |
|---|---|
| `64500:100:R` | Learned by router `R`. |
| `64500:101:V` | Learned at venue `V`: an Internet exchange, a transit provider, or customers. |
| `64500:102:S` | Learned over interconnection `S`. |
Each router adds them with `add_large_communities` on every EBGP session:
```textproto
peers {
name: "exchange-a-rs1-v4"
import {
add_large_communities: "64500:100:1"
add_large_communities: "64500:101:1"
add_large_communities: "64500:102:1001"
}
}
```
The communities are retained on IBGP sessions, so that each router sees
where the other routers learned their routes. Because Ferrix removes the
local AS's communities from routes from EBGP neighbors (see
[Communities of the Local AS](#communities-of-the-local-as)), a neighbor
cannot set them: every such community on a route was added by a router of
the local AS.
## Import Rules
Import rules apply local preference and communities to some routes only:
routes with certain communities, routes for certain prefixes, or both.
Ferrix evaluates the rules of a session in order, and applies the first
rule that matches a route. The communities that the rules match are those
of the route after the session's other import statements are applied, so
rules can match the session's own informational communities.
The following example, on an IBGP session, prefers routes that the other
router learned at exchange A over routes that it learned at exchange B,
and marks the routes of one prefix:
```textproto
peers {
name: "router-2"
remote_asn: 64500
import {
rules { match_communities: "64500:101:1" local_pref: 200 }
rules { match_communities: "64500:101:2" local_pref: 130 }
rules {
match_prefixes { prefix: "192.0.2.0/24" }
add_large_communities: "64500:31:10"
}
}
}
```
Routes that the rules do not match keep the local preference that they
were advertised with. Routes for one prefix that arrive in one UPDATE
message can match different rules; Ferrix stores each group separately.
> [!TIP]
> Choose local preferences so that the local AS's outbound traffic leaves
> through the same venue through which other networks send their traffic
> in. If a route is preferred at venue A for outbound traffic, use control
> communities to prepend the local AS number at the venues that are less
> preferred. Otherwise traffic to and from a network uses different
> venues, which makes troubleshooting harder and can overload a link that
> was meant as a fallback.
## Communities of the Local AS
By default, Ferrix removes the local AS's communities from every route that
it receives from an EBGP neighbor, before `add_communities` and
`add_large_communities` are applied. The following communities are
removed, where `L` is the local AS number:
- Standard communities `L:n`, if `L` is 65535 or lower.
- Large communities `L:n:m`.
- Standard communities `0:n`, if `control_communities` is enabled.
These communities are instructions and records of the local AS: control
communities decide where Ferrix advertises a route, and informational
communities record where a route was learned. A neighbor at an Internet
exchange must not be able to set them. Ferrix retains them on routes from
IBGP neighbors.
A customer that sets control communities on its own routes requires the
following statement:
```textproto
peers {
name: "customer-a"
role: CUSTOMER
import { accept_control_communities: true }
}
```
Ferrix then retains all communities of the local AS on the customer's
routes. The customer can affect only its own routes. Use
`remove_communities` to remove informational communities that the
customer must not set, for example `64500:2:*`.
Ferrix examines the communities of every UPDATE message from an EBGP
neighbor. A message without communities of the local AS is accepted
unchanged.
## Performance
Loaded 100 of 116 files, more files were not shown because too many files have changed in this diff. Show more