Hygiene: scripts, CI, docs and manifests

- scripts/in-namespace.sh set CARGO_TARGET_DIR to the empty string when
  it was unset, which cargo refuses, and dropped RUSTFLAGS, so the run
  in the namespace rebuilt what kernel-tests.sh had just built. Each is
  passed on only where set.
- The README's list of unsupported features still had route reflection
  and graceful restart, both of which exist; the command-line reference
  lacked SIGUSR1 and exit status 75; the HTTP API reference did not say
  what errors look like.
- The event stream said "missed 0 events" for the best-routes stream,
  which does not count what it missed; it leaves the count out now.
- CI lints bgp-bench with its count-memory feature, which nothing built.
- bgp-bench had two unused dev-dependencies; bgp-integration-tests had
  three dependencies that only its tests and examples use.
- install.sh updates apt before installing; two-routers.sh loses an
  unused variable; small things in the web crate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.github/workflows/ci.yml

@@ -24,6 +24,10 @@ jobs:
         run: cargo fmt --all -- --check
       - name: Lint
         run: cargo clippy --locked --workspace --all-targets -- -D warnings
+      # The memory-counting allocator of the benchmarks is behind a feature
+      # that the workspace build does not turn on.
+      - name: Lint the benchmarks with memory counting
+        run: cargo clippy --locked -p bgp-bench --all-targets --features count-memory -- -D warnings
       - name: Test
         run: cargo test --locked --workspace
       # A link in the documentation to something that is not there.

Cargo.lock

@@ -183,8 +183,6 @@ dependencies = [
  "flate2",
  "stats_alloc",
  "tokio",
- "tracing",
- "tracing-subscriber",
 ]
 
 [[package]]

README.md

@@ -12,9 +12,9 @@ Ferrix has not been deployed in a production network.
 
 ### Unsupported Features
 
-- Route reflection, per-client best path selection for route servers, and ADD-PATH for advertised routes
+- Per-client best path selection for route servers, and ADD-PATH for advertised routes
 - TCP-AO, TCP MD5, and GTSM
-- Graceful restart, BFD, and configuration changes without a restart
+- Long-Lived Graceful Restart, BFD, and configuration changes without a restart
 - Recursive resolution of next hops that are not directly connected
 
 ### Layout
@@ -28,6 +28,8 @@ Ferrix has not been deployed in a production network.
 | `crates/fib` | The best routes in the Linux kernel, over netlink |
 | `crates/server` | Sessions, the state machine, policy, configuration |
 | `crates/web` | The status page, HTTP API and metrics |
+| `crates/integration-tests` | A test server and test peers, and the tests that run the daemon end to end |
+| `crates/bench` | Benchmarks at the scale of the Internet routing table |
 | `bin/bgp` | The daemon |
 
 ## Documentation
@@ -39,7 +41,8 @@ The documentation is a website built from [`docs/`](docs/) with
 zola --root docs serve     # http://127.0.0.1:1111
 ```
 
-- [Overview](docs/content/docs/getting-started/introduction.md) and
+- [Overview](docs/content/docs/getting-started/introduction.md),
+  [installation](docs/content/docs/getting-started/installation.md) and
   [quick start](docs/content/docs/getting-started/quick-start.md)
 - User guide: [BGP neighbors](docs/content/docs/guides/peers.md),
   [import policy](docs/content/docs/guides/import-policy.md),
@@ -47,15 +50,18 @@ zola --root docs serve     # http://127.0.0.1:1111
   [RPKI](docs/content/docs/guides/rpki.md),
   [kernel route installation](docs/content/docs/guides/kernel.md),
   [route advertisement](docs/content/docs/guides/passing-routes-on.md),
-  [configuration examples](docs/examples/)
+  [graceful restart](docs/content/docs/guides/graceful-restart.md),
+  [configuration examples](docs/content/docs/guides/examples.md)
 - [Deployment](docs/content/docs/operations/deployment.md) and
   [monitoring](docs/content/docs/operations/monitoring.md)
 - Reference: [configuration](docs/content/docs/reference/configuration.md),
+  [command line](docs/content/docs/reference/command-line.md),
   [HTTP API](docs/content/docs/reference/http-api.md),
   [metrics](docs/content/docs/reference/metrics.md),
   [supported standards](docs/content/docs/reference/rfcs.md)
-- [Architecture](docs/content/docs/development/architecture.md) and
-  [testing](docs/content/docs/development/testing.md)
+- [Architecture](docs/content/docs/development/architecture.md),
+  [testing](docs/content/docs/development/testing.md) and
+  [contributing](docs/content/docs/development/contributing.md)
 
 ## Development
 

crates/bench/Cargo.toml

@@ -21,8 +21,6 @@ bgp-integration-tests.workspace = true
 bgp-rib.workspace = true
 bgp-server.workspace = true
 tokio.workspace = true
-tracing.workspace = true
-tracing-subscriber.workspace = true
 
 [features]
 # Counts every allocation, to report how much memory a table holds. The

crates/bench/benches/full_table.rs

@@ -17,7 +17,7 @@ use std::sync::atomic::{AtomicUsize, Ordering};
 use std::time::{Duration, Instant};
 
 use bgp_bench::memory::{Region, describe};
-use bgp_bench::{Prefixes, Shape, announcements, encoded, wire_context};
+use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context};
 use bgp_integration_tests::test_peer::TestPeer;
 use bgp_integration_tests::test_server::TestServer;
 use bgp_packet::message::{Message, PathAttribute};
@@ -25,13 +25,6 @@ use bgp_primitives::ip_prefix::IpPrefix;
 use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
 use bytes::BytesMut;
 
-fn env(name: &str, default: usize) -> usize {
-    std::env::var(name)
-        .ok()
-        .and_then(|v| v.parse().ok())
-        .unwrap_or(default)
-}
-
 fn mib(bytes: usize) -> f64 {
     bytes as f64 / (1024.0 * 1024.0)
 }

crates/bench/benches/transit.rs

@@ -7,7 +7,7 @@ use std::net::Ipv4Addr;
 use std::sync::Arc;
 use std::time::{Duration, Instant};
 
-use bgp_bench::{Prefixes, Shape, announcements, encoded};
+use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
 use bgp_integration_tests::test_peer::TestPeer;
 use bgp_integration_tests::test_server::TestServer;
 use bgp_packet::message::PathAttribute;
@@ -15,13 +15,6 @@ use bgp_server::marks::{Control, Marks};
 use bgp_server::policy::ExportPolicy;
 use bytes::BytesMut;
 
-fn env(name: &str, default: usize) -> usize {
-    std::env::var(name)
-        .ok()
-        .and_then(|v| v.parse().ok())
-        .unwrap_or(default)
-}
-
 /// How a server passes routes on in one run.
 #[derive(Clone, Copy)]
 enum Passing {

crates/bench/src/lib.rs

@@ -1,9 +1,10 @@
 //! A synthetic copy of the Internet routing table, as the UPDATE messages a
 //! peer would send to announce it.
 //!
-//! It is made up, not captured, so that the benchmarks need no data files
-//! and give the same input every run. It is shaped like the real thing where
-//! that matters for cost:
+//! It is made up, not captured, so that the benchmarks of a synthetic table
+//! need no data files and give the same input every run; the one of a real
+//! table reads an MRT dump instead, see [`mrt`]. It is shaped like the real
+//! thing where that matters for cost:
 //!
 //! - about a million IPv4 prefixes, most of them /24s, and about a quarter
 //!   of a million IPv6 prefixes, most of them /48s;
@@ -163,7 +164,7 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
         let hops = 2 + rng.below(4) as usize;
         let mut path = vec![peer_asn];
         path.extend((0..hops).map(|_| 1 + rng.below(400_000) as u32));
-        let mut attributes = vec![
+        let attributes = vec![
             PathAttribute::Origin(OriginPathAttribute::IGP),
             PathAttribute::AsPath(AsPathAttribute {
                 segments: vec![AsPathSegment {
@@ -199,7 +200,7 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
         } else {
             None
         };
-        (std::mem::take(&mut attributes), later, large)
+        (attributes, later, large)
     };
 
     let mut updates = Vec::new();
@@ -283,3 +284,12 @@ pub fn wire_context() -> ParserContext {
         add_path: Vec::new(),
     }
 }
+
+/// The value of the environment variable `name` as a number, or `default`
+/// when it is not set or is not one: how the benchmarks take their size.
+pub fn env(name: &str, default: usize) -> usize {
+    std::env::var(name)
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(default)
+}

crates/integration-tests/Cargo.toml

@@ -17,11 +17,14 @@ bgp-rpki.workspace = true
 bgp-server.workspace = true
 bgp-web.workspace = true
 bytes.workspace = true
-clap.workspace = true
 eyre.workspace = true
+tokio.workspace = true
+
+# Used by the examples and the tests only.
+[dev-dependencies]
+clap.workspace = true
 reqwest.workspace = true
 serde_json.workspace = true
-tokio.workspace = true
 
 [lints]
 workspace = true

crates/web/src/api.rs

@@ -1,12 +1,11 @@
 //! JSON handlers and their view types. Everything here is read-only: the
 //! daemon is configured through its configuration file.
 
-use std::collections::{BTreeMap, BinaryHeap, HashMap};
+use std::collections::{BTreeMap, BinaryHeap, HashMap, HashSet};
 use std::net::{IpAddr, Ipv4Addr};
 use std::time::{SystemTime, UNIX_EPOCH};
 
 use axum::Json;
-
 use axum::extract::{Query, State};
 use axum::http::StatusCode;
 use axum::response::{IntoResponse, Response};
@@ -210,7 +209,7 @@ impl PeerView {
 pub fn peer_views(registry: &SessionRegistry, restarts: &Restarts) -> Vec<PeerView> {
     let kept = restarts.kept();
     let sessions = registry.list();
-    let mut claimed = std::collections::HashSet::new();
+    let mut claimed = HashSet::new();
     let mut views: Vec<PeerView> = registry
         .configured_peers()
         .iter()
@@ -698,15 +697,16 @@ enum Scope {
     Within(IpPrefix),
 }
 
+/// A prefix given in a query.
+fn parse_prefix(text: &str) -> Result<IpPrefix, QueryError> {
+    IpPrefix::try_from(text).map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
+}
+
 impl Scope {
     fn of(query: &RouteQuery) -> Result<Self, QueryError> {
-        let prefix = |text: &str| {
-            IpPrefix::try_from(text)
-                .map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
-        };
         match (&query.prefix, &query.address, &query.within) {
             (None, None, None) => Ok(Self::All),
-            (Some(text), None, None) => Ok(Self::Exactly(prefix(text)?)),
+            (Some(text), None, None) => Ok(Self::Exactly(parse_prefix(text)?)),
             (None, Some(text), None) => {
                 let address: IpAddr = text
                     .parse()
@@ -719,7 +719,7 @@ impl Scope {
                 host.map(Self::Covering)
                     .map_err(|e| QueryError::Bad(e.to_string()))
             }
-            (None, None, Some(text)) => Ok(Self::Within(prefix(text)?)),
+            (None, None, Some(text)) => Ok(Self::Within(parse_prefix(text)?)),
             _ => Err(QueryError::Bad(
                 "give at most one of prefix, address and within".into(),
             )),
@@ -753,14 +753,10 @@ enum Place {
 
 impl Place {
     fn of(query: &RouteQuery) -> Result<Self, QueryError> {
-        let prefix = |text: &str| {
-            IpPrefix::try_from(text)
-                .map_err(|e| QueryError::Bad(format!("{text:?} is not a prefix: {e}")))
-        };
         match (&query.after, &query.before) {
             (None, None) => Ok(Self::Start),
-            (Some(after), None) => Ok(Self::After(prefix(after)?)),
-            (None, Some(before)) => Ok(Self::Before(prefix(before)?)),
+            (Some(after), None) => Ok(Self::After(parse_prefix(after)?)),
+            (None, Some(before)) => Ok(Self::Before(parse_prefix(before)?)),
             (Some(_), Some(_)) => Err(QueryError::Bad(
                 "give at most one of after and before".into(),
             )),
@@ -988,8 +984,8 @@ pub async fn received_routes(
 }
 
 /// Validation with the RPKI: where the data comes from and how fresh it
-/// is.
-#[derive(Debug, Clone, Serialize)]
+/// is. The default is the view of a router that does not validate.
+#[derive(Debug, Clone, Default, Serialize)]
 pub struct RpkiView {
     /// Whether validation is configured at all.
     pub enabled: bool,
@@ -1012,21 +1008,9 @@ pub struct RpkiView {
 }
 
 pub async fn rpki(State(state): State<AppState>) -> Json<RpkiView> {
+    // Validation not configured: nothing to say, and `enabled` false.
     let Some(status) = state.rpki.as_ref().map(|rpki| rpki.status()) else {
-        return Json(RpkiView {
-            enabled: false,
-            cache: None,
-            state: None,
-            session_id: None,
-            serial: None,
-            protocol_version: None,
-            vrps_ipv4: 0,
-            vrps_ipv6: 0,
-            aspas: 0,
-            updated_at_unix: None,
-            expires_at_unix: None,
-            last_error: None,
-        });
+        return Json(RpkiView::default());
     };
     Json(RpkiView {
         enabled: true,

crates/web/src/events.rs

@@ -52,7 +52,8 @@ pub enum WireEvent {
     /// table.
     Lagged {
         stream: &'static str,
-        missed: u64,
+        #[serde(skip_serializing_if = "Option::is_none")]
+        missed: Option<u64>,
     },
 }
 
@@ -71,7 +72,7 @@ impl WireEvent {
             },
             Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
                 stream: "sessions",
-                missed: n,
+                missed: Some(n),
             },
         }
     }
@@ -86,7 +87,7 @@ impl WireEvent {
             },
             Err(BroadcastStreamRecvError::Lagged(n)) => Self::Lagged {
                 stream: "local_routes",
-                missed: n,
+                missed: Some(n),
             },
         }
     }
@@ -153,9 +154,11 @@ fn best_routes(state: &AppState) -> impl Stream<Item = WireEvent> + use<> {
                         Change::Withdrawn(prefix) => WireEvent::BestWithdrawn {
                             prefix: prefix.to_string(),
                         },
+                        // The journal does not say how many changes went
+                        // by before the reader was given the table again.
                         Change::Lagged => WireEvent::Lagged {
                             stream: "best_routes",
-                            missed: 0,
+                            missed: None,
                         },
                     };
                     if events.send(event).await.is_err() {

crates/web/src/lib.rs

@@ -14,8 +14,7 @@ use std::sync::Arc;
 use std::time::{Instant, SystemTime};
 
 use axum::Router;
-use axum::http::header;
-use axum::response::{Html, IntoResponse, Redirect};
+use axum::response::{Html, Redirect};
 use axum::routing::get;
 use bgp_fib::Fib;
 use bgp_rib::LocalRib;
@@ -111,11 +110,8 @@ pub struct AppState {
 /// The status page, embedded so the binary is self-contained.
 const STATUSZ_HTML: &str = include_str!("../static/statusz.html");
 
-async fn statusz() -> impl IntoResponse {
-    (
-        [(header::CONTENT_TYPE, "text/html; charset=utf-8")],
-        Html(STATUSZ_HTML),
-    )
+async fn statusz() -> Html<&'static str> {
+    Html(STATUSZ_HTML)
 }
 
 /// Builds the router. Exposed so tests can drive it without a socket.

crates/web/src/metrics.rs

@@ -26,10 +26,11 @@ pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
     RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len()));
 
     let mut buf = Vec::new();
-    match TextEncoder::new().encode(&prometheus::gather(), &mut buf) {
+    let encoder = TextEncoder::new();
+    match encoder.encode(&prometheus::gather(), &mut buf) {
         Ok(()) => (
             StatusCode::OK,
-            [(header::CONTENT_TYPE, "text/plain; version=0.0.4")],
+            [(header::CONTENT_TYPE, encoder.format_type())],
             buf,
         )
             .into_response(),

crates/web/static/statusz.html

@@ -495,7 +495,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
   function describe(ev) {
     switch (ev.type) {
       case 'session_up': return `<span class="ok">session UP</span> with ${esc(ev.peer.name || ev.peer.peer_addr)} AS${ev.peer.peer_asn} (opened by ${ev.peer.direction === 'outbound' ? 'us' : 'peer'})`;
-      case 'route_refresh_requested': return `${esc(ev.peer.name || ev.peer.peer_addr)} AS${ev.peer.peer_asn} asked for a route refresh; re-sent all originated routes`;
+      case 'route_refresh_requested': return `${esc(ev.peer.name || ev.peer.peer_addr)} AS${ev.peer.peer_asn} asked for a route refresh; re-sent its routes`;
       case 'session_down': return `<span class="bad">session DOWN</span> with ${esc(ev.peer_addr)} AS${ev.peer_asn}`;
       case 'local_announced': { const r = ev.route; return `originating ${esc(r.prefix)}` +
         (r.communities.length ? ` [${names(r.communities)}]` : '') + (r.large_communities.length ? ` [${names(r.large_communities)}]` : '') +
@@ -503,7 +503,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
       case 'local_withdrawn': return `<span class="warn">no longer originating</span> ${esc(ev.prefix)}`;
       case 'best_changed': return `best route for ${esc(ev.route.prefix)} is now from AS${ev.route.peer_asn} path [${ev.route.summary.as_path.join(' ')}]`;
       case 'best_withdrawn': return `${esc(ev.prefix)} has <span class="warn">no route</span> any more`;
-      case 'lagged': return `<span class="bad">missed ${ev.missed} events</span> on ${esc(ev.stream)}`;
+      case 'lagged': return `<span class="bad">missed ${ev.missed == null ? '' : ev.missed + ' '}events</span> on ${esc(ev.stream)}`;
       default: return esc(JSON.stringify(ev));
     }
   }

docs/content/docs/reference/command-line.md

@@ -32,8 +32,9 @@ validates the file.
 
 | Status | Meaning |
 |---|---|
-| 0 | With `--check`: the configuration is valid. Otherwise: the daemon stopped in response to a signal. |
+| 0 | With `--check`: the configuration is valid. Otherwise: the daemon stopped in response to `SIGTERM` or `SIGINT`. |
 | 1 | The configuration is invalid, or the daemon failed. The error is written to standard error. |
+| 75 | The daemon exited in response to `SIGUSR1`, to be started again. The systemd unit restarts it on this status (`RestartForceExitStatus=75`). See [Upgrading Ferrix](@/docs/operations/deployment.md#upgrading-ferrix). |
 
 ## Environment Variables
 
@@ -45,7 +46,8 @@ validates the file.
 
 | Signal | Action |
 |---|---|
-| `SIGTERM`, `SIGINT` | Sends a Cease NOTIFICATION message with the subcode Administrative Shutdown to every neighbor, removes the daemon's routes from the kernel, and exits. |
+| `SIGTERM`, `SIGINT` | Sends a Cease NOTIFICATION message with the subcode Administrative Shutdown to every neighbor, removes the daemon's routes from the kernel, and exits with status 0. |
+| `SIGUSR1` | Starts a graceful restart (RFC 4724): neighbors on which graceful restart is in use retain the routes of Ferrix for the restart time, other neighbors receive a Cease NOTIFICATION message with the subcode Administrative Shutdown, the routes stay in the kernel, and the daemon exits with status 75. See [Upgrading Ferrix](@/docs/operations/deployment.md#upgrading-ferrix). |
 
 Ferrix does not reload its configuration while it runs. To apply a new
 configuration, restart the daemon. See

docs/content/docs/reference/http-api.md

@@ -8,9 +8,16 @@ nav = "HTTP API"
 +++
 
 Ferrix serves its HTTP interface on the address configured with
-`http_endpoint`. All endpoints are read-only and use the `GET` method. If a
-query parameter is invalid, the server returns status 400 (Bad Request)
-with a description of the error.
+`http_endpoint`. All endpoints are read-only and use the `GET` method.
+
+An error response has a `text/plain` body that describes the error. The
+server returns the following error statuses:
+
+| Status | Meaning |
+|---|---|
+| 400 (Bad Request) | A query parameter is invalid. |
+| 500 (Internal Server Error) | The server could not produce the response, for example because the metrics could not be encoded. |
+| 503 (Service Unavailable) | A table is unavailable because the daemon is shutting down. |
 
 > [!CAUTION]
 > The HTTP interface does not require authentication. Bind it to a loopback
@@ -229,7 +236,7 @@ identifies the event, as listed in Table 2.
 | `local_announced`, `local_withdrawn` | An originated route was advertised or withdrawn. |
 | `best_changed` | The best path for a prefix changed. The `route` field contains the new best path. |
 | `best_withdrawn` | A prefix no longer has a route. |
-| `lagged` | The client did not read events quickly enough and missed events on `stream`. Reload the corresponding table. |
+| `lagged` | The client did not read events quickly enough and missed events on `stream`. Reload the corresponding table. The `missed` field gives the number of missed events for the `sessions` and `local_routes` streams; the `best_routes` stream omits it because the number is not known. |
 
 The following command displays the event stream:
 

scripts/in-namespace.sh

@@ -11,8 +11,14 @@ sudo modprobe dummy 2>/dev/null || true
 sudo ip netns add "$namespace"
 trap 'sudo ip netns del "$namespace"' EXIT
 sudo ip netns exec "$namespace" ip link set lo up
+# Only the variables that are set are passed on: cargo refuses an empty
+# CARGO_TARGET_DIR, and the build settings have to be the same as the
+# prebuild's in scripts/kernel-tests.sh for it to be reused.
 sudo ip netns exec "$namespace" \
     setpriv --reuid "$(id -u)" --regid "$(id -g)" --init-groups \
     --inh-caps +net_admin --ambient-caps +net_admin \
-    env PATH="$PATH" HOME="$HOME" CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-}" \
-    FERRIX_ROUTES="${FERRIX_ROUTES:-}" "$@"
+    env PATH="$PATH" HOME="$HOME" \
+    ${CARGO_TARGET_DIR:+CARGO_TARGET_DIR="$CARGO_TARGET_DIR"} \
+    ${RUSTFLAGS:+RUSTFLAGS="$RUSTFLAGS"} \
+    ${CARGO_TERM_COLOR:+CARGO_TERM_COLOR="$CARGO_TERM_COLOR"} \
+    ${FERRIX_ROUTES:+FERRIX_ROUTES="$FERRIX_ROUTES"} "$@"

scripts/interop/install.sh

@@ -3,6 +3,7 @@
 # BIRD from the distribution, Routinator from crates.io (a few minutes to
 # build), and radvd, which tells FRR who is on a link.
 set -eu
+sudo apt-get update
 sudo DEBIAN_FRONTEND=noninteractive apt-get install -y frr bird2 jq curl radvd
 # They are run by the lab, in namespaces, not as services of this machine.
 sudo systemctl disable --now frr bird radvd 2>/dev/null || true

scripts/two-routers.sh

@@ -97,7 +97,6 @@ start() {
     pids="$pids $!"
 }
 start "$a" a
-pid_a=$!
 start "$b" b
 
 # Waits until `ip route` in a namespace shows, or no longer shows, a route.