rayhaan/ferrix
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

packet, fib, bench: framing helpers, one netlink reader, a mesh with named phases

Browse Source 
A code-quality pass over the periphery. The packet crate frames a
message with `begin_message` and `finish_message`, which the
exporter's hand-written UPDATE header can use instead of a bare type
code, and keeps the NLRI reader and writer together as a pair; the
rule that the body's prefixes are IPv4 unicast lives once; a session
with two-octet AS numbers no longer reserves the attributes' wire
copy it never keeps. The TCP diagnostics read netlink with the
netlink module's own attribute walk and helpers instead of copies of
them. The kernel follower learns what the kernel has in one named
step used at start and on relearning, keeps its batch buffers between
wakes, and builds its task in a constructor. The mesh benchmark,
which had grown six phases in one main, is a `Mesh` with named phases
and a `Receiver`, with one report of the sessions for both deadlines;
the collector capabilities, the KEEPALIVE bytes, the draining of a
session and the printing of a table row are helpers shared by the
benches, the examples and the integration tests. The MRT reader uses
the packet crate's attribute splitter.

Testing: a message framed by hand is the one `to_wire` writes; a
short tcp_info from an old kernel leaves the later fields zero; the
synthetic table is the same every run and every announcement decodes
to itself. The packing example gives the same figures on the RIS dump
before and after. The crates' tests, the integration tests, the
kernel tests and every lab phase pass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Rayhaan JaufeerallyandClaude Fable 5.1 committed 2026-10-10 22:44:32 +02:00
1 parent bcfd6e013f
commit 25597ab3ef
16 files changed
+933 -565
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

crates/bench/benches/full_table.rs
+3 -17
View File
@@ -18,6 +18,7 @@ use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::memory::{Region, describe};
use bgp_bench::report::{mib, per_second, row};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, wire_context};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_server::TestServer;
@@ -26,23 +27,6 @@ use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{PeerIndex, PeerInfo, RibTable};
use bytes::BytesMut;
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
}
fn main() {
let peers = env("BENCH_PEERS", 4) as u32;
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
@@ -110,6 +94,8 @@ fn main() {
.handshake(64512 + peer, Ipv4Addr::new(10, 0, 0, peer as u8), 90)
.await
.expect("a session");
// The same bytes `table_stream` would make, from the messages
// already encoded for the decoding above.
let stream: Vec<u8> = messages.iter().flat_map(|m| m.iter().copied()).collect();
senders.push((session, stream));
}
crates/bench/benches/mesh.rs
+433 -243
View File
@@ -32,17 +32,21 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
use bgp_bench::report::row;
use bgp_bench::{Prefixes, Shape, announcements, encoded, env, table_stream};
use bgp_integration_tests::peer_config;
use bgp_integration_tests::test_peer::TestPeer;
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_integration_tests::test_server::TestServer;
use bgp_packet::attributes::PathAttribute;
use bgp_packet::open::{AddPathEntry, Capability, SendReceive};
use bgp_packet::message::msg_type;
use bgp_packet::open::SendReceive;
use bgp_server::config::PeerConfig;
use bgp_server::family::SUPPORTED;
use bgp_server::policy::{ExportPolicy, ImportPolicy, PeerRole};
use bgp_server::session::SessionRegistry;
use tokio::io::{AsyncReadExt, AsyncWriteExt, BufReader};
use tokio::net::tcp::OwnedWriteHalf;
use tokio::task::JoinHandle;
/// An AS number that is in none of the synthetic paths.
const ASN: u32 = 4_200_000_000;
@@ -58,6 +62,9 @@ fn collector_address(n: u32) -> Ipv4Addr {
/// The collectors' AS numbers start here.
const COLLECTOR_ASN: u32 = 65000;
/// How often the tables are looked at while they fill.
const POLL: Duration = Duration::from_millis(5);
/// What the routers send each other, and the receiver: every path.
fn every_path() -> Arc<ExportPolicy> {
Arc::new(ExportPolicy {
@@ -77,12 +84,12 @@ fn free_port() -> u16 {
/// A router of the mesh, listening for feeders on IPv4 loopback and for
/// the other router on IPv6 loopback at `port`, which it dials at
/// `other`.
/// `other_port`, knowing it as `other_name`.
async fn router(
name: &'static str,
other_name: &'static str,
id: u8,
port: u16,
other: u16,
other_port: u16,
with_receiver: bool,
collectors: u32,
) -> TestServer {
@@ -99,8 +106,8 @@ async fn router(
import: ImportPolicy::permissive(),
export: every_path(),
..peer_config(
name,
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), other),
other_name,
SocketAddr::new(Ipv6Addr::LOCALHOST.into(), other_port),
ASN,
)
}];
@@ -144,20 +151,22 @@ async fn feeders(
) -> Vec<(TestPeer, Vec<u8>)> {
let mut sessions = Vec::new();
for peer in first..first + count {
let mut updates = announcements(prefixes, shape, peer);
if marked {
let stream = if marked {
// As having come across from a peer (RFC 9234): not for the
// collectors, which are peers too.
let mut updates = announcements(prefixes, shape, peer);
for update in &mut updates {
update
.path_attributes
.push(PathAttribute::OnlyToCustomer(64512 + peer));
}
}
let stream: Vec<u8> = encoded(&updates)
.iter()
.flat_map(|m| m.iter().copied())
.collect();
encoded(&updates)
.iter()
.flat_map(|m| m.iter().copied())
.collect()
} else {
table_stream(prefixes, shape, peer)
};
let mut session = TestPeer::connect(router.addr).await.expect("a connection");
// No hold timer: a feeder has nothing to say after its table.
session
@@ -169,39 +178,112 @@ async fn feeders(
sessions
}
fn main() {
// The routers' logs, with `BENCH_LOG` as a `RUST_LOG` filter.
if let Ok(filter) = std::env::var("BENCH_LOG") {
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.init();
}
let peers_a = env("BENCH_PEERS", 4) as u32;
let peers_b = env("BENCH_PEERS_B", 2) as u32;
let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
let per_peer = shape.prefixes();
println!(
"== Two routers of AS{ASN}, every path both ways: A fed by {peers_a} peers, B by {peers_b}, {} prefixes each",
per_peer
);
let prefixes = Prefixes::generate(shape);
/// Every session of `registry`, one line each, for the panic when a
/// deadline passes: where the routes have got to, and what each session
/// was to be sent.
fn sessions_report(registry: &SessionRegistry) -> Vec<String> {
registry
.list()
.iter()
.map(|s| {
format!(
"{} {:?} export {:?} add_path_send {:?} routes {} rejected {} advertised {}",
s.peer,
s.name,
s.export,
s.add_path_send,
s.counters.routes(),
s.counters.rejected(),
s.counters.advertised()
)
})
.collect()
}
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
/// How big a run is, from the environment.
struct Settings {
peers_a: u32,
peers_b: u32,
shape: Shape,
collectors: u32,
}
impl Settings {
fn from_env() -> Self {
Self {
peers_a: env("BENCH_PEERS", 4) as u32,
peers_b: env("BENCH_PEERS_B", 2) as u32,
shape: Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1)),
collectors: env("BENCH_COLLECTORS", 0) as u32,
}
}
/// How many routes A's own feeders bring, and B's.
fn own_routes(&self) -> (usize, usize) {
let per_peer = self.shape.prefixes();
(
per_peer * self.peers_a as usize,
per_peer * self.peers_b as usize,
)
}
/// How many routes the receiver is to be sent: A's IPv4 routes from
/// its own peers.
fn receiver_routes(&self) -> usize {
self.shape.ipv4_prefixes * self.peers_a as usize
}
}
/// The two routers, up and peered, with the collectors' sessions once
/// they are connected.
struct Mesh {
settings: Settings,
prefixes: Prefixes,
a: TestServer,
b: TestServer,
/// The collectors' connections, drained, held until the end.
collectors: Vec<(OwnedWriteHalf, JoinHandle<()>)>,
}
/// The feeders sending, from when they began.
struct Feeding {
started: Instant,
/// Each hands its session back once its table is written.
sending: Vec<JoinHandle<TestPeer>>,
}
/// When each router had what: its own peers' routes, and the other's too.
struct Tables {
a_own: Duration,
b_own: Duration,
a_all: Duration,
b_all: Duration,
}
impl Mesh {
/// Starts the routers, with the collectors and receiver configured at
/// A, and waits for them to peer.
async fn start(settings: Settings, prefixes: Prefixes) -> Self {
let (port_a, port_b) = (free_port(), free_port());
let collectors = env("BENCH_COLLECTORS", 0) as u32;
let a = router("b", 1, port_a, port_b, true, collectors).await;
let a = router("b", 1, port_a, port_b, true, settings.collectors).await;
let b = router("a", 2, port_b, port_a, false, 0).await;
a.wait_for_sessions(1).await.expect("the mesh session at A");
b.wait_for_sessions(1).await.expect("the mesh session at B");
// The collectors are there before the feeders, as a router's
// peers are before its tables arrive after a restart.
let mut draining = Vec::new();
for n in 1..=collectors {
Self {
settings,
prefixes,
a,
b,
collectors: Vec::new(),
}
}
/// Connects the collectors to A. They are there before the feeders, as
/// a router's peers are before its tables arrive after a restart. An
/// address that is not this machine's ends the connecting, with a
/// word.
async fn connect_collectors(&mut self) {
for n in 1..=self.settings.collectors {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(collector_address(n).into(), 0)) {
println!(
@@ -210,61 +292,82 @@ fn main() {
);
break;
}
let stream = socket.connect(a.addr).await.expect("A listens");
let stream = socket.connect(self.a.addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake(COLLECTOR_ASN + n, Ipv4Addr::new(192, 0, 3, n as u8), 0)
.await
.expect("A takes a collector's session");
let (mut reader, writer) = session.into_stream().into_split();
draining.push((
writer,
tokio::spawn(async move {
let mut sink = vec![0u8; 1 << 16];
while matches!(reader.read(&mut sink).await, Ok(n) if n > 0) {}
}),
));
self.collectors.push(session.drain());
}
if collectors > 0 {
a.wait_for_sessions(1 + draining.len())
if !self.collectors.is_empty() {
self.a
.wait_for_sessions(1 + self.collectors.len())
.await
.expect("the collectors' sessions at A");
}
}
let feed_a = feeders(&a, &prefixes, shape, 1, peers_a, collectors > 0).await;
let feed_b = feeders(&b, &prefixes, shape, 1 + peers_a, peers_b, false).await;
let own_a = per_peer * peers_a as usize;
let own_b = per_peer * peers_b as usize;
let receiver_routes = shape.ipv4_prefixes * peers_a as usize;
/// Connects the feeders to both routers and starts them sending.
async fn feed(&self) -> Feeding {
let Settings {
peers_a,
peers_b,
shape,
..
} = self.settings;
let marked = !self.collectors.is_empty();
let feed_a = feeders(&self.a, &self.prefixes, shape, 1, peers_a, marked).await;
let feed_b = feeders(&self.b, &self.prefixes, shape, 1 + peers_a, peers_b, false).await;
let started = Instant::now();
// Early: as the feeders start, or (2) once A has its own peers'
// routes and B's are still coming, as a router's sessions come
// up after a restart while its peers' tables arrive.
let when = env("BENCH_RECEIVER_EARLY", 0);
let early_receiver = (when > 0).then(|| {
let (addr, registry, rib) = (a.addr, a.registry.clone(), Arc::clone(&a.rib));
tokio::spawn(async move {
while when == 2 && rib.len() < own_a {
tokio::time::sleep(Duration::from_millis(5)).await;
}
resync(addr, registry, receiver_routes).await
let sending = feed_a
.into_iter()
.chain(feed_b)
.map(|(mut session, stream)| {
tokio::spawn(async move {
session.send_raw(&stream).await.expect("the router reads");
session
})
})
});
let mut sending = Vec::new();
for (mut session, stream) in feed_a.into_iter().chain(feed_b) {
sending.push(tokio::spawn(async move {
session.send_raw(&stream).await.expect("the router reads");
session
}));
.collect();
Feeding { started, sending }
}
/// The receiver taking A's table while it fills, if `BENCH_RECEIVER_EARLY`
/// asks for that: as the feeders start (1), or once A has its own
/// peers' routes and B's are still coming (2), as a router's sessions
/// come up after a restart while its peers' tables arrive.
fn early_receiver(&self, deadline: Duration) -> Option<JoinHandle<()>> {
let when = env("BENCH_RECEIVER_EARLY", 0);
if when == 0 {
return None;
}
let (own_a, _) = self.settings.own_routes();
let routes = self.settings.receiver_routes();
let (addr, registry, rib) = (
self.a.addr,
self.a.registry.clone(),
Arc::clone(&self.a.rib),
);
Some(tokio::spawn(async move {
while when == 2 && rib.len() < own_a {
tokio::time::sleep(POLL).await;
}
resync(addr, registry, routes, deadline).await
}))
}
/// Waits until both routers have every route, noting when each had
/// what; or panics once `deadline` has passed since `started`.
async fn wait_for_tables(&self, started: Instant, deadline: Duration) -> Tables {
let (own_a, own_b) = self.settings.own_routes();
let mut a_own = None;
let mut b_own = None;
let mut a_all = None;
let mut b_all = None;
while a_all.is_none() || b_all.is_none() {
tokio::time::sleep(Duration::from_millis(5)).await;
let (at_a, at_b) = (a.rib.len(), b.rib.len());
let (a_all, b_all) = loop {
tokio::time::sleep(POLL).await;
let (at_a, at_b) = (self.a.rib.len(), self.b.rib.len());
let now = started.elapsed();
if a_own.is_none() && at_a >= own_a {
a_own = Some(now);
@@ -278,192 +381,279 @@ fn main() {
if b_all.is_none() && at_b >= own_a + own_b {
b_all = Some(now);
}
if now > Duration::from_secs(env("BENCH_DEADLINE", 600) as u64) {
let sessions = |router: &TestServer| -> Vec<String> {
router
.registry
.list()
.iter()
.map(|s| {
format!(
"{} routes {} rejected {}",
s.peer,
s.counters.routes(),
s.counters.rejected()
)
})
.collect()
};
if let (Some(a_all), Some(b_all)) = (a_all, b_all) {
break (a_all, b_all);
}
if now > deadline {
panic!(
"not all routes arrived: A has {at_a} {:?}, B has {at_b} {:?}",
sessions(&a),
sessions(&b)
sessions_report(&self.a.registry),
sessions_report(&self.b.registry)
);
}
}
let row = |what: &str, when: Option<Duration>, routes: usize| {
let when = when.unwrap_or_default();
println!(
"{what:<52} {:>8.3} s {:.2} M routes/s",
when.as_secs_f64(),
routes as f64 / when.as_secs_f64() / 1e6
);
};
row("A has its own peers' routes", a_own, own_a);
row("B has its own peers' routes", b_own, own_b);
Tables {
a_own: a_own.unwrap_or_default(),
b_own: b_own.unwrap_or_default(),
a_all,
b_all,
}
}
/// One row for each of the four moments of `tables`, with the rate the
/// routes that made the moment came at.
fn report(&self, tables: &Tables) {
let (own_a, own_b) = self.settings.own_routes();
let rate = |when: Duration, routes: usize| {
format!("{:.2} M routes/s", routes as f64 / when.as_secs_f64() / 1e6)
};
row(
"A has its own peers' routes",
tables.a_own,
rate(tables.a_own, own_a),
);
row(
"B has its own peers' routes",
tables.b_own,
rate(tables.b_own, own_b),
);
row(
"A has B's routes too (B's exporter, A's session)",
a_all,
own_b,
tables.a_all,
rate(tables.a_all, own_b),
);
row(
"B has A's routes too (A's exporter, B's session)",
b_all,
own_a,
tables.b_all,
rate(tables.b_all, own_a),
);
// The feeders' sessions stay up until the end: a feeder going
// would have its routes withdrawn from both tables meanwhile.
}
}
impl Feeding {
/// Waits for every feeder to have written its table, and returns the
/// sessions, which are to stay up until the end: a feeder going would
/// have its routes withdrawn from both tables meanwhile.
async fn finished(self) -> Vec<TestPeer> {
let mut feeders = Vec::new();
for session in sending {
for session in self.sending {
feeders.push(session.await.expect("a feeder is done"));
}
feeders
}
}
fn main() {
// The routers' logs, with `BENCH_LOG` as a `RUST_LOG` filter.
if let Ok(filter) = std::env::var("BENCH_LOG") {
tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.init();
}
let settings = Settings::from_env();
let deadline = Duration::from_secs(env("BENCH_DEADLINE", 600) as u64);
println!(
"== Two routers of AS{ASN}, every path both ways: A fed by {} peers, B by {}, {} prefixes each",
settings.peers_a,
settings.peers_b,
settings.shape.prefixes()
);
let prefixes = Prefixes::generate(settings.shape);
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.expect("a runtime");
runtime.block_on(async {
let mut mesh = Mesh::start(settings, prefixes).await;
mesh.connect_collectors().await;
let early_receiver = mesh.early_receiver(deadline);
let feeding = mesh.feed().await;
let tables = mesh.wait_for_tables(feeding.started, deadline).await;
mesh.report(&tables);
let feeders = feeding.finished().await;
match early_receiver {
Some(receiver) => receiver.await.expect("the receiver is done"),
None => resync(a.addr, a.registry.clone(), receiver_routes).await,
None => {
resync(
mesh.a.addr,
mesh.a.registry.clone(),
mesh.settings.receiver_routes(),
deadline,
)
.await
}
}
drop(feeders);
drop(draining);
drop(mesh);
});
}
/// The new peer taking a router's table: counting the UPDATEs it is sent
/// while keeping the session alive.
struct Receiver {
started: Instant,
/// When the first UPDATE arrived, in microseconds after `started`; 0
/// until it has.
first: Arc<AtomicU64>,
updates: Arc<AtomicU64>,
counting: JoinHandle<()>,
keepalives: JoinHandle<()>,
}
impl Receiver {
/// A session with the router at `addr` from [`RECEIVER`], as a
/// collector of every path; or `None`, with a word, where that address
/// is not this machine's to connect from.
async fn connect(addr: SocketAddr) -> Option<TestPeer> {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(RECEIVER.into(), 0)) {
println!("A new peer taking A's routes: skipped, {RECEIVER} is not ours ({e})");
return None;
}
let stream = socket.connect(addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
session
.handshake_with(
ASN,
Ipv4Addr::new(192, 0, 2, 9),
90,
collector_capabilities(ASN, &SUPPORTED, false),
)
.await
.expect("A takes the receiver's session");
Some(session)
}
/// Starts counting what `session` is sent, from now. Every millionth
/// UPDATE is noted, as is a NOTIFICATION or the connection closing:
/// what went wrong, if the table never all arrives.
fn count_updates(session: TestPeer) -> Self {
let started = Instant::now();
let (reader, mut writer) = session.into_stream().into_split();
// KEEPALIVEs, so that the session outlives the hold time.
let keepalives = tokio::spawn(async move {
let keepalive = TestPeer::keepalive_bytes();
loop {
tokio::time::sleep(Duration::from_secs(20)).await;
if writer.write_all(&keepalive).await.is_err() {
break;
}
}
});
let first = Arc::new(AtomicU64::new(0));
let updates = Arc::new(AtomicU64::new(0));
let counting = tokio::spawn({
let (first, updates) = (Arc::clone(&first), Arc::clone(&updates));
async move {
let mut reader = BufReader::with_capacity(1 << 20, reader);
let mut header = [0u8; 19];
let mut body = vec![0u8; 4096];
while reader.read_exact(&mut header).await.is_ok() {
let length = usize::from(u16::from_be_bytes([header[16], header[17]]));
if reader.read_exact(&mut body[..length - 19]).await.is_err() {
break;
}
match header[18] {
msg_type::UPDATE => {
let _ = first.compare_exchange(
0,
started.elapsed().as_micros() as u64,
Ordering::Relaxed,
Ordering::Relaxed,
);
let n = updates.fetch_add(1, Ordering::Relaxed) + 1;
if n % 1_000_000 == 0 {
println!(
" the receiver has read {n} UPDATEs after {:.2} s",
started.elapsed().as_secs_f64()
);
}
}
msg_type::NOTIFICATION => println!(
"the receiver was sent a NOTIFICATION, error {} subcode {}, after {} UPDATEs",
body[0],
body[1],
updates.load(Ordering::Relaxed)
),
_ => {}
}
}
println!(
"the receiver's connection closed after {} UPDATEs",
updates.load(Ordering::Relaxed)
);
}
});
Self {
started,
first,
updates,
counting,
keepalives,
}
}
/// Waits until the router whose sessions are in `registry` has
/// advertised `routes` routes to the receiver, and returns how many it
/// has; or panics once `deadline` has passed.
async fn wait_for_advertised(
&self,
registry: &SessionRegistry,
routes: usize,
deadline: Duration,
) -> u64 {
loop {
tokio::time::sleep(POLL).await;
let advertised = registry
.list()
.iter()
.find(|s| s.peer.addr.ip() == RECEIVER)
.map_or(0, |s| s.counters.advertised());
if advertised as usize >= routes {
return advertised;
}
if self.started.elapsed() > deadline {
panic!(
"the receiver has {advertised} of {routes} routes and read {} UPDATEs; sessions at A: {:#?}",
self.updates.load(Ordering::Relaxed),
sessions_report(registry)
);
}
}
}
/// Prints the row for the table being all there, `advertised` routes
/// of it, and stops counting.
fn report(self, advertised: u64) {
let took = self.started.elapsed();
self.counting.abort();
self.keepalives.abort();
let first = Duration::from_micros(self.first.load(Ordering::Relaxed));
row(
"A new peer has A's IPv4 routes (A's exporter)",
took,
format!(
"{:.2} M routes/s first UPDATE after {:.0} ms, {} UPDATEs",
advertised as f64 / took.as_secs_f64() / 1e6,
first.as_secs_f64() * 1e3,
self.updates.load(Ordering::Relaxed),
),
);
}
}
/// A new peer takes the table of the router at `addr`, whose sessions
/// are in `registry`: `routes` routes with every path. How long until
/// the first UPDATE, and until all of it is sent.
async fn resync(addr: SocketAddr, registry: SessionRegistry, routes: usize) {
let socket = tokio::net::TcpSocket::new_v4().expect("a socket");
if let Err(e) = socket.bind(SocketAddr::new(RECEIVER.into(), 0)) {
println!("A new peer taking A's routes: skipped, {RECEIVER} is not ours ({e})");
async fn resync(addr: SocketAddr, registry: SessionRegistry, routes: usize, deadline: Duration) {
let Some(session) = Receiver::connect(addr).await else {
return;
}
let stream = socket.connect(addr).await.expect("A listens");
let mut session = TestPeer::from_stream(stream);
let mut capabilities = vec![Capability::FourByteAsn { asn: ASN }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: SUPPORTED
.iter()
.map(|family| AddPathEntry {
afi: family.afi,
safi: family.safi,
send_recv: SendReceive::Receive,
})
.collect(),
});
session
.handshake_with(ASN, Ipv4Addr::new(192, 0, 2, 9), 90, capabilities)
.await
.expect("A takes the receiver's session");
let started = Instant::now();
let (reader, mut writer) = session.into_stream().into_split();
// KEEPALIVEs, so that the session outlives the hold time.
let keepalives = tokio::spawn(async move {
let mut keepalive = [0xffu8; 19];
keepalive[16..19].copy_from_slice(&[0, 19, 4]);
loop {
tokio::time::sleep(Duration::from_secs(20)).await;
if writer.write_all(&keepalive).await.is_err() {
break;
}
}
});
// When the first UPDATE arrived, in microseconds; 0 until it has.
let first = Arc::new(AtomicU64::new(0));
let updates = Arc::new(AtomicU64::new(0));
let counting = tokio::spawn({
let (first, updates) = (Arc::clone(&first), Arc::clone(&updates));
async move {
let mut reader = BufReader::with_capacity(1 << 20, reader);
let mut header = [0u8; 19];
let mut body = vec![0u8; 4096];
while reader.read_exact(&mut header).await.is_ok() {
let length = usize::from(u16::from_be_bytes([header[16], header[17]]));
if reader.read_exact(&mut body[..length - 19]).await.is_err() {
break;
}
match header[18] {
// An UPDATE (RFC 4271 §4.1).
2 => {
let _ = first.compare_exchange(
0,
started.elapsed().as_micros() as u64,
Ordering::Relaxed,
Ordering::Relaxed,
);
let n = updates.fetch_add(1, Ordering::Relaxed) + 1;
if n % 1_000_000 == 0 {
println!(
" the receiver has read {n} UPDATEs after {:.2} s",
started.elapsed().as_secs_f64()
);
}
}
3 => println!(
"the receiver was sent a NOTIFICATION, error {} subcode {}, after {} UPDATEs",
body[0],
body[1],
updates.load(Ordering::Relaxed)
),
_ => {}
}
}
println!(
"the receiver's connection closed after {} UPDATEs",
updates.load(Ordering::Relaxed)
);
}
});
let deadline = Duration::from_secs(env("BENCH_DEADLINE", 600) as u64);
let advertised = loop {
tokio::time::sleep(Duration::from_millis(5)).await;
let advertised = registry
.list()
.iter()
.find(|s| s.peer.addr.ip() == RECEIVER)
.map_or(0, |s| s.counters.advertised());
if advertised as usize >= routes {
break advertised;
}
if started.elapsed() > deadline {
let sessions: Vec<String> = registry
.list()
.iter()
.map(|s| {
format!(
"{} {:?} export {:?} add_path_send {:?} advertised {}",
s.peer,
s.name,
s.export,
s.add_path_send,
s.counters.advertised()
)
})
.collect();
panic!(
"the receiver has {advertised} of {routes} routes and read {} UPDATEs; sessions at A: {sessions:#?}",
updates.load(Ordering::Relaxed)
);
}
};
let took = started.elapsed();
counting.abort();
keepalives.abort();
let first = Duration::from_micros(first.load(Ordering::Relaxed));
println!(
"{:<52} {:>8.3} s {:.2} M routes/s first UPDATE after {:.0} ms, {} UPDATEs",
"A new peer has A's IPv4 routes (A's exporter)",
took.as_secs_f64(),
advertised as f64 / took.as_secs_f64() / 1e6,
first.as_secs_f64() * 1e3,
updates.load(Ordering::Relaxed),
);
let receiver = Receiver::count_updates(session);
let advertised = receiver
.wait_for_advertised(&registry, routes, deadline)
.await;
receiver.report(advertised);
}
crates/bench/benches/real_table.rs
+6 -16
View File
@@ -11,30 +11,20 @@ use std::net::Ipv4Addr;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::time::{Duration, Instant};
use std::time::Instant;
use bgp_bench::memory::Region;
use bgp_bench::mrt::{Dump, parse_attributes};
use bgp_bench::report::{mib, per_second, row};
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_primitives::ip_prefix::IpPrefix;
use bgp_rib::table::{Event, PeerIndex, PeerInfo, RibTable};
/// Bytes allocated and not freed since `region` began; zero when memory
/// is not being counted (see `bgp_bench::memory`).
fn retained(region: &Region) -> i64 {
region.held().unwrap_or(0)
}
fn mib(bytes: i64) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
fn per_second(count: usize, elapsed: Duration) -> String {
format!("{:.2} M/s", count as f64 / elapsed.as_secs_f64() / 1e6)
}
fn row(what: &str, elapsed: Duration, detail: String) {
println!("{what:<44} {:>9.3} s {detail}", elapsed.as_secs_f64());
/// is not being counted (see `bgp_bench::memory`), and zero too in the
/// odd case of more freed than allocated, which is nothing held.
fn retained(region: &Region) -> usize {
usize::try_from(region.held().unwrap_or(0)).unwrap_or(0)
}
fn dump_path() -> Option<PathBuf> {
crates/bench/examples/feed.rs
+26 -68
View File
@@ -32,32 +32,18 @@
use std::net::{Ipv4Addr, SocketAddr};
use std::time::{Duration, Instant};
use bgp_bench::{Prefixes, Shape, announcements, encoded, env};
use bgp_integration_tests::test_peer::TestPeer;
use bgp_packet::open::{AddPathEntry, Capability, ExtendedNextHopEntry, SendReceive};
use bgp_primitives::constants::{Afi, AfiSafi, Safi};
use bgp_server::family::SUPPORTED;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use bgp_bench::report::mib;
use bgp_bench::{Prefixes, Shape, env, table_stream};
use bgp_integration_tests::test_peer::{TestPeer, collector_capabilities};
use bgp_primitives::constants::AfiSafi;
use tokio::io::AsyncWriteExt;
use tokio::net::TcpSocket;
use tokio::net::tcp::OwnedWriteHalf;
/// A KEEPALIVE message: the marker, the length (19) and the type (4).
const KEEPALIVE: [u8; 19] = {
let mut bytes = [0xff; 19];
bytes[16] = 0;
bytes[17] = 19;
bytes[18] = 4;
bytes
};
/// How often KEEPALIVEs go out while the sessions are held open: well
/// inside the 90 s hold time.
const KEEPALIVE_EVERY: Duration = Duration::from_secs(20);
fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
#[tokio::main]
async fn main() {
let to: SocketAddr = std::env::var("FEED_TO")
@@ -73,12 +59,7 @@ async fn main() {
let started = Instant::now();
let prefixes = Prefixes::generate(shape);
let tables: Vec<Vec<u8>> = (1..=peers)
.map(|peer| {
encoded(&announcements(&prefixes, shape, peer))
.iter()
.flat_map(|message| message.iter().copied())
.collect()
})
.map(|peer| table_stream(&prefixes, shape, peer))
.collect();
let routes = shape.prefixes() * peers as usize;
let bytes: usize = tables.iter().map(Vec::len).sum();
@@ -109,38 +90,21 @@ async fn main() {
}
let stream = socket.connect(to).await.expect("the daemon listens");
let mut session = TestPeer::from_stream(stream);
let mut capabilities = vec![Capability::FourByteAsn { asn: 212_232 }];
capabilities.extend(SUPPORTED.into_iter().map(Capability::multiprotocol));
capabilities.push(Capability::AddPath {
entries: [AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST]
.into_iter()
.map(|family| AddPathEntry {
afi: family.afi,
safi: family.safi,
send_recv: SendReceive::Receive,
})
.collect(),
});
capabilities.push(Capability::ExtendedNextHop {
entries: vec![ExtendedNextHopEntry {
afi: Afi::IPV4,
safi: Safi::UNICAST,
nh_afi: Afi::IPV6,
}],
});
session
.handshake_with(212_232, Ipv4Addr::new(10, 0, 0, 200), 90, capabilities)
.handshake_with(
212_232,
Ipv4Addr::new(10, 0, 0, 200),
90,
collector_capabilities(
212_232,
&[AfiSafi::IPV4_UNICAST, AfiSafi::IPV6_UNICAST],
true,
),
)
.await
.expect("the daemon takes the collector's session");
let (mut reader, writer) = session.into_stream().into_split();
draining.push(tokio::spawn(async move {
let mut sink = [0u8; 65536];
while let Ok(n) = reader.read(&mut sink).await {
if n == 0 {
break;
}
}
}));
let (writer, sink) = session.drain();
draining.push(sink);
println!("collector session established");
collector_writer = Some(writer);
}
@@ -166,7 +130,11 @@ async fn main() {
)
.await
.expect("the daemon takes the session");
sessions.push(session.into_stream());
// Whatever the daemon sends back, KEEPALIVEs as a rule, is read
// and dropped, so that the writer is never held up.
let (writer, sink) = session.drain();
draining.push(sink);
sessions.push(writer);
}
println!("{peers} session(s) established; sending");
@@ -174,19 +142,8 @@ async fn main() {
let senders: Vec<_> = sessions
.into_iter()
.zip(tables)
.map(|(stream, table)| {
.map(|(mut writer, table)| {
tokio::spawn(async move {
let (mut reader, mut writer) = stream.into_split();
// Whatever the daemon sends back, KEEPALIVEs as a rule, is
// read and dropped, so that its writer is never held up.
tokio::spawn(async move {
let mut sink = [0u8; 65536];
while let Ok(n) = reader.read(&mut sink).await {
if n == 0 {
break;
}
}
});
let started = Instant::now();
writer.write_all(&table).await.expect("the daemon reads");
(writer, started.elapsed())
@@ -210,12 +167,13 @@ async fn main() {
routes as f64 / elapsed.as_secs_f64() / 1e6
);
let keepalive = TestPeer::keepalive_bytes();
let deadline = Instant::now() + hold;
while Instant::now() < deadline {
tokio::time::sleep(KEEPALIVE_EVERY.min(deadline - Instant::now())).await;
for writer in &mut writers {
// A session the daemon closed is left alone.
let _ = writer.write_all(&KEEPALIVE).await;
let _ = writer.write_all(&keepalive).await;
}
}
println!("sessions closed after {} s", hold.as_secs());
crates/bench/src/lib.rs
+76
View File
@@ -16,6 +16,7 @@
pub mod memory;
pub mod mrt;
pub mod report;
// The benchmarks allocate as the daemon does (bin/bgp/src/main.rs), with
// jemalloc; the C library's allocator would put its own locks in the
@@ -266,6 +267,16 @@ pub fn announcements(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<Update
updates
}
/// The whole table as the peer numbered `peer` sends it, one stream of
/// bytes: [`announcements`], [`encoded`] and run together, for a session
/// that writes the table in one go.
pub fn table_stream(prefixes: &Prefixes, shape: Shape, peer: u32) -> Vec<u8> {
encoded(&announcements(prefixes, shape, peer))
.iter()
.flat_map(|message| message.iter().copied())
.collect()
}
/// `updates` as they are on the wire, on a session with four-octet AS
/// numbers.
pub fn encoded(updates: &[UpdateMessage]) -> Vec<BytesMut> {
@@ -300,3 +311,68 @@ pub fn env(name: &str, default: usize) -> usize {
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
#[cfg(test)]
mod tests {
use super::*;
/// A small table, enough for every prefix length and attribute to
/// come up.
fn shape() -> Shape {
Shape::INTERNET.scaled_down(1000)
}
/// The benchmarks are compared from one run to the next, so the table
/// they are fed must not change between runs: the generator is
/// seeded, and nothing in it depends on where things are in memory.
#[test]
fn the_table_is_the_same_every_run() {
let shape = shape();
let once = Prefixes::generate(shape);
let again = Prefixes::generate(shape);
assert_eq!(once.ipv4, again.ipv4);
assert_eq!(once.ipv6, again.ipv6);
assert_eq!(once.ipv4.len(), shape.ipv4_prefixes);
assert_eq!(once.ipv6.len(), shape.ipv6_prefixes);
for peer in [1, 2] {
assert_eq!(
table_stream(&once, shape, peer),
table_stream(&again, shape, peer)
);
}
assert_ne!(
table_stream(&once, shape, 1),
table_stream(&once, shape, 2),
"each peer has attributes of its own"
);
}
/// What a peer of the benchmarks sends is well formed, and says what
/// it was meant to: every UPDATE reads back as it was made.
#[test]
fn every_announcement_decodes_to_itself() {
let shape = shape();
let prefixes = Prefixes::generate(shape);
let updates = announcements(&prefixes, shape, 1);
let ctx = wire_context();
let announced: usize = updates
.iter()
.map(|update| {
update.announced_routes.len()
+ update
.path_attributes
.iter()
.map(|attribute| match attribute {
PathAttribute::MpReachNlri(reach) => reach.prefixes.len(),
_ => 0,
})
.sum::<usize>()
})
.sum();
assert_eq!(announced, shape.prefixes());
for (update, wire) in updates.iter().zip(encoded(&updates)) {
let decoded = Message::decode(&ctx, &wire).expect("well formed");
assert_eq!(decoded, Message::Update(update.clone()));
}
}
}
crates/bench/src/mrt.rs
+16 -26
View File
@@ -10,7 +10,7 @@ use std::io::{self, BufReader, Read};
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
use std::path::Path;
use bgp_packet::message::{NlriNextHop, PathAttribute};
use bgp_packet::message::{NlriNextHop, PathAttribute, attr_type, split_attribute};
use bgp_primitives::constants::IpVersion;
use bgp_primitives::ip_prefix::IpPrefix;
use flate2::read::MultiGzDecoder;
@@ -21,7 +21,6 @@ const TABLE_DUMP_V2: u16 = 13;
const PEER_INDEX_TABLE: u16 = 1;
const RIB_IPV4_UNICAST: u16 = 2;
const RIB_IPV6_UNICAST: u16 = 4;
const MP_REACH_NLRI: u8 = 14;
/// A peer of the collector.
#[derive(Debug, Clone)]
@@ -205,32 +204,23 @@ pub fn parse_attributes(mut wire: &[u8]) -> Option<(Vec<PathAttribute>, Option<N
let mut attributes = Vec::new();
let mut next_hop = None;
while !wire.is_empty() {
if wire.get(1) == Some(&MP_REACH_NLRI) {
let extended = wire[0] & 0x10 != 0;
let (length, header) = if extended {
(
usize::from(u16::from_be_bytes([*wire.get(2)?, *wire.get(3)?])),
4,
)
} else {
(usize::from(*wire.get(2)?), 3)
let (raw, rest) = split_attribute(wire).ok()?;
if raw.type_code == attr_type::MP_REACH_NLRI {
// The value is the next hop's length and the next hop, and
// nothing of the family, reserved octet and prefixes that
// follow in an UPDATE, so the attribute's own parser would
// refuse it.
let (length, address) = raw.value.split_first()?;
if usize::from(*length) != address.len() {
return None;
}
next_hop = Some(NlriNextHop::from_bytes(address)?);
} else {
let ([], attribute) = PathAttribute::from_wire(&ctx, raw.wire).ok()? else {
return None;
};
let value = wire.get(header..header + length)?;
let address = |bytes: &[u8]| Some(Ipv6Addr::from(<[u8; 16]>::try_from(bytes).ok()?));
next_hop = match value {
[4, a, b, c, d] => Some(NlriNextHop::Ipv4(Ipv4Addr::new(*a, *b, *c, *d))),
[16, rest @ ..] => Some(NlriNextHop::Ipv6(address(rest)?)),
[32, rest @ ..] if rest.len() == 32 => Some(NlriNextHop::Ipv6WithLl {
global: address(&rest[..16])?,
link_local: address(&rest[16..])?,
}),
_ => return None,
};
wire = &wire[header + length..];
continue;
attributes.push(attribute);
}
let (rest, attribute) = PathAttribute::from_wire(&ctx, wire).ok()?;
attributes.push(attribute);
wire = rest;
}
Some((attributes, next_hop))
crates/bench/src/report.rs
+31
View File
@@ -0,0 +1,31 @@
//! How the benchmarks print what they measured: one row a measurement,
//! what was measured in a column wide enough for the longest of them, the
//! time, and whatever else there is to say about it. The rows are what a
//! run is compared against, so their shape is settled here once.
use std::time::Duration;
/// How wide the column that says what was measured is. The widest row is
/// the mesh benchmark's, so that is what every benchmark lines up to.
pub const WIDTH: usize = 52;
/// `bytes` in mebibytes.
pub fn mib(bytes: usize) -> f64 {
bytes as f64 / (1024.0 * 1024.0)
}
/// `count` things in `elapsed`, as a rate: millions a second where there
/// are that many, thousands otherwise.
pub fn per_second(count: usize, elapsed: Duration) -> String {
let rate = count as f64 / elapsed.as_secs_f64();
if rate >= 1e6 {
format!("{:.2} M/s", rate / 1e6)
} else {
format!("{:.0} k/s", rate / 1e3)
}
}
/// Prints one row: `what` took `elapsed`, and `detail` says more.
pub fn row(what: &str, elapsed: Duration, detail: impl std::fmt::Display) {
println!("{what:<WIDTH$} {:>8.3} s {detail}", elapsed.as_secs_f64());
}