ferrix/docs/content/docs/development/testing.md

+++ title = "Testing" description = "Describes the Ferrix test suites: unit and integration tests, property tests, kernel tests, the interoperability lab, fuzz testing, and benchmarks." weight = 2 +++

Ferrix is tested at several levels, from individual functions to networks of routers. Continuous integration runs all levels on every push and fails on any compiler warning. Table 1 summarizes the test suites.

Suite	Scope	Command
Unit and integration tests	All crates, and the server against in-process peers	cargo test --workspace
Property tests	Encoding and decoding of generated messages	Included in cargo test
Kernel tests	Route installation in a Linux kernel, in network namespaces	scripts/kernel-tests.sh
Interoperability lab	The daemon with FRR, BIRD, and Routinator	scripts/interop/lab.sh
Fuzz tests	All parsers of external input	cargo fuzz run <target>
Benchmarks	Route processing performance	cargo bench -p bgp-bench

Unit and Integration Tests

To run the unit and integration tests:

cargo test --workspace

The crates/integration-tests crate runs the server in-process against test peers that exchange BGP messages over sockets. It covers session establishment, import policy, RPKI validation, BGP roles, route advertisement, connection collisions, and prefix limits. The BGP finite state machine has no input or output, so its state transitions are tested with direct function calls.

Property Tests

The property tests verify that generated messages are decoded to the same value after encoding, and that no input causes a parser to panic. They run as part of cargo test in approximately 1 second. To run more cases:

PROPTEST_CASES=1000000 cargo test --release -p bgp-packet --test properties

Kernel Tests

The tests that modify a kernel routing table are not run by cargo test. The script scripts/kernel-tests.sh runs them on Linux, each in its own network namespace. The script scripts/in-namespace.sh runs a single command in the same way. The kernel tests include:

• Route installation and removal, verified with ip route.
• The daemon with a BGP peer.
• Two daemons that exchange routes, verified with ping (scripts/two-routers.sh).

On macOS, the script scripts/linux-vm.sh runs the kernel tests, or any other command, in a Lima virtual machine. The script creates the virtual machine on first use.

scripts/linux-vm.sh                  # kernel tests
scripts/linux-vm.sh cargo test       # all tests, on Linux
scripts/linux-vm.sh bash             # interactive shell

Interoperability Lab

The script scripts/interop/lab.sh tests the daemon with FRR, BIRD, and Routinator. Each router runs in its own network namespace, connected to a shared exchange LAN. Routinator serves RPKI data defined for the lab, and does not retrieve data from the Internet.

The lab requires Linux, sudo, and a release build of the daemon. The script scripts/interop/install.sh installs FRR, BIRD, radvd, and Routinator on Ubuntu.

cargo build --release -p bgp
scripts/interop/lab.sh                          # all phases
PHASES="basics transit" scripts/interop/lab.sh  # selected phases
scripts/interop/lab.sh up                       # set up the lab and leave it running

On macOS, prefix each command with scripts/linux-vm.sh. The lab was last run with FRR 10.5.1, BIRD 2.18, and Routinator 0.15.2. Table 2 lists the test phases.

Phase	Verified behavior
basics	IPv4 and IPv6 sessions; routes in both directions with AS path, next hop, communities, and large communities preserved; four-octet AS numbers; ADD-PATH from BIRD; forwarding with ping over the installed routes
roles	BGP role negotiation with FRR (peer) and BIRD (provider); OTC attribute added and verified in both directions
rpki	RTR version 2 with Routinator; an RPKI invalid route from FRR is rejected and held, and accepted after Routinator restarts with data that authorizes it
refresh	Routes added and withdrawn by FRR; ROUTE-REFRESH from FRR; route readvertisement by BIRD
flaps	Five session resets; FRR and BIRD terminated and restarted; link failure detected by the hold timer
restart	Daemon stopped normally (kernel routes removed, neighbors notified) and terminated (routes retained, stale route removed at the next start)
scale	A routing table from BIRD (SCALE_ROUTES; 1 million routes for a full table) installed in the routing table and the kernel; daemon terminated and restarted under load; table withdrawn
limits	Session rejected on role mismatch; Cease NOTIFICATION with Maximum Number of Prefixes Reached, and hold-down
hijacks	FRR advertises a local prefix, a route with BIRD as third-party next hop, and the exchange LAN prefix; all are rejected
nexthops	A preferred route from FRR with an unresolvable next hop is held, and the route from BIRD is selected and installed; the selection changes when the next hop becomes resolvable, and reverts when it becomes unresolvable
transit	FRR as customer and BIRD as provider: FRR's routes are advertised to BIRD with the local AS prepended; BIRD's routes are advertised to FRR with the OTC attribute; a control community prevents the advertisement of one FRR route to BIRD; traffic is forwarded between their networks
v6only	Exchange LAN without IPv4 addresses: one IPv6 session with each router for both address families; IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel; IPv4 forwarding verified with ping
linklocal	Exchange LAN with link-local addresses only, with the same addresses on a second interface: sessions with FRR (by interface, with radvd) and BIRD at fe80::…%ix0; both address families in BGP and in the kernel; forwarding verified with ping; interface disabled and enabled

The script scripts/interop/real-rpki.sh tests the RTR client against a Routinator instance with the global RPKI data, including ASPAs. The test verifies that the daemon receives the same number of VRPs and ASPAs that the validator reports.

The lab does not cover routers other than FRR and BIRD, IPv6 routes over IPv4 sessions (which Ferrix does not advertise), an interface that is deleted and recreated during a session, or long-running operation.

Fuzz Testing

The fuzz/ directory contains coverage-guided fuzz targets for all parsers of external input. Table 3 lists the targets.

Target	Input
message	Raw bytes received on a BGP connection
message_body	A message body with a valid header, to reach the body parsers
attrs	Path attribute bytes stored in the routing table
prefix_text	A prefix in text form
config	The configuration file
rtr	A stream of RTR PDUs from an RPKI cache
netlink	Messages from the kernel routing socket

For the message and message_body targets, a failure is either a panic or a message that is decoded but cannot be encoded and decoded back to the same value.

Fuzz testing requires a nightly toolchain and cargo-fuzz:

rustup toolchain install nightly
cargo install cargo-fuzz
cd fuzz
cargo +nightly fuzz run message_body corpus/message_body seeds/message_body -- -max_total_time=300

The first directory is the working corpus, which is not stored in the repository. The seeds/ directory contains valid inputs. Continuous integration runs each target for 1 minute.

When a target fails, the input is saved in artifacts/<target>/. To reproduce the failure, run cargo +nightly fuzz run <target> <file>. After you fix the defect, add a unit test with the input next to the affected code.

Benchmarks

Table 4 lists the benchmarks.

Benchmark	Measurement
full_table	Processing of a full routing table, from received bytes to best path selection. BENCH_PEERS sets the number of neighbors (default: 4); BENCH_SCALE_DOWN reduces the table size.
transit	Route advertisement: one neighbor advertises the table, and a second neighbor receives every route. The time is measured from the first byte sent to the last route received.
real_table	The full_table measurement on a RIPE RIS routing table dump. See crates/bench/data/README.md for how to obtain a dump.

cargo bench -p bgp-bench --bench full_table
cargo bench -p bgp-bench --bench transit
cargo bench -p bgp-bench --bench real_table

The full_table and transit benchmarks use a synthetic routing table, so that they require no data files and produce reproducible results. The table has the characteristics of the Internet routing table that affect performance: approximately 1 million IPv4 prefixes, mostly /24, and 250,000 IPv6 prefixes, mostly /48; prefixes with shared attributes grouped in UPDATE messages; AS paths of several hops; and communities on most routes.

Table 5 lists full_table results on an Apple M-series computer with a release build: 1.23 million prefixes from each of four neighbors, 4.92 million routes.

Measurement	Result
Decoding the UPDATE messages, one thread	0.43 seconds, 11 million routes per second
Decoding, interning attributes, inserting and selecting, one thread	1.9 seconds, 2.5 million routes per second
The same, 12 threads	0.7 seconds, 6.9 million routes per second
Four neighbors send the table over TCP to the in-process server	0.9 seconds, 5.2 million routes per second
Memory per route, attribute sets included (--features count-memory)	77 bytes

Profiling

The scripts/profile/full-table.sh script measures the daemon itself, as built for production, while one or more neighbors send it a full routing table, and records where its CPU time goes. It requires Linux with perf, inferno (cargo install inferno), curl, jq, and sudo. On another operating system, run it in the Linux virtual machine:

scripts/linux-vm.sh scripts/profile/full-table.sh

The script builds the daemon with the profiling Cargo profile, which is the release profile with line tables, and with frame pointers, so that perf can attribute samples to functions. It starts the daemon in a network namespace of its own, connected by a virtual Ethernet pair to a second namespace in which the load generator (crates/bench/examples/feed.rs) connects as the neighbors and sends the synthetic table. Routes that the daemon installs go into the first namespace's routing table and do not affect the host. Table 6 lists the environment variables that control the run.

Variable	Effect	Default
PEERS	The number of neighbors that send the table	1
KERNEL	1 installs the best routes in the kernel; 0 keeps them in the routing table only	1
SCALE_DOWN	Divides the size of the table	1
HOLD_SECS	How long the sessions stay open after the table is sent	30
FREQ	The sampling frequency in hertz	1999
OUT	The directory for the results	~/profile

When run through scripts/linux-vm.sh, pass the variables with env, because the virtual machine does not receive the host's environment:

scripts/linux-vm.sh env PEERS=4 KERNEL=0 scripts/profile/full-table.sh

The script prints how long the routes take to arrive in the routing table and in the kernel, and the daemon's resident memory. It writes the following files to the results directory:

File	Contents
flame.svg	An interactive flame graph of the daemon's threads
top-self.txt	Functions by the time spent in their own code
top-children.txt	Functions by the time spent in them and everything they call
stacks.folded	The collapsed stacks the flame graph is drawn from
perf.data, daemon.log, feed.log	The raw samples and the logs

Table 7 lists results in the Linux virtual machine (8 virtual CPUs on an Apple M-series computer) with the synthetic table of 1.23 million prefixes.

Measurement	Result
One neighbor; routes in the routing table	0.7 seconds, 1.7 million routes per second
One neighbor; the best routes installed in the kernel	2.0 seconds after the first route
One neighbor, without the kernel; routes in the routing table	0.6 seconds, 2.0 million routes per second
Four neighbors; 4.92 million routes in the routing table	1.1 seconds, 4.5 million routes per second
Four neighbors; the best routes installed in the kernel	3.0 seconds after the first route
Resident memory with one neighbor's table installed	520 MB
Resident memory with four neighbors' tables installed	680 MB

A session is processed on one thread, so one neighbor's table arrives at the rate of one CPU core, and several neighbors' tables arrive in parallel. Installing routes in the kernel takes longer than receiving them, and continues after the last route has arrived.