ferrix/docs/content/docs/development/testing.md

+++
title = "Testing"
description = "Describes the Ferrix test suites: unit and integration tests, property tests, kernel tests, the interoperability lab, fuzz testing, and benchmarks."
weight = 2
+++

Ferrix is tested at several levels, from individual functions to networks
of routers. Continuous integration runs all levels on every push and fails
on any compiler warning. Table 1 summarizes the test suites.

| Suite | Scope | Command |
|---|---|---|
| Unit and integration tests | All crates, and the server against in-process peers | `cargo test --workspace` |
| Property tests | Encoding and decoding of generated messages | Included in `cargo test` |
| Kernel tests | Route installation in a Linux kernel, in network namespaces | `scripts/kernel-tests.sh` |
| Interoperability lab | The daemon with FRR, BIRD, and Routinator | `scripts/interop/lab.sh` |
| Fuzz tests | All parsers of external input | `cargo fuzz run <target>` |
| Benchmarks | Route processing performance | `cargo bench -p bgp-bench` |

## Unit and Integration Tests

To run the unit and integration tests:

```bash
cargo test --workspace
```

The `crates/integration-tests` crate runs the server in-process against
test peers that exchange BGP messages over sockets. It covers session
establishment, import policy, RPKI validation, BGP roles, route
advertisement, connection collisions, and prefix limits. The BGP finite
state machine has no input or output, so its state transitions are tested
with direct function calls.

## Property Tests

The property tests verify that generated messages are decoded to the same
value after encoding, and that no input causes a parser to panic. They run
as part of `cargo test` in approximately 1 second. To run more cases:

```bash
PROPTEST_CASES=1000000 cargo test --release -p bgp-packet --test properties
```

## Kernel Tests

The tests that modify a kernel routing table are not run by `cargo test`.
The script `scripts/kernel-tests.sh` runs them on Linux, each in its own
network namespace. The script `scripts/in-namespace.sh` runs a single
command in the same way. The kernel tests include:

- Route installation and removal, verified with `ip route`.
- The daemon with a BGP peer.
- Two daemons that exchange routes, verified with `ping`
  (`scripts/two-routers.sh`).

On macOS, the script `scripts/linux-vm.sh` runs the kernel tests, or any
other command, in a [Lima](https://lima-vm.io) virtual machine. The script
creates the virtual machine on first use.

```bash
scripts/linux-vm.sh                  # kernel tests
scripts/linux-vm.sh cargo test       # all tests, on Linux
scripts/linux-vm.sh bash             # interactive shell
```

## Interoperability Lab

The script `scripts/interop/lab.sh` tests the daemon with FRR, BIRD, and
Routinator. Each router runs in its own network namespace, connected to a
shared exchange LAN. Routinator serves RPKI data defined for the lab, and
does not retrieve data from the Internet.

The lab requires Linux, `sudo`, and a release build of the daemon. The
script `scripts/interop/install.sh` installs FRR, BIRD, radvd, and
Routinator on Ubuntu.

```bash
cargo build --release -p bgp
scripts/interop/lab.sh                          # all phases
PHASES="basics transit" scripts/interop/lab.sh  # selected phases
scripts/interop/lab.sh up                       # set up the lab and leave it running
```

On macOS, prefix each command with `scripts/linux-vm.sh`. The lab was last
run with FRR 10.5.1, BIRD 2.18, and Routinator 0.15.2. Table 2 lists the
test phases.

| Phase | Verified behavior |
|---|---|
| `basics` | IPv4 and IPv6 sessions; routes in both directions with AS path, next hop, communities, and large communities preserved; four-octet AS numbers; ADD-PATH from BIRD; forwarding with `ping` over the installed routes |
| `roles` | BGP role negotiation with FRR (peer) and BIRD (provider); OTC attribute added and verified in both directions |
| `rpki` | RTR version 2 with Routinator; an RPKI invalid route from FRR is rejected and held, and accepted after Routinator restarts with data that authorizes it |
| `refresh` | Routes added and withdrawn by FRR; ROUTE-REFRESH from FRR; route readvertisement by BIRD |
| `flaps` | Five session resets; FRR and BIRD terminated and restarted; link failure detected by the hold timer |
| `restart` | Daemon stopped normally (kernel routes removed, neighbors notified) and terminated (routes retained, stale route removed at the next start) |
| `scale` | A routing table from BIRD (`SCALE_ROUTES`; 1 million routes for a full table) installed in the routing table and the kernel; daemon terminated and restarted under load; table withdrawn |
| `limits` | Session rejected on role mismatch; Cease NOTIFICATION with Maximum Number of Prefixes Reached, and hold-down |
| `hijacks` | FRR advertises a local prefix, a route with BIRD as third-party next hop, and the exchange LAN prefix; all are rejected |
| `nexthops` | A preferred route from FRR with an unresolvable next hop is held, and the route from BIRD is selected and installed; the selection changes when the next hop becomes resolvable, and reverts when it becomes unresolvable |
| `transit` | FRR as customer and BIRD as provider: FRR's routes are advertised to BIRD with the local AS prepended; BIRD's routes are advertised to FRR with the OTC attribute; a control community prevents the advertisement of one FRR route to BIRD; traffic is forwarded between their networks |
| `v6only` | Exchange LAN without IPv4 addresses: one IPv6 session with each router for both address families; IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel; IPv4 forwarding verified with `ping` |
| `linklocal` | Exchange LAN with link-local addresses only, with the same addresses on a second interface: sessions with FRR (by interface, with radvd) and BIRD at `fe80::…%ix0`; both address families in BGP and in the kernel; forwarding verified with `ping`; interface disabled and enabled |

The script `scripts/interop/real-rpki.sh` tests the RTR client against a
Routinator instance with the global RPKI data, including ASPAs. The test
verifies that the daemon receives the same number of VRPs and ASPAs that
the validator reports.

The lab does not cover routers other than FRR and BIRD, IPv6 routes over
IPv4 sessions (which Ferrix does not advertise), an interface that is
deleted and recreated during a session, or long-running operation.

## Fuzz Testing

The `fuzz/` directory contains coverage-guided fuzz targets for all parsers
of external input. Table 3 lists the targets.

| Target | Input |
|---|---|
| `message` | Raw bytes received on a BGP connection |
| `message_body` | A message body with a valid header, to reach the body parsers |
| `attrs` | Path attribute bytes stored in the routing table |
| `prefix_text` | A prefix in text form |
| `config` | The configuration file |
| `rtr` | A stream of RTR PDUs from an RPKI cache |
| `netlink` | Messages from the kernel routing socket |

For the `message` and `message_body` targets, a failure is either a panic or
a message that is decoded but cannot be encoded and decoded back to the
same value.

Fuzz testing requires a nightly toolchain and `cargo-fuzz`:

```bash
rustup toolchain install nightly
cargo install cargo-fuzz
cd fuzz
cargo +nightly fuzz run message_body corpus/message_body seeds/message_body -- -max_total_time=300
```

The first directory is the working corpus, which is not stored in the
repository. The `seeds/` directory contains valid inputs. Continuous
integration runs each target for 1 minute.

When a target fails, the input is saved in `artifacts/<target>/`. To
reproduce the failure, run `cargo +nightly fuzz run <target> <file>`. After
you fix the defect, add a unit test with the input next to the affected
code.

## Benchmarks

Table 4 lists the benchmarks.

| Benchmark | Measurement |
|---|---|
| `full_table` | Processing of a full routing table, from received bytes to best path selection. `BENCH_PEERS` sets the number of neighbors (default: 4); `BENCH_SCALE_DOWN` reduces the table size. |
| `transit` | Route advertisement: one neighbor advertises the table, and a second neighbor receives every route. The time is measured from the first byte sent to the last route received. |
| `real_table` | The `full_table` measurement on a RIPE RIS routing table dump. See `crates/bench/data/README.md` for how to obtain a dump. |

```bash
cargo bench -p bgp-bench --bench full_table
cargo bench -p bgp-bench --bench transit
cargo bench -p bgp-bench --bench real_table
```

The `full_table` and `transit` benchmarks use a synthetic routing table, so
that they require no data files and produce reproducible results. The
table has the characteristics of the Internet routing table that affect
performance: approximately 1 million IPv4 prefixes, mostly /24, and
250,000 IPv6 prefixes, mostly /48; prefixes with shared attributes grouped
in UPDATE messages; AS paths of several hops; and communities on most
routes.

Table 5 lists results on an Apple M-series computer with a release build.

| Measurement | Result |
|---|---|
| 1.23 million routes from one neighbor, accepted into the routing table | 0.47 seconds |
| The same routes received by a second neighbor (`transit`) | 2.1 seconds, approximately 4 routes per UPDATE message |
| 1 million IPv4 routes installed in the kernel (Linux virtual machine) | Slightly more than 1 second |
| The same routes removed from the kernel | 0.5 seconds |