Pass learned routes on to peers, with marks and control communities

What was to be scripted is configuration, done natively:

- import { add_communities, add_large_communities, remove_communities,
  local_pref }: every route taken from a peer is marked (where it was
  learned, say), with nobody else's word for our marks taken, and given
  its preference, which selection goes by.
- export { add_communities, add_large_communities, remove_communities,
  as_path_prepend, learned_routes }, for a peer or for all that have none
  of their own.
- control_communities: the communities by which routes say which peers
  they are passed on to, as exchanges' route servers have them (0:P, L:P,
  0:L, and the large forms L:0:P, L:1:P, L:0:0), taken off as the route
  goes on.

And what the last of these is for: with export.learned_routes a peer is
told the best route of every prefix the others announced. The rules are
BGP's and cannot be turned off: not back where it came from, not to a
peer whose AS is on the path, not between two internal peers, not past
NO_EXPORT or NO_ADVERTISE; our AS on the path and ourselves as next hop
to another network, LOCAL_PREF to our own; to a route server's client
the path, MED and next hop as they came (RFC 7947). Which routes go where
follows from the roles of RFC 9234: a route that came down or across goes
on down only, so that with roles on every peer this makes us transit for
our customers and nothing else. A peer without a role is told everything,
and the daemon says so at startup.

Each session follows the table's best routes with a reader and keeps the
prefixes it has announced, so that it can withdraw them. Routes that go
out alike go in one UPDATE: changes are gathered for 50 ms first, and a
table that is given out anew is gathered whole (at most four sessions at
a time; it is tens of megabytes). For that to work the table's journal
had to give out together what was written together: every change now
carries a time on a table-wide clock, a reader's pass reads every shard
up to one time, no further than the changes that are complete in every
shard, and a resync takes its places in the journals as of one time.

One peer sending 1.23 M routes over a session, another told of them:
in our table after 0.47 s, all heard after 2.1 s, four routes an UPDATE.
The lab has a phase, transit: FRR as our customer and BIRD as our
provider, routes each way with the right paths and marks, one kept back
by a community, and pings between their networks through us.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

README.md

@@ -6,7 +6,7 @@ The aim of this project is to provide a modern, high performance, secure, progra
 
 ## Current status
 
-The daemon holds sessions with configured peers, takes their routes through an import policy, validates them against the RPKI (route origin validation and ASPA), picks the best route for each prefix and, on Linux, installs it in the kernel. It announces the routes it originates. It does not yet pass routes learned from one peer on to another.
+The daemon holds sessions with configured peers, takes their routes through an import policy, validates them against the RPKI (route origin validation and ASPA), picks the best route for each prefix and, on Linux, installs it in the kernel. It announces the routes it originates and, to the peers configured for it, the best of what it learned from the others, as far as RFC 9234's roles and the routes' communities allow.
 
 - [`bin/bgp/README.md`](bin/bgp/README.md): what the daemon does, how to configure and run it, and how it is tested against FRR, BIRD and Routinator.
 - [`docs/RFCs.md`](docs/RFCs.md): which parts of which RFCs are implemented.
@@ -16,7 +16,7 @@ It has not yet run in a production network.
 
 ### Not there yet
 
-- Announcing learned routes to other peers, and an export policy for it
+- Route reflection, per-client selection for a route server, ADD-PATH for learned routes
 - TCP-AO / TCP MD5 and GTSM
 - Graceful restart, BFD, changing the configuration without a restart
 - Next hops that are not directly connected

bin/bgp/README.md

@@ -42,8 +42,8 @@ connections, with RFC 4271 collision detection), announces its originated
 routes to every peer with itself as next hop, judges what peers send by each
 peer's import policy, keeps what passes, chooses a best route for each
 prefix, and on Linux can put those best routes in the kernel so that the
-machine forwards by them. It does not yet pass routes learned from one peer
-on to another, and has no export policy.
+machine forwards by them. Peers whose `export` says so are told the best of
+what the others announced, as the roles of RFC 9234 allow.
 
 ## Who may connect
 
@@ -89,6 +89,88 @@ A refused route is dropped, along with the route it would have replaced.
 `bgp_routes_rejected_total` counts them by reason, and the status page shows
 how many routes each peer has and how many were refused.
 
+## Passing routes on
+
+A peer whose `export` has `learned_routes: true` is told the best route of
+every prefix the others announced, not only what we originate. The rules
+are the usual ones and cannot be turned off: not back to the peer a route
+came from, not to a peer whose AS is on its path already, not from one
+peer in our own AS to another, not past NO_EXPORT or NO_ADVERTISE. A
+route goes out with our AS added to its path and ourselves as next hop,
+without the MED it came with; to a peer in our own AS the path is as it
+was and LOCAL_PREF goes along; to a `ROUTE_SERVER_CLIENT` path, MED and
+(where the client can use it) next hop are left as they came, as a route
+server sends them (RFC 7947).
+
+**What goes where is decided by roles (RFC 9234).** A route learned from a
+provider, a peer or a route server is marked "only to customer" on the way
+in and goes on to customers only; a customer's route goes to everyone. So
+with roles on every peer, `learned_routes` makes us transit for our
+customers and nothing else. A peer without a role has no such rule, and is
+told every route; the daemon warns about that at startup. The top-level
+`export` is for every peer that has none of its own:
+
+```
+export { learned_routes: true }
+peers { name: "customer-a" role: CUSTOMER ... }
+peers { name: "transit-a" role: PROVIDER ... }
+```
+
+**Marks.** `import` puts communities on every route taken from a peer, and
+takes some off first; `export` does the same for every route announced to
+one. Where a route was learned, which router it went through:
+
+```
+peers {
+  name: "ixp-a-rs"
+  import {
+    remove_communities: "64601:*"        # nobody else's word for our marks
+    add_communities: "64601:1001"        # learned at IXP A
+    add_large_communities: "64601:1:1001"
+    local_pref: 200
+  }
+}
+export { add_large_communities: "64601:2:1" }   # went through router 1
+```
+
+Patterns for `remove_communities` are `64601:100`, `64601:*`, `*:*`, or
+with three parts for large communities. `import.local_pref` is what route
+selection goes by: of two routes for a prefix the one with the higher is
+chosen, whatever else they have.
+
+**Control communities.** With `control_communities: true`, a route's
+communities say which peers it is passed on to, in the form Internet
+exchanges' route servers use (`L` our AS, `P` a peer's):
+
+| Standard | Large | |
+|---|---|---|
+| `0:P` | `L:0:P` | not to the peer with AS `P` |
+| `L:P` | `L:1:P` | to the peer with AS `P`, and then only to peers named so |
+| `0:L` | `L:0:0` | to nobody |
+
+"Not" wins over "to"; a route with none of them goes to everyone it
+otherwise would. Standard communities only have room for AS numbers below
+65536; the large forms work for any. They are taken off a route as it is
+passed on, and they hold for routes we originate too. With this on,
+standard communities `L:n` mean "to the peer with AS n", so mark routes
+with large communities (`L:2:n` and up) instead.
+
+**Cost.** A decision is a few comparisons on the route's communities and
+path. Routes that go out alike go in one UPDATE: changes are gathered for
+50 ms before they go, and a peer that is given the whole table (when it
+comes up, or asks for it again) has it gathered first, so that the
+prefixes that share attributes, which are spread over the table's shards,
+go together. At most four sessions gather a table at once; the others wait
+their turn. `cargo bench -p bgp-bench --bench transit`, with one peer
+sending 1.23 million routes over a session and another being told of them,
+on a laptop: in our table after 0.47 s, all of it heard by the other peer
+after 2.1 s, in UPDATEs of four routes each on average.
+
+**Not there yet.** Per-client best path selection for a route server (every
+client is told the one best route); ADD-PATH for learned routes (only the
+best is sent); route reflection; a way to send pre-built messages to
+several peers at once.
+
 ## Route origin validation (RPKI)
 
 With `rpki { caches { address: "..." port: 3323 } }` in the configuration,
@@ -369,6 +451,7 @@ Routinator 0.15.2. What it covers:
 | `limits` | A peer whose role does not fit gets no session; a peer over its prefix limit gets Cease / Maximum Number of Prefixes and is held down |
 | `hijacks` | FRR announces our own prefix, a route whose next hop is BIRD, and the exchange LAN's own prefix: none is taken |
 | `nexthops` | A better route from FRR whose next hop cannot be reached is held and BIRD's is chosen and installed; they swap when the next hop becomes a neighbor, and swap back when it stops being one |
+| `transit` | FRR as our customer and BIRD as our provider: FRR's routes reach BIRD with our AS on the path, BIRD's reach FRR marked only to customer, one of FRR's is kept from BIRD by a community, and packets go between their networks through us |
 | `v6only` | The exchange LAN without IPv4 addresses: one IPv6 session with each router for both families, IPv4 routes with IPv6 next hops (RFC 8950) in BGP and in the kernel, IPv4 pings across |
 | `linklocal` | The exchange LAN with link-local addresses only, and the same ones on a second interface: sessions with FRR (by interface, with radvd beside the daemon) and BIRD at `fe80::…%ix0`, both families in BGP and in the kernel, pings across, the interface taken down and up |
 
@@ -416,6 +499,7 @@ For alerting, `/metricz` has:
 | `bgp_peer_max_prefixes{peer}` | The peer's limit, where it has one |
 | `bgp_prefix_limit_exceeded_total{peer}` | Sessions ended for going over the limit |
 | `bgp_connections_refused_total{reason}` | Connections closed without a session |
+| `bgp_routes_exported_total{change}` | Learned routes announced to, and withdrawn from, other peers |
 | `bgp_peer_up{peer}` | 1 while the session with a configured peer is Established, else 0 |
 | `bgp_build_info{version}` | Always 1; says which version is running |
 

crates/bench/Cargo.toml

@@ -38,5 +38,9 @@ harness = false
 name = "real_table"
 harness = false
 
+[[bench]]
+name = "transit"
+harness = false
+
 [lints]
 workspace = true

crates/bench/benches/transit.rs

@@ -0,0 +1,152 @@
+//! What passing routes on costs: one peer sends the table over a session,
+//! and another is told of every route, over its own. The time is from the
+//! first byte sent to the last route heard; alongside, the time for the
+//! same table to be in our own table and no further.
+
+use std::net::Ipv4Addr;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+use bgp_bench::{Prefixes, Shape, announcements, encoded};
+use bgp_integration_tests::test_peer::TestPeer;
+use bgp_integration_tests::test_server::TestServer;
+use bgp_packet::message::PathAttribute;
+use bgp_server::marks::{Control, Marks};
+use bgp_server::policy::ExportPolicy;
+use bytes::BytesMut;
+
+fn env(name: &str, default: usize) -> usize {
+    std::env::var(name)
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(default)
+}
+
+/// How a server passes routes on in one run.
+#[derive(Clone, Copy)]
+enum Passing {
+    Not,
+    Plain,
+    /// With the communities that say who is told, and a mark put on every
+    /// route.
+    Dressed,
+}
+
+/// Sends the table from one peer and, where it is passed on, hears all of
+/// it at another. Returns when the table was in, and when it was all
+/// heard.
+fn run(
+    wire: &[BytesMut],
+    routes: usize,
+    heard_routes: usize,
+    passing: Passing,
+) -> (Duration, Option<Duration>) {
+    let runtime = tokio::runtime::Builder::new_multi_thread()
+        .enable_all()
+        .build()
+        .expect("a runtime");
+    runtime.block_on(async {
+        let server = TestServer::start_with(4_200_000_000, |config| {
+            config.export = Arc::new(ExportPolicy {
+                learned_routes: !matches!(passing, Passing::Not),
+                marks: match passing {
+                    Passing::Dressed => Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
+                    _ => Marks::default(),
+                },
+                prepend: 0,
+            });
+            config.control = matches!(passing, Passing::Dressed).then_some(Control {
+                local_asn: 4_200_000_000,
+            });
+        })
+        .await
+        .expect("a server");
+        let mut hearer = TestPeer::connect(server.addr).await.expect("a connection");
+        hearer
+            // An AS number that is on none of the generated paths: a route
+            // is not passed to a peer whose AS it has been through.
+            .handshake(4_100_000_000, Ipv4Addr::new(10, 0, 0, 100), 90)
+            .await
+            .expect("a session");
+        let mut sender = TestPeer::connect(server.addr).await.expect("a connection");
+        sender
+            .handshake(64513, Ipv4Addr::new(10, 0, 0, 1), 90)
+            .await
+            .expect("a session");
+        let stream: Vec<u8> = wire.iter().flat_map(|m| m.iter().copied()).collect();
+        let started = Instant::now();
+        let sending = tokio::spawn(async move {
+            sender.send_raw(&stream).await.expect("the server reads");
+            sender
+        });
+        let hearing = tokio::spawn(async move {
+            if matches!(passing, Passing::Not) {
+                return (hearer, None, 0);
+            }
+            let mut heard = 0usize;
+            let mut messages = 0usize;
+            while heard < heard_routes {
+                let update = hearer.recv_update().await.expect("an UPDATE");
+                messages += 1;
+                heard += update.announced_routes.len();
+                heard += update
+                    .path_attributes
+                    .iter()
+                    .map(|attribute| match attribute {
+                        PathAttribute::MpReachNlri(mp) => mp.prefixes.len(),
+                        _ => 0,
+                    })
+                    .sum::<usize>();
+            }
+            (hearer, Some(started.elapsed()), messages)
+        });
+        while server.rib.len() < routes {
+            tokio::time::sleep(Duration::from_millis(2)).await;
+        }
+        let in_table = started.elapsed();
+        let (hearer, heard, messages) = hearing.await.expect("the hearer");
+        if messages > 0 {
+            println!(
+                "    ({messages} UPDATEs, {:.1} routes each)",
+                heard_routes as f64 / messages as f64
+            );
+        }
+        drop(sending.await);
+        drop(hearer);
+        (in_table, heard)
+    })
+}
+
+fn main() {
+    let shape = Shape::INTERNET.scaled_down(env("BENCH_SCALE_DOWN", 1));
+    let prefixes = Prefixes::generate(shape);
+    let wire = encoded(&announcements(&prefixes, shape, 1));
+    let routes = shape.prefixes();
+    // Over a session on IPv4 only the IPv4 routes go.
+    let heard_routes = shape.ipv4_prefixes;
+    println!(
+        "== One peer sends {routes} routes in {} UPDATEs over TCP; another, over IPv4, is told of the {heard_routes} IPv4 ones",
+        wire.len()
+    );
+    for (what, passing) in [
+        ("not passed on", Passing::Not),
+        ("passed on", Passing::Plain),
+        (
+            "passed on, with control communities and a mark",
+            Passing::Dressed,
+        ),
+    ] {
+        let (in_table, heard) = run(&wire, routes, heard_routes, passing);
+        let heard = heard.map_or(String::new(), |heard| {
+            format!(
+                "   all heard by the other peer after {:>6.3} s ({:.2} M routes/s)",
+                heard.as_secs_f64(),
+                heard_routes as f64 / heard.as_secs_f64() / 1e6
+            )
+        });
+        println!(
+            "{what:<48} in the table after {:>6.3} s{heard}",
+            in_table.as_secs_f64()
+        );
+    }
+}

crates/integration-tests/src/test_server.rs

@@ -67,6 +67,8 @@ impl TestServer {
             http_endpoint: None,
             rpki: None,
             kernel: None,
+            export: Default::default(),
+            control: None,
         };
         customize(&mut config);
 

crates/integration-tests/tests/add_path.rs

@@ -50,6 +50,7 @@ async fn server_offering(
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
         c.originate = originate;
         // Keep the dialer out of the way.

crates/integration-tests/tests/admission.rs

@@ -28,6 +28,7 @@ fn peer_at(endpoint: SocketAddr) -> PeerConfig {
         import: ImportPolicy::permissive(),
         role: None,
         require_role: false,
+        export: Default::default(),
     }
 }
 

crates/integration-tests/tests/aspa.rs

@@ -67,6 +67,7 @@ async fn server_with(
             import,
             role,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;

crates/integration-tests/tests/collision.rs

@@ -46,6 +46,7 @@ async fn server_dialing(listener: &TcpListener) -> Result<TestServer> {
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await

crates/integration-tests/tests/debugging.rs

@@ -180,6 +180,7 @@ async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
             },
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;

crates/integration-tests/tests/dialer.rs

@@ -37,6 +37,7 @@ async fn server_dialing(listener: &TcpListener, remote_asn: u32) -> Result<TestS
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await
@@ -194,6 +195,7 @@ async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;
@@ -245,6 +247,7 @@ async fn test_local_address_is_used_as_the_source() -> Result<()> {
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;
@@ -265,6 +268,7 @@ async fn test_local_address_is_used_as_the_source() -> Result<()> {
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;

crates/integration-tests/tests/import_policy.rs

@@ -51,6 +51,7 @@ async fn server_with(import: ImportPolicy) -> Result<(TestServer, TcpListener)>
             import,
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;
@@ -281,3 +282,63 @@ async fn test_a_peer_over_its_limit_is_cut_off_and_kept_down() -> Result<()> {
         .await?;
     Ok(())
 }
+
+/// Routes from a peer are marked with where they were learned and given
+/// a preference, as the peer's `import` says; and marks that mean
+/// something to us are not taken on the peer's word.
+#[tokio::test]
+async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()> {
+    use bgp_packet::message::{CommunitiesPathAttribute, LocalPrefPathAttribute};
+    use bgp_server::marks::Marks;
+    let (server, _listener) = server_with(ImportPolicy {
+        marks: Marks::parse(
+            &["1299:*".to_string()],
+            &["1299:1001".to_string()],
+            &["1299:9:7".to_string()],
+        )
+        .unwrap(),
+        local_pref: Some(250),
+        ..default_policy()
+    })
+    .await?;
+    let mut peer = established(&server).await?;
+    peer.send(&Message::Update(UpdateMessage {
+        withdrawn_routes: vec![],
+        path_attributes: vec![
+            PathAttribute::Origin(OriginPathAttribute::IGP),
+            AsPathAttribute::from_asns(vec![PEER_ASN]),
+            PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
+            PathAttribute::Communities(CommunitiesPathAttribute(vec![(3333, 5), (1299, 2002)])),
+        ],
+        announced_routes: vec![prefix("193.0.0.0/21").into(), prefix("193.0.8.0/22").into()],
+    }))
+    .await?;
+    table_after(&server, 2, 0).await?;
+    let routes = server.received_routes();
+    for route in &routes {
+        assert!(
+            route
+                .attributes
+                .contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
+                    (3333, 5),
+                    (1299, 1001)
+                ]))),
+            "{route:?}"
+        );
+        assert!(
+            route
+                .attributes
+                .contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(250))),
+            "{route:?}"
+        );
+        assert!(
+            route.attributes.iter().any(|attribute| matches!(
+                attribute,
+                PathAttribute::LargeCommunities(large) if large.communities.len() == 1
+            )),
+            "{route:?}"
+        );
+    }
+    assert_eq!(server.rib.attrs().len(), 1, "they share their attributes");
+    Ok(())
+}

crates/integration-tests/tests/negotiation.rs

@@ -193,6 +193,7 @@ async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
             import: ImportPolicy::permissive(),
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
         // Keep the dialer out of the way.
         c.connect_retry = Duration::from_secs(3600);

crates/integration-tests/tests/roles.rs

@@ -58,6 +58,7 @@ async fn server_with(role: Option<PeerRole>, require_role: bool) -> Result<Setup
             import: default_policy(),
             role,
             require_role,
+            export: Default::default(),
         }];
     })
     .await?;

crates/integration-tests/tests/rpki.rs

@@ -77,6 +77,7 @@ async fn server_with(
             import,
             role: None,
             require_role: false,
+            export: Default::default(),
         }];
     })
     .await?;

crates/integration-tests/tests/transit.rs

@@ -0,0 +1,389 @@
+//! Passing routes on: what one peer announced reaches the others, as far
+//! as the rules for that let it.
+
+use std::collections::BTreeSet;
+use std::net::Ipv4Addr;
+use std::sync::Arc;
+use std::time::Duration;
+
+use bgp_integration_tests::test_peer::TestPeer;
+use bgp_integration_tests::test_server::TestServer;
+use bgp_packet::message::{
+    AsPathAttribute, CommunitiesPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute,
+    PathAttribute, RouteRefreshMessage, UpdateMessage,
+};
+use bgp_primitives::constants::AfiSafi;
+use bgp_primitives::ip_prefix::IpPrefix;
+use bgp_rib::{LocalRoute, LocalRouteAttrs};
+use bgp_server::marks::{Control, Marks};
+use bgp_server::policy::ExportPolicy;
+use eyre::Result;
+
+const LOCAL_ASN: u32 = 64601;
+const TIMEOUT: Duration = Duration::from_secs(5);
+
+fn prefix(s: &str) -> IpPrefix {
+    IpPrefix::try_from(s).unwrap()
+}
+
+/// A server that passes on to every peer what it learned from the others.
+async fn transit(customize: impl FnOnce(&mut bgp_server::config::Config)) -> Result<TestServer> {
+    TestServer::start_with(LOCAL_ASN, |config| {
+        config.export = Arc::new(ExportPolicy {
+            learned_routes: true,
+            ..ExportPolicy::default()
+        });
+        customize(config);
+    })
+    .await
+}
+
+/// The peer with AS `64700 + n`, its session up.
+async fn peer(server: &TestServer, n: u8) -> Result<TestPeer> {
+    let before = server.registry.list().len();
+    let mut peer = TestPeer::connect(server.addr).await?;
+    peer.handshake(64700 + u32::from(n), Ipv4Addr::new(10, 0, 0, n), 90)
+        .await?;
+    tokio::time::timeout(TIMEOUT, async {
+        while server.registry.list().len() != before + 1 {
+            tokio::time::sleep(Duration::from_millis(5)).await;
+        }
+    })
+    .await?;
+    Ok(peer)
+}
+
+async fn announce(
+    peer: &mut TestPeer,
+    prefixes: &[&str],
+    path: &[u32],
+    communities: &[(u16, u16)],
+) -> Result<()> {
+    let mut path_attributes = vec![
+        PathAttribute::Origin(OriginPathAttribute::IGP),
+        AsPathAttribute::from_asns(path.to_vec()),
+        PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
+    ];
+    if !communities.is_empty() {
+        path_attributes.push(PathAttribute::Communities(CommunitiesPathAttribute(
+            communities.to_vec(),
+        )));
+    }
+    peer.send(&Message::Update(UpdateMessage {
+        withdrawn_routes: vec![],
+        path_attributes,
+        announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
+    }))
+    .await
+}
+
+async fn withdraw(peer: &mut TestPeer, prefixes: &[&str]) -> Result<()> {
+    peer.send(&Message::Update(UpdateMessage {
+        withdrawn_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
+        path_attributes: vec![],
+        announced_routes: vec![],
+    }))
+    .await
+}
+
+/// What a peer has been told so far, as the prefixes it would now have
+/// and the path each came with.
+#[derive(Default, Debug)]
+struct Told {
+    routes: std::collections::BTreeMap<String, Vec<u32>>,
+    updates: usize,
+}
+
+impl Told {
+    fn take(&mut self, update: &UpdateMessage) {
+        self.updates += 1;
+        for route in &update.withdrawn_routes {
+            self.routes.remove(&route.prefix.to_string());
+        }
+        let path: Vec<u32> = update
+            .path_attributes
+            .iter()
+            .find_map(|attribute| match attribute {
+                PathAttribute::AsPath(path) => Some(
+                    path.segments
+                        .iter()
+                        .flat_map(|segment| segment.path.clone())
+                        .collect(),
+                ),
+                _ => None,
+            })
+            .unwrap_or_default();
+        for route in &update.announced_routes {
+            self.routes.insert(route.prefix.to_string(), path.clone());
+        }
+    }
+
+    fn prefixes(&self) -> BTreeSet<&str> {
+        self.routes.keys().map(String::as_str).collect()
+    }
+}
+
+/// Reads what the peer is sent until it has exactly `expected`.
+async fn told_becomes(peer: &mut TestPeer, told: &mut Told, expected: &[&str]) -> Result<()> {
+    let expected: BTreeSet<&str> = expected.iter().copied().collect();
+    let reading = async {
+        while told.prefixes() != expected {
+            let update = peer.recv_update().await?;
+            told.take(&update);
+        }
+        Ok::<(), eyre::Report>(())
+    };
+    match tokio::time::timeout(TIMEOUT, reading).await {
+        Ok(read) => read,
+        Err(_) => Err(eyre::eyre!(
+            "the peer has {:?}; expected {expected:?}",
+            told.prefixes()
+        )),
+    }
+}
+
+/// Nothing more is sent to the peer for a while.
+async fn told_nothing(peer: &mut TestPeer) {
+    let more = tokio::time::timeout(Duration::from_millis(300), peer.recv_update()).await;
+    assert!(more.is_err(), "the peer was sent {more:?}");
+}
+
+/// What one peer announces the others are told, with our AS on the path
+/// and ourselves as next hop; and what it takes back, they are told too.
+#[tokio::test]
+async fn test_routes_from_one_peer_reach_the_others() -> Result<()> {
+    let server = transit(|_| {}).await?;
+    let mut one = peer(&server, 1).await?;
+    let mut two = peer(&server, 2).await?;
+
+    announce(
+        &mut one,
+        &["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"],
+        &[64701, 65000],
+        &[],
+    )
+    .await?;
+    let mut told = Told::default();
+    told_becomes(
+        &mut two,
+        &mut told,
+        &["10.1.0.0/16", "10.2.0.0/16", "10.3.0.0/16"],
+    )
+    .await?;
+    assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64701, 65000]);
+    assert_eq!(
+        told.updates, 1,
+        "routes that go out alike go in one message"
+    );
+    // Not back to where they came from.
+    told_nothing(&mut one).await;
+
+    // The next hop is ourselves: the address of our end of the session.
+    announce(&mut one, &["10.4.0.0/16"], &[64701], &[]).await?;
+    let update = tokio::time::timeout(TIMEOUT, two.recv_update()).await??;
+    assert!(
+        update
+            .path_attributes
+            .contains(&PathAttribute::NextHop(NextHopPathAttribute(
+                Ipv4Addr::LOCALHOST
+            ))),
+        "{update:?}"
+    );
+    told.take(&update);
+
+    // One is taken back; then the peer goes, and its routes with it.
+    withdraw(&mut one, &["10.2.0.0/16"]).await?;
+    told_becomes(
+        &mut two,
+        &mut told,
+        &["10.1.0.0/16", "10.3.0.0/16", "10.4.0.0/16"],
+    )
+    .await?;
+    drop(one);
+    told_becomes(&mut two, &mut told, &[]).await?;
+    Ok(())
+}
+
+/// A peer is told of the best route, and of the next best when that one
+/// goes; and a peer that comes later is told what there is.
+#[tokio::test]
+async fn test_a_peer_is_kept_at_the_best_route() -> Result<()> {
+    let server = transit(|_| {}).await?;
+    let mut near = peer(&server, 1).await?;
+    let mut far = peer(&server, 2).await?;
+    let mut watcher = peer(&server, 3).await?;
+    let mut told = Told::default();
+
+    announce(&mut far, &["10.1.0.0/16"], &[64702, 65000, 65001], &[]).await?;
+    told_becomes(&mut watcher, &mut told, &["10.1.0.0/16"]).await?;
+    assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64702, 65000, 65001]);
+
+    // A better route for it comes: that is the one told now.
+    announce(&mut near, &["10.1.0.0/16"], &[64701, 65001], &[]).await?;
+    let update = tokio::time::timeout(TIMEOUT, watcher.recv_update()).await??;
+    told.take(&update);
+    assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64701, 65001]);
+    assert!(
+        update.withdrawn_routes.is_empty(),
+        "replaced, not withdrawn"
+    );
+
+    // And goes again: back to the other, with no gap.
+    withdraw(&mut near, &["10.1.0.0/16"]).await?;
+    let update = tokio::time::timeout(TIMEOUT, watcher.recv_update()).await??;
+    told.take(&update);
+    assert_eq!(told.routes["10.1.0.0/16"], [LOCAL_ASN, 64702, 65000, 65001]);
+
+    // The other peer was told of the better route while it was the best,
+    // and then that it is gone: the one left is its own, which it is not
+    // told of.
+    let mut told_far = Told::default();
+    told_becomes(&mut far, &mut told_far, &["10.1.0.0/16"]).await?;
+    let gone = tokio::time::timeout(TIMEOUT, far.recv_update()).await??;
+    assert_eq!(gone.withdrawn_routes, vec![prefix("10.1.0.0/16")]);
+    told_nothing(&mut far).await;
+
+    // A peer that comes now is told what there is; and again if it asks.
+    let mut late = peer(&server, 4).await?;
+    let mut late_told = Told::default();
+    told_becomes(&mut late, &mut late_told, &["10.1.0.0/16"]).await?;
+    late.send(&Message::RouteRefresh(RouteRefreshMessage::request(
+        AfiSafi::IPV4_UNICAST,
+    )))
+    .await?;
+    let again = tokio::time::timeout(TIMEOUT, late.recv_update()).await??;
+    assert_eq!(again.announced_routes, vec![prefix("10.1.0.0/16")]);
+    Ok(())
+}
+
+/// A route is not sent to a peer whose AS is on its path already, nor
+/// anywhere if it says NO_EXPORT; and what we originate goes to everyone
+/// beside what was learned.
+#[tokio::test]
+async fn test_routes_stop_where_they_must() -> Result<()> {
+    let server = transit(|config| {
+        config.originate = vec![LocalRoute::new(
+            prefix("10.99.0.0/16"),
+            LocalRouteAttrs::default(),
+        )];
+    })
+    .await?;
+    let mut one = peer(&server, 1).await?;
+    let mut two = peer(&server, 2).await?;
+    let mut told = Told::default();
+    told_becomes(&mut two, &mut told, &["10.99.0.0/16"]).await?;
+    assert_eq!(told.routes["10.99.0.0/16"], [LOCAL_ASN]);
+
+    // Through the second peer's AS already; and not for other networks.
+    announce(&mut one, &["10.1.0.0/16"], &[64701, 64702, 65000], &[]).await?;
+    announce(&mut one, &["10.2.0.0/16"], &[64701], &[(0xFFFF, 0xFF01)]).await?;
+    announce(&mut one, &["10.3.0.0/16"], &[64701], &[]).await?;
+    told_becomes(&mut two, &mut told, &["10.99.0.0/16", "10.3.0.0/16"]).await?;
+    told_nothing(&mut two).await;
+    assert_eq!(server.rib.len(), 3, "all three were taken");
+    Ok(())
+}
+
+/// Communities on a route say which peers it is passed on to, and are
+/// gone when it is; and every route to a peer gets the marks of `export`.
+#[tokio::test]
+async fn test_communities_say_who_is_told() -> Result<()> {
+    let server = transit(|config| {
+        config.control = Some(Control {
+            local_asn: LOCAL_ASN,
+        });
+        config.export = Arc::new(ExportPolicy {
+            learned_routes: true,
+            marks: Marks::parse(&[], &["64601:7".to_string()], &[]).unwrap(),
+            prepend: 1,
+        });
+        config.originate = vec![LocalRoute::new(
+            prefix("10.99.0.0/16"),
+            LocalRouteAttrs {
+                // Our own route, and not for the second peer.
+                communities: vec![(0, 64702)],
+                ..LocalRouteAttrs::default()
+            },
+        )];
+    })
+    .await?;
+    let mut one = peer(&server, 1).await?;
+    let mut two = peer(&server, 2).await?;
+    let mut three = peer(&server, 3).await?;
+    let (mut told_two, mut told_three) = (Told::default(), Told::default());
+
+    // To everyone but the second; to the second only; to nobody; and
+    // with nothing said.
+    announce(
+        &mut one,
+        &["10.1.0.0/16"],
+        &[64701],
+        &[(0, 64702), (3356, 3)],
+    )
+    .await?;
+    announce(&mut one, &["10.2.0.0/16"], &[64701], &[(64601, 64702)]).await?;
+    announce(&mut one, &["10.3.0.0/16"], &[64701], &[(0, 64601)]).await?;
+    announce(&mut one, &["10.4.0.0/16"], &[64701], &[]).await?;
+
+    told_becomes(&mut two, &mut told_two, &["10.2.0.0/16", "10.4.0.0/16"]).await?;
+    told_becomes(
+        &mut three,
+        &mut told_three,
+        &["10.99.0.0/16", "10.1.0.0/16", "10.4.0.0/16"],
+    )
+    .await?;
+    told_nothing(&mut two).await;
+    told_nothing(&mut three).await;
+    // Our AS once for passing it on, and once more by `export`.
+    assert_eq!(
+        told_three.routes["10.1.0.0/16"],
+        [LOCAL_ASN, LOCAL_ASN, 64701]
+    );
+    assert_eq!(told_three.routes["10.99.0.0/16"], [LOCAL_ASN, LOCAL_ASN]);
+
+    // What was for us to act on is gone, and the router's mark is on.
+    announce(
+        &mut one,
+        &["10.5.0.0/16"],
+        &[64701],
+        &[(0, 64702), (3356, 3)],
+    )
+    .await?;
+    let update = tokio::time::timeout(TIMEOUT, three.recv_update()).await??;
+    assert!(
+        update
+            .path_attributes
+            .contains(&PathAttribute::Communities(CommunitiesPathAttribute(vec![
+                (3356, 3),
+                (64601, 7)
+            ]))),
+        "{update:?}"
+    );
+    Ok(())
+}
+
+/// Without `learned_routes` a peer is told what we originate and nothing
+/// else, as before there was such a thing.
+#[tokio::test]
+async fn test_nothing_learned_is_passed_on_unless_asked_for() -> Result<()> {
+    let server = TestServer::start_with(LOCAL_ASN, |config| {
+        config.originate = vec![LocalRoute::new(
+            prefix("10.99.0.0/16"),
+            LocalRouteAttrs::default(),
+        )];
+    })
+    .await?;
+    let mut one = peer(&server, 1).await?;
+    let mut two = peer(&server, 2).await?;
+    let mut told = Told::default();
+    told_becomes(&mut two, &mut told, &["10.99.0.0/16"]).await?;
+    announce(&mut one, &["10.1.0.0/16"], &[64701], &[]).await?;
+    tokio::time::timeout(TIMEOUT, async {
+        while server.rib.len() != 1 {
+            tokio::time::sleep(Duration::from_millis(5)).await;
+        }
+    })
+    .await?;
+    told_nothing(&mut two).await;
+    Ok(())
+}

crates/rib/src/table/journal.rs

@@ -87,6 +87,11 @@ impl Default for Limits {
 struct Entry {
     seq: u64,
     prefix: IpPrefix,
+    /// When, by the table's clock: the same for every change one call
+    /// made, in whatever shard. A reader gives out all the changes up to
+    /// a time or none of them, so that what was changed together is read
+    /// together.
+    stamp: u64,
 }
 
 /// One journal of one shard.
@@ -111,6 +116,8 @@ struct Journal {
     /// How many reader slots are in use.
     readers: usize,
     limits: Limits,
+    /// The time the entries written now are given; see [`Entry::stamp`].
+    stamp: u64,
 }
 
 impl Journal {
@@ -124,6 +131,7 @@ impl Journal {
             gone: HashMap::default(),
             cursors: Vec::new(),
             readers: 0,
+            stamp: 0,
         }
     }
 
@@ -135,6 +143,7 @@ impl Journal {
             self.entries.push_back(Entry {
                 seq: self.seq,
                 prefix,
+                stamp: self.stamp,
             });
         }
         self.seq
@@ -156,6 +165,7 @@ impl Journal {
             self.entries.push_back(Entry {
                 seq: self.seq,
                 prefix,
+                stamp: self.stamp,
             });
             self.gone.insert(prefix, self.seq);
         }
@@ -289,6 +299,13 @@ impl Journals {
         }
     }
 
+    /// The time by the table's clock that the changes made from now on
+    /// are given; see [`Entry::stamp`].
+    pub(super) fn set_stamp(&mut self, stamp: u64) {
+        self.best.stamp = stamp;
+        self.all.stamp = stamp;
+    }
+
     /// Notes that `prefix` has its first route, and so a row. Returns the
     /// sequence numbers for the new row.
     pub(super) fn appeared(&mut self, prefix: IpPrefix) -> Seqs {
@@ -409,10 +426,22 @@ pub struct Progress {
 
 enum Phase {
     /// Giving out the whole table; `next` is the shard to do next, and
-    /// `announced` whether [`Event::Reset`] has been given.
-    Resync { next: usize, announced: bool },
-    /// Following the journals; `next` is the shard to read next.
-    Follow { next: usize },
+    /// `announced` whether [`Event::Reset`] has been given. `upto` is the
+    /// time on the table's clock the table is given out as of: the shards
+    /// are gone through one at a time while the table changes, and the
+    /// place taken in each shard's journal is as of that one time, so that
+    /// what follows is consistent from shard to shard. A change after it
+    /// that a later shard's rows already show is read again from the
+    /// journal; that is harmless.
+    Resync {
+        next: usize,
+        announced: bool,
+        upto: u64,
+    },
+    /// Following the journals; `next` is the shard to read next, and
+    /// `upto` the time by the table's clock up to which this pass reads
+    /// (see [`Entry::stamp`]).
+    Follow { next: usize, upto: u64 },
 }
 
 impl Phase {
@@ -420,6 +449,7 @@ impl Phase {
     const RESYNC: Self = Self::Resync {
         next: 0,
         announced: false,
+        upto: 0,
     };
 }
 
@@ -453,7 +483,7 @@ impl RibTable {
     /// those who want the news and not the table. If it falls too far
     /// behind it is reset like any other, and is then given the table.
     pub fn reader_from_now(self: &Arc<Self>) -> Reader {
-        let mut reader = self.register(Kind::Best, Phase::Follow { next: 0 });
+        let mut reader = self.register(Kind::Best, Phase::Follow { next: 0, upto: 0 });
         for (index, shard) in self.shards.iter().enumerate() {
             let shard = read(shard);
             reader.set_cursor(index, &shard, shard.journals.best.seq);
@@ -567,54 +597,65 @@ impl Reader {
         let mut events = 0;
         loop {
             match self.phase {
-                Phase::Resync { next, announced } => {
-                    if !announced {
+                Phase::Resync {
+                    next,
+                    announced,
+                    upto,
+                } => {
+                    let upto = if announced {
+                        upto
+                    } else {
                         self.forget_place();
                         visit(Event::Reset);
                         events += 1;
-                    }
+                        self.table.settled()
+                    };
                     if next == shards {
                         visit(Event::Synced);
                         events += 1;
-                        self.phase = Phase::Follow { next: 0 };
+                        self.phase = Phase::Follow { next: 0, upto: 0 };
                         continue;
                     }
-                    events += self.scan(next, &mut visit);
+                    events += self.scan(next, upto, &mut visit);
                     self.phase = Phase::Resync {
                         next: next + 1,
                         announced: true,
+                        upto,
                     };
                     if events >= budget {
                         return self.progress(events, false);
                     }
                 }
-                Phase::Follow { next } => {
+                Phase::Follow { next, upto } => {
                     // Read before the pass: a change made during it shows as
                     // a newer version, and `changed` then returns at once.
-                    let version = if next == 0 {
-                        self.table.version.load(Ordering::Acquire)
+                    // And the pass reads up to one time in every shard
+                    // alike, chosen so that about the budget is read.
+                    let (version, upto) = if next == 0 {
+                        (
+                            self.table.version.load(Ordering::Acquire),
+                            self.time_for(budget - events),
+                        )
                     } else {
-                        self.seen
+                        (self.seen, upto)
                     };
                     self.seen = version;
                     let mut shard = next;
                     let mut lagged = false;
+                    let mut more_somewhere = false;
                     while shard < shards {
                         if events >= budget {
-                            self.phase = Phase::Follow { next: shard };
+                            self.phase = Phase::Follow { next: shard, upto };
                             return self.progress(events, false);
                         }
-                        match self.drain(shard, budget - events, &mut visit) {
+                        match self.drain(shard, upto, &mut visit) {
                             Drained::Lagged => {
                                 lagged = true;
                                 break;
                             }
                             Drained::Read { events: read, more } => {
                                 events += read;
-                                if more {
-                                    self.phase = Phase::Follow { next: shard };
-                                    return self.progress(events, false);
-                                }
+                                more_somewhere |= more;
                             }
                         }
                         shard += 1;
@@ -623,11 +664,16 @@ impl Reader {
                         self.phase = Phase::RESYNC;
                         continue;
                     }
-                    self.phase = Phase::Follow { next: 0 };
-                    if next != 0 {
-                        // This pass took up where an earlier call stopped.
-                        // The shards before that point have not been looked
-                        // at in this call, so go round again from the first.
+                    self.phase = Phase::Follow { next: 0, upto: 0 };
+                    if next != 0 || more_somewhere {
+                        // This pass took up where an earlier call stopped,
+                        // so the shards before that point have not been
+                        // looked at in this call; or some shard had more
+                        // than its share, or changes later than this
+                        // pass's time. Go round again from the first.
+                        if events >= budget {
+                            return self.progress(events, false);
+                        }
                         continue;
                     }
                     return self.progress(events, true);
@@ -691,22 +737,68 @@ impl Reader {
         shard.journals.of(self.kind).cursors[self.slot].store(cursor, Ordering::Release);
     }
 
-    /// Gives out every prefix in a shard and takes a place at the end of
-    /// its journal, in one hold of the shard: what changes afterwards is in
-    /// the journal after that place.
-    fn scan(&mut self, index: usize, visit: &mut impl FnMut(Event<'_>)) -> usize {
+    /// Gives out every prefix in a shard and takes a place in its journal
+    /// as of `upto` on the table's clock, in one hold of the shard: what
+    /// changed after that time is in the journal after that place, whether
+    /// or not the rows given out show it already.
+    fn scan(&mut self, index: usize, upto: u64, visit: &mut impl FnMut(Event<'_>)) -> usize {
         let table = Arc::clone(&self.table);
         let shard = read(&table.shards[index]);
         for row in &shard.rows {
             visit(self.event_for(row));
         }
-        self.set_cursor(index, &shard, shard.journals.of(self.kind).seq);
+        let journal = shard.journals.of(self.kind);
+        let beyond = journal.entries.partition_point(|entry| entry.stamp <= upto);
+        let cursor = match (beyond.checked_sub(1), journal.entries.get(beyond)) {
+            // Nothing in the journal is later than `upto`.
+            (_, None) => journal.seq,
+            // The last entry that is not later; or, if every entry is
+            // later, just before the first.
+            (Some(last), Some(_)) => journal.entries[last].seq,
+            (None, Some(first)) => first.seq - 1,
+        };
+        self.set_cursor(index, &shard, cursor);
         shard.rows.len()
     }
 
-    /// Reads a shard's journal from this reader's place, for at most `max`
-    /// events.
-    fn drain(&mut self, index: usize, max: usize, visit: &mut impl FnMut(Event<'_>)) -> Drained {
+    /// The time on the table's clock up to which a pass over the shards
+    /// is to read, for about `budget` events in all: changes made together
+    /// are read together, in whatever shards they are (see
+    /// [`Entry::stamp`]). Taken shard by shard to the budget, the first
+    /// shards would be read far ahead of the last, and the routes of one
+    /// UPDATE, spread over the shards by their prefixes, given out far
+    /// apart.
+    fn time_for(&self, budget: usize) -> u64 {
+        let shards = self.table.shards.len();
+        let share = (budget / shards).max(1);
+        // No further than the changes that are in every shard already.
+        let settled = self.table.settled();
+        let mut upto = settled;
+        // The earliest change not yet read: at least that one is, however
+        // many changes were made with it.
+        let mut earliest = None;
+        for (index, shard) in self.table.shards.iter().enumerate() {
+            let shard = read(shard);
+            let journal = shard.journals.of(self.kind);
+            let cursor = self.cursors[index];
+            if cursor < journal.floor || cursor == journal.seq {
+                continue;
+            }
+            let start = journal.entries.partition_point(|entry| entry.seq <= cursor);
+            if let Some(next) = journal.entries.get(start) {
+                earliest = Some(earliest.map_or(next.stamp, |e: u64| e.min(next.stamp)));
+            }
+            // Everything before the change that would go over the share.
+            if let Some(beyond) = journal.entries.get(start + share) {
+                upto = upto.min(beyond.stamp - 1);
+            }
+        }
+        upto.max(earliest.unwrap_or(upto)).min(settled)
+    }
+
+    /// Reads a shard's journal from this reader's place, no further than
+    /// the changes made by `upto` on the table's clock.
+    fn drain(&mut self, index: usize, upto: u64, visit: &mut impl FnMut(Event<'_>)) -> Drained {
         let table = Arc::clone(&self.table);
         let shard = read(&table.shards[index]);
         let journal = shard.journals.of(self.kind);
@@ -725,7 +817,7 @@ impl Reader {
         let mut events = 0;
         let mut more = false;
         for entry in journal.entries.range(start..) {
-            if events == max {
+            if entry.stamp > upto {
                 more = true;
                 break;
             }

crates/rib/src/table/journal_tests.rs

@@ -711,3 +711,56 @@ proptest! {
         }
     }
 }
+
+/// Routes that were announced together, with one set of attributes, are
+/// spread over the shards by their prefixes. A reader that takes its
+/// budget from the shards evenly is given them together all the same,
+/// which is what lets whoever passes them on send them in one message.
+#[test]
+fn changes_made_together_are_read_together() {
+    let fixture = Fixture::on(RibTable::new(128), &[external(1)]);
+    let mut reader = fixture.table.reader();
+    assert!(reader.poll(10, |_| {}).caught_up);
+
+    // 6000 UPDATEs of six prefixes each, every one with attributes of its
+    // own.
+    let groups = 6000u32;
+    for group in 0..groups {
+        let attrs = fixture
+            .table
+            .attrs()
+            .intern(&Route::via(&[1, 100_000 + group]).attributes(), None)
+            .unwrap();
+        let prefixes: Vec<IpPrefix> = (0..6).map(|n| nth_prefix(group * 6 + n)).collect();
+        fixture.table.announce(fixture.peers[0], &attrs, &prefixes);
+    }
+
+    // Read in batches a tenth of that size: each should hold whole groups,
+    // but for the ones cut by its edges.
+    let mut batches = 0;
+    let mut split_groups = 0;
+    let mut seen_total = 0;
+    while seen_total < groups * 6 {
+        let mut groups_in_batch: HashMap<usize, usize> = HashMap::new();
+        let progress = reader.poll(3600, |event| {
+            if let Event::Best { attrs, .. } = event {
+                *groups_in_batch
+                    .entry(Arc::as_ptr(attrs) as usize)
+                    .or_default() += 1;
+            }
+        });
+        seen_total += groups_in_batch.values().sum::<usize>() as u32;
+        split_groups += groups_in_batch.values().filter(|n| **n < 6).count();
+        batches += 1;
+        if progress.events == 0 {
+            break;
+        }
+    }
+    assert_eq!(seen_total, groups * 6);
+    // A few are, where a batch's time falls inside one: the reader cannot
+    // know what was written together until it is all there.
+    assert!(
+        split_groups < 100,
+        "{split_groups} of {groups} groups were split over {batches} batches"
+    );
+}

crates/rib/src/table/mod.rs

@@ -30,6 +30,7 @@ mod select;
 #[cfg(test)]
 mod tests;
 
+use std::collections::BTreeSet;
 use std::hash::BuildHasher;
 use std::net::{IpAddr, Ipv4Addr};
 use std::sync::atomic::{AtomicU64, AtomicUsize};
@@ -301,6 +302,13 @@ pub struct RibTable {
     /// Goes up whenever something is written to a journal that has
     /// readers.
     version: AtomicU64,
+    /// Counts the calls that change the table, so that a change can say
+    /// when it was made relative to others (see the journal's `Entry`).
+    clock: AtomicU64,
+    /// The changes being made right now, by their time on the clock: a
+    /// reader does not read past the earliest of them, which may not have
+    /// reached every shard yet.
+    in_progress: Mutex<BTreeSet<u64>>,
     wake: Notify,
 }
 
@@ -339,6 +347,8 @@ impl RibTable {
             reader_slots: Mutex::default(),
             readers: AtomicUsize::new(0),
             version: AtomicU64::new(0),
+            clock: AtomicU64::new(0),
+            in_progress: Mutex::default(),
             wake: Notify::new(),
         }
     }
@@ -395,8 +405,10 @@ impl RibTable {
     pub fn remove_peer(&self, peer: PeerIndex) -> Outcome {
         let mut outcome = Outcome::default();
         let mut news = false;
+        let change = self.begin_change();
         for shard in &self.shards {
             let mut shard = write(shard);
+            shard.journals.set_stamp(change.stamp);
             let before = shard.journals.mark();
             outcome += shard.remove_peer(peer);
             // From here on the shard takes no more routes from the peer.
@@ -491,8 +503,11 @@ impl RibTable {
         placed.sort_unstable_by_key(|(hash, _)| self.shard_of(*hash));
 
         let mut news = false;
+        // One time for all of this call's changes, in every shard.
+        let change = self.begin_change();
         for run in placed.chunk_by(|a, b| self.shard_of(a.0) == self.shard_of(b.0)) {
             let mut shard = write(&self.shards[self.shard_of(run[0].0)]);
+            shard.journals.set_stamp(change.stamp);
             let before = shard.journals.mark();
             apply(&mut shard, run);
             shard.tidy_journals();
@@ -503,6 +518,30 @@ impl RibTable {
         }
     }
 
+    /// Takes a time on the clock for a change that is about to be made,
+    /// and notes that it is under way until the guard drops.
+    fn begin_change(&self) -> Change<'_> {
+        let stamp = self.clock.fetch_add(1, std::sync::atomic::Ordering::AcqRel) + 1;
+        self.in_progress
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner)
+            .insert(stamp);
+        Change { table: self, stamp }
+    }
+
+    /// The time on the clock up to which every change is complete in every
+    /// shard: everything before the earliest one still under way.
+    pub(super) fn settled(&self) -> u64 {
+        let in_progress = self
+            .in_progress
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner);
+        match in_progress.first() {
+            Some(earliest) => earliest - 1,
+            None => self.clock.load(std::sync::atomic::Ordering::Acquire),
+        }
+    }
+
     /// How many routes the table holds, from all peers.
     pub fn len(&self) -> usize {
         self.shards.iter().map(|shard| read(shard).routes).sum()
@@ -620,6 +659,22 @@ fn write(shard: &RwLock<Shard>) -> RwLockWriteGuard<'_, Shard> {
     shard.write().unwrap_or_else(PoisonError::into_inner)
 }
 
+/// A change to the table that is under way; see [`RibTable::begin_change`].
+struct Change<'a> {
+    table: &'a RibTable,
+    stamp: u64,
+}
+
+impl Drop for Change<'_> {
+    fn drop(&mut self) {
+        self.table
+            .in_progress
+            .lock()
+            .unwrap_or_else(PoisonError::into_inner)
+            .remove(&self.stamp);
+    }
+}
+
 /// Picks one of `count` buckets by a hash, for spreading keys over shards
 /// or lock stripes that each hold a hash table keyed by the same hash.
 ///

crates/server/proto/config.proto

@@ -53,6 +53,32 @@ message BgpConfig {
   // that the machine forwards packets by them. Left out, the daemon only
   // speaks BGP and the machine's forwarding is not touched.
   Kernel kernel = 11;
+
+  // What is announced, and how, to every peer that has no `export` of its
+  // own.
+  ExportPolicy export = 12;
+
+  // Let routes say, with communities, which peers they are passed on to,
+  // as the route servers of Internet exchanges do. With L our own AS
+  // number and P a peer's:
+  //
+  //   0:P   or  L:0:P   not to the peer with AS P
+  //   L:P   or  L:1:P   to the peer with AS P, and then only to peers
+  //                     named so
+  //   0:L   or  L:0:0   to nobody
+  //
+  // A route with none of these goes to everyone it otherwise would. "Not"
+  // wins over "to". The two-part form is a standard community, with room
+  // for AS numbers below 65536; the three-part form is a large community
+  // and works for any. They are taken off a route as it is passed on.
+  //
+  // They are honoured on routes we originate too (`communities` of
+  // `originate`), and on what import.add_communities puts on.
+  //
+  // With this on, standard communities L:n are taken for "to the peer
+  // with AS n": mark routes with something else (large communities
+  // L:2:n and up, say).
+  bool control_communities = 13;
 }
 
 // Installing routes in the kernel. Linux only; the daemon needs the
@@ -211,6 +237,55 @@ message Peer {
   // that says nothing (one that does not implement RFC 9234) is taken at
   // this configuration's word for it. Needs role.
   bool require_role = 10;
+
+  // What is announced to the peer, and how. Left out, the top-level
+  // `export` holds, and without that the routes we originate are
+  // announced as they are.
+  ExportPolicy export = 11;
+}
+
+// What is announced to a peer.
+//
+// The routes we originate are always announced. Routes learned from other
+// peers are passed on only if learned_routes says so, and then only as
+// far as these let them:
+//
+// - Never back to the peer a route came from, nor to a peer whose AS is
+//   on its path already.
+// - A route from a peer in our own AS is not passed to another such peer
+//   (there is no route reflection).
+// - Where peers have roles (RFC 9234): a route learned from a provider, a
+//   peer or a route server goes on to customers only. So with roles on
+//   all peers, turning this on makes us transit for our customers and for
+//   nobody else. Without roles there is no such rule, and every route
+//   goes to every peer: do not turn this on for a peer without a role
+//   unless that is what is wanted.
+// - The communities NO_EXPORT and NO_ADVERTISE, and control_communities.
+//
+// A route is passed on with our AS added to its path and ourselves as
+// next hop, without the MED it came with. To a peer in our own AS the
+// path is left as it is and LOCAL_PREF goes along. To a peer whose role
+// is ROUTE_SERVER_CLIENT the route goes as a route server sends it
+// (RFC 7947): path, MED and, where the client can use it, next hop
+// unchanged.
+message ExportPolicy {
+  // Pass on the best routes learned from other peers.
+  bool learned_routes = 1;
+
+  // Communities put on every route announced to the peer: which router
+  // it went through, say. "64601:100", or a well-known name.
+  repeated string add_communities = 2;
+
+  // Large communities put on, as "64601:1:2".
+  repeated string add_large_communities = 3;
+
+  // Communities taken off first: one ("64601:100"), all of an AS
+  // ("64601:*"), or large ones with three parts ("64601:1:*").
+  repeated string remove_communities = 4;
+
+  // How many more times our AS is put on the path of every route
+  // announced to the peer, to make them look further away. At most 16.
+  uint32 as_path_prepend = 5;
 }
 
 // What a peer is to us.
@@ -296,6 +371,23 @@ message ImportPolicy {
   // to. Not needed for a peer whose role is ROUTE_SERVER, whose routes
   // have its clients as next hops.
   bool accept_third_party_next_hop = 14;
+
+  // Communities put on every route taken from the peer: where it was
+  // learned, say, for other routers of ours and for `export` to go by.
+  // "64601:100", or a well-known name.
+  repeated string add_communities = 15;
+
+  // Large communities put on, as "64601:1:2".
+  repeated string add_large_communities = 16;
+
+  // Communities taken off first: one ("64601:100"), all of an AS
+  // ("64601:*"), or large ones with three parts ("64601:1:*"). Marks that
+  // mean something to us are not to be taken on a peer's word.
+  repeated string remove_communities = 17;
+
+  // The LOCAL_PREF of the peer's routes: of two routes for a prefix the
+  // one with the higher is chosen, whatever else they have. Left out, 100.
+  optional uint32 local_pref = 18;
 }
 
 // A prefix, and optionally the more specific prefixes inside it.

crates/server/src/advertise.rs

@@ -18,6 +18,7 @@
 use std::collections::{HashMap, HashSet};
 use std::net::{IpAddr, Ipv6Addr};
 use std::sync::Arc;
+use std::time::Duration;
 
 use bgp_packet::message::{
     AsPathAttribute, AsPathSegment, CommunitiesPathAttribute, LargeCommunitiesPathAttribute,
@@ -26,14 +27,19 @@ use bgp_packet::message::{
 };
 use bgp_primitives::constants::{AfiSafi, IpVersion};
 use bgp_primitives::ip_prefix::{IpPrefix, Nlri};
-use bgp_rib::table::DEFAULT_LOCAL_PREF;
+use bgp_primitives::parser::ParserContext;
+use bgp_rib::table::{AttrSet, DEFAULT_LOCAL_PREF, Event, PeerIndex, Reader, RibTable};
 use bgp_rib::{LocalRib, LocalRibEvent, LocalRoute};
-use tokio::sync::Notify;
+use bytes::BytesMut;
 use tokio::sync::broadcast::Receiver;
 use tokio::sync::broadcast::error::RecvError;
+use tokio::sync::{Notify, OwnedSemaphorePermit, Semaphore};
 use tokio::task::JoinHandle;
 use tracing::{debug, info, warn};
 
+use crate::export::{self, Egress, Outgoing, Source};
+use crate::marks::Control;
+use crate::metrics::ROUTES_EXPORTED;
 use crate::policy::PeerRole;
 use crate::session::{PeerId, SessionClosed, SessionEvent, SessionHandle, SessionRegistry};
 
@@ -41,14 +47,19 @@ use crate::session::{PeerId, SessionClosed, SessionEvent, SessionHandle, Session
 /// this returns, so call it before anything that can bring one up.
 pub fn spawn_advertiser(
     local_rib: LocalRib,
+    rib: Arc<RibTable>,
     registry: SessionRegistry,
     local_asn: u32,
+    control: Option<Control>,
 ) -> JoinHandle<()> {
     let advertiser = Advertiser {
         session_events: registry.subscribe(),
         local_rib,
+        rib,
         registry,
         local_asn,
+        control,
+        resyncs: Arc::new(Semaphore::new(RESYNCS_AT_ONCE)),
         exporters: HashMap::new(),
     };
     tokio::spawn(advertiser.run())
@@ -70,8 +81,14 @@ impl Drop for Running {
 
 struct Advertiser {
     local_rib: LocalRib,
+    /// The routes learned from peers, of which the best are passed on.
+    rib: Arc<RibTable>,
     registry: SessionRegistry,
     local_asn: u32,
+    control: Option<Control>,
+    /// Leave for a session to gather the whole table before it sends it;
+    /// see `Learned::gathered`.
+    resyncs: Arc<Semaphore>,
     session_events: Receiver<SessionEvent>,
     exporters: HashMap<PeerId, Running>,
 }
@@ -113,6 +130,15 @@ impl Advertiser {
             // routes, so that no change between the two is missed.
             changes: self.local_rib.subscribe(),
             local_rib: self.local_rib.clone(),
+            // A follower of the best routes, if the peer is to be told
+            // of them: it is given the whole table first.
+            learned: session
+                .export
+                .learned_routes
+                .then(|| Learned::new(self.rib.reader())),
+            rib: Arc::clone(&self.rib),
+            resyncs: Arc::clone(&self.resyncs),
+            control: self.control,
             session: session.clone(),
             local_asn: self.local_asn,
             resend: Arc::clone(&resend),
@@ -147,8 +173,13 @@ impl Advertiser {
     }
 }
 
-/// Keeps one session's peer up to date with the routes we originate.
+/// Keeps one session's peer up to date with the routes we originate and,
+/// if it is to be told of them, the best of those learned from others.
 struct Exporter {
+    rib: Arc<RibTable>,
+    resyncs: Arc<Semaphore>,
+    control: Option<Control>,
+    learned: Option<Learned>,
     local_rib: LocalRib,
     changes: Receiver<LocalRibEvent>,
     session: SessionHandle,
@@ -159,6 +190,137 @@ struct Exporter {
     announced: HashSet<IpPrefix>,
 }
 
+/// How many changes of best route are taken from the table at a time.
+const EXPORT_BATCH: usize = 65_536;
+
+/// How long changes are gathered before they go out, once there are any.
+const COALESCE: Duration = Duration::from_millis(50);
+
+/// How many sessions may hold a whole table in hand at once while it is
+/// given out to them anew: some tens of megabytes each for the Internet.
+const RESYNCS_AT_ONCE: usize = 4;
+
+/// How many routes ready to go out are remembered, by the attributes they
+/// came with. A table has a few hundred thousand distinct sets.
+const DRESSED_KEPT: usize = 262_144;
+
+fn version_index(version: IpVersion) -> usize {
+    match version {
+        IpVersion::Ipv4 => 0,
+        IpVersion::Ipv6 => 1,
+    }
+}
+
+/// A change of best route, as taken out of the table to be dealt with
+/// once the table is let go of.
+enum News {
+    Reset,
+    Synced,
+    Best {
+        prefix: IpPrefix,
+        peer: PeerIndex,
+        attrs: Arc<AttrSet>,
+    },
+    Withdrawn(IpPrefix),
+}
+
+/// What a learned route goes out as depends on: the attributes it came
+/// with (by where they are), who it came from, and its IP version.
+type DressedKey = (usize, Source, IpVersion);
+
+/// A learned route as this session announces it.
+struct Dressed {
+    outgoing: Outgoing,
+    /// How many octets its attributes take on this session's wire.
+    octets: usize,
+}
+
+/// What an exporter knows of the learned routes it has passed on.
+struct Learned {
+    reader: Reader,
+    /// The verdicts so far on a table that is being given out anew, kept
+    /// until the table is all there; and the leave to keep them.
+    gathered: Option<HashMap<IpPrefix, Option<Arc<Dressed>>>>,
+    permit: Option<OwnedSemaphorePermit>,
+    /// The prefixes announced to the peer and not withdrawn since.
+    announced: HashSet<IpPrefix>,
+    /// While the table is being gone through anew: what was announced
+    /// before and has not been heard of again yet.
+    stale: Option<HashSet<IpPrefix>>,
+    /// Routes as they go out, by the attributes they came with, who they
+    /// came from and their IP version: a table's worth of routes share a
+    /// few sets of attributes, and each is worked out once. The key holds
+    /// the attributes, so that it stays theirs.
+    dressed: HashMap<DressedKey, (Arc<AttrSet>, Option<Arc<Dressed>>)>,
+}
+
+impl Learned {
+    fn new(reader: Reader) -> Self {
+        Self {
+            reader,
+            gathered: None,
+            permit: None,
+            announced: HashSet::new(),
+            stale: None,
+            dressed: HashMap::new(),
+        }
+    }
+
+    /// The route with `attrs` from `source` as it goes out on `egress`
+    /// for prefixes of IP version `version`; `None` if it does not.
+    fn dressed(
+        &mut self,
+        attrs: &Arc<AttrSet>,
+        source: Source,
+        version: IpVersion,
+        egress: &Egress<'_>,
+    ) -> Option<Arc<Dressed>> {
+        let key = (Arc::as_ptr(attrs) as usize, source, version);
+        if let Some((_, dressed)) = self.dressed.get(&key) {
+            return dressed.clone();
+        }
+        let dressed = attrs
+            .attributes()
+            .and_then(|(attributes, next_hop)| {
+                egress.outgoing(attributes, next_hop, source, version)
+            })
+            .map(|outgoing| {
+                // As long as they will be on the wire, to know how many
+                // prefixes fit beside them.
+                let ctx = ParserContext {
+                    four_octet_asn: egress.towards.four_octet_asn,
+                    ibgp: egress.towards.ibgp,
+                    add_path: Vec::new(),
+                };
+                let mut wire = BytesMut::new();
+                let mut attributes = outgoing.attributes.clone();
+                export::finish(&mut attributes, &egress.towards);
+                let octets = attributes
+                    .iter()
+                    .try_for_each(|attribute| attribute.to_wire(&ctx, &mut wire))
+                    .map_or(usize::MAX, |()| wire.len());
+                Arc::new(Dressed { outgoing, octets })
+            })
+            // One that cannot be written is one that does not go.
+            .filter(|dressed| dressed.octets < 3072);
+        if self.dressed.len() >= DRESSED_KEPT {
+            self.dressed.clear();
+        }
+        self.dressed
+            .insert(key, (Arc::clone(attrs), dressed.clone()));
+        dressed
+    }
+}
+
+/// Waits until there may be changes of best route to pass on; for ever if
+/// the peer is not told of them.
+async fn changed(learned: &mut Option<Learned>) {
+    match learned {
+        Some(learned) => learned.reader.changed().await,
+        None => std::future::pending().await,
+    }
+}
+
 /// Why an exporter has nothing more to do.
 struct Finished;
 
@@ -193,9 +355,207 @@ impl Exporter {
                     }
                     Err(RecvError::Closed) => return Err(Finished),
                 },
-                () = self.resend.notified() => self.send_all().await?,
+                () = self.resend.notified() => {
+                    self.send_all().await?;
+                    // Everything, again: a new follower starts with the
+                    // whole table.
+                    if let Some(learned) = &mut self.learned {
+                        learned.reader = self.rib.reader();
                     }
                 }
+                () = changed(&mut self.learned) => {
+                    // Changes come in runs: a table arriving, a peer
+                    // going. Gathered for a moment, the routes that go out
+                    // alike go in one message, and the whole run costs one
+                    // look at what each peer is told.
+                    tokio::time::sleep(COALESCE).await;
+                    self.follow().await?;
+                }
+            }
+        }
+    }
+
+    /// The session as far as it decides what goes out on it and how.
+    fn towards(&self) -> Towards {
+        let session = &self.session;
+        Towards {
+            local_asn: self.local_asn,
+            next_hop: session.local_addr.ip(),
+            four_octet_asn: session.four_octet_asn,
+            ibgp: session.ibgp,
+            role: session.role,
+            extended_next_hop: session.extended_next_hop,
+        }
+    }
+
+    /// Takes the next changes of best route from the table and tells the
+    /// peer of those that are for it.
+    async fn follow(&mut self) -> Result<(), Finished> {
+        let towards = self.towards();
+        let Some(learned) = &mut self.learned else {
+            return Ok(());
+        };
+        // Only what is needed is taken while the table is held; and all
+        // of what there is, up to a limit, so that it goes out together.
+        let mut news = Vec::new();
+        while news.len() < EXPORT_BATCH {
+            let progress = learned.reader.poll(EXPORT_BATCH - news.len(), |event| {
+                news.push(match event {
+                    Event::Reset => News::Reset,
+                    Event::Synced => News::Synced,
+                    Event::Best {
+                        prefix,
+                        peer,
+                        attrs,
+                        ..
+                    } => News::Best {
+                        prefix,
+                        peer,
+                        attrs: Arc::clone(attrs),
+                    },
+                    Event::Withdrawn { prefix } => News::Withdrawn(prefix),
+                    // Not given to a follower of the best routes.
+                    Event::Routes { .. } => return,
+                });
+            });
+            if progress.caught_up || progress.events == 0 {
+                break;
+            }
+        }
+
+        let session = &self.session;
+        let egress = Egress {
+            towards,
+            peer_asn: session.peer.asn,
+            rib_peer: session.rib_peer,
+            policy: &session.export,
+            control: self.control,
+        };
+        // What the peer is to end up with for each prefix heard of: the
+        // last word on it counts. While the whole table is being given out
+        // anew, which is shard by shard, this is kept until it is all
+        // there: the routes that share attributes are in every shard, and
+        // are only sent together if gathered first.
+        let mut verdicts = learned.gathered.take().unwrap_or_default();
+        let mut synced = false;
+        // Who the routes are from, asked of the table once for each.
+        let mut sources: HashMap<PeerIndex, Option<bool>> = HashMap::new();
+        for item in news {
+            match item {
+                News::Reset => {
+                    // Whatever was announced and is not heard of again by
+                    // the time the table is through is withdrawn then.
+                    let known = std::mem::take(&mut learned.announced);
+                    learned.stale.get_or_insert_default().extend(known);
+                    verdicts.clear();
+                    synced = false;
+                    // Gathering a table takes memory: only so many sessions
+                    // do it at once, the others wait their turn.
+                    if learned.permit.is_none() {
+                        learned.permit = self.resyncs.clone().acquire_owned().await.ok();
+                    }
+                }
+                News::Synced => {
+                    synced = true;
+                    learned.permit = None;
+                }
+                News::Withdrawn(prefix) => {
+                    verdicts.insert(prefix, None);
+                }
+                News::Best {
+                    prefix,
+                    peer,
+                    attrs,
+                } => {
+                    let ibgp = *sources
+                        .entry(peer)
+                        .or_insert_with(|| self.rib.peer(peer).map(|peer| peer.ibgp));
+                    let dressed = match ibgp {
+                        // What we originate ourselves is announced as
+                        // that, and not as someone else's route for it.
+                        Some(ibgp)
+                            if session.families.carries(&prefix)
+                                && !self.announced.contains(&prefix) =>
+                        {
+                            let source = Source { peer, ibgp };
+                            learned.dressed(&attrs, source, prefix.address_family(), &egress)
+                        }
+                        // The peer has gone, and its routes with it.
+                        _ => None,
+                    };
+                    verdicts.insert(prefix, dressed);
+                }
+            }
+        }
+
+        if learned.permit.is_some() && !synced {
+            // Not all there yet.
+            learned.gathered = Some(verdicts);
+            return Ok(());
+        }
+
+        // Withdrawals first, then announcements, routes that go out alike
+        // together.
+        let mut withdrawn: [Vec<Nlri>; 2] = Default::default();
+        let mut announced: HashMap<(usize, IpVersion), (Arc<Dressed>, Vec<Nlri>)> = HashMap::new();
+        for (prefix, verdict) in verdicts {
+            let version = prefix.address_family();
+            // Heard of again: no longer in doubt, whichever way it goes.
+            let was_stale = learned
+                .stale
+                .as_mut()
+                .is_some_and(|stale| stale.remove(&prefix));
+            match verdict {
+                Some(dressed) => {
+                    learned.announced.insert(prefix);
+                    announced
+                        .entry((Arc::as_ptr(&dressed) as usize, version))
+                        .or_insert_with(|| (dressed, Vec::new()))
+                        .1
+                        .push(prefix.into());
+                }
+                None => {
+                    if learned.announced.remove(&prefix) || was_stale {
+                        withdrawn[version_index(version)].push(prefix.into());
+                    }
+                }
+            }
+        }
+        if synced && let Some(stale) = learned.stale.take() {
+            for prefix in stale {
+                withdrawn[version_index(prefix.address_family())].push(prefix.into());
+            }
+        }
+        for (version, routes) in [IpVersion::Ipv4, IpVersion::Ipv6]
+            .into_iter()
+            .zip(withdrawn)
+        {
+            if routes.is_empty() {
+                continue;
+            }
+            ROUTES_EXPORTED
+                .with_label_values(&["withdrawn"])
+                .inc_by(routes.len() as u64);
+            for update in export::withdrawals(version, &routes, &towards) {
+                session.send(Message::Update(update)).await?;
+            }
+        }
+        for ((_, version), (dressed, routes)) in announced {
+            ROUTES_EXPORTED
+                .with_label_values(&["announced"])
+                .inc_by(routes.len() as u64);
+            let updates = export::announcements(
+                &dressed.outgoing,
+                version,
+                &routes,
+                &towards,
+                dressed.octets,
+            );
+            for update in updates {
+                session.send(Message::Update(update)).await?;
+            }
+        }
+        Ok(())
     }
 
     /// Brings the peer to the routes as they now are: announces every one
@@ -227,15 +587,8 @@ impl Exporter {
             );
             return Ok(());
         }
-        let towards = Towards {
-            local_asn: self.local_asn,
-            next_hop: session.local_addr.ip(),
-            four_octet_asn: session.four_octet_asn,
-            ibgp: session.ibgp,
-            role: session.role,
-            extended_next_hop: session.extended_next_hop,
-        };
-        let Some(update) = build_announce(route, &towards) else {
+        let towards = self.towards();
+        let Some(mut update) = build_announce(route, &towards) else {
             debug!(
                 peer = %session.peer,
                 prefix = %route.prefix,
@@ -243,6 +596,24 @@ impl Exporter {
             );
             return Ok(());
         };
+        // What every route on its way to this peer goes through: whether
+        // its communities let it go there, and what it is marked with.
+        let egress = Egress {
+            towards,
+            peer_asn: session.peer.asn,
+            rib_peer: session.rib_peer,
+            policy: &session.export,
+            control: self.control,
+        };
+        if !egress.dress(&mut update.path_attributes) {
+            debug!(
+                peer = %session.peer,
+                prefix = %route.prefix,
+                "Not announcing route: its communities say not to this peer"
+            );
+            return Ok(());
+        }
+        export::finish(&mut update.path_attributes, &towards);
         debug!(peer = %session.peer, prefix = %route.prefix, "Announcing local route");
         session.send(Message::Update(update)).await?;
         self.announced.insert(route.prefix);
@@ -402,7 +773,7 @@ pub fn build_announce(route: &LocalRoute, towards: &Towards) -> Option<UpdateMes
 /// global address would be and where the link-local one is (RFC 2545 §3
 /// has no form for a link-local address alone). FRR sends it so, and takes
 /// the route for unreachable if it comes by itself; BIRD takes either.
-fn ipv6_next_hop(ours: Ipv6Addr) -> NlriNextHop {
+pub(crate) fn ipv6_next_hop(ours: Ipv6Addr) -> NlriNextHop {
     if ours.is_unicast_link_local() {
         NlriNextHop::Ipv6WithLl {
             global: ours,
@@ -845,7 +1216,13 @@ mod tests {
         fn new() -> Self {
             let registry = SessionRegistry::new();
             let local_rib = LocalRib::spawn();
-            spawn_advertiser(local_rib.clone(), registry.clone(), 65000);
+            spawn_advertiser(
+                local_rib.clone(),
+                Arc::new(RibTable::new(4)),
+                registry.clone(),
+                65000,
+                None,
+            );
             Self {
                 registry,
                 local_rib,

crates/server/src/config.rs

@@ -13,9 +13,10 @@ use bgp_rib::{LocalRoute, LocalRouteAttrs};
 use bgp_rpki::RpkiConfig;
 
 use crate::address::{Endpoint, PeerAddress};
-use crate::error::{ConfigError, ImportProblem, OriginateProblem, PeerProblem};
+use crate::error::{ConfigError, ExportProblem, ImportProblem, OriginateProblem, PeerProblem};
 use crate::family::FamilySet;
-use crate::policy::{ImportPolicy, PeerRole, PrefixSet};
+use crate::marks::{Control, Marks};
+use crate::policy::{ExportPolicy, ImportPolicy, PeerRole, PrefixSet};
 use crate::proto::config::{self as proto, BgpConfig, OriginatedRoute, Peer};
 
 const DEFAULT_BGP_PORT: u16 = 179;
@@ -52,6 +53,11 @@ pub struct Config {
     /// Where route origin validation gets its data; `None` for no
     /// validation.
     pub rpki: Option<RpkiConfig>,
+    /// What a peer is told that has no `export` of its own.
+    pub export: Arc<ExportPolicy>,
+    /// The communities by which routes say who is told of them, if they
+    /// are honoured.
+    pub control: Option<Control>,
 }
 
 #[derive(Debug, Clone)]
@@ -76,6 +82,8 @@ pub struct PeerConfig {
     /// Refuse the peer if it does not say what it is itself (RFC 9234
     /// §4.2, "strict mode").
     pub require_role: bool,
+    /// What the peer is told, and how.
+    pub export: Arc<ExportPolicy>,
 }
 
 impl Config {
@@ -111,6 +119,10 @@ impl Config {
                 .collect::<Result<_, _>>()?
         };
 
+        let export = Arc::new(match proto.export.as_ref() {
+            Some(export) => parse_export(export)?,
+            None => ExportPolicy::default(),
+        });
         let mut peers: Vec<PeerConfig> = Vec::with_capacity(proto.peers.len());
         for peer in proto.peers {
             let name = peer.name.clone();
@@ -118,7 +130,7 @@ impl Config {
                 peer: name.clone(),
                 problem,
             };
-            let peer = parse_peer(peer).map_err(rejected)?;
+            let peer = parse_peer(peer, &export).map_err(rejected)?;
             // RFC 9234 is about what networks are to each other.
             if peer.role.is_some() && peer.remote_asn == proto.local_asn {
                 return Err(rejected(PeerProblem::RoleInOwnAs));
@@ -194,6 +206,10 @@ impl Config {
             http_endpoint,
             kernel: proto.kernel.as_ref().map(parse_kernel),
             rpki,
+            export,
+            control: proto.control_communities.then_some(Control {
+                local_asn: proto.local_asn,
+            }),
         })
     }
 }
@@ -214,7 +230,22 @@ fn parse_kernel(kernel: &proto::Kernel) -> FibConfig {
     }
 }
 
-fn parse_peer(peer: Peer) -> Result<PeerConfig, PeerProblem> {
+fn parse_export(export: &proto::ExportPolicy) -> Result<ExportPolicy, ExportProblem> {
+    Ok(ExportPolicy {
+        learned_routes: export.learned_routes,
+        marks: Marks::parse(
+            &export.remove_communities,
+            &export.add_communities,
+            &export.add_large_communities,
+        )?,
+        prepend: u8::try_from(export.as_path_prepend)
+            .ok()
+            .filter(|prepend| u32::from(*prepend) <= MAX_AS_PATH_PREPEND)
+            .ok_or(ExportProblem::Prepend(export.as_path_prepend))?,
+    })
+}
+
+fn parse_peer(peer: Peer, export: &Arc<ExportPolicy>) -> Result<PeerConfig, PeerProblem> {
     let ep = peer.endpoint.as_ref().ok_or(PeerProblem::MissingEndpoint)?;
     let endpoint = parse_port(ep.port)
         .and_then(|port| Ok(Endpoint::parse(&ep.address, port)?))
@@ -299,6 +330,10 @@ fn parse_peer(peer: Peer) -> Result<PeerConfig, PeerProblem> {
         import,
         role,
         require_role: peer.require_role,
+        export: match peer.export.as_ref() {
+            Some(own) => Arc::new(parse_export(own)?),
+            None => Arc::clone(export),
+        },
     })
 }
 
@@ -387,6 +422,12 @@ fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProb
         accept_as_sets: import.accept_as_sets,
         accept_third_party_next_hop: import.accept_third_party_next_hop,
         own_prefixes: Arc::default(),
+        marks: Marks::parse(
+            &import.remove_communities,
+            &import.add_communities,
+            &import.add_large_communities,
+        )?,
+        local_pref: import.local_pref,
         max_as_path_length: match import.max_as_path_length {
             0 => defaults.max_as_path_length,
             length => u16::try_from(length).map_err(|_| ImportProblem::PathLength(length))?,

crates/server/src/connection.rs

@@ -45,7 +45,7 @@ use crate::metrics::{
     SESSIONS_ESTABLISHED_TOTAL, UPDATE_ERRORS, message_type_name,
 };
 use crate::panic::contained;
-use crate::policy::{self, HeldDown, HoldDowns, ImportPolicy, PeerRole};
+use crate::policy::{self, ExportPolicy, HeldDown, HoldDowns, ImportPolicy, PeerRole};
 use crate::reach::{Reach, neighbor_at};
 use crate::refusals::{PeerRefusals, RefusalLog};
 use crate::routes::{self, Held};
@@ -107,13 +107,15 @@ pub struct PeerSettings {
     pub role: Option<PeerRole>,
     /// Refuse the peer if it does not say what it is (RFC 9234).
     pub require_role: bool,
+    /// What the peer is told, and how.
+    pub export: Arc<ExportPolicy>,
 }
 
 impl PeerSettings {
     /// For a peer that connected to us and is not in the configuration.
     /// Such peers are taken only where that was asked for, on a closed
     /// test network, and there everything they announce is taken.
-    pub fn unconfigured(hold_time_secs: u16) -> Self {
+    pub fn unconfigured(hold_time_secs: u16, export: Arc<ExportPolicy>) -> Self {
         Self {
             name: None,
             expected_asn: None,
@@ -123,6 +125,7 @@ impl PeerSettings {
             import: Arc::new(ImportPolicy::permissive()),
             role: None,
             require_role: false,
+            export,
         }
     }
 }
@@ -138,6 +141,7 @@ impl From<&PeerConfig> for PeerSettings {
             import: Arc::new(peer.import.clone()),
             role: peer.role,
             require_role: peer.require_role,
+            export: Arc::clone(&peer.export),
         }
     }
 }
@@ -448,6 +452,8 @@ struct Session {
     import: Arc<ImportPolicy>,
     /// What the peer is to us, if that is known.
     role: Option<PeerRole>,
+    /// What the peer is told, and how.
+    export: Arc<ExportPolicy>,
     /// Where what is refused is noted.
     refusals: Arc<PeerRefusals>,
 }
@@ -503,6 +509,7 @@ impl Session {
             peer_name: peer.name,
             import: peer.import,
             role: peer.role,
+            export: peer.export,
             said: Said::new(),
         };
         Ok((session, first_actions))
@@ -782,6 +789,7 @@ impl Session {
         );
         handle.role = self.role;
         handle.role_confirmed = negotiated.role_confirmed;
+        handle.export = Arc::clone(&self.export);
         // Routes from other networks are validated, where there is data
         // to validate against and the peer's policy does not say to take
         // invalid ones anyway.
@@ -1181,7 +1189,7 @@ mod tests {
             peer_addr,
             Direction::Inbound,
             server,
-            PeerSettings::unconfigured(90),
+            PeerSettings::unconfigured(90, Arc::default()),
         )
         .unwrap();
 
@@ -1230,7 +1238,7 @@ mod tests {
             peer_addr,
             Direction::Inbound,
             server,
-            PeerSettings::unconfigured(90),
+            PeerSettings::unconfigured(90, Arc::default()),
         )
         .unwrap();
         session.send_hold_time = Duration::from_millis(100);

crates/server/src/error.rs

@@ -176,6 +176,8 @@ pub enum ConfigError {
     HttpPort,
     #[error("rpki: no caches are listed")]
     RpkiNoCaches,
+    #[error("export: {0}")]
+    Export(#[from] ExportProblem),
     #[error("rpki: the port of cache {0:?} is required")]
     RpkiCachePort(String),
     #[error("peer {peer:?}: {problem}")]
@@ -218,6 +220,8 @@ pub enum PeerProblem {
     Role,
     #[error("import: {0}")]
     Import(#[from] ImportProblem),
+    #[error("export: {0}")]
+    Export(#[from] ExportProblem),
 }
 
 /// What is wrong with a peer's `import` policy.
@@ -244,6 +248,17 @@ pub enum ImportProblem {
     },
     #[error("max_as_path_length of {0} is more than a path can hold")]
     PathLength(u32),
+    #[error(transparent)]
+    Marks(#[from] crate::marks::MarksProblem),
+}
+
+/// What is wrong with an `export` policy.
+#[derive(Debug, Clone, PartialEq, Eq, Error)]
+pub enum ExportProblem {
+    #[error(transparent)]
+    Marks(#[from] crate::marks::MarksProblem),
+    #[error("as_path_prepend of {0} is more than 16")]
+    Prepend(u32),
 }
 
 /// What is wrong with one `originate` entry.

crates/server/src/export.rs

@@ -0,0 +1,777 @@
+//! Whether a route goes to a peer, and as what.
+//!
+//! This is asked for every route and every peer it might go to, so it is
+//! plain comparisons on what the route carries; nothing here waits or
+//! looks anything up. [`crate::advertise`] does the sending.
+//!
+//! A route we originate always may go, as far as the rules below let it.
+//! A route learned from a peer is passed on only to peers whose `export`
+//! says so, and then:
+//!
+//! - not back to the peer it came from, and not to a peer whose AS is on
+//!   its path already (it would refuse it as a loop);
+//! - not from one peer in our own AS to another (RFC 4271 §9.2.1: there is
+//!   no route reflection here);
+//! - not up or across if it came down or across: a route marked "only to
+//!   customer" goes to customers only (RFC 9234 §5, egress, 2);
+//! - not where its communities say it is not to go: NO_EXPORT and
+//!   NO_ADVERTISE (RFC 1997), and those of [`Control`].
+//!
+//! What goes is changed as BGP has it: our AS on the path and ourselves as
+//! next hop towards another network, the MED it came with dropped; the
+//! path as it was and a LOCAL_PREF towards a peer in our own AS. To a
+//! route server's client the route goes as it came (RFC 7947 §2.2).
+
+use std::net::IpAddr;
+
+use bgp_packet::message::{
+    AsPathAttribute, AsPathSegment, LocalPrefPathAttribute, MpReachNlriPathAttribute,
+    MpUnreachNlriPathAttribute, NextHopPathAttribute, NlriNextHop, PathAttribute, UpdateMessage,
+};
+use bgp_primitives::constants::{AfiSafi, IpVersion};
+use bgp_primitives::ip_prefix::Nlri;
+use bgp_rib::table::{DEFAULT_LOCAL_PREF, PeerIndex};
+
+use crate::advertise::{Towards, ipv6_next_hop};
+use crate::marks::{self, Control, communities_of, well_known_allow};
+use crate::policy::{ExportPolicy, PeerRole};
+
+/// The session a route may go out on, as far as that decides whether it
+/// does and as what.
+#[derive(Debug, Clone, Copy)]
+pub struct Egress<'a> {
+    pub towards: Towards,
+    pub peer_asn: u32,
+    /// The peer's number in the routing table: what its own routes there
+    /// are marked with.
+    pub rib_peer: PeerIndex,
+    pub policy: &'a ExportPolicy,
+    /// The communities by which routes say who is told of them, if they
+    /// are honoured.
+    pub control: Option<Control>,
+}
+
+/// Where a learned route came from.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub struct Source {
+    pub peer: PeerIndex,
+    /// Whether that peer is in our own AS.
+    pub ibgp: bool,
+}
+
+/// A route ready to be announced on a session: its attributes, but for
+/// the ones that depend on which prefixes go with them.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Outgoing {
+    pub attributes: Vec<PathAttribute>,
+    pub next_hop: NlriNextHop,
+}
+
+impl Egress<'_> {
+    fn transparent(&self) -> bool {
+        !self.towards.ibgp && self.towards.role == Some(PeerRole::RouteServerClient)
+    }
+
+    /// What is common to every route on its way out, originated or
+    /// learned: whether the communities that are for us to act on let it
+    /// go to this peer, and what it is marked with when it does. Returns
+    /// whether it goes.
+    pub fn dress(&self, attributes: &mut Vec<PathAttribute>) -> bool {
+        let towards = &self.towards;
+        let (communities, large) = communities_of(attributes);
+        if let Some(control) = self.control {
+            if !control.allows(communities, large, self.peer_asn) {
+                return false;
+            }
+            control.strip(attributes);
+        }
+        self.policy.marks.apply(attributes);
+        if self.policy.prepend > 0 && !towards.ibgp {
+            let ours = std::iter::repeat_n(towards.local_asn, usize::from(self.policy.prepend));
+            for attribute in attributes.iter_mut() {
+                if let PathAttribute::AsPath(path) = attribute {
+                    prepend(path, ours);
+                    break;
+                }
+            }
+        }
+        // A route that goes down to a customer or across to a peer may not
+        // come up again from there: it is marked, for every network after
+        // us to see (RFC 9234 §5, egress, 1).
+        let marked = attributes
+            .iter()
+            .any(|attribute| matches!(attribute, PathAttribute::OnlyToCustomer(_)));
+        if !marked
+            && let Some(marked_by) = towards
+                .role
+                .filter(|_| !towards.ibgp)
+                .and_then(|role| role.only_to_customer_on_export(towards.local_asn))
+        {
+            marks::put(attributes, PathAttribute::OnlyToCustomer(marked_by));
+        }
+        true
+    }
+
+    /// The route with `attributes` and `next_hop`, learned from `source`,
+    /// as it is announced on this session for prefixes of IP version
+    /// `routes`; `None` if it is not to be.
+    pub fn outgoing(
+        &self,
+        mut attributes: Vec<PathAttribute>,
+        next_hop: Option<NlriNextHop>,
+        source: Source,
+        routes: IpVersion,
+    ) -> Option<Outgoing> {
+        let towards = &self.towards;
+        if !self.policy.learned_routes || source.peer == self.rib_peer {
+            return None;
+        }
+        if source.ibgp && towards.ibgp {
+            return None;
+        }
+        let mut came_with = next_hop;
+        let mut marked = false;
+        let mut on_path = false;
+        for attribute in &attributes {
+            match attribute {
+                PathAttribute::NextHop(next_hop) => {
+                    came_with = came_with.or(Some(NlriNextHop::Ipv4(next_hop.0)));
+                }
+                PathAttribute::OnlyToCustomer(_) => marked = true,
+                PathAttribute::AsPath(path) => {
+                    on_path = path
+                        .segments
+                        .iter()
+                        .any(|segment| segment.path.contains(&self.peer_asn));
+                }
+                _ => {}
+            }
+        }
+        // The peer would refuse it as having been there before.
+        if on_path && !towards.ibgp {
+            return None;
+        }
+        // Whoever sent it said how far it may go (RFC 1997). On a route
+        // of our own the same communities are for those who get it from
+        // us, and it goes out with them.
+        if !well_known_allow(communities_of(&attributes).0, towards.ibgp) {
+            return None;
+        }
+        // It came down or across once: it goes on down only.
+        if marked
+            && !towards.ibgp
+            && towards
+                .role
+                .is_some_and(|role| !role.may_be_sent_only_to_customer_routes())
+        {
+            return None;
+        }
+
+        let transparent = self.transparent();
+        attributes.retain_mut(|attribute| match attribute {
+            // Ours to say, each way.
+            PathAttribute::NextHop(_)
+            | PathAttribute::MpReachNlri(_)
+            | PathAttribute::MpUnreachNlri(_)
+            | PathAttribute::As4Path(_) => false,
+            // For the routers of the AS it was learned in, and for ours
+            // (RFC 4271 §5.1.4, §5.1.5).
+            PathAttribute::MultiExitDisc(_) => towards.ibgp || transparent,
+            PathAttribute::LocalPref(_) => towards.ibgp,
+            // What we do not understand goes on if it says it may, marked
+            // as having passed a router that did not look at it; and no
+            // further if not (RFC 4271 §5).
+            PathAttribute::UnknownPathAttribute { flags, .. } => {
+                let passes = flags.optional() && flags.transitive();
+                if passes {
+                    flags.set_partial(true);
+                }
+                passes
+            }
+            _ => true,
+        });
+        if towards.ibgp {
+            if !attributes
+                .iter()
+                .any(|attribute| matches!(attribute, PathAttribute::LocalPref(_)))
+            {
+                marks::put(
+                    &mut attributes,
+                    PathAttribute::LocalPref(LocalPrefPathAttribute(DEFAULT_LOCAL_PREF)),
+                );
+            }
+        } else if !transparent {
+            let ours = std::iter::once(towards.local_asn);
+            match attributes.iter_mut().find_map(|attribute| match attribute {
+                PathAttribute::AsPath(path) => Some(path),
+                _ => None,
+            }) {
+                Some(path) => prepend(path, ours),
+                None => marks::put(
+                    &mut attributes,
+                    AsPathAttribute::from_asns(vec![towards.local_asn]),
+                ),
+            }
+        }
+        if !self.dress(&mut attributes) {
+            return None;
+        }
+
+        // Ourselves, unless we are a route server and the client can use
+        // the next hop the route came with.
+        let ours = match (routes, towards.next_hop) {
+            (IpVersion::Ipv4, IpAddr::V4(ours)) => NlriNextHop::Ipv4(ours),
+            (IpVersion::Ipv6, IpAddr::V6(ours)) => ipv6_next_hop(ours),
+            (IpVersion::Ipv4, IpAddr::V6(ours)) if towards.extended_next_hop => ipv6_next_hop(ours),
+            _ => return None,
+        };
+        let same_kind = |theirs: &NlriNextHop| {
+            matches!(theirs, NlriNextHop::Ipv4(_)) == matches!(ours, NlriNextHop::Ipv4(_))
+        };
+        let next_hop = came_with
+            .filter(|theirs| transparent && same_kind(theirs))
+            .unwrap_or(ours);
+        Some(Outgoing {
+            attributes,
+            next_hop,
+        })
+    }
+}
+
+/// Puts `ours` at the front of `path`.
+fn prepend(path: &mut AsPathAttribute, ours: impl Iterator<Item = u32>) {
+    match path.segments.first_mut() {
+        Some(first) if first.ordered => {
+            first.path.splice(0..0, ours);
+        }
+        _ => path.segments.insert(
+            0,
+            AsPathSegment {
+                ordered: true,
+                path: ours.collect(),
+            },
+        ),
+    }
+}
+
+/// Makes `attributes` ready for the wire of a session: the path also as
+/// AS4_PATH where the session cannot carry its numbers (RFC 6793 §4.2.2),
+/// and everything in the order of the type codes.
+pub(crate) fn finish(attributes: &mut Vec<PathAttribute>, towards: &Towards) {
+    attributes.retain(|attribute| !matches!(attribute, PathAttribute::As4Path(_)));
+    if !towards.four_octet_asn {
+        let as4_path = attributes.iter().find_map(|attribute| match attribute {
+            PathAttribute::AsPath(path) => path.as4_path(),
+            _ => None,
+        });
+        attributes.extend(as4_path);
+    }
+    attributes.sort_by_key(PathAttribute::type_code);
+}
+
+/// How many octets a message has room for after its header and the
+/// lengths every UPDATE has.
+const UPDATE_ROOM: usize = 4096 - 19 - 4;
+
+/// The most octets a prefix takes in an UPDATE: a path identifier, a
+/// length, and the address.
+fn nlri_octets(routes: IpVersion) -> usize {
+    4 + 1 + usize::from(routes.bits()) / 8
+}
+
+/// The UPDATEs that announce `routes`, all of IP version `version`, as
+/// `outgoing`: as few as they fit in. `attributes_octets` is how long
+/// `outgoing`'s attributes are on this session's wire.
+pub(crate) fn announcements(
+    outgoing: &Outgoing,
+    version: IpVersion,
+    routes: &[Nlri],
+    towards: &Towards,
+    attributes_octets: usize,
+) -> Vec<UpdateMessage> {
+    // What the next hop takes, in its own attribute or with the prefixes,
+    // and a little to spare.
+    const NEXT_HOP_ROOM: usize = 64;
+    let room = UPDATE_ROOM.saturating_sub(attributes_octets + NEXT_HOP_ROOM);
+    let at_once = (room / nlri_octets(version)).max(1);
+    let in_body = matches!(outgoing.next_hop, NlriNextHop::Ipv4(_)) && version == IpVersion::Ipv4;
+    routes
+        .chunks(at_once)
+        .map(|routes| {
+            let mut attributes = outgoing.attributes.clone();
+            let mut announced_routes = Vec::new();
+            match outgoing.next_hop {
+                NlriNextHop::Ipv4(next_hop) if in_body => {
+                    attributes.push(PathAttribute::NextHop(NextHopPathAttribute(next_hop)));
+                    announced_routes = routes.to_vec();
+                }
+                next_hop => attributes.push(PathAttribute::MpReachNlri(MpReachNlriPathAttribute {
+                    family: AfiSafi::unicast(version),
+                    next_hop,
+                    prefixes: routes.to_vec(),
+                })),
+            }
+            finish(&mut attributes, towards);
+            UpdateMessage {
+                withdrawn_routes: vec![],
+                path_attributes: attributes,
+                announced_routes,
+            }
+        })
+        .collect()
+}
+
+/// The UPDATEs that withdraw `routes`, all of IP version `version`.
+pub(crate) fn withdrawals(
+    version: IpVersion,
+    routes: &[Nlri],
+    towards: &Towards,
+) -> Vec<UpdateMessage> {
+    let at_once = (UPDATE_ROOM - 16) / nlri_octets(version);
+    // IPv4 routes that went in MP_REACH_NLRI are withdrawn the same way.
+    let in_body =
+        version == IpVersion::Ipv4 && !(towards.extended_next_hop && towards.next_hop.is_ipv6());
+    routes
+        .chunks(at_once)
+        .map(|routes| {
+            if in_body {
+                UpdateMessage {
+                    withdrawn_routes: routes.to_vec(),
+                    path_attributes: vec![],
+                    announced_routes: vec![],
+                }
+            } else {
+                UpdateMessage {
+                    withdrawn_routes: vec![],
+                    path_attributes: vec![PathAttribute::MpUnreachNlri(
+                        MpUnreachNlriPathAttribute {
+                            family: AfiSafi::unicast(version),
+                            prefixes: routes.to_vec(),
+                        },
+                    )],
+                    announced_routes: vec![],
+                }
+            }
+        })
+        .collect()
+}
+
+#[cfg(test)]
+mod tests {
+    use std::net::Ipv4Addr;
+
+    use bgp_packet::message::{
+        CommunitiesPathAttribute, MultiExitDiscPathAttribute, OriginPathAttribute,
+        PathAttributeFlags,
+    };
+    use bgp_primitives::ip_prefix::IpPrefix;
+    use bgp_rib::table::{PeerInfo, RibTable};
+
+    use super::*;
+    use crate::marks::Marks;
+
+    const LOCAL: u32 = 64601;
+    const THEM: u32 = 64602;
+
+    /// Two peers' numbers in a table: the one a route is from, and the one
+    /// it might go to.
+    fn peers() -> (PeerIndex, PeerIndex) {
+        let table = RibTable::new(1);
+        let peer = |n: u8| {
+            table
+                .add_peer(PeerInfo {
+                    asn: 64600 + u32::from(n),
+                    router_id: Ipv4Addr::new(10, 0, 0, n),
+                    address: Ipv4Addr::new(192, 0, 2, n).into(),
+                    interface: 0,
+                    ibgp: false,
+                })
+                .unwrap()
+        };
+        (peer(1), peer(2))
+    }
+
+    fn towards(role: Option<PeerRole>, ibgp: bool) -> Towards {
+        Towards {
+            local_asn: LOCAL,
+            next_hop: Ipv4Addr::new(192, 0, 2, 1).into(),
+            four_octet_asn: true,
+            ibgp,
+            role,
+            extended_next_hop: false,
+        }
+    }
+
+    fn all_routes() -> ExportPolicy {
+        ExportPolicy {
+            learned_routes: true,
+            ..ExportPolicy::default()
+        }
+    }
+
+    /// A route from AS 64700 by way of AS 3356, next hop 192.0.2.9.
+    fn route(more: Vec<PathAttribute>) -> Vec<PathAttribute> {
+        let mut attributes = vec![
+            PathAttribute::Origin(OriginPathAttribute::IGP),
+            AsPathAttribute::from_asns(vec![3356, 64700]),
+            PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 9))),
+        ];
+        attributes.extend(more);
+        attributes
+    }
+
+    fn communities(communities: &[(u16, u16)]) -> PathAttribute {
+        PathAttribute::Communities(CommunitiesPathAttribute(communities.to_vec()))
+    }
+
+    struct Case<'a> {
+        policy: &'a ExportPolicy,
+        role: Option<PeerRole>,
+        to_ibgp: bool,
+        from_ibgp: bool,
+        control: Option<Control>,
+    }
+
+    impl Case<'_> {
+        fn out(&self, attributes: Vec<PathAttribute>) -> Option<Outgoing> {
+            let (from, to) = peers();
+            let egress = Egress {
+                towards: towards(self.role, self.to_ibgp),
+                peer_asn: if self.to_ibgp { LOCAL } else { THEM },
+                rib_peer: to,
+                policy: self.policy,
+                control: self.control,
+            };
+            let source = Source {
+                peer: from,
+                ibgp: self.from_ibgp,
+            };
+            egress.outgoing(attributes, None, source, IpVersion::Ipv4)
+        }
+    }
+
+    fn plain(policy: &ExportPolicy) -> Case<'_> {
+        Case {
+            policy,
+            role: None,
+            to_ibgp: false,
+            from_ibgp: false,
+            control: None,
+        }
+    }
+
+    /// To another network: our AS on the path, ourselves as next hop, and
+    /// nothing that was between the route's last AS and us.
+    #[test]
+    fn a_route_is_passed_on_to_another_network_as_bgp_has_it() {
+        let policy = all_routes();
+        let came = route(vec![
+            PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
+            PathAttribute::LocalPref(LocalPrefPathAttribute(300)),
+            communities(&[(3356, 3)]),
+            // Understood by nobody here: one that may go on, one that may
+            // not.
+            PathAttribute::UnknownPathAttribute {
+                flags: PathAttributeFlags::new(true, true, false, false),
+                type_code: 200,
+                payload: vec![1],
+            },
+            PathAttribute::UnknownPathAttribute {
+                flags: PathAttributeFlags::new(true, false, false, false),
+                type_code: 201,
+                payload: vec![2],
+            },
+        ]);
+        let outgoing = plain(&policy).out(came).unwrap();
+        assert_eq!(
+            outgoing.next_hop,
+            NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 1))
+        );
+        assert_eq!(
+            outgoing.attributes,
+            [
+                PathAttribute::Origin(OriginPathAttribute::IGP),
+                AsPathAttribute::from_asns(vec![LOCAL, 3356, 64700]),
+                communities(&[(3356, 3)]),
+                PathAttribute::UnknownPathAttribute {
+                    // Marked as having passed a router that did not know it.
+                    flags: PathAttributeFlags::new(true, true, true, false),
+                    type_code: 200,
+                    payload: vec![1],
+                },
+            ]
+        );
+    }
+
+    #[test]
+    fn a_route_is_passed_on_only_where_it_may_go() {
+        let policy = all_routes();
+        let (from, _) = peers();
+        // Only if the peer is told of learned routes at all.
+        let originated_only = ExportPolicy::default();
+        assert!(plain(&originated_only).out(route(vec![])).is_none());
+        assert!(plain(&policy).out(route(vec![])).is_some());
+
+        // Not back to where it came from.
+        let egress = Egress {
+            towards: towards(None, false),
+            peer_asn: THEM,
+            rib_peer: from,
+            policy: &policy,
+            control: None,
+        };
+        let source = Source {
+            peer: from,
+            ibgp: false,
+        };
+        assert!(
+            egress
+                .outgoing(route(vec![]), None, source, IpVersion::Ipv4)
+                .is_none()
+        );
+
+        // Not to a peer whose AS it has been through.
+        let mut through_them = route(vec![]);
+        through_them[1] = AsPathAttribute::from_asns(vec![3356, THEM, 64700]);
+        assert!(plain(&policy).out(through_them).is_none());
+
+        // Not where the communities it came with say not to (RFC 1997).
+        let no_export = route(vec![communities(&[(0xFFFF, 0xFF01)])]);
+        assert!(plain(&policy).out(no_export.clone()).is_none());
+        let internal = Case {
+            to_ibgp: true,
+            ..plain(&policy)
+        };
+        assert!(internal.out(no_export).is_some());
+        let no_advertise = route(vec![communities(&[(0xFFFF, 0xFF02)])]);
+        assert!(internal.out(no_advertise).is_none());
+    }
+
+    /// Inside our own AS the path is as it came and LOCAL_PREF goes along;
+    /// and a route does not go from one internal peer to another.
+    #[test]
+    fn a_route_goes_to_a_peer_in_our_own_as_unchanged_but_for_the_next_hop() {
+        let policy = all_routes();
+        let internal = Case {
+            to_ibgp: true,
+            ..plain(&policy)
+        };
+        let came = route(vec![PathAttribute::MultiExitDisc(
+            MultiExitDiscPathAttribute(50),
+        )]);
+        let outgoing = internal.out(came).unwrap();
+        assert_eq!(
+            outgoing.attributes,
+            [
+                PathAttribute::Origin(OriginPathAttribute::IGP),
+                AsPathAttribute::from_asns(vec![3356, 64700]),
+                PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
+                PathAttribute::LocalPref(LocalPrefPathAttribute(DEFAULT_LOCAL_PREF)),
+            ]
+        );
+        // The preference import policy gave it is what the others hear.
+        let preferred = route(vec![PathAttribute::LocalPref(LocalPrefPathAttribute(300))]);
+        assert!(
+            internal
+                .out(preferred)
+                .unwrap()
+                .attributes
+                .contains(&PathAttribute::LocalPref(LocalPrefPathAttribute(300)))
+        );
+
+        let between_internal = Case {
+            to_ibgp: true,
+            from_ibgp: true,
+            ..plain(&policy)
+        };
+        assert!(between_internal.out(route(vec![])).is_none());
+        let from_internal = Case {
+            from_ibgp: true,
+            ..plain(&policy)
+        };
+        assert!(from_internal.out(route(vec![])).is_some());
+    }
+
+    /// RFC 9234 §5, egress: what came down or across goes on down only,
+    /// and what goes down or across is marked.
+    #[test]
+    fn a_route_only_for_customers_goes_to_customers_only() {
+        use PeerRole::*;
+        let policy = all_routes();
+        let marked = || route(vec![PathAttribute::OnlyToCustomer(3356)]);
+        for (role, goes) in [
+            (Customer, true),
+            (RouteServerClient, true),
+            (Provider, false),
+            (Peer, false),
+            (RouteServer, false),
+        ] {
+            let case = Case {
+                role: Some(role),
+                ..plain(&policy)
+            };
+            assert_eq!(case.out(marked()).is_some(), goes, "{role:?}");
+        }
+        // The mark it has is kept; one it lacks is put on, with our AS,
+        // on the way down or across.
+        let mark = |role, came: Vec<PathAttribute>| {
+            let case = Case {
+                role: Some(role),
+                ..plain(&policy)
+            };
+            case.out(came)
+                .unwrap()
+                .attributes
+                .iter()
+                .find_map(|attribute| match attribute {
+                    PathAttribute::OnlyToCustomer(asn) => Some(*asn),
+                    _ => None,
+                })
+        };
+        assert_eq!(mark(Customer, marked()), Some(3356));
+        assert_eq!(mark(Customer, route(vec![])), Some(LOCAL));
+        assert_eq!(mark(Peer, route(vec![])), Some(LOCAL));
+        assert_eq!(mark(Provider, route(vec![])), None);
+        // A peer that is nothing in particular: no rule, no mark.
+        assert!(plain(&policy).out(marked()).is_some());
+    }
+
+    /// RFC 7947 §2.2: a route server is not on the path, and not in the
+    /// way of the packets.
+    #[test]
+    fn a_route_server_passes_routes_on_as_they_came() {
+        let policy = all_routes();
+        let client = Case {
+            role: Some(PeerRole::RouteServerClient),
+            ..plain(&policy)
+        };
+        let came = route(vec![PathAttribute::MultiExitDisc(
+            MultiExitDiscPathAttribute(50),
+        )]);
+        let outgoing = client.out(came).unwrap();
+        assert_eq!(
+            outgoing.next_hop,
+            NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 9))
+        );
+        assert_eq!(
+            outgoing.attributes,
+            [
+                PathAttribute::Origin(OriginPathAttribute::IGP),
+                AsPathAttribute::from_asns(vec![3356, 64700]),
+                PathAttribute::MultiExitDisc(MultiExitDiscPathAttribute(50)),
+                PathAttribute::OnlyToCustomer(LOCAL),
+            ]
+        );
+    }
+
+    /// The marks of `export`, and the communities that say who is told.
+    #[test]
+    fn communities_decide_and_are_changed_on_the_way_out() {
+        let strings =
+            |texts: &[&str]| -> Vec<String> { texts.iter().map(|text| text.to_string()).collect() };
+        let policy = ExportPolicy {
+            learned_routes: true,
+            marks: Marks::parse(&strings(&["64601:900"]), &strings(&["64601:5"]), &[]).unwrap(),
+            prepend: 2,
+        };
+        let control = Some(Control { local_asn: LOCAL });
+        let case = Case {
+            control,
+            ..plain(&policy)
+        };
+        // Not to this peer; to this peer only; and with nothing said.
+        assert!(case.out(route(vec![communities(&[(0, 64602)])])).is_none());
+        let outgoing = case
+            .out(route(vec![communities(&[
+                (64601, 64602),
+                (3356, 3),
+                (64601, 900),
+            ])]))
+            .unwrap();
+        assert_eq!(
+            outgoing.attributes,
+            [
+                PathAttribute::Origin(OriginPathAttribute::IGP),
+                // Ours once for passing it on, and twice more by policy.
+                AsPathAttribute::from_asns(vec![LOCAL, LOCAL, LOCAL, 3356, 64700]),
+                // What was for us is gone; the router's mark is on.
+                communities(&[(3356, 3), (64601, 5)]),
+            ]
+        );
+        assert!(
+            case.out(route(vec![communities(&[(64601, 64999)])]))
+                .is_none(),
+            "for another peer only"
+        );
+        // Without control communities the same are only communities.
+        assert!(
+            plain(&policy)
+                .out(route(vec![communities(&[(0, 64602)])]))
+                .is_some()
+        );
+    }
+
+    /// As many prefixes to a message as fit, and the path as AS4_PATH too
+    /// where the session cannot carry it.
+    #[test]
+    fn routes_that_go_out_alike_share_their_messages() {
+        let outgoing = Outgoing {
+            attributes: vec![
+                PathAttribute::Origin(OriginPathAttribute::IGP),
+                AsPathAttribute::from_asns(vec![LOCAL, 4_200_000_001]),
+            ],
+            next_hop: NlriNextHop::Ipv4(Ipv4Addr::new(192, 0, 2, 1)),
+        };
+        let routes: Vec<Nlri> = (0..3000u32)
+            .map(|n| {
+                let [_, _, a, b] = n.to_be_bytes();
+                IpPrefix::new(IpVersion::Ipv4, [100, a, b, 0], 24)
+                    .unwrap()
+                    .into()
+            })
+            .collect();
+        let narrow = Towards {
+            four_octet_asn: false,
+            ..towards(None, false)
+        };
+        let updates = announcements(&outgoing, IpVersion::Ipv4, &routes, &narrow, 40);
+        assert!(updates.len() < 8, "{} messages", updates.len());
+        let told: usize = updates.iter().map(|u| u.announced_routes.len()).sum();
+        assert_eq!(told, routes.len());
+        let ctx = bgp_primitives::parser::ParserContext {
+            four_octet_asn: false,
+            ibgp: false,
+            add_path: Vec::new(),
+        };
+        for update in &updates {
+            let codes: Vec<u8> = update
+                .path_attributes
+                .iter()
+                .map(PathAttribute::type_code)
+                .collect();
+            assert_eq!(codes, [1, 2, 3, 17], "AS4_PATH with it");
+            let mut wire = bytes::BytesMut::new();
+            bgp_packet::message::Message::Update(update.clone())
+                .to_wire(&ctx, &mut wire)
+                .expect("it fits a message");
+        }
+        // IPv6 routes, and IPv4 ones with an IPv6 next hop, go in
+        // MP_REACH_NLRI; and are withdrawn in MP_UNREACH_NLRI.
+        let v6 = Outgoing {
+            next_hop: NlriNextHop::Ipv6("2001:db8::1".parse().unwrap()),
+            ..outgoing
+        };
+        let update = &announcements(&v6, IpVersion::Ipv4, &routes[..2], &narrow, 40)[0];
+        assert!(update.announced_routes.is_empty());
+        assert!(update.path_attributes.iter().any(|attribute| matches!(
+            attribute,
+            PathAttribute::MpReachNlri(mp) if mp.prefixes.len() == 2
+        )));
+        let gone = withdrawals(IpVersion::Ipv4, &routes, &narrow);
+        assert_eq!(
+            gone.iter().map(|u| u.withdrawn_routes.len()).sum::<usize>(),
+            routes.len()
+        );
+        assert!(gone.len() < 8);
+    }
+}

crates/server/src/lib.rs

@@ -5,8 +5,10 @@ pub mod config;
 mod connection;
 mod dialer;
 pub mod error;
+pub mod export;
 pub mod family;
 mod fsm;
+pub mod marks;
 pub mod metrics;
 mod panic;
 pub mod policy;

crates/server/src/marks.rs

@@ -0,0 +1,497 @@
+//! What routes are marked with, and what marks tell us to do.
+//!
+//! A network marks routes with communities to remember things about
+//! them: where a route was learned, which router passed it on. And a
+//! route's communities can say who is to be told of it. Both are fixed
+//! rules with a few numbers in them, so they are configuration, and what
+//! they cost is a look through a route's communities:
+//!
+//! - [`Marks`]: communities taken off and put on every route from a peer
+//!   (`import`) or to one (`export`).
+//! - [`Control`]: communities on a route that say which peers it is, and
+//!   is not, passed on to.
+
+use bgp_packet::community::{CommunityError, parse_large, parse_standard};
+use bgp_packet::message::{
+    CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity, PathAttribute,
+};
+
+/// A standard community, or all those with one half or both left open:
+/// `64601:100`, `64601:*`, `*:*`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct Pattern(Option<u16>, Option<u16>);
+
+/// The same for a large community: `64601:1:*`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct LargePattern(Option<u32>, Option<u32>, Option<u32>);
+
+fn fits<T: PartialEq>(wanted: Option<T>, has: T) -> bool {
+    wanted.is_none_or(|wanted| wanted == has)
+}
+
+impl Pattern {
+    fn matches(&self, community: (u16, u16)) -> bool {
+        fits(self.0, community.0) && fits(self.1, community.1)
+    }
+}
+
+impl LargePattern {
+    fn matches(&self, community: &LargeCommunity) -> bool {
+        fits(self.0, community.global_admin)
+            && fits(self.1, community.data_1)
+            && fits(self.2, community.data_2)
+    }
+}
+
+/// What is wrong with the communities of a policy.
+#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)]
+pub enum MarksProblem {
+    #[error(transparent)]
+    Community(#[from] CommunityError),
+    #[error(
+        "{0:?} is not a community, or a pattern of them: 64601:100, 64601:*, or with three \
+         numbers for large communities"
+    )]
+    Pattern(String),
+}
+
+/// One part of a pattern: a number, or `*` for any.
+fn part<T: std::str::FromStr>(text: &str, whole: &str) -> Result<Option<T>, MarksProblem> {
+    match text.trim() {
+        "*" => Ok(None),
+        number => number
+            .parse()
+            .map(Some)
+            .map_err(|_| MarksProblem::Pattern(whole.to_string())),
+    }
+}
+
+/// What is done to the communities of every route that goes one way on
+/// one session: some are taken off, then some are put on.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct Marks {
+    remove: Vec<Pattern>,
+    remove_large: Vec<LargePattern>,
+    add: Vec<(u16, u16)>,
+    add_large: Vec<LargeCommunity>,
+}
+
+impl Marks {
+    /// From the configuration. `remove` has patterns of standard
+    /// communities (two parts) and of large ones (three).
+    pub fn parse(
+        remove: &[String],
+        add: &[String],
+        add_large: &[String],
+    ) -> Result<Self, MarksProblem> {
+        let mut marks = Self::default();
+        for text in remove {
+            let parts: Vec<&str> = text.split(':').collect();
+            match parts.as_slice() {
+                [high, low] => marks
+                    .remove
+                    .push(Pattern(part(high, text)?, part(low, text)?)),
+                [admin, first, second] => marks.remove_large.push(LargePattern(
+                    part(admin, text)?,
+                    part(first, text)?,
+                    part(second, text)?,
+                )),
+                _ => return Err(MarksProblem::Pattern(text.clone())),
+            }
+        }
+        for text in add {
+            let community = parse_standard(text)?;
+            if !marks.add.contains(&community) {
+                marks.add.push(community);
+            }
+        }
+        for text in add_large {
+            let community = parse_large(text)?;
+            if !marks.add_large.contains(&community) {
+                marks.add_large.push(community);
+            }
+        }
+        Ok(marks)
+    }
+
+    pub fn is_empty(&self) -> bool {
+        *self == Self::default()
+    }
+
+    /// Does it to `attributes`.
+    pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
+        if !self.remove.is_empty() || !self.add.is_empty() {
+            let mut communities = take_communities(attributes);
+            communities.retain(|community| !self.remove.iter().any(|p| p.matches(*community)));
+            for community in &self.add {
+                if !communities.contains(community) {
+                    communities.push(*community);
+                }
+            }
+            put_communities(attributes, communities);
+        }
+        if !self.remove_large.is_empty() || !self.add_large.is_empty() {
+            let mut large = take_large_communities(attributes);
+            large.retain(|community| !self.remove_large.iter().any(|p| p.matches(community)));
+            for community in &self.add_large {
+                if !large.contains(community) {
+                    large.push(community.clone());
+                }
+            }
+            put_large_communities(attributes, large);
+        }
+    }
+}
+
+/// Takes the standard communities out of `attributes`.
+fn take_communities(attributes: &mut Vec<PathAttribute>) -> Vec<(u16, u16)> {
+    let at = attributes
+        .iter()
+        .position(|attribute| matches!(attribute, PathAttribute::Communities(_)));
+    match at.map(|at| attributes.remove(at)) {
+        Some(PathAttribute::Communities(communities)) => communities.0,
+        _ => Vec::new(),
+    }
+}
+
+fn take_large_communities(attributes: &mut Vec<PathAttribute>) -> Vec<LargeCommunity> {
+    let at = attributes
+        .iter()
+        .position(|attribute| matches!(attribute, PathAttribute::LargeCommunities(_)));
+    match at.map(|at| attributes.remove(at)) {
+        Some(PathAttribute::LargeCommunities(large)) => large.communities,
+        _ => Vec::new(),
+    }
+}
+
+fn put_communities(attributes: &mut Vec<PathAttribute>, communities: Vec<(u16, u16)>) {
+    if !communities.is_empty() {
+        put(
+            attributes,
+            PathAttribute::Communities(CommunitiesPathAttribute(communities)),
+        );
+    }
+}
+
+fn put_large_communities(attributes: &mut Vec<PathAttribute>, communities: Vec<LargeCommunity>) {
+    if !communities.is_empty() {
+        put(
+            attributes,
+            PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute { communities }),
+        );
+    }
+}
+
+/// Puts `attribute` among `attributes`, in place of one of its type if
+/// there is one, and else where it belongs in the order of type codes.
+pub(crate) fn put(attributes: &mut Vec<PathAttribute>, attribute: PathAttribute) {
+    let type_code = attribute.type_code();
+    match attributes
+        .iter()
+        .position(|other| other.type_code() >= type_code)
+    {
+        Some(at) if attributes[at].type_code() == type_code => attributes[at] = attribute,
+        Some(at) => attributes.insert(at, attribute),
+        None => attributes.push(attribute),
+    }
+}
+
+/// The communities by which a route says who is told of it, as Internet
+/// exchanges' route servers have them. With `L` our own AS number and `P`
+/// a peer's:
+///
+/// | Standard | Large | |
+/// |---|---|---|
+/// | `0:P` | `L:0:P` | not to the peer with AS `P` |
+/// | `L:P` | `L:1:P` | to the peer with AS `P`, and with that only to peers named so |
+/// | `0:L` | `L:0:0` | to nobody |
+///
+/// A route with none of them goes to everyone. What says "not" wins over
+/// what says "to". A standard community has room for AS numbers below
+/// 65536 only; the large ones are for the others, and work for all.
+///
+/// These are taken off a route as it is passed on: they were for us.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct Control {
+    pub local_asn: u32,
+}
+
+/// The well-known communities that limit how far a route goes
+/// (RFC 1997).
+const NO_EXPORT: (u16, u16) = (0xFFFF, 0xFF01);
+const NO_ADVERTISE: (u16, u16) = (0xFFFF, 0xFF02);
+const NO_EXPORT_SUBCONFED: (u16, u16) = (0xFFFF, 0xFF03);
+
+/// Whether a route with these standard communities may go to a peer at
+/// all, as far as the well-known ones say: NO_ADVERTISE keeps it from
+/// everyone, NO_EXPORT from other networks. (There are no confederations
+/// here, so NO_EXPORT_SUBCONFED is the same as NO_EXPORT.)
+pub fn well_known_allow(communities: &[(u16, u16)], peer_is_internal: bool) -> bool {
+    !communities.iter().any(|community| {
+        *community == NO_ADVERTISE
+            || (!peer_is_internal && matches!(*community, NO_EXPORT | NO_EXPORT_SUBCONFED))
+    })
+}
+
+impl Control {
+    /// Our AS number as half a standard community, if it fits.
+    fn local_16(&self) -> Option<u16> {
+        u16::try_from(self.local_asn).ok()
+    }
+
+    /// Whether a route with these communities is passed on to the peer
+    /// with AS number `peer_asn`.
+    pub fn allows(
+        &self,
+        communities: &[(u16, u16)],
+        large: &[LargeCommunity],
+        peer_asn: u32,
+    ) -> bool {
+        let local_16 = self.local_16();
+        let peer_16 = u16::try_from(peer_asn).ok();
+        let (mut not_to_peer, mut to_peer, mut to_nobody, mut to_some) =
+            (false, false, false, false);
+        for (high, low) in communities {
+            if *high == 0 {
+                not_to_peer |= Some(*low) == peer_16;
+                to_nobody |= Some(*low) == local_16;
+            } else if Some(*high) == local_16 {
+                to_peer |= Some(*low) == peer_16;
+                // `L:L` is an exchange's "to everyone", which is what a
+                // route with nothing on it gets anyway.
+                to_some |= Some(*low) != local_16;
+            }
+        }
+        for community in large {
+            if community.global_admin != self.local_asn {
+                continue;
+            }
+            match (community.data_1, community.data_2) {
+                (0, 0) => to_nobody = true,
+                (0, asn) => not_to_peer |= asn == peer_asn,
+                (1, asn) => {
+                    to_peer |= asn == peer_asn;
+                    to_some = true;
+                }
+                _ => {}
+            }
+        }
+        !not_to_peer && (to_peer || !(to_nobody || to_some))
+    }
+
+    /// Takes off `attributes` the communities that were for us to act on.
+    pub fn strip(&self, attributes: &mut Vec<PathAttribute>) {
+        let local_16 = self.local_16();
+        let ours = |community: &(u16, u16)| community.0 == 0 || Some(community.0) == local_16;
+        let has = attributes.iter().any(|attribute| match attribute {
+            PathAttribute::Communities(communities) => communities.0.iter().any(ours),
+            _ => false,
+        });
+        if has {
+            let mut communities = take_communities(attributes);
+            communities.retain(|community| !ours(community));
+            put_communities(attributes, communities);
+        }
+        let ours_large = |community: &LargeCommunity| {
+            community.global_admin == self.local_asn && community.data_1 <= 1
+        };
+        let has_large = attributes.iter().any(|attribute| match attribute {
+            PathAttribute::LargeCommunities(large) => large.communities.iter().any(ours_large),
+            _ => false,
+        });
+        if has_large {
+            let mut large = take_large_communities(attributes);
+            large.retain(|community| !ours_large(community));
+            put_large_communities(attributes, large);
+        }
+    }
+}
+
+/// The communities of `attributes`, standard and large, for looking at.
+pub fn communities_of(attributes: &[PathAttribute]) -> (&[(u16, u16)], &[LargeCommunity]) {
+    let (mut standard, mut large): (&[(u16, u16)], &[LargeCommunity]) = (&[], &[]);
+    for attribute in attributes {
+        match attribute {
+            PathAttribute::Communities(communities) => standard = &communities.0,
+            PathAttribute::LargeCommunities(communities) => large = &communities.communities,
+            _ => {}
+        }
+    }
+    (standard, large)
+}
+
+// Used by the tests of other modules too.
+#[cfg(test)]
+pub(crate) fn large(global_admin: u32, data_1: u32, data_2: u32) -> LargeCommunity {
+    LargeCommunity {
+        global_admin,
+        data_1,
+        data_2,
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use bgp_packet::message::{AsPathAttribute, OriginPathAttribute};
+
+    use super::*;
+
+    fn strings(texts: &[&str]) -> Vec<String> {
+        texts.iter().map(|text| text.to_string()).collect()
+    }
+
+    fn route(
+        communities: &[(u16, u16)],
+        large_communities: &[LargeCommunity],
+    ) -> Vec<PathAttribute> {
+        let mut attributes = vec![
+            PathAttribute::Origin(OriginPathAttribute::IGP),
+            AsPathAttribute::from_asns(vec![64602]),
+        ];
+        put_communities(&mut attributes, communities.to_vec());
+        put_large_communities(&mut attributes, large_communities.to_vec());
+        attributes
+    }
+
+    /// Where a route was learned is put on it, and nobody else's word
+    /// for that is taken.
+    #[test]
+    fn marks_are_taken_off_and_put_on() {
+        let marks = Marks::parse(
+            &strings(&["64601:*", "64601:9:*"]),
+            &strings(&["64601:1001", "64601:1001"]),
+            &strings(&["64601:9:7"]),
+        )
+        .unwrap();
+        let mut attributes = route(
+            &[(3356, 3), (64601, 2002), (64601, 1001)],
+            &[large(64601, 9, 1), large(64601, 8, 1)],
+        );
+        marks.apply(&mut attributes);
+        assert_eq!(
+            attributes,
+            route(
+                &[(3356, 3), (64601, 1001)],
+                &[large(64601, 8, 1), large(64601, 9, 7)]
+            )
+        );
+
+        // On a route with none, in the order of the type codes.
+        let mut bare = route(&[], &[]);
+        marks.apply(&mut bare);
+        let codes: Vec<u8> = bare.iter().map(PathAttribute::type_code).collect();
+        assert_eq!(codes, [1, 2, 8, 32]);
+
+        // Taking off the last leaves no empty attribute behind.
+        let scrub = Marks::parse(&strings(&["*:*", "*:*:*"]), &[], &[]).unwrap();
+        let mut all = route(&[(1, 2)], &[large(1, 2, 3)]);
+        scrub.apply(&mut all);
+        assert_eq!(all, route(&[], &[]));
+
+        assert!(Marks::default().is_empty());
+        assert!(!marks.is_empty());
+    }
+
+    #[test]
+    fn communities_that_are_not_are_refused() {
+        for bad in ["64601", "64601:1:2:3", "x:1", "70000:1"] {
+            assert_eq!(
+                Marks::parse(&strings(&[bad]), &[], &[]),
+                Err(MarksProblem::Pattern(bad.to_string())),
+                "{bad}"
+            );
+        }
+        // What is put on is one community, not a pattern.
+        assert!(Marks::parse(&[], &strings(&["64601:*"]), &[]).is_err());
+        assert!(Marks::parse(&[], &[], &strings(&["64601:1"])).is_err());
+        // A well-known one by its name.
+        assert!(Marks::parse(&[], &strings(&["NO_EXPORT"]), &[]).is_ok());
+    }
+
+    /// A route says who is told of it.
+    #[test]
+    fn communities_say_which_peers_a_route_goes_to() {
+        let control = Control { local_asn: 64601 };
+        let to = |communities: &[(u16, u16)], large: &[LargeCommunity], peer: u32| {
+            control.allows(communities, large, peer)
+        };
+        // Nothing said: everyone.
+        assert!(to(&[(3356, 3)], &[], 64602));
+        // Not to one peer.
+        assert!(!to(&[(0, 64602)], &[], 64602));
+        assert!(to(&[(0, 64602)], &[], 64603));
+        // Only to those named.
+        assert!(to(&[(64601, 64602)], &[], 64602));
+        assert!(!to(&[(64601, 64602)], &[], 64603));
+        assert!(to(&[(64601, 64602), (64601, 64603)], &[], 64603));
+        // To nobody; and to nobody but one, as exchanges have it.
+        assert!(!to(&[(0, 64601)], &[], 64602));
+        assert!(to(&[(0, 64601), (64601, 64602)], &[], 64602));
+        assert!(!to(&[(0, 64601), (64601, 64602)], &[], 64603));
+        // "Not" wins over "to".
+        assert!(!to(&[(64601, 64602), (0, 64602)], &[], 64602));
+        // An exchange's "to everyone" changes nothing.
+        assert!(to(&[(64601, 64601)], &[], 64602));
+
+        // The same in large communities, which have room for any AS.
+        let wide = 4_200_000_001;
+        assert!(!to(&[], &[large(64601, 0, wide)], wide));
+        assert!(to(&[], &[large(64601, 0, wide)], 64602));
+        assert!(to(&[], &[large(64601, 1, wide)], wide));
+        assert!(!to(&[], &[large(64601, 1, wide)], 64602));
+        assert!(!to(&[], &[large(64601, 0, 0)], 64602));
+        assert!(to(
+            &[],
+            &[large(64601, 0, 0), large(64601, 1, 64602)],
+            64602
+        ));
+        // Someone else's, or ours for something else, say nothing.
+        assert!(to(
+            &[],
+            &[large(3356, 0, 64602), large(64601, 2, 64602)],
+            64602
+        ));
+        // A peer whose AS number does not fit a standard community is not
+        // meant by one.
+        assert!(to(&[(0, 1)], &[], 65537));
+
+        // Our own AS number may not fit either: then only the large ones
+        // can name us.
+        let control = Control { local_asn: wide };
+        assert!(control.allows(&[(0, 64602)], &[], 64603));
+        assert!(!control.allows(&[(0, 64602)], &[], 64602));
+        assert!(!control.allows(&[], &[large(wide, 0, 0)], 64602));
+    }
+
+    #[test]
+    fn what_was_for_us_is_taken_off_as_a_route_is_passed_on() {
+        let control = Control { local_asn: 64601 };
+        let mut attributes = route(
+            &[(0, 64602), (3356, 3), (64601, 64603)],
+            &[
+                large(64601, 0, 5),
+                large(64601, 1, 6),
+                large(64601, 2, 7),
+                large(3356, 0, 5),
+            ],
+        );
+        control.strip(&mut attributes);
+        assert_eq!(
+            attributes,
+            route(&[(3356, 3)], &[large(64601, 2, 7), large(3356, 0, 5)])
+        );
+        let mut all = route(&[(0, 1)], &[large(64601, 0, 0)]);
+        control.strip(&mut all);
+        assert_eq!(all, route(&[], &[]));
+    }
+
+    #[test]
+    fn the_well_known_communities_keep_a_route_in() {
+        assert!(well_known_allow(&[(3356, 3)], false));
+        assert!(!well_known_allow(&[NO_ADVERTISE], true));
+        assert!(!well_known_allow(&[NO_ADVERTISE], false));
+        assert!(well_known_allow(&[NO_EXPORT], true));
+        assert!(!well_known_allow(&[NO_EXPORT], false));
+        assert!(!well_known_allow(&[NO_EXPORT_SUBCONFED], false));
+    }
+}

crates/server/src/metrics.rs

@@ -164,6 +164,15 @@ pub static RIB_JOURNAL_ENTRIES: LazyLock<IntGauge> = LazyLock::new(|| {
     .expect("metric registration")
 });
 
+pub static ROUTES_EXPORTED: LazyLock<IntCounterVec> = LazyLock::new(|| {
+    register_int_counter_vec!(
+        "bgp_routes_exported_total",
+        "Routes learned from one peer and announced to, or withdrawn from, another",
+        &["change"]
+    )
+    .expect("metric registration")
+});
+
 /// Whether each configured peer's session is up: what to alert on when a
 /// peer goes away, which a count of sessions cannot say.
 pub static PEER_UP: LazyLock<IntGaugeVec> = LazyLock::new(|| {
@@ -207,6 +216,7 @@ pub fn init() {
     LazyLock::force(&RIB_PREFIXES);
     LazyLock::force(&RIB_ATTRIBUTE_SETS);
     LazyLock::force(&RIB_JOURNAL_ENTRIES);
+    LazyLock::force(&ROUTES_EXPORTED);
     LazyLock::force(&PEER_UP);
     BUILD_INFO
         .with_label_values(&[env!("CARGO_PKG_VERSION")])

crates/server/src/policy.rs

@@ -29,12 +29,15 @@ use std::net::{IpAddr, Ipv4Addr, Ipv6Addr};
 use std::sync::{Arc, LazyLock, Mutex};
 use std::time::{Duration, Instant};
 
-use bgp_packet::message::{AS_TRANS, AsPathAttribute, NlriNextHop, PathAttribute};
+use bgp_packet::message::{
+    AS_TRANS, AsPathAttribute, LocalPrefPathAttribute, NlriNextHop, PathAttribute,
+};
 use bgp_packet::open::BgpRole;
 use bgp_primitives::constants::IpVersion;
 use bgp_primitives::ip_prefix::IpPrefix;
 
 use crate::address::PeerAddress;
+use crate::marks::{self, Marks};
 use crate::sync::IgnorePoison;
 
 /// The longest prefixes routed between networks: anything more specific is
@@ -96,6 +99,11 @@ pub struct ImportPolicy {
     /// a customer with a part of our space. Empty for a peer in our own
     /// AS.
     pub own_prefixes: Arc<PrefixSet>,
+    /// Communities taken off and put on every route that is taken.
+    pub marks: Marks,
+    /// The LOCAL_PREF every route that is taken is given; `None` leaves
+    /// it at the default.
+    pub local_pref: Option<u32>,
 }
 
 impl Default for ImportPolicy {
@@ -118,6 +126,8 @@ impl Default for ImportPolicy {
             accept_as_sets: false,
             accept_third_party_next_hop: false,
             own_prefixes: Arc::default(),
+            marks: Marks::default(),
+            local_pref: None,
         }
     }
 }
@@ -143,6 +153,37 @@ impl ImportPolicy {
     }
 }
 
+impl ImportPolicy {
+    /// Whether routes are changed on their way in.
+    pub fn changes_routes(&self) -> bool {
+        !self.marks.is_empty() || self.local_pref.is_some()
+    }
+
+    /// Makes of `attributes` what the route is kept with: marked, and
+    /// with its preference.
+    pub fn change(&self, attributes: &mut Vec<PathAttribute>) {
+        self.marks.apply(attributes);
+        if let Some(local_pref) = self.local_pref {
+            marks::put(
+                attributes,
+                PathAttribute::LocalPref(LocalPrefPathAttribute(local_pref)),
+            );
+        }
+    }
+}
+
+/// What is announced to one peer, and how.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct ExportPolicy {
+    /// Pass on the best routes learned from other peers, and not only
+    /// what we originate.
+    pub learned_routes: bool,
+    /// Communities taken off and put on every route announced.
+    pub marks: Marks,
+    /// How many more times our AS goes on the path.
+    pub prepend: u8,
+}
+
 /// Why a route was not taken.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
 pub enum Rejection {

crates/server/src/routes.rs

@@ -128,6 +128,7 @@ pub(crate) fn apply(
         path_invalid: OnceCell::new(),
         as_path: OnceCell::new(),
         mark: only_to_customer_mark(import, update),
+        changed: OnceCell::new(),
     };
     applying.withdraw();
     // Routes in the body have their next hop in the NEXT_HOP attribute,
@@ -181,6 +182,9 @@ struct Applying<'a> {
     as_path: OnceCell<Arc<[u32]>>,
     /// See [`only_to_customer_mark`].
     mark: Option<u32>,
+    /// The UPDATE's attributes as the policy changes them, if it does:
+    /// made when the first route is taken.
+    changed: OnceCell<Vec<PathAttribute>>,
 }
 
 impl Applying<'_> {
@@ -335,6 +339,27 @@ impl Applying<'_> {
     /// The attribute set the UPDATE's routes go into the table with.
     fn attributes(&self, next_hop: Option<&NlriNextHop>) -> Result<Arc<AttrSet>, EncodeError> {
         let store = self.table.attrs();
+        // What the policy puts on the peer's routes (communities that say
+        // where they were learned, a preference) is the same for all of
+        // an UPDATE's routes, and worked out once for them.
+        if self.import.policy.changes_routes() {
+            let changed = self.changed.get_or_init(|| {
+                let mut attributes = self.update.path_attributes.clone();
+                // Not the route's own: they hold the prefixes.
+                attributes.retain(|attribute| {
+                    !matches!(
+                        attribute,
+                        PathAttribute::MpReachNlri(_) | PathAttribute::MpUnreachNlri(_)
+                    )
+                });
+                self.import.policy.change(&mut attributes);
+                if let Some(marked_by) = self.mark {
+                    attributes.push(PathAttribute::OnlyToCustomer(marked_by));
+                }
+                attributes
+            });
+            return store.intern(changed, next_hop);
+        }
         let Some(marked_by) = self.mark else {
             if let Some(set) = self
                 .attributes_wire

crates/server/src/server.rs

@@ -179,6 +179,18 @@ impl BgpServer {
                     "No limit on the routes taken from this peer (import.max_prefixes)"
                 );
             }
+            // Without a role there is no rule about which learned routes
+            // go to the peer: all of them do.
+            if peer.export.learned_routes
+                && peer.role.is_none()
+                && peer.remote_asn != self.config.local_asn
+            {
+                warn!(
+                    peer = %peer.name,
+                    "Every route learned from any peer is passed on to this one: it has no role \
+                     (export.learned_routes without role)"
+                );
+            }
         }
         if self.config.accept_unconfigured_peers {
             warn!("Taking sessions and routes from any host (accept_unconfigured_peers)");
@@ -204,8 +216,10 @@ impl BgpServer {
             "advertiser",
             spawn_advertiser(
                 self.local_rib.clone(),
+                Arc::clone(&self.rib),
                 self.registry.clone(),
                 self.config.local_asn,
+                self.config.control,
             ),
         ));
         tasks.push(named("sweeper", spawn_sweeper(Arc::clone(&self.rib))));
@@ -354,7 +368,7 @@ fn admit(
     let peer = match config.peer_at(address) {
         Some(peer) => PeerSettings::from(peer),
         None if config.accept_unconfigured_peers => {
-            PeerSettings::unconfigured(config.hold_time_secs)
+            PeerSettings::unconfigured(config.hold_time_secs, Arc::clone(&config.export))
         }
         None => return Err(Refusal::NotAPeer),
     };

crates/server/src/session.rs

@@ -24,7 +24,7 @@ use crate::collision::Direction;
 use crate::family::FamilySet;
 use crate::fsm::Negotiated;
 use crate::metrics::{PEER_UP, SESSIONS_ESTABLISHED};
-use crate::policy::PeerRole;
+use crate::policy::{ExportPolicy, PeerRole};
 use crate::rpki::Subject;
 use crate::sync::IgnorePoisonRw;
 
@@ -145,6 +145,8 @@ pub struct SessionHandle {
     pub rib_peer: PeerIndex,
     /// How many routes the peer has, and how many were refused.
     pub counters: Arc<SessionCounters>,
+    /// What the peer is told, and how.
+    pub export: Arc<ExportPolicy>,
     outbound: mpsc::Sender<Message>,
     /// Which of the RPKI's checks the peer's routes are subject to, if
     /// any, and where the session is told to look at its routes again.
@@ -181,6 +183,7 @@ impl SessionHandle {
             add_path_send: negotiated.add_path_send.clone(),
             rib_peer,
             counters: Arc::default(),
+            export: Arc::default(),
             outbound,
             validation: None,
         }

docs/RFCs.md

@@ -37,7 +37,10 @@ The base protocol specification.
 | §9.1.2.1 | Next hops resolved through other routes | ❌ (only next hops that are neighbors count as reachable) |
 | §9 | Adj-RIB-In (per-peer table of received routes) | ✅ |
 | §9 | Locally originated routes, advertised to all established peers | ✅ |
-| §9 | Loc-RIB / Adj-RIB-Out for routes learned from peers (re-advertisement) | ❌ |
+| §9 | Loc-RIB / Adj-RIB-Out: the best route of each prefix passed on to peers (`export.learned_routes`), with our AS on the path and ourselves as next hop; MED and LOCAL_PREF not sent to other networks; LOCAL_PREF to our own (§5.1.4, §5.1.5, §9.1.3) | ✅ |
+| §9.2.1 | No route learned from one internal peer passed to another (no route reflection) | ✅ |
+| §9.1.3 | A route not sent to a peer whose AS is on its path | ✅ |
+| §5 | Unknown optional transitive attributes passed on with the Partial bit set; non-transitive ones dropped | ✅ |
 | §9.2 | Hold timer reset on any received message (not only KEEPALIVE) | ✅ |
 
 ---
@@ -148,6 +151,8 @@ https://datatracker.ietf.org/doc/html/rfc1997
 |---------|--------|
 | `COMMUNITIES` path attribute (parse, serialise) | ✅ |
 | Attach to locally originated routes | ✅ |
+| NO_EXPORT, NO_EXPORT_SUBCONFED and NO_ADVERTISE honoured when learned routes are passed on | ✅ |
+| Communities put on and taken off routes by configuration (`import` and `export`) | ✅ |
 
 ### RFC 4360 — BGP Extended Communities Attribute
 https://datatracker.ietf.org/doc/html/rfc4360
@@ -219,6 +224,15 @@ Implemented from version 24 of the draft.
 | Paths with an AS_SET invalid | ✅ |
 | Verification on export | ❌ (learned routes are not exported yet) |
 
+### RFC 7947 — Internet Exchange BGP Route Server
+https://datatracker.ietf.org/doc/html/rfc7947
+
+| Feature | Status |
+|---------|--------|
+| §2.2.1 / §2.2.2: to a `ROUTE_SERVER_CLIENT` the path is not prepended and the next hop and MED are left as they came | ✅ |
+| §2.2.2.2 / §4.6: communities by which a client says which other clients are told of its route (`control_communities`) | ✅ |
+| Per-client best path selection (a different best route for each client) | ❌ (every client is told the one best route) |
+
 ### RFC 9234 — Route Leak Prevention and Detection Using Roles in UPDATE and OPEN Messages
 https://datatracker.ietf.org/doc/html/rfc9234
 
@@ -231,7 +245,7 @@ https://datatracker.ietf.org/doc/html/rfc9234
 | Ingress: marked routes from customers, route server clients and (unless marked by the peer itself) peers refused | ✅ |
 | Ingress: routes from providers, peers and route servers marked | ✅ |
 | Egress: originated routes to customers, peers and route server clients marked | ✅ |
-| Egress: marked routes not sent to providers, peers or route servers | ❌ (learned routes are not exported yet) |
+| Egress: marked routes not sent to providers, peers or route servers | ✅ |
 
 ### RFC 8210 — The RPKI to Router Protocol, Version 1 (RFC 6810, version 0; draft-ietf-sidrops-8210bis, version 2)
 https://datatracker.ietf.org/doc/html/rfc8210

scripts/interop/lab.sh

@@ -29,7 +29,7 @@ cd "$(dirname "$0")/../.."
 bgp="${CARGO_TARGET_DIR:-target}/release/bgp"
 work=/tmp/ferrix-lab
 pathspace=ferrixlab
-all_phases="basics roles rpki refresh flaps restart scale limits hijacks nexthops v6only linklocal"
+all_phases="basics roles rpki refresh flaps restart scale limits hijacks nexthops transit v6only linklocal"
 phases="${PHASES:-$all_phases}"
 scale_routes="${SCALE_ROUTES:-200000}"
 
@@ -768,7 +768,7 @@ phase_hijacks() {
     frr_configure \
         "router bgp 64602" "address-family ipv4 unicast" \
         "no network 10.1.0.0/24" "no network 10.9.0.0/24" "no network 10.2.7.0/24" \
-        "no neighbor 10.9.0.1 route-map TO-DUT out"
+        "no neighbor 10.9.0.1 route-map TO-DUT out" "no route-map TO-DUT"
     converged
     ok "back to normal"
 }
@@ -818,7 +818,7 @@ phase_nexthops() {
 
     frr_configure \
         "router bgp 64602" "address-family ipv4 unicast" \
-        "no network 10.77.0.0/24" "no neighbor 10.9.0.1 route-map TO-DUT out"
+        "no network 10.77.0.0/24" "no neighbor 10.9.0.1 route-map TO-DUT out" "no route-map TO-DUT"
     bird_config
     bird_ctl configure >/dev/null
     dut_down TERM
@@ -828,6 +828,55 @@ phase_nexthops() {
     ok "back to normal"
 }
 
+# Routes are passed on: FRR becomes our customer and BIRD stays our
+# provider, so what FRR announces goes up to BIRD with our AS on the path,
+# what BIRD announces goes down to FRR marked "only to customer", and
+# packets go between their networks through us. A community on one of
+# FRR's routes keeps it from BIRD.
+phase_transit() {
+    dut_down TERM
+    dut_config CUSTOMER 2000000 "" "" "" \
+        'export { learned_routes: true }
+control_communities: true'
+    in_ns dut sysctl -qw net.ipv4.ip_forward=1
+    dut_up
+    frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role customer" \
+        "ip prefix-list KEPT seq 5 permit 10.2.9.0/24" \
+        "route-map TO-DUT-TRANSIT permit 10" "match ip address prefix-list KEPT" "set large-community 64601:0:4200000003" \
+        "route-map TO-DUT-TRANSIT permit 20" \
+        "router bgp 64602" "address-family ipv4 unicast" \
+        "neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "network 10.2.9.0/24"
+    wait_for "four sessions" established 4
+    wait_for "FRR's routes at the daemon" dut_has 10.2.9.0/24 frr4
+
+    # Up to the provider, with our AS on the path; not the one that says
+    # not to.
+    wait_for "FRR's route at BIRD, by way of us" bash -c "sudo birdc -s $work/bird.ctl show route for 10.2.0.0/24 all | grep -q 'BGP.as_path: 64601 64602'"
+    sleep 1
+    bird_ctl show route for 10.2.9.0/24 all | grep -q "64601 64602" && fail "BIRD was told the route whose community says not to"
+    ok "the customer's routes reach the provider with our AS on the path; one kept back by its community"
+
+    # Down to the customer, marked.
+    wait_for "BIRD's route at FRR, by way of us" bash -c "sudo ip netns exec lab-frr vtysh -N $pathspace -c 'show bgp ipv4 unicast 10.3.0.0/24 json' 2>/dev/null | jq -e '.paths[0].aspath.string == \"64601 4200000003\"'"
+    # Marked on the way in from the provider, with the provider's AS, and
+    # the mark kept as it goes down (RFC 9234 §5).
+    vty -c "show bgp ipv4 unicast 10.3.0.0/24" | grep -q "otc 4200000003" || fail "the route to the customer is not marked only to customer"
+    ok "the provider's routes reach the customer, marked only to customer"
+
+    # And packets follow them, through us.
+    in_ns frr ping -q -c 2 -W 2 -I 10.2.0.1 10.3.0.1 >/dev/null || fail "no ping from the customer's network to the provider's through us"
+    in_ns bird ping -q -c 2 -W 2 -I 10.3.0.1 10.2.0.1 >/dev/null || fail "no ping from the provider's network to the customer's through us"
+    ok "packets go between the customer's and the provider's networks through us"
+
+    frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role peer" \
+        "address-family ipv4 unicast" "no network 10.2.9.0/24" "no neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "no route-map TO-DUT-TRANSIT"
+    dut_down TERM
+    dut_config
+    dut_up
+    converged
+    ok "back to normal"
+}
+
 # The exchange LAN loses its IPv4 addresses. One session over IPv6 with
 # each router carries both families, and IPv4 routes have IPv6 next hops
 # (RFC 8950), in BGP and in the kernel. Run after the rest: the lab stays