Control communities: session groups, prepends, and only ours to give

Control communities could say which peers a route goes to, by AS. Now
they can also name sessions by group, and ask for prepends:

  L:11:P .. L:13:P   our AS once to three times more towards AS P
                     (P 0: towards every peer in another AS)
  L:20:G / L:21:G    not to / only to the sessions of group G
  L:31:G .. L:33:G   our AS once to three times more towards group G

A group is a number given to sessions in the configuration (`groups` of
a peer): those at one exchange, or those over one link. The same network
at two exchanges is one AS but two sessions, which is what steering
inbound traffic between exchanges needs. Of several prepends that apply
the most counts, on top of the session's own as_path_prepend. Functions
2 to 9, as the documentation already promised, and 100 and up stay
informational; 10 to 99 are kept for actions to come.

They are acted on where a route leaves our AS, and taken off there; our
informational communities are not. To a peer in our own AS a route goes
with them, unchanged, so that a mark put on at one router is acted on at
another. Before, they were acted on and taken off on internal sessions
too, so a route marked "to nobody" did not even reach our own routers.

And they are ours to give: the communities of our own AS (L:n, L:n:m,
and 0:n where control communities are on) are taken off every route
from a peer in another AS as it comes in, before import marks are put
on. A peer at an exchange could otherwise decide where its routes go
from us, and set our informational marks. A customer that is to steer
its own routes is given import.accept_control_communities. An UPDATE
without any of ours is still taken as it came, without being written
out again.

This changes what a customer that set control communities gets: it now
needs accept_control_communities. The transit example and the interop
lab say so; the lab also has FRR, as a customer, ask for a prepend
towards BIRD's group, and BIRD, as a provider, have its "not to" taken
off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/integration-tests/tests/add_path.rs

@@ -41,6 +41,7 @@ async fn server_offering(
     TestServer::start_with(LOCAL_ASN, |c| {
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "route-server-client".into(),
             endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/admission.rs

@@ -19,6 +19,7 @@ const TIMEOUT: Duration = Duration::from_secs(5);
 fn peer_at(endpoint: SocketAddr) -> PeerConfig {
     PeerConfig {
         graceful_restart: Default::default(),
+        groups: Vec::new(),
         name: "r1".into(),
         endpoint: endpoint.into(),
         remote_asn: PEER_ASN,

crates/integration-tests/tests/aspa.rs

@@ -58,6 +58,7 @@ async fn server_with(
         });
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/collision.rs

@@ -37,6 +37,7 @@ async fn server_dialing(listener: &TcpListener) -> Result<TestServer> {
         c.connect_retry = Duration::from_millis(200);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/debugging.rs

@@ -168,6 +168,7 @@ async fn test_refused_routes_can_be_seen_and_counted() -> Result<()> {
         c.accept_unconfigured_peers = false;
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "refused-demo".into(),
             endpoint: endpoint.into(),
             remote_asn: 3333,

crates/integration-tests/tests/dialer.rs

@@ -28,6 +28,7 @@ async fn server_dialing(listener: &TcpListener, remote_asn: u32) -> Result<TestS
         c.connect_retry = Duration::from_millis(200);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn,
@@ -187,6 +188,7 @@ async fn test_inbound_from_configured_address_uses_config() -> Result<()> {
         c.connect_retry = Duration::from_secs(60);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
             remote_asn: PEER_ASN,
@@ -240,6 +242,7 @@ async fn test_local_address_is_used_as_the_source() -> Result<()> {
         c.connect_retry = Duration::from_millis(200);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,
@@ -262,6 +265,7 @@ async fn test_local_address_is_used_as_the_source() -> Result<()> {
         c.connect_retry = Duration::from_millis(200);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/import_policy.rs

@@ -10,12 +10,13 @@ use bgp_integration_tests::test_peer::TestPeer;
 use bgp_integration_tests::test_server::TestServer;
 use bgp_packet::errors::{BgpError, CeaseSubcode};
 use bgp_packet::message::{
-    AsPathAttribute, Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute,
+    AsPathAttribute, CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
-    UpdateMessage,
+    Message, NextHopPathAttribute, OriginPathAttribute, PathAttribute, UpdateMessage,
 };
 use bgp_primitives::ip_prefix::IpPrefix;
 use bgp_server::config::PeerConfig;
 use bgp_server::family::FamilySet;
+use bgp_server::marks::Scrub;
 use bgp_server::policy::{ImportPolicy, PrefixSet};
 use bgp_server::session::DialState;
 use eyre::Result;
@@ -42,6 +43,7 @@ async fn server_with(import: ImportPolicy) -> Result<(TestServer, TcpListener)>
         c.connect_retry = Duration::from_millis(50);
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,
@@ -343,3 +345,92 @@ async fn test_a_peers_routes_are_marked_and_ranked_as_configured() -> Result<()>
     assert_eq!(server.rib.attrs().len(), 1, "they share their attributes");
     Ok(())
 }
+
+/// The communities of our own AS are taken off what another network
+/// sends, before the route is kept, and nothing else is; from a peer that
+/// is to steer its routes with them (a customer), they are kept.
+#[tokio::test]
+async fn test_our_communities_are_not_taken_from_other_networks() -> Result<()> {
+    let ours_and_theirs = || {
+        vec![
+            PathAttribute::Communities(CommunitiesPathAttribute(vec![
+                (0, 13335),
+                (3356, 3),
+                (1299, 5),
+            ])),
+            PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
+                communities: vec![large(1299, 0, 0), large(1299, 31, 2), large(3356, 1, 2)],
+            }),
+        ]
+    };
+    let kept = |server: &TestServer| -> Vec<PathAttribute> {
+        let routes = server.received_routes();
+        routes[0]
+            .attributes
+            .iter()
+            .filter(|attribute| {
+                matches!(
+                    attribute,
+                    PathAttribute::Communities(_) | PathAttribute::LargeCommunities(_)
+                )
+            })
+            .cloned()
+            .collect()
+    };
+
+    let scrubbed = ImportPolicy {
+        scrub: Some(Scrub {
+            local_asn: LOCAL_ASN,
+            zero: true,
+        }),
+        ..default_policy()
+    };
+    let (server, _listener) = server_with(scrubbed).await?;
+    let mut peer = established(&server).await?;
+    announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
+    table_after(&server, 1, 0).await?;
+    assert_eq!(
+        kept(&server),
+        [
+            PathAttribute::Communities(CommunitiesPathAttribute(vec![(3356, 3)])),
+            PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
+                communities: vec![large(3356, 1, 2)],
+            }),
+        ]
+    );
+
+    let (server, _listener) = server_with(default_policy()).await?;
+    let mut peer = established(&server).await?;
+    announce_with(&mut peer, &["193.0.0.0/21"], ours_and_theirs()).await?;
+    table_after(&server, 1, 0).await?;
+    assert_eq!(kept(&server), ours_and_theirs());
+    Ok(())
+}
+
+fn large(global_admin: u32, data_1: u32, data_2: u32) -> LargeCommunity {
+    LargeCommunity {
+        global_admin,
+        data_1,
+        data_2,
+    }
+}
+
+/// Announces IPv4 `prefixes` from the peer itself, with `more` attributes.
+async fn announce_with(
+    peer: &mut TestPeer,
+    prefixes: &[&str],
+    more: Vec<PathAttribute>,
+) -> Result<()> {
+    let mut path_attributes = vec![
+        PathAttribute::Origin(OriginPathAttribute::IGP),
+        AsPathAttribute::from_asns(vec![PEER_ASN]),
+        PathAttribute::NextHop(NextHopPathAttribute(Ipv4Addr::new(192, 0, 2, 1))),
+    ];
+    path_attributes.extend(more);
+    peer.send(&Message::Update(UpdateMessage {
+        withdrawn_routes: vec![],
+        path_attributes,
+        announced_routes: prefixes.iter().map(|p| prefix(p).into()).collect(),
+    }))
+    .await
+}

crates/integration-tests/tests/negotiation.rs

@@ -184,6 +184,7 @@ async fn test_no_family_in_common_refuses_the_session() -> Result<()> {
         // Inbound connections from this address inherit the peer's settings.
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "v6-only".into(),
             endpoint: SocketAddr::new(Ipv4Addr::LOCALHOST.into(), 1).into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/roles.rs

@@ -49,6 +49,7 @@ async fn server_with(role: Option<PeerRole>, require_role: bool) -> Result<Setup
         c.originate = vec![LocalRoute::new(prefix("193.0.0.0/21"), Default::default())];
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,

crates/integration-tests/tests/rpki.rs

@@ -68,6 +68,7 @@ async fn server_with(
         });
         c.peers = vec![PeerConfig {
             graceful_restart: Default::default(),
+            groups: Vec::new(),
             name: "r1".into(),
             endpoint: endpoint.into(),
             remote_asn: PEER_ASN,

crates/server/proto/config.proto

@@ -58,26 +58,43 @@ message BgpConfig {
   // own.
   ExportPolicy export = 12;
 
-  // Let routes say, with communities, which peers they are passed on to,
+  // Let routes say, with communities, which peers they are passed on to
-  // as the route servers of Internet exchanges do. With L our own AS
+  // and how far away we look to them, as the route servers of Internet
-  // number and P a peer's:
+  // exchanges and transit networks do. With L our own AS number, P a
+  // peer's, and G a group of sessions (`groups` of a peer):
   //
-  //   0:P   or  L:0:P   not to the peer with AS P
+  //   0:P   or  L:0:P    not to the peer with AS P
-  //   L:P   or  L:1:P   to the peer with AS P, and then only to peers
+  //   L:P   or  L:1:P    to the peer with AS P, and then only to peers
-  //                     named so
+  //                      named so
-  //   0:L   or  L:0:0   to nobody
+  //   0:L   or  L:0:0    to nobody
+  //               L:11:P  with our AS once more on the path to the peer
+  //               L:12:P  with AS P; twice, three times. P 0 is every
+  //               L:13:P  peer in another AS.
+  //               L:20:G  not to the sessions of group G
+  //               L:21:G  to the sessions of group G, and then only to
+  //                       sessions named so, by group or AS
+  //               L:31:G  with our AS once more on the path to the
+  //               L:32:G  sessions of group G; twice, three times
+  //               L:33:G
   //
   // A route with none of these goes to everyone it otherwise would. "Not"
-  // wins over "to". The two-part form is a standard community, with room
+  // wins over "to". Of several prepends that apply, the most counts, on
-  // for AS numbers below 65536; the three-part form is a large community
+  // top of the session's export.as_path_prepend. The two-part form is a
-  // and works for any. They are taken off a route as it is passed on.
+  // standard community, with room for AS numbers below 65536; the
+  // three-part forms are large communities and work for any.
   //
-  // They are honoured on routes we originate too (`communities` of
+  // They are acted on where a route leaves our AS, and taken off there:
-  // `originate`), and on what import.add_communities puts on.
+  // to a peer in our own AS a route goes with them, for that router to act
+  // on. They are honoured on routes we originate too (`communities` and
+  // `large_communities` of `originate`), on what import.add_communities
+  // puts on, and on what a peer in our own AS, or one with
+  // import.accept_control_communities, sends. From any other peer the
+  // communities of our own AS are taken off as its routes come in, so that
+  // nobody outside can say what is done with our routes.
   //
   // With this on, standard communities L:n are taken for "to the peer
   // with AS n": mark routes with something else (large communities
-  // L:2:n and up, say).
+  // L:100:n and up, say; functions 2 to 9 are left for marks too).
   bool control_communities = 13;
 
   // Graceful restart, for every peer that has no `graceful_restart` of its
@@ -280,6 +297,13 @@ message Peer {
   // Graceful restart with this peer. What is set here takes the place of
   // what the top-level `graceful_restart` says.
   GracefulRestart graceful_restart = 12;
+
+  // The groups the session is in, by number, for control communities to
+  // name sessions by (see control_communities): those at one Internet
+  // exchange, say, or those over one link, whatever AS the peers are. Use
+  // the same numbers on all routers of the AS: a route marked on one is
+  // acted on by another. From 1.
+  repeated uint32 groups = 13;
 }
 
 // What is announced to a peer.
@@ -426,6 +450,15 @@ message ImportPolicy {
   // The LOCAL_PREF of the peer's routes: of two routes for a prefix the
   // one with the higher is chosen, whatever else they have. Left out, 100.
   optional uint32 local_pref = 18;
+
+  // By default the communities of our own AS (L:n and L:n:m; and 0:n,
+  // with control_communities on) are taken off the routes of a peer in
+  // another AS as they come in, before add_communities puts ours on: what
+  // they ask of us is not for others to ask. True keeps them, for a
+  // customer that is to steer its own routes with control communities, or
+  // a route server's client. Not for a peer in our own AS, whose routes
+  // keep them in any case.
+  bool accept_control_communities = 19;
 }
 
 // A prefix, and optionally the more specific prefixes inside it.

crates/server/src/advertise.rs

@@ -496,6 +496,7 @@ impl Exporter {
             peer_asn: session.peer.asn,
             rib_peer: session.rib_peer,
             policy: &session.export,
+            groups: &session.groups,
             control: self.control,
         };
         // What the peer is to end up with for each prefix heard of: the
@@ -673,6 +674,7 @@ impl Exporter {
             peer_asn: session.peer.asn,
             rib_peer: session.rib_peer,
             policy: &session.export,
+            groups: &session.groups,
             control: self.control,
         };
         if !egress.dress(&mut update.path_attributes) {

crates/server/src/config.rs

@@ -15,7 +15,7 @@ use bgp_rpki::RpkiConfig;
 use crate::address::{Endpoint, PeerAddress};
 use crate::error::{ConfigError, ExportProblem, ImportProblem, OriginateProblem, PeerProblem};
 use crate::family::FamilySet;
-use crate::marks::{Control, Marks};
+use crate::marks::{Control, Marks, Scrub};
 use crate::policy::{ExportPolicy, ImportPolicy, PeerRole, PrefixSet};
 use crate::proto::config::{self as proto, BgpConfig, OriginatedRoute, Peer};
 use crate::restart::GracefulRestartConfig;
@@ -91,6 +91,9 @@ pub struct PeerConfig {
     pub export: Arc<ExportPolicy>,
     /// Graceful restart with the peer (RFC 4724).
     pub graceful_restart: GracefulRestartConfig,
+    /// The groups the session is in, for control communities to name it
+    /// by.
+    pub groups: Vec<u32>,
 }
 
 impl Config {
@@ -189,6 +192,15 @@ impl Config {
                 peer.import.originated = Arc::clone(&originated);
             } else {
                 peer.import.own_prefixes = Arc::clone(&own_prefixes);
+                // What is ours to say is not taken from other networks,
+                // unless they are to say it: customers that steer their
+                // own routes.
+                if !peer.import.accept_control_communities {
+                    peer.import.scrub = Some(Scrub {
+                        local_asn: proto.local_asn,
+                        zero: proto.control_communities,
+                    });
+                }
             }
         }
 
@@ -334,6 +346,9 @@ fn parse_peer(
     if peer.require_role && role.is_none() {
         return Err(PeerProblem::RequireRoleWithoutRole);
     }
+    if peer.groups.contains(&0) {
+        return Err(PeerProblem::Group);
+    }
 
     Ok(PeerConfig {
         // A peer without a name goes by its address.
@@ -359,6 +374,7 @@ fn parse_peer(
             Some(own) => graceful_restart.overridden_by(own)?,
             None => graceful_restart,
         },
+        groups: peer.groups,
     })
 }
 
@@ -448,6 +464,8 @@ fn parse_import(import: &proto::ImportPolicy) -> Result<ImportPolicy, ImportProb
         accept_third_party_next_hop: import.accept_third_party_next_hop,
         own_prefixes: Arc::default(),
         originated: Arc::default(),
+        accept_control_communities: import.accept_control_communities,
+        scrub: None,
         marks: Marks::parse(
             &import.remove_communities,
             &import.add_communities,
@@ -570,6 +588,56 @@ mod tests {
 
     const BASE: &str = "router_id: \"192.0.2.1\" local_asn: 65000\n";
 
+    /// Our communities are taken off what other networks send, unless a
+    /// peer is to steer its routes with them; not off what our own
+    /// routers send. `0:n` too where control communities make it ours.
+    #[test]
+    fn our_communities_are_taken_only_from_whom_they_are_ours() {
+        let config = parse(&format!(
+            "{BASE}control_communities: true
+             peers {{ name: \"peer\" endpoint {{ address: \"192.0.2.2\" }} remote_asn: 64496 }}
+             peers {{ name: \"customer\" endpoint {{ address: \"192.0.2.3\" }} remote_asn: 64497
+                     import {{ accept_control_communities: true }} }}
+             peers {{ name: \"internal\" endpoint {{ address: \"192.0.2.4\" }} remote_asn: 65000 }}"
+        ))
+        .unwrap();
+        let scrubs: Vec<_> = config.peers.iter().map(|peer| peer.import.scrub).collect();
+        assert_eq!(
+            scrubs,
+            [
+                Some(Scrub {
+                    local_asn: 65000,
+                    zero: true
+                }),
+                None,
+                None
+            ]
+        );
+    }
+
+    #[test]
+    fn sessions_are_put_in_groups() {
+        let config = parse(&format!(
+            "{BASE}peers {{ name: \"r1\" endpoint {{ address: \"192.0.2.2\" }} remote_asn: 1
+                     groups: 2 groups: 10 }}"
+        ))
+        .unwrap();
+        assert_eq!(config.peers[0].groups, [2, 10]);
+        let zero = rejected(&format!(
+            "{BASE}peers {{ name: \"r1\" endpoint {{ address: \"192.0.2.2\" }} remote_asn: 1 groups: 0 }}"
+        ));
+        assert!(
+            matches!(
+                &zero,
+                ConfigError::Peer {
+                    problem: PeerProblem::Group,
+                    ..
+                }
+            ),
+            "{zero:?}"
+        );
+    }
+
     /// What we originate is refused from other networks, with what is
     /// inside it; from a router of our own it is ignored, and only the
     /// prefix itself.
@@ -845,8 +913,16 @@ mod tests {
     /// checks; what is said overrides them one by one.
     #[test]
     fn an_import_policy_starts_from_the_general_checks() {
-        assert_eq!(import_of("").unwrap(), ImportPolicy::default());
+        // From another network, without our communities.
-        assert_eq!(import_of("import {}").unwrap(), ImportPolicy::default());
+        let general = ImportPolicy {
+            scrub: Some(Scrub {
+                local_asn: 65000,
+                zero: false,
+            }),
+            ..ImportPolicy::default()
+        };
+        assert_eq!(import_of("").unwrap(), general);
+        assert_eq!(import_of("import {}").unwrap(), general);
 
         let policy = import_of(
             "import {

crates/server/src/connection.rs

@@ -114,6 +114,8 @@ pub struct PeerSettings {
     pub require_role: bool,
     /// What the peer is told, and how.
     pub export: Arc<ExportPolicy>,
+    /// The groups the session is in, for control communities.
+    pub groups: Arc<[u32]>,
     /// Graceful restart with the peer.
     pub graceful_restart: GracefulRestartConfig,
 }
@@ -137,6 +139,7 @@ impl PeerSettings {
             role: None,
             require_role: false,
             export,
+            groups: Arc::default(),
             graceful_restart,
         }
     }
@@ -154,6 +157,7 @@ impl From<&PeerConfig> for PeerSettings {
             role: peer.role,
             require_role: peer.require_role,
             export: Arc::clone(&peer.export),
+            groups: peer.groups.as_slice().into(),
             graceful_restart: peer.graceful_restart,
         }
     }
@@ -567,6 +571,8 @@ struct Session {
     role: Option<PeerRole>,
     /// What the peer is told, and how.
     export: Arc<ExportPolicy>,
+    /// The groups the session is in, for control communities.
+    groups: Arc<[u32]>,
     /// Where what is refused is noted.
     refusals: Arc<PeerRefusals>,
     /// Graceful restart with the peer, as configured.
@@ -629,6 +635,7 @@ impl Session {
             import: peer.import,
             role: peer.role,
             export: peer.export,
+            groups: peer.groups,
             said: Said::new(),
             graceful_restart: peer.graceful_restart,
             leave,
@@ -930,6 +937,7 @@ impl Session {
         handle.role = self.role;
         handle.role_confirmed = negotiated.role_confirmed;
         handle.export = Arc::clone(&self.export);
+        handle.groups = Arc::clone(&self.groups);
         // Routes from other networks are validated, where there is data
         // to validate against and the peer's policy does not say to take
         // invalid ones anyway.

crates/server/src/error.rs

@@ -220,6 +220,8 @@ pub enum PeerProblem {
     RequireRoleWithoutRole,
     #[error("role has a value that is not one of its settings")]
     Role,
+    #[error("groups has a 0 in it; groups are numbered from 1")]
+    Group,
     #[error("import: {0}")]
     Import(#[from] ImportProblem),
     #[error("export: {0}")]

crates/server/src/export.rs

@@ -15,7 +15,9 @@
 //! - not up or across if it came down or across: a route marked "only to
 //!   customer" goes to customers only (RFC 9234 §5, egress, 2);
 //! - not where its communities say it is not to go: NO_EXPORT and
-//!   NO_ADVERTISE (RFC 1997), and those of [`Control`].
+//!   NO_ADVERTISE (RFC 1997), and, where it leaves our AS, those of
+//!   [`Control`], which also say how many more times our AS goes on its
+//!   path.
 //!
 //! What goes is changed as BGP has it: our AS on the path and ourselves as
 //! next hop towards another network, the MED it came with dropped; the
@@ -33,7 +35,7 @@ use bgp_primitives::ip_prefix::Nlri;
 use bgp_rib::table::{DEFAULT_LOCAL_PREF, PeerIndex};
 
 use crate::advertise::{Towards, ipv6_next_hop};
-use crate::marks::{self, Control, communities_of, well_known_allow};
+use crate::marks::{self, Control, Named, communities_of, well_known_allow};
 use crate::policy::{ExportPolicy, PeerRole};
 
 /// The session a route may go out on, as far as that decides whether it
@@ -46,6 +48,9 @@ pub struct Egress<'a> {
     /// are marked with.
     pub rib_peer: PeerIndex,
     pub policy: &'a ExportPolicy,
+    /// The groups the session is in, for control communities to name it
+    /// by.
+    pub groups: &'a [u32],
     /// The communities by which routes say who is told of them, if they
     /// are honoured.
     pub control: Option<Control>,
@@ -78,16 +83,25 @@ impl Egress<'_> {
     /// whether it goes.
     pub fn dress(&self, attributes: &mut Vec<PathAttribute>) -> bool {
         let towards = &self.towards;
-        let (communities, large) = communities_of(attributes);
+        let mut prepends = self.policy.prepend;
-        if let Some(control) = self.control {
+        // Control communities are acted on where a route leaves our AS. To
-            if !control.allows(communities, large, self.peer_asn) {
+        // a peer in it the route goes with them, for that router to act
+        // on where it passes the route on.
+        if let Some(control) = self.control.filter(|_| !towards.ibgp) {
+            let (communities, large) = communities_of(attributes);
+            let session = Named {
+                asn: self.peer_asn,
+                groups: self.groups,
+            };
+            if !control.allows(communities, large, session) {
                 return false;
             }
+            prepends = prepends.saturating_add(control.prepends(large, session));
             control.strip(attributes);
         }
         self.policy.marks.apply(attributes);
-        if self.policy.prepend > 0 && !towards.ibgp {
+        if prepends > 0 && !towards.ibgp {
-            let ours = std::iter::repeat_n(towards.local_asn, usize::from(self.policy.prepend));
+            let ours = std::iter::repeat_n(towards.local_asn, usize::from(prepends));
             for attribute in attributes.iter_mut() {
                 if let PathAttribute::AsPath(path) = attribute {
                     prepend(path, ours);
@@ -361,14 +375,14 @@ mod tests {
     use std::net::Ipv4Addr;
 
     use bgp_packet::message::{
-        CommunitiesPathAttribute, MultiExitDiscPathAttribute, OriginPathAttribute,
+        CommunitiesPathAttribute, LargeCommunitiesPathAttribute, LargeCommunity,
-        PathAttributeFlags,
+        MultiExitDiscPathAttribute, OriginPathAttribute, PathAttributeFlags,
     };
     use bgp_primitives::ip_prefix::IpPrefix;
     use bgp_rib::table::{PeerInfo, RibTable};
 
     use super::*;
-    use crate::marks::Marks;
+    use crate::marks::{Marks, large};
 
     const LOCAL: u32 = 64601;
     const THEM: u32 = 64602;
@@ -429,6 +443,7 @@ mod tests {
         role: Option<PeerRole>,
         to_ibgp: bool,
         from_ibgp: bool,
+        groups: &'a [u32],
         control: Option<Control>,
     }
 
@@ -440,6 +455,7 @@ mod tests {
                 peer_asn: if self.to_ibgp { LOCAL } else { THEM },
                 rib_peer: to,
                 policy: self.policy,
+                groups: self.groups,
                 control: self.control,
             };
             let source = Source {
@@ -456,6 +472,7 @@ mod tests {
             role: None,
             to_ibgp: false,
             from_ibgp: false,
+            groups: &[],
             control: None,
         }
     }
@@ -518,6 +535,7 @@ mod tests {
             peer_asn: THEM,
             rib_peer: from,
             policy: &policy,
+            groups: &[],
             control: None,
         };
         let source = Source {
@@ -711,6 +729,98 @@ mod tests {
         );
     }
 
+    fn large_communities(communities: &[LargeCommunity]) -> PathAttribute {
+        PathAttribute::LargeCommunities(LargeCommunitiesPathAttribute {
+            communities: communities.to_vec(),
+        })
+    }
+
+    /// Control communities are for where a route leaves our AS. To a
+    /// router of our own the route goes with them, whatever they say, for
+    /// it to act on; to another network they decide, by group as well as
+    /// by AS, and are taken off, while our marks go on.
+    #[test]
+    fn control_communities_are_acted_on_where_routes_leave_our_as() {
+        let policy = all_routes();
+        let control = Some(Control { local_asn: LOCAL });
+        let mark = large(LOCAL, 100, 7);
+        let tagged = |actions: &[LargeCommunity]| {
+            route(vec![large_communities(
+                &[actions, std::slice::from_ref(&mark)].concat(),
+            )])
+        };
+
+        // Inside: as it came, though it says "to nobody".
+        let inside = Case {
+            to_ibgp: true,
+            control,
+            ..plain(&policy)
+        };
+        let nobody = [large(LOCAL, 0, 0), large(LOCAL, 31, 2)];
+        let kept = inside
+            .out(tagged(&nobody))
+            .expect("our own router hears of it");
+        assert!(
+            kept.attributes.contains(&large_communities(
+                &[&nobody[..], std::slice::from_ref(&mark)].concat()
+            )),
+            "{:?}",
+            kept.attributes
+        );
+        // Outside, it is done as it says.
+        let at_two = Case {
+            groups: &[2],
+            control,
+            ..plain(&policy)
+        };
+        assert!(at_two.out(tagged(&nobody)).is_none());
+
+        // Prepended where the group is named, and not elsewhere; the
+        // community that asked is gone, the mark is not.
+        let prepend_at_two = [large(LOCAL, 31, 2)];
+        let elsewhere = Case {
+            groups: &[1],
+            control,
+            ..plain(&policy)
+        };
+        for (case, path) in [
+            (&at_two, vec![LOCAL, LOCAL, 3356, 64700]),
+            (&elsewhere, vec![LOCAL, 3356, 64700]),
+        ] {
+            let outgoing = case.out(tagged(&prepend_at_two)).unwrap();
+            assert_eq!(
+                outgoing.attributes,
+                [
+                    PathAttribute::Origin(OriginPathAttribute::IGP),
+                    AsPathAttribute::from_asns(path),
+                    large_communities(std::slice::from_ref(&mark)),
+                ]
+            );
+        }
+        // A route without one, as an anycast prefix would be, is not.
+        let bare = at_two.out(route(vec![])).unwrap();
+        assert_eq!(
+            bare.attributes[1],
+            AsPathAttribute::from_asns(vec![LOCAL, 3356, 64700])
+        );
+
+        // The session's own prepend and the route's add up.
+        let prepending = ExportPolicy {
+            prepend: 1,
+            ..all_routes()
+        };
+        let both = Case {
+            groups: &[2],
+            control,
+            ..plain(&prepending)
+        };
+        let outgoing = both.out(tagged(&prepend_at_two)).unwrap();
+        assert_eq!(
+            outgoing.attributes[1],
+            AsPathAttribute::from_asns(vec![LOCAL, LOCAL, LOCAL, 3356, 64700])
+        );
+    }
+
     /// As many prefixes to a message as fit, and the path as AS4_PATH too
     /// where the session cannot carry it.
     #[test]

crates/server/src/marks.rs

@@ -9,7 +9,9 @@
 //! - [`Marks`]: communities taken off and put on every route from a peer
 //!   (`import`) or to one (`export`).
 //! - [`Control`]: communities on a route that say which peers it is, and
-//!   is not, passed on to.
+//!   is not, passed on to, and how far away we look to them.
+//! - [`Scrub`]: the communities of our own AS taken off what other
+//!   networks send, so that only we and those we let can ask for that.
 
 use bgp_packet::community::{CommunityError, parse_large, parse_standard};
 use bgp_packet::message::{
@@ -196,26 +198,52 @@ pub(crate) fn put(attributes: &mut Vec<PathAttribute>, attribute: PathAttribute)
     }
 }
 
-/// The communities by which a route says who is told of it, as Internet
+/// The communities by which a route says who is told of it, and with how
-/// exchanges' route servers have them. With `L` our own AS number and `P`
+/// many prepends, as Internet exchanges' route servers and transit
-/// a peer's:
+/// networks have them. With `L` our own AS number, `P` a peer's and `G` a
+/// group of sessions (`groups` of a peer):
 ///
 /// | Standard | Large | |
 /// |---|---|---|
 /// | `0:P` | `L:0:P` | not to the peer with AS `P` |
 /// | `L:P` | `L:1:P` | to the peer with AS `P`, and with that only to peers named so |
 /// | `0:L` | `L:0:0` | to nobody |
+/// | | `L:11:P`, `L:12:P`, `L:13:P` | our AS once, twice, three times more on the path to AS `P`; `P` 0 for everyone |
+/// | | `L:20:G` | not to the sessions of group `G` |
+/// | | `L:21:G` | to the sessions of group `G`, and with that only to sessions named so |
+/// | | `L:31:G`, `L:32:G`, `L:33:G` | our AS once, twice, three times more on the path to group `G` |
 ///
 /// A route with none of them goes to everyone. What says "not" wins over
-/// what says "to". A standard community has room for AS numbers below
+/// what says "to"; of several prepends, the most counts. A standard
-/// 65536 only; the large ones are for the others, and work for all.
+/// community has room for AS numbers below 65536 only; the large ones
+/// are for the others, and work for all.
 ///
-/// These are taken off a route as it is passed on: they were for us.
+/// They are acted on where a route leaves our AS, and taken off there:
+/// they were for us. Other large communities of ours are marks, which go
+/// on.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub struct Control {
     pub local_asn: u32,
 }
 
+/// What a large community of ours asks for, by its second part.
+const NOT_TO_AS: u32 = 0;
+const ONLY_TO_AS: u32 = 1;
+const PREPEND_TO_AS: [u32; 3] = [11, 12, 13];
+const NOT_TO_GROUP: u32 = 20;
+const ONLY_TO_GROUP: u32 = 21;
+const PREPEND_TO_GROUP: [u32; 3] = [31, 32, 33];
+
+/// Whether a large community of ours with `function` as its second part
+/// asks for something, and is not a mark.
+fn is_action(function: u32) -> bool {
+    matches!(
+        function,
+        NOT_TO_AS | ONLY_TO_AS | NOT_TO_GROUP | ONLY_TO_GROUP
+    ) || PREPEND_TO_AS.contains(&function)
+        || PREPEND_TO_GROUP.contains(&function)
+}
+
 /// The well-known communities that limit how far a route goes
 /// (RFC 1997).
 const NO_EXPORT: (u16, u16) = (0xFFFF, 0xFF01);
@@ -233,22 +261,38 @@ pub fn well_known_allow(communities: &[(u16, u16)], peer_is_internal: bool) -> b
     })
 }
 
+/// A session a route may go out on, as control communities name it.
+#[derive(Debug, Clone, Copy)]
+pub struct Named<'a> {
+    pub asn: u32,
+    pub groups: &'a [u32],
+}
+
+impl Named<'_> {
+    fn is(&self, asn: u32) -> bool {
+        asn == self.asn
+    }
+
+    fn in_group(&self, group: u32) -> bool {
+        self.groups.contains(&group)
+    }
+}
+
 impl Control {
     /// Our AS number as half a standard community, if it fits.
     fn local_16(&self) -> Option<u16> {
         u16::try_from(self.local_asn).ok()
     }
 
-    /// Whether a route with these communities is passed on to the peer
+    /// Whether a route with these communities is passed on to `session`.
-    /// with AS number `peer_asn`.
     pub fn allows(
         &self,
         communities: &[(u16, u16)],
         large: &[LargeCommunity],
-        peer_asn: u32,
+        session: Named<'_>,
     ) -> bool {
         let local_16 = self.local_16();
-        let peer_16 = u16::try_from(peer_asn).ok();
+        let peer_16 = u16::try_from(session.asn).ok();
         let (mut not_to_peer, mut to_peer, mut to_nobody, mut to_some) =
             (false, false, false, false);
         for (high, low) in communities {
@@ -267,10 +311,15 @@ impl Control {
                 continue;
             }
             match (community.data_1, community.data_2) {
-                (0, 0) => to_nobody = true,
+                (NOT_TO_AS, 0) => to_nobody = true,
-                (0, asn) => not_to_peer |= asn == peer_asn,
+                (NOT_TO_AS, asn) => not_to_peer |= session.is(asn),
-                (1, asn) => {
+                (ONLY_TO_AS, asn) => {
-                    to_peer |= asn == peer_asn;
+                    to_peer |= session.is(asn);
+                    to_some = true;
+                }
+                (NOT_TO_GROUP, group) => not_to_peer |= session.in_group(group),
+                (ONLY_TO_GROUP, group) => {
+                    to_peer |= session.in_group(group);
                     to_some = true;
                 }
                 _ => {}
@@ -279,31 +328,103 @@ impl Control {
         !not_to_peer && (to_peer || !(to_nobody || to_some))
     }
 
+    /// How many more times our AS goes on the path of a route with these
+    /// large communities to `session`: the most that one of them asks.
+    pub fn prepends(&self, large: &[LargeCommunity], session: Named<'_>) -> u8 {
+        let mut most = 0;
+        for community in large {
+            if community.global_admin != self.local_asn {
+                continue;
+            }
+            let (function, target) = (community.data_1, community.data_2);
+            let times = if let Some(at) = PREPEND_TO_AS.iter().position(|f| *f == function) {
+                (target == 0 || session.is(target)).then_some(at + 1)
+            } else if let Some(at) = PREPEND_TO_GROUP.iter().position(|f| *f == function) {
+                session.in_group(target).then_some(at + 1)
+            } else {
+                None
+            };
+            // At most three: the position in a list of three, plus one.
+            most = most.max(times.map_or(0, |times| times as u8));
+        }
+        most
+    }
+
     /// Takes off `attributes` the communities that were for us to act on.
     pub fn strip(&self, attributes: &mut Vec<PathAttribute>) {
         let local_16 = self.local_16();
-        let ours = |community: &(u16, u16)| community.0 == 0 || Some(community.0) == local_16;
+        remove_where(
-        let has = attributes.iter().any(|attribute| match attribute {
+            attributes,
-            PathAttribute::Communities(communities) => communities.0.iter().any(ours),
+            |community| community.0 == 0 || Some(community.0) == local_16,
-            _ => false,
+            |community| community.global_admin == self.local_asn && is_action(community.data_1),
-        });
+        );
-        if has {
+    }
-            let mut communities = take_communities(attributes);
+}
-            communities.retain(|community| !ours(community));
+
-            put_communities(attributes, communities);
+/// The communities of our own AS, taken off the routes of a peer in
-        }
+/// another AS as they come in. What they ask of us (see [`Control`]), and
-        let ours_large = |community: &LargeCommunity| {
+/// what our marks say of a route, is not for others to say: a peer at an
-            community.global_admin == self.local_asn && community.data_1 <= 1
+/// exchange that put `L:0:0` on its routes would decide which of our
-        };
+/// customers hear of them. The standard form of "not to" (`0:P`) is only
-        let has_large = attributes.iter().any(|attribute| match attribute {
+/// ours to act on where control communities are, and so taken off only
-            PathAttribute::LargeCommunities(large) => large.communities.iter().any(ours_large),
+/// then.
-            _ => false,
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-        });
+pub struct Scrub {
-        if has_large {
+    pub local_asn: u32,
-            let mut large = take_large_communities(attributes);
+    /// Take off `0:n` too.
-            large.retain(|community| !ours_large(community));
+    pub zero: bool,
-            put_large_communities(attributes, large);
+}
-        }
+
+impl Scrub {
+    fn ours(&self, community: &(u16, u16)) -> bool {
+        (self.zero && community.0 == 0) || u32::from(community.0) == self.local_asn
+    }
+
+    fn ours_large(&self, community: &LargeCommunity) -> bool {
+        community.global_admin == self.local_asn
+    }
+
+    /// Whether `attributes` have a community to take off: asked of every
+    /// UPDATE, so that those without one are taken as they came.
+    pub fn finds(&self, attributes: &[PathAttribute]) -> bool {
+        let (standard, large) = communities_of(attributes);
+        standard.iter().any(|community| self.ours(community))
+            || large.iter().any(|community| self.ours_large(community))
+    }
+
+    pub fn apply(&self, attributes: &mut Vec<PathAttribute>) {
+        remove_where(
+            attributes,
+            |community| self.ours(community),
+            |community| self.ours_large(community),
+        );
+    }
+}
+
+/// Takes off `attributes` the standard communities for which `standard`
+/// holds and the large ones for which `large` does, leaving the
+/// attributes as they were if there are none.
+fn remove_where(
+    attributes: &mut Vec<PathAttribute>,
+    standard: impl Fn(&(u16, u16)) -> bool,
+    large: impl Fn(&LargeCommunity) -> bool,
+) {
+    let (has, has_large) = {
+        let (communities, large_communities) = communities_of(attributes);
+        (
+            communities.iter().any(&standard),
+            large_communities.iter().any(&large),
+        )
+    };
+    if has {
+        let mut communities = take_communities(attributes);
+        communities.retain(|community| !standard(community));
+        put_communities(attributes, communities);
+    }
+    if has_large {
+        let mut large_communities = take_large_communities(attributes);
+        large_communities.retain(|community| !large(community));
+        put_large_communities(attributes, large_communities);
     }
 }
 
@@ -413,7 +534,7 @@ mod tests {
     fn communities_say_which_peers_a_route_goes_to() {
         let control = Control { local_asn: 64601 };
         let to = |communities: &[(u16, u16)], large: &[LargeCommunity], peer: u32| {
-            control.allows(communities, large, peer)
+            control.allows(communities, large, named(peer, &[]))
         };
         // Nothing said: everyone.
         assert!(to(&[(3356, 3)], &[], 64602));
@@ -451,6 +572,7 @@ mod tests {
             &[large(3356, 0, 64602), large(64601, 2, 64602)],
             64602
         ));
+        assert!(to(&[], &[large(64601, 100, 64602)], 64602));
         // A peer whose AS number does not fit a standard community is not
         // meant by one.
         assert!(to(&[(0, 1)], &[], 65537));
@@ -458,27 +580,122 @@ mod tests {
         // Our own AS number may not fit either: then only the large ones
         // can name us.
         let control = Control { local_asn: wide };
-        assert!(control.allows(&[(0, 64602)], &[], 64603));
+        assert!(control.allows(&[(0, 64602)], &[], named(64603, &[])));
-        assert!(!control.allows(&[(0, 64602)], &[], 64602));
+        assert!(!control.allows(&[(0, 64602)], &[], named(64602, &[])));
-        assert!(!control.allows(&[], &[large(wide, 0, 0)], 64602));
+        assert!(!control.allows(&[], &[large(wide, 0, 0)], named(64602, &[])));
+    }
+
+    fn named(asn: u32, groups: &[u32]) -> Named<'_> {
+        Named { asn, groups }
+    }
+
+    /// Sessions are named by group as well as by AS: the same network at
+    /// two exchanges is two sessions in two groups.
+    #[test]
+    fn communities_name_sessions_by_group() {
+        let control = Control { local_asn: 64601 };
+        let at_one = named(13335, &[1]);
+        let at_two = named(13335, &[2, 10]);
+        let allows = |large: &[LargeCommunity], session| control.allows(&[], large, session);
+        // Not to group 2: the same peer elsewhere still hears of it.
+        assert!(!allows(&[large(64601, 20, 2)], at_two));
+        assert!(allows(&[large(64601, 20, 2)], at_one));
+        // Only to group 1, or to group 1 and AS 13335 anywhere.
+        assert!(allows(&[large(64601, 21, 1)], at_one));
+        assert!(!allows(&[large(64601, 21, 1)], at_two));
+        assert!(!allows(&[large(64601, 21, 1)], named(64602, &[])));
+        assert!(allows(
+            &[large(64601, 21, 1), large(64601, 1, 13335)],
+            at_two
+        ));
+        // A session in several groups is in each of them.
+        assert!(!allows(&[large(64601, 20, 10)], at_two));
+        // "Not" wins, whichever way the session is named.
+        assert!(!allows(
+            &[large(64601, 1, 13335), large(64601, 20, 2)],
+            at_two
+        ));
+        assert!(!allows(
+            &[large(64601, 21, 2), large(64601, 0, 13335)],
+            at_two
+        ));
+    }
+
+    #[test]
+    fn communities_say_how_far_away_we_look() {
+        let control = Control { local_asn: 64601 };
+        let at_two = named(13335, &[2]);
+        let prepends = |large: &[LargeCommunity], session| control.prepends(large, session);
+        assert_eq!(prepends(&[], at_two), 0);
+        // To a group, to an AS, to everyone.
+        assert_eq!(prepends(&[large(64601, 31, 2)], at_two), 1);
+        assert_eq!(prepends(&[large(64601, 33, 2)], at_two), 3);
+        assert_eq!(prepends(&[large(64601, 31, 1)], at_two), 0);
+        assert_eq!(prepends(&[large(64601, 12, 13335)], at_two), 2);
+        assert_eq!(prepends(&[large(64601, 12, 64602)], at_two), 0);
+        assert_eq!(prepends(&[large(64601, 11, 0)], at_two), 1);
+        // The most counts, not the sum.
+        assert_eq!(
+            prepends(
+                &[
+                    large(64601, 31, 2),
+                    large(64601, 13, 13335),
+                    large(64601, 12, 0)
+                ],
+                at_two
+            ),
+            3
+        );
+        // Group 0 is no group; someone else's say nothing.
+        assert_eq!(prepends(&[large(64601, 31, 0)], named(13335, &[])), 0);
+        assert_eq!(prepends(&[large(3356, 13, 0)], at_two), 0);
+    }
+
+    /// From another network, ours are taken off, and nothing else is.
+    #[test]
+    fn our_communities_are_not_taken_from_others() {
+        let scrub = Scrub {
+            local_asn: 64601,
+            zero: true,
+        };
+        let mut attributes = route(
+            &[(0, 64602), (3356, 3), (64601, 900)],
+            &[large(64601, 0, 0), large(64601, 100, 1), large(3356, 0, 5)],
+        );
+        assert!(scrub.finds(&attributes));
+        scrub.apply(&mut attributes);
+        assert_eq!(attributes, route(&[(3356, 3)], &[large(3356, 0, 5)]));
+        assert!(!scrub.finds(&attributes));
+
+        // `0:n` only where it means something to us.
+        let scrub = Scrub {
+            zero: false,
+            ..scrub
+        };
+        assert!(!scrub.finds(&route(&[(0, 64602)], &[])));
+        // An AS number too wide for a standard community has none of them.
+        let wide = Scrub {
+            local_asn: 4_200_000_001,
+            zero: false,
+        };
+        assert!(!wide.finds(&route(&[(64601, 1)], &[large(64601, 0, 0)])));
+        assert!(wide.finds(&route(&[], &[large(4_200_000_001, 100, 1)])));
     }
 
     #[test]
     fn what_was_for_us_is_taken_off_as_a_route_is_passed_on() {
         let control = Control { local_asn: 64601 };
+        let actions = [0, 1, 11, 12, 13, 20, 21, 31, 32, 33].map(|f| large(64601, f, 5));
+        let marks = [2, 3, 9, 10, 14, 30, 100].map(|f| large(64601, f, 7));
         let mut attributes = route(
             &[(0, 64602), (3356, 3), (64601, 64603)],
-            &[
+            &[&actions[..], &marks[..], &[large(3356, 0, 5)]].concat(),
-                large(64601, 0, 5),
-                large(64601, 1, 6),
-                large(64601, 2, 7),
-                large(3356, 0, 5),
-            ],
         );
         control.strip(&mut attributes);
+        // Our marks go on; what was for us to do does not.
         assert_eq!(
             attributes,
-            route(&[(3356, 3)], &[large(64601, 2, 7), large(3356, 0, 5)])
+            route(&[(3356, 3)], &[&marks[..], &[large(3356, 0, 5)]].concat())
         );
         let mut all = route(&[(0, 1)], &[large(64601, 0, 0)]);
         control.strip(&mut all);

crates/server/src/policy.rs

@@ -37,7 +37,7 @@ use bgp_primitives::constants::IpVersion;
 use bgp_primitives::ip_prefix::IpPrefix;
 
 use crate::address::PeerAddress;
-use crate::marks::{self, Marks};
+use crate::marks::{self, Marks, Scrub};
 use crate::sync::IgnorePoison;
 
 /// The longest prefixes routed between networks: anything more specific is
@@ -106,6 +106,14 @@ pub struct ImportPolicy {
     /// counted as rejected. Empty for a peer in another AS, whose routes
     /// for them `own_prefixes` refuses.
     pub originated: Arc<PrefixSet>,
+    /// Keep the communities of our own AS on the peer's routes, for it to
+    /// steer them with control communities: a customer. Without it, those
+    /// of a peer in another AS are taken off by `scrub`.
+    pub accept_control_communities: bool,
+    /// The communities of our own AS taken off the peer's routes as they
+    /// come in, first; `None` for a peer in our own AS and for one with
+    /// `accept_control_communities`.
+    pub scrub: Option<Scrub>,
     /// Communities taken off and put on every route that is taken.
     pub marks: Marks,
     /// The LOCAL_PREF every route that is taken is given; `None` leaves
@@ -134,6 +142,8 @@ impl Default for ImportPolicy {
             accept_third_party_next_hop: false,
             own_prefixes: Arc::default(),
             originated: Arc::default(),
+            accept_control_communities: false,
+            scrub: None,
             marks: Marks::default(),
             local_pref: None,
         }
@@ -162,14 +172,21 @@ impl ImportPolicy {
 }
 
 impl ImportPolicy {
-    /// Whether routes are changed on their way in.
+    /// Whether routes with `attributes` are changed on their way in.
-    pub fn changes_routes(&self) -> bool {
+    /// Asked of every UPDATE: those that come as they are to be kept are
-        !self.marks.is_empty() || self.local_pref.is_some()
+    /// taken as they came, without being written out again.
+    pub fn changes(&self, attributes: &[PathAttribute]) -> bool {
+        !self.marks.is_empty()
+            || self.local_pref.is_some()
+            || self.scrub.is_some_and(|scrub| scrub.finds(attributes))
     }
 
-    /// Makes of `attributes` what the route is kept with: marked, and
+    /// Makes of `attributes` what the route is kept with: without what
-    /// with its preference.
+    /// others may not say for us, marked, and with its preference.
     pub fn change(&self, attributes: &mut Vec<PathAttribute>) {
+        if let Some(scrub) = self.scrub {
+            scrub.apply(attributes);
+        }
         self.marks.apply(attributes);
         if let Some(local_pref) = self.local_pref {
             marks::put(

crates/server/src/routes.rs

@@ -370,7 +370,7 @@ impl Applying<'_> {
         // What the policy puts on the peer's routes (communities that say
         // where they were learned, a preference) is the same for all of
         // an UPDATE's routes, and worked out once for them.
-        if self.import.policy.changes_routes() {
+        if self.import.policy.changes(&self.update.path_attributes) {
             let changed = self.changed.get_or_init(|| {
                 let mut attributes = self.update.path_attributes.clone();
                 // Not the route's own: they hold the prefixes.

crates/server/src/session.rs

@@ -165,6 +165,8 @@ pub struct SessionHandle {
     pub counters: Arc<SessionCounters>,
     /// What the peer is told, and how.
     pub export: Arc<ExportPolicy>,
+    /// The groups the session is in, for control communities.
+    pub groups: Arc<[u32]>,
     outbound: mpsc::Sender<Message>,
     /// Which of the RPKI's checks the peer's routes are subject to, if
     /// any, and where the session is told to look at its routes again.
@@ -203,6 +205,7 @@ impl SessionHandle {
             graceful_restart: negotiated.graceful_restart.clone(),
             counters: Arc::default(),
             export: Arc::default(),
+            groups: Arc::default(),
             outbound,
             validation: None,
         }

docs/content/docs/guides/import-policy.md

@@ -235,8 +235,7 @@ set their local preference:
 peers {
   name: "ixp-a-rs"
   import {
-    remove_communities: "64500:2:*"
+    remove_communities: "64496:*"
-    remove_communities: "64500:3:*"
     add_large_communities: "64500:2:1001"
     local_pref: 150
   }
@@ -246,7 +245,9 @@ peers {
 The statements have the following effect:
 
 - `remove_communities` removes matching communities. Ferrix applies it
-  before adding communities. A pattern has two fields for a standard
+  after it removes the communities of the local AS (see
+  [Communities of the Local AS](#communities-of-the-local-as)), and before
+  adding communities. A pattern has two fields for a standard
   community and three for a large community; each field is a number or `*`.
   Examples: `64500:100`, `64500:*`, `64500:2:*`, `*:*`.
 - `add_communities` and `add_large_communities` add communities to every
@@ -257,17 +258,52 @@ The statements have the following effect:
   value is preferred. Default: 100.
 
 > [!TIP]
-> Remove the local AS's informational communities from received routes, so
+> A common local preference scheme assigns 200 to customer routes, 150 to
-> that a neighbor cannot set them. A common local preference scheme assigns
+> peer and route server routes, and 100 to transit routes.
-> 200 to customer routes, 150 to peer and route server routes, and 100 to
-> transit routes.
 
 The examples in this documentation use large communities `64500:2:x` to
 record where a route was learned and `64500:3:r` to record the router that
 advertised it. These values do not overlap the
 [control communities](@/docs/guides/passing-routes-on.md#control-communities),
-which use `64500:0:*` and `64500:1:*`, or the standard communities
+or the standard communities `64500:n`, which control communities interpret
-`64500:n`, which control communities interpret as "advertise to AS n".
+as "advertise to AS n".
+
+## Communities of the Local AS
+
+By default, Ferrix removes the local AS's communities from every route that
+it receives from an EBGP neighbor, before `add_communities` and
+`add_large_communities` are applied. The following communities are
+removed, where `L` is the local AS number:
+
+- Standard communities `L:n`, if `L` is 65535 or lower.
+- Large communities `L:n:m`.
+- Standard communities `0:n`, if `control_communities` is enabled.
+
+These communities are instructions and records of the local AS: control
+communities decide where Ferrix advertises a route, and informational
+communities record where a route was learned. A neighbor at an Internet
+exchange must not be able to set them. Ferrix retains them on routes from
+IBGP neighbors.
+
+A customer that sets control communities on its own routes requires the
+following statement:
+
+```textproto
+peers {
+  name: "customer-a"
+  role: CUSTOMER
+  import { accept_control_communities: true }
+}
+```
+
+Ferrix then retains all communities of the local AS on the customer's
+routes. The customer can affect only its own routes. Use
+`remove_communities` to remove informational communities that the
+customer must not set, for example `64500:2:*`.
+
+Ferrix examines the communities of every UPDATE message from an EBGP
+neighbor. A message without communities of the local AS is accepted
+unchanged.
 
 ## Performance
 

docs/content/docs/guides/passing-routes-on.md

@@ -111,52 +111,130 @@ export {
 ## Control Communities
 
 Control communities allow a route's communities to restrict the neighbors
-to which Ferrix advertises it. The format is the one used by Internet
+to which Ferrix advertises it, and to prepend the local AS number to its AS
-exchange route servers. To enable control communities, configure the
+path on its way to them. The formats are the ones used by Internet exchange
-following top-level statement:
+route servers and transit providers. To enable control communities,
+configure the following top-level statement:
 
 ```textproto
 control_communities: true
 ```
 
-Table 2 lists the control communities, where `L` is the local AS number and
+Table 2 lists the control communities, where `L` is the local AS number,
-`P` is a neighbor's AS number.
+`P` is a neighbor's AS number, and `G` is a session group (see
+[Session Groups](#session-groups)).
 
 | Standard community | Large community | Action |
 |---|---|---|
 | `0:P` | `L:0:P` | Do not advertise to the neighbor with AS `P`. |
 | `L:P` | `L:1:P` | Advertise to the neighbor with AS `P`. If present, advertise only to neighbors named in such communities. |
 | `0:L` | `L:0:0` | Do not advertise to any neighbor. |
+| | `L:11:P`, `L:12:P`, `L:13:P` | Prepend the local AS number once, twice, or three times when advertising to the neighbor with AS `P`. If `P` is 0, prepend to every EBGP neighbor. |
+| | `L:20:G` | Do not advertise to the sessions in group `G`. |
+| | `L:21:G` | Advertise to the sessions in group `G`. If present, advertise only to sessions named in such communities, by group or by AS. |
+| | `L:31:G`, `L:32:G`, `L:33:G` | Prepend the local AS number once, twice, or three times when advertising to the sessions in group `G`. |
 
 Control communities are evaluated as follows:
 
 - "Do not advertise" takes precedence over "advertise".
 - A route without control communities is advertised as the other rules
   permit.
+- If several prepend communities apply to a session, the largest number of
+  prepends applies. It is added to the session's `as_path_prepend`.
 - Standard communities can express AS numbers up to 65535 only. Use the
   large community forms for four-octet AS numbers.
-- Ferrix removes control communities from a route when it advertises the
-  route.
-- Control communities apply to originated routes and to communities added
-  by the import policy.
 
-The following example prevents the advertisement of an originated route to
+### Where Control Communities Apply
-AS 64496:
+
+Ferrix acts on control communities when it advertises a route to an EBGP
+neighbor, and removes them from the route at that point. Informational
+communities of the local AS are not removed. To an IBGP neighbor, Ferrix
+advertises the route with its control communities, and without acting on
+them: the other router acts on them when it advertises the route outside
+the local AS. A control community can therefore be set on one router and
+take effect on another.
+
+Ferrix acts on control communities that come from the following sources:
+
+- The `communities` and `large_communities` statements of an originated
+  route.
+- The `add_communities` and `add_large_communities` statements of an
+  import policy.
+- Routes from IBGP neighbors.
+- Routes from EBGP neighbors with `accept_control_communities`, usually
+  customers that steer their own routes.
+
+Ferrix removes the local AS's communities from routes from all other
+neighbors when it receives them. See
+[Communities of the Local AS](@/docs/guides/import-policy.md#communities-of-the-local-as).
+
+### Session Groups
+
+A control community can name sessions by group instead of by AS. A group
+is a number that the configuration assigns to sessions, for example the
+sessions at one Internet exchange. A neighbor present at two exchanges has
+the same AS number at both, but its sessions are in different groups. A
+session can be in several groups:
 
 ```textproto
-control_communities: true
+peers {
-originate {
+  name: "exchange-b-rs1"
-  prefix: "198.51.100.0/24"
+  role: ROUTE_SERVER
-  large_communities: "64500:0:64496"
+  groups: 2
+  groups: 10
 }
 ```
 
+Use the same group numbers on all routers in the local AS, because a
+control community set on one router can take effect on another.
+
+### Example: Inbound Traffic Engineering
+
+In the following example, the router is connected to exchange A over a
+fast link and to exchange B over a slow one. Routes of the customer are to
+reach networks at exchange B with a longer AS path, so that networks
+present at both exchanges send traffic for them through exchange A. The
+originated anycast prefix is advertised at both exchanges unchanged.
+
+```textproto
+local_asn: 64500
+control_communities: true
+originate { prefix: "198.51.100.0/24" }
+
+peers {
+  name: "customer-a"
+  role: CUSTOMER
+  import {
+    add_large_communities: "64500:31:2"
+    accept_control_communities: true
+  }
+}
+
+peers {
+  name: "exchange-a-rs1"
+  role: ROUTE_SERVER
+  groups: 1
+}
+
+peers {
+  name: "exchange-b-rs1"
+  role: ROUTE_SERVER
+  groups: 2
+}
+```
+
+Ferrix adds `64500:31:2` to every route of the customer, and prepends the
+local AS number once to those routes on the sessions in group 2. With
+`accept_control_communities`, the customer can also set control
+communities on individual prefixes, for example `64500:20:2` to withdraw a
+prefix from exchange B.
+
 > [!IMPORTANT]
 > When control communities are enabled, Ferrix interprets the standard
-> community `L:n` as "advertise to AS n", and the large communities `L:0:n`
+> community `L:n` as "advertise to AS n", and the large communities of
-> and `L:1:n` as listed in Table 2. Use large communities `L:2:n` and higher
+> Table 2 as listed. Use large communities with a second field of 2
-> for informational communities. Do not remove customers' control
+> through 9, or 100 and higher, for informational communities. Values 10
-> communities with `remove_communities`.
+> through 99 are reserved for control communities.
 
 ## Update Generation
 

docs/content/docs/reference/configuration.md

@@ -54,7 +54,7 @@ Top-level statements.
 | `rpki` <span class="type">[Rpki](#rpki)</span> | Disabled | Enables RPKI route origin validation and ASPA verification. |
 | `kernel` <span class="type">[Kernel](#kernel)</span> | Disabled | Enables kernel route installation (Linux only). |
 | `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Originated routes only | Specifies the export policy for neighbors that have no `export` statement. |
-| `control_communities` <span class="type">bool</span> | `false` | Enables control communities, which restrict the neighbors to which a route is advertised. See [Control Communities](@/docs/guides/passing-routes-on.md#control-communities). |
+| `control_communities` <span class="type">bool</span> | `false` | Enables control communities, which restrict the neighbors to which a route is advertised and prepend the local AS number on its way to them. See [Control Communities](@/docs/guides/passing-routes-on.md#control-communities). |
 | `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Enabled | Specifies graceful restart for neighbors that have no `graceful_restart` statement. Its `restart_time_seconds` also limits how long Ferrix waits at startup for neighbors to send their routes. |
 
 ## Peer
@@ -75,6 +75,7 @@ Statements for a BGP neighbor.
 | `require_role` <span class="type">bool</span> | `false` | Rejects the session if the neighbor does not advertise a BGP role. Requires `role`. |
 | `export` <span class="type">[ExportPolicy](#exportpolicy)</span> | Top-level `export` | Specifies the export policy for the neighbor. Replaces the top-level `export` statement. |
 | `graceful_restart` <span class="type">[GracefulRestart](#gracefulrestart)</span> | Top-level `graceful_restart` | Specifies graceful restart for the neighbor. Each statement that is set replaces the corresponding top-level statement. |
+| `groups` <span class="type">uint32, repeated</span> | None | Specifies the groups of the session, by number, for control communities that name sessions by group. Use the same numbers on all routers in the local AS. Range: 1 through 4,294,967,295. See [Session Groups](@/docs/guides/passing-routes-on.md#session-groups). |
 
 ## TcpEndpoint
 
@@ -110,6 +111,7 @@ accepted. See [Import Policy](@/docs/guides/import-policy.md).
 | `add_large_communities` <span class="type">string, repeated</span> | None | Specifies large communities added to accepted routes, for example `64500:2:1`. |
 | `remove_communities` <span class="type">string, repeated</span> | None | Specifies patterns of communities removed from accepted routes, for example `64500:100`, `64500:*`, or `64500:2:*`. Applied before communities are added. |
 | `local_pref` <span class="type">uint32, optional</span> | 100 | Specifies the local preference of accepted routes. |
+| `accept_control_communities` <span class="type">bool</span> | `false` | Retains the local AS's communities on routes from an EBGP neighbor, so that the neighbor can set control communities on its own routes. By default, Ferrix removes them. Has no effect for an IBGP neighbor. See [Communities of the Local AS](@/docs/guides/import-policy.md#communities-of-the-local-as). |
 
 ## PrefixRange
 

docs/examples/transit.textpb

@@ -10,7 +10,9 @@ export { learned_routes: true }
 # Customers can restrict the advertisement of their routes with the large
 # communities 64500:0:P (not to AS P), 64500:1:P (only to AS P), and
 # 64500:0:0 (to no neighbor), or the standard communities 0:P, 64500:P, and
-# 0:64500.
+# 0:64500, and prepend 64500 towards AS P with 64500:11:P to 64500:13:P.
+# Ferrix removes the communities of AS 64500 from the routes of other
+# neighbors, such as the providers below.
 control_communities: true
 
 peers {
@@ -20,7 +22,6 @@ peers {
   role: PROVIDER
   import {
     max_prefixes: 1500000
-    remove_communities: "64500:2:*"
     add_large_communities: "64500:2:100"
   }
 }
@@ -32,7 +33,6 @@ peers {
   role: PROVIDER
   import {
     max_prefixes: 1500000
-    remove_communities: "64500:2:*"
     add_large_communities: "64500:2:200"
   }
   # Prepend to make the path through transit-b less preferred. This
@@ -43,6 +43,8 @@ peers {
 # Customer with a four-octet AS number. The prefix and origin AS lists
 # restrict its routes to its registered address space. Local preference
 # 200 makes its routes preferred over the same prefixes from providers.
+# The customer may set control communities; the informational communities
+# 64500:2:* are removed, so that it cannot set those.
 peers {
   name: "customer-65536"
   endpoint { address: "203.0.113.5" }
@@ -54,6 +56,7 @@ peers {
     prefixes { prefix: "192.0.2.0/24" }
     prefixes { prefix: "2001:db8:a00::/40" max_length: 48 }
     origin_asns: 65536
+    accept_control_communities: true
     remove_communities: "64500:2:*"
     add_large_communities: "64500:2:300"
     local_pref: 200

scripts/interop/lab.sh

@@ -890,23 +890,31 @@ phase_nexthops() {
 # Routes are passed on: FRR becomes our customer and BIRD stays our
 # provider, so what FRR announces goes up to BIRD with our AS on the path,
 # what BIRD announces goes down to FRR marked "only to customer", and
-# packets go between their networks through us. A community on one of
+# packets go between their networks through us. FRR, as a customer, steers
-# FRR's routes keeps it from BIRD.
+# its routes with our control communities: one keeps a route from BIRD,
+# another has us prepend on the way to BIRD's session group. BIRD, as a
+# provider, may not: its "not to FRR" is taken off as its route comes in.
 phase_transit() {
     dut_down TERM
-    dut_config CUSTOMER 2000000 "" "" "" \
+    dut_config CUSTOMER 2000000 "accept_control_communities: true" "" "groups: 7" \
         'export { learned_routes: true }
 control_communities: true'
+    echo "route 10.3.8.0/24 blackhole { bgp_large_community.add((64601, 0, 64602)); };" > "$work/bird-transit.conf"
+    bird_config "$work/bird-transit.conf"
+    bird_ctl configure >/dev/null
     in_ns dut sysctl -qw net.ipv4.ip_forward=1
     dut_up
     frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role customer" \
         "ip prefix-list KEPT seq 5 permit 10.2.9.0/24" \
+        "ip prefix-list FAR seq 5 permit 10.2.10.0/24" \
         "route-map TO-DUT-TRANSIT permit 10" "match ip address prefix-list KEPT" "set large-community 64601:0:4200000003" \
+        "route-map TO-DUT-TRANSIT permit 15" "match ip address prefix-list FAR" "set large-community 64601:31:7" \
         "route-map TO-DUT-TRANSIT permit 20" \
         "router bgp 64602" "address-family ipv4 unicast" \
-        "neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "network 10.2.9.0/24"
+        "neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "network 10.2.9.0/24" "network 10.2.10.0/24"
     wait_for "four sessions" established 4
     wait_for "FRR's routes at the daemon" dut_has 10.2.9.0/24 frr4
+    wait_for "FRR's routes at the daemon" dut_has 10.2.10.0/24 frr4
 
     # Up to the provider, with our AS on the path; not the one that says
     # not to.
@@ -915,6 +923,19 @@ control_communities: true'
     bird_ctl show route for 10.2.9.0/24 all | grep -q "64601 64602" && fail "BIRD was told the route whose community says not to"
     ok "the customer's routes reach the provider with our AS on the path; one kept back by its community"
 
+    # Prepended once more towards BIRD's group, at the customer's word,
+    # and the community that asked for it gone.
+    wait_for "the prepended route at BIRD" bash -c "sudo birdc -s $work/bird.ctl show route for 10.2.10.0/24 all | grep -q 'BGP.as_path: 64601 64601 64602'"
+    bird_ctl show route for 10.2.10.0/24 all | grep -q "(64601, 31, 7)" && fail "the control community went on to BIRD"
+    ok "a customer's control community has us prepend towards a session group, and is taken off"
+
+    # The provider's "not to FRR" is not ours to act on: taken off as the
+    # route comes in, and the route goes down to FRR all the same.
+    wait_for "BIRD's route with our community at FRR" bash -c "sudo ip netns exec lab-frr vtysh -N $pathspace -c 'show bgp ipv4 unicast 10.3.8.0/24 json' 2>/dev/null | jq -e '.paths[0].aspath.string == \"64601 4200000003\"'"
+    api "/api/routes/received?prefix=10.3.8.0/24" | jq -e '.routes | length > 0 and all(.summary.large_communities | index("64601:0:64602") | not)' >/dev/null \
+        || fail "the provider's control community was kept: $(api "/api/routes/received?prefix=10.3.8.0/24")"
+    ok "a provider's control community is taken off as its route comes in"
+
     # Down to the customer, marked.
     wait_for "BIRD's route at FRR, by way of us" bash -c "sudo ip netns exec lab-frr vtysh -N $pathspace -c 'show bgp ipv4 unicast 10.3.0.0/24 json' 2>/dev/null | jq -e '.paths[0].aspath.string == \"64601 4200000003\"'"
     # Marked on the way in from the provider, with the provider's AS, and
@@ -928,7 +949,9 @@ control_communities: true'
     ok "packets go between the customer's and the provider's networks through us"
 
     frr_configure "router bgp 64602" "neighbor 10.9.0.1 local-role peer" \
-        "address-family ipv4 unicast" "no network 10.2.9.0/24" "no neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "no route-map TO-DUT-TRANSIT"
+        "address-family ipv4 unicast" "no network 10.2.9.0/24" "no network 10.2.10.0/24" "no neighbor 10.9.0.1 route-map TO-DUT-TRANSIT out" "no route-map TO-DUT-TRANSIT"
+    bird_config
+    bird_ctl configure >/dev/null
     dut_down TERM
     dut_config
     dut_up