Rayhaan JaufeerallyandClaude Fable 5.1
d7d7d39bb9
The kernel's view of a session's TCP connection, which the status page shows on request since the last change, could not be looked at after the fact: a feed that crawled overnight left no record of whether the peer's window or our own side held it up. This change samples it at every scrape of `/metricz`: for each established session, one `NETLINK_SOCK_DIAG` round trip, and twenty gauges labelled by peer, `bgp_peer_tcp_*`: smoothed, least and variance of the round-trip time; the congestion window and segment size; the peer's receive window and ours; the send queue, the part of it not yet sent, the receive queue and the segments in flight; retransmitted and lost segments; the busy time and the time held up by the peer's window and by our send buffer, cumulative, so that their rates give the share of time a session is held up; octets sent and received; and the delivery and pacing rates. A session the kernel has no connection for loses its series, as does one that ends. `SessionHandle::metric_label` is the one place a peer's metric label is made (its name, or its address), used by the advertiser's gauge and here. Documentation: the metrics reference lists them; the monitoring guide gives the query for the share of time a session is held up by the neighbor's receive window, which is the question the overnight feed raised. Testing: a daemon in the Linux VM with one session shows the series and drops them when the session ends; the lab's basics phase checks the scrape for the FRR session's round-trip time; the workspace tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
119 lines
4.5 KiB
Rust
119 lines
4.5 KiB
Rust
//! Prometheus scrape endpoint. Counters are updated by the server crate as
|
|
//! messages flow; table-size gauges are refreshed at scrape time.
|
|
|
|
use axum::extract::State;
|
|
use axum::http::{StatusCode, header};
|
|
use axum::response::IntoResponse;
|
|
use bgp_server::metrics::{
|
|
LOCAL_ROUTES, PEER_TCP, RIB_ATTRIBUTE_SETS, RIB_JOURNAL_ENTRIES, RIB_PREFIXES, RIB_ROUTES,
|
|
};
|
|
use prometheus::{Encoder, TextEncoder};
|
|
|
|
use crate::AppState;
|
|
|
|
pub async fn metricz(State(state): State<AppState>) -> impl IntoResponse {
|
|
// A scrape that cannot read a table fails. Serving the last known
|
|
// size would hide that the daemon is broken from the one system whose
|
|
// job is to notice.
|
|
let gauge = |count: usize| i64::try_from(count).unwrap_or(i64::MAX);
|
|
match state.local_rib.len().await {
|
|
Ok(local) => LOCAL_ROUTES.set(gauge(local)),
|
|
Err(e) => return (StatusCode::SERVICE_UNAVAILABLE, e.to_string()).into_response(),
|
|
}
|
|
RIB_ROUTES.set(gauge(state.rib.len()));
|
|
RIB_PREFIXES.set(gauge(state.rib.prefix_count()));
|
|
RIB_ATTRIBUTE_SETS.set(gauge(state.rib.attrs().len()));
|
|
RIB_JOURNAL_ENTRIES.set(gauge(state.rib.journal_len()));
|
|
sample_tcp(&state).await;
|
|
|
|
let mut buf = Vec::new();
|
|
let encoder = TextEncoder::new();
|
|
match encoder.encode(&prometheus::gather(), &mut buf) {
|
|
Ok(()) => (
|
|
StatusCode::OK,
|
|
[(header::CONTENT_TYPE, encoder.format_type())],
|
|
buf,
|
|
)
|
|
.into_response(),
|
|
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(),
|
|
}
|
|
}
|
|
|
|
/// Asks the kernel about every established session's TCP connection and
|
|
/// sets the peer's gauges: one netlink round trip a session, at scrape
|
|
/// time, so that there is history to look back on. A session the kernel
|
|
/// has no connection for any more loses its series.
|
|
async fn sample_tcp(state: &AppState) {
|
|
let tcp = &*PEER_TCP;
|
|
for session in state.registry.list() {
|
|
let label = session.metric_label();
|
|
let connection =
|
|
bgp_fib::sockdiag::tcp_connection(session.local_addr, session.peer.addr).await;
|
|
let Ok(Some(c)) = connection else {
|
|
tcp.remove(&label);
|
|
continue;
|
|
};
|
|
let seconds = |us: u64| us as f64 / 1e6;
|
|
let int = |n: u64| i64::try_from(n).unwrap_or(i64::MAX);
|
|
let label = [label.as_str()];
|
|
tcp.rtt_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.rtt.into()));
|
|
tcp.min_rtt_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.min_rtt.into()));
|
|
tcp.rtt_variance_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.rtt_variance.into()));
|
|
tcp.cwnd_segments
|
|
.with_label_values(&label)
|
|
.set(c.cwnd.into());
|
|
tcp.mss_bytes.with_label_values(&label).set(c.mss.into());
|
|
tcp.send_window_bytes
|
|
.with_label_values(&label)
|
|
.set(c.send_window.into());
|
|
tcp.receive_window_bytes
|
|
.with_label_values(&label)
|
|
.set(c.receive_window.into());
|
|
tcp.send_queue_bytes
|
|
.with_label_values(&label)
|
|
.set(c.send_queue.into());
|
|
tcp.not_sent_bytes
|
|
.with_label_values(&label)
|
|
.set(c.not_sent.into());
|
|
tcp.receive_queue_bytes
|
|
.with_label_values(&label)
|
|
.set(c.recv_queue.into());
|
|
tcp.unacked_segments
|
|
.with_label_values(&label)
|
|
.set(c.unacked.into());
|
|
tcp.retransmitted_segments
|
|
.with_label_values(&label)
|
|
.set(c.total_retrans.into());
|
|
tcp.lost_segments
|
|
.with_label_values(&label)
|
|
.set(c.lost.into());
|
|
tcp.busy_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.busy));
|
|
tcp.receive_window_limited_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.limited_by_receive_window));
|
|
tcp.send_buffer_limited_seconds
|
|
.with_label_values(&label)
|
|
.set(seconds(c.limited_by_send_buffer));
|
|
tcp.sent_bytes
|
|
.with_label_values(&label)
|
|
.set(int(c.bytes_sent));
|
|
tcp.received_bytes
|
|
.with_label_values(&label)
|
|
.set(int(c.bytes_received));
|
|
tcp.delivery_rate_bytes_per_second
|
|
.with_label_values(&label)
|
|
.set(int(c.delivery_rate));
|
|
tcp.pacing_rate_bytes_per_second
|
|
.with_label_values(&label)
|
|
.set(int(c.pacing_rate));
|
|
}
|
|
}
|