ferrix/docs/content/docs/operations/monitoring.md

+++ title = "Monitoring Ferrix" description = "Describes the status page, route lookup, rejected route records, and the metrics and alerting rules for monitoring Ferrix." weight = 2

[extra] nav = "Monitoring" +++

Ferrix provides a status page, a JSON API, an event stream, and Prometheus metrics over HTTP. To enable the HTTP interface, configure http_endpoint:

http_endpoint { address: "127.0.0.1" port: 8080 }

Table 1 lists the HTTP endpoints. All endpoints are read-only.

Path	Description
/statusz	Status page
/api/...	JSON API. See HTTP API Reference.
/api/events	Server-sent events for session and route changes
/metricz	Prometheus metrics. See Metrics Reference.

Status Page

The status page, /statusz, opens with a status line that lists what needs attention: neighbors whose sessions are down, with the reason; neighbors that are restarting; the kernel not in step with the routing table, or refusing routes; and RPKI data that is missing or about to expire. When nothing needs attention, the line says so. A second line summarizes the normal state: sessions established, routes installed, and the RPKI state.

Below the status line, the page displays the following information:

• The configuration that was read at startup.
• Each neighbor, with its session state, time established, next hop, session properties, number of accepted routes, rejected announcements, and held routes, and its last error. Neighbors that need attention are listed first; the others are grouped by role. The problems only option hides the neighbors that need no attention.
• RPKI validation: the cache, the RTR version, and the age of the data.
• The originated routes.
• Kernel route installation: the number of installed routes, the number of routes not installed and the reason, the last error reported by the kernel, and the interfaces.
• The received routes, in order of path selection, with a lookup by prefix, address, origin AS, or neighbor name. The lookup is above the table that it filters.
• Rejected routes, by neighbor, with the number of rejections for each reason, and the most recent rejected routes. Rejections of the same prefix for the same reason by the same AS path are displayed as one row with a count.
• A live event log.

The page uses color as follows: red indicates a failure that requires action, such as a session that is down; amber indicates a degraded state, such as held or stale routes; green indicates a function that is operating normally, such as the RPKI cache being synchronized; gray indicates an inactive or expected state, such as a passive neighbor waiting for a connection. Normal values are displayed in black.

Properties that repeat on many rows are displayed as glyphs. Table 2 lists them. Point to a glyph to display its meaning.

Glyph	Meaning
v4, v6	The address families of the session.
GR	Graceful restart is in use.
ENH	IPv4 routes are exchanged with IPv6 next hops (RFC 8950).
AP↓, AP↑	ADD-PATH: several paths per prefix are received, or sent.
2B	The neighbor supports two-octet AS numbers only.
↑, ↓, ↔, RS, RSC, iBGP	The role of the neighbor: provider, customer, peer, route server, route server client, or a router of the local AS.
★, 2, 3, …	The position of the route in path selection: the best path, the path that would be selected if the best path were withdrawn, and so on.
×N	After an AS number in an AS path: the AS appears N times in a row (prepending).
OTC, AA, AGG, AS4, OID, CL, EXT	Path attributes without a column of their own: Only to Customer, ATOMIC_AGGREGATE, AGGREGATOR, AS4_PATH, ORIGINATOR_ID, CLUSTER_LIST, and extended communities.
✓, ✗, –	RPKI or ASPA validation state: valid, invalid, or not found.

An AS path or a list of communities that runs to more than a few lines is displayed shortened, with a more control. Click the control, or double-click the cell, to display the whole value.

Communities for which community_names specifies a name are displayed as a badge with that name; the community itself is displayed when you point to the badge. Communities of the local AS that are control communities display their effect in the same way. Well-known communities are displayed by name.

To display the name of the holder of an AS number, point to the number. The browser requests the name from RIPEstat; Ferrix does not send any requests itself. If the browser cannot reach RIPEstat, the page displays "name not available".

Route Lookup

The status page provides a lookup of received routes. The HTTP endpoint /api/routes/received accepts the same queries. Table 3 lists the query parameters.

Parameter	Result
prefix=193.0.0.0/21	All routes for the prefix, in order of path selection
address=193.0.0.1	All prefixes that contain the address, most specific first
within=193.0.0.0/16	The prefix and all more specific prefixes
peer=NAME	Only routes from the neighbor, identified by name or address
origin=AS64496	Only routes originated by the AS, the last AS of the AS path. Routes with an empty AS path are originated by the local AS.
best=true	Only the best path for each prefix
limit=N	At most N routes. Default: 1000. Maximum: 100000.

The following example displays the best path for the destination 193.0.0.1:

curl -s '127.0.0.1:8080/api/routes/received?address=193.0.0.1&best=true' | jq

Note

The within, peer, origin, and best parameters, when used without prefix or address, scan the entire routing table. With a full routing table, such a query takes noticeably longer.

Rejected Routes

Ferrix does not store rejected routes in the routing table. For each neighbor, it records the number of rejected routes for each reason, and the 200 most recent rejected routes with all the path attributes they were received with. The status page displays this information under "Rejected routes", with the attributes of each rejected route in the same columns as the received routes. The HTTP endpoint /api/routes/rejected returns the same information as JSON. To display the records for one neighbor, add ?peer=NAME, or select the neighbor's name on the status page.

The numbers count rejected announcements, not prefixes. When a neighbor announces a route again, for example because an attribute changed, and Ferrix rejects it again, the count increases again. Two neighbors that advertise the same routes can therefore have different counts. The number of routes currently held, because they are RPKI or ASPA invalid or their next hop is unreachable, is displayed in the neighbor table as held.

[{"peer":"transit-a","total":3,"by_reason":{"RPKI invalid":2,"bogon prefix":1},
  "recent":[{"prefix":"203.0.113.0/24","reason":"RPKI invalid","as_path":[64496,64511]}]}]

The records are retained after the session ends. Ferrix logs the first rejected route from each neighbor, and at most one rejected route per minute thereafter.

For a description of each rejection reason, see Import Policy.

Alerting Rules

The following Prometheus alerting rules cover the conditions that require operator action: route hijack and route leak attempts, prefix limits, session failures, loss of RPKI data, and kernel errors.

groups:
  - name: bgp
    rules:
      # A neighbor advertises routes that its import policy rejects. The
      # cause is a route leak or hijack at the neighbor, or an outdated
      # prefix list.
      - alert: BgpRoutesRejected
        expr: sum by (peer, reason) (increase(bgp_routes_rejected_total[10m])) > 0
        labels: { severity: warning }

      # A route that contains the local AS number was received: a routing
      # loop, or another network using the local AS number.
      - alert: BgpOwnAsInPath
        expr: increase(bgp_routes_rejected_total{reason="own AS in path"}[10m]) > 0
        labels: { severity: page }

      # A neighbor advertised a prefix of the local network, or a route
      # with a third-party next hop.
      - alert: BgpHijackAttempt
        expr: increase(bgp_routes_rejected_total{reason=~"prefix is our own|next hop is not the peer"}[10m]) > 0
        labels: { severity: page }

      # A neighbor is approaching its prefix limit.
      - alert: BgpPeerNearPrefixLimit
        expr: bgp_peer_routes / on (peer) bgp_peer_max_prefixes > 0.8
        for: 10m
        labels: { severity: warning }

      # A session was torn down because the neighbor exceeded its prefix
      # limit.
      - alert: BgpPrefixLimitExceeded
        expr: increase(bgp_prefix_limit_exceeded_total[1h]) > 0
        labels: { severity: page }

      # A session was torn down because more routes were to be advertised
      # to the neighbor than its export limit permits: a probable route
      # leak, which Ferrix did not advertise.
      - alert: BgpExportLimitExceeded
        expr: increase(bgp_export_limit_exceeded_total[1h]) > 0
        labels: { severity: page }

      # A configured neighbor has no established session.
      - alert: BgpPeerDown
        expr: bgp_peer_up == 0
        for: 5m
        labels: { severity: page }

      # No RPKI data is available. Routes are not validated.
      - alert: BgpRpkiNoData
        expr: sum(bgp_rpki_vrps) == 0
        for: 15m
        labels: { severity: page }

      # The kernel rejected route installation requests.
      - alert: BgpKernelErrors
        expr: increase(bgp_kernel_errors_total[10m]) > 0
        labels: { severity: warning }

Dashboard Queries

Table 3 lists queries for a monitoring dashboard.

Panel	Query
Established sessions	sum(bgp_peer_up), of count(bgp_peer_up)
Routes by neighbor	bgp_peer_routes
Prefixes with a best path	bgp_rib_prefixes
Rejected routes by reason	sum by (reason) (rate(bgp_routes_rejected_total[5m]))
UPDATE messages received and sent	rate(bgp_messages_received_total{type="update"}[5m]), rate(bgp_messages_sent_total{type="update"}[5m])
Kernel routes	bgp_kernel_routes, bgp_kernel_routes_unresolved
RPKI data	bgp_rpki_vrps, bgp_rpki_aspas

The metric bgp_build_info{version} reports the running version.