Graceful restart (RFC 4724, RFC 8538)

A peer that restarts keeps its routes with us, and we keep ours with our
peers while we restart, so that a restart withdraws nothing across the
network and the machine goes on forwarding. On by default for every
peer; `graceful_restart` sets the restart time we give, caps how long a
peer's routes are kept, or turns it off.

When a peer restarts: its routes are marked stale in the routing table,
and chosen as before, for at most its restart time. When it is back, what
it sends again is fresh, and the stale routes of a family go at its
End-of-RIB marker, or after the cap; at once for a family whose
forwarding it did not keep. Only a connection that failed counts as a
restart, and, with RFC 8538 agreed, an expired hold timer or a
NOTIFICATION other than a Hard Reset. The stale mark is a bit of the
route's word that selection never looks at.

When we restart: SIGUSR1 leaves the kernel's routes where they are,
closes the sessions without a word where graceful restart is agreed, and
exits with status 75, on which the systemd unit starts us again. The next
start finds the routes in the kernel, says so in its OPENs (R, and F where
forwarding was kept), and has them go once every peer has sent its
End-of-RIB. Our own End-of-RIB markers for learned routes wait for that
too. SIGTERM still stops for good, now with a Hard Reset where RFC 8538 is
agreed, as are errors of ours.

The status page and API show what was agreed with each peer, the routes
kept while a peer restarts, and which routes are stale. Tested against
FRR and BIRD in a new phase of the interoperability lab, and with routes
kept in a real kernel over a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

crates/web/static/statusz.html

@@ -193,14 +193,20 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
     if (!state.peers.length) { setHTML(b, '<tr><td colspan="11" class="dim">No peers configured and none connected.</td></tr>'); return; }
     setHTML(b, state.peers.map(p => {
       const up = p.state === 'Established';
-      const detail = p.state === 'Backoff' ? `${esc(p.last_error || '')}; retry in ${until(p.retry_at_unix)}` : '';
+      const restarting = !up && p.stale_until_unix
+        ? `<span class="warn" title="Graceful restart: the peer's routes are kept, and used, until it is back and has sent them again">restarting: ${p.stale_routes} routes kept for ${until(p.stale_until_unix)}</span>` : '';
+      const detail = [restarting, p.state === 'Backoff' ? `${esc(p.last_error || '')}; retry in ${until(p.retry_at_unix)}` : '']
+        .filter(Boolean).join('; ');
+      const gr = p.graceful_restart;
+      const graceful = gr ? ` &middot; <span title="Graceful restart (RFC 4724): the peer says a restart takes ${gr.restart_time_secs}s${gr.notification ? ', and NOTIFICATIONs other than Hard Reset count as restarts (RFC 8538)' : ''}">graceful restart</span>` : '';
+      const stale = p.stale_routes ? ` <span class="warn" title="Kept from the peer's last session, and not sent again yet">(${p.stale_routes} stale)</span>` : '';
       return `<tr>
         <td>${p.name ? esc(p.name) : '<span class="dim">(inbound)</span>'}</td><td class="mono">${esc(p.peer_addr)}</td><td>${asn(p.peer_asn)}${p.role ? ` <span class="dim" title="${p.role_confirmed ? 'The peer says so too (RFC 9234)' : 'By our configuration; the peer has not said what it is'}">${esc(p.role)}${p.role_confirmed ? ' &check;' : ''}</span>` : ''}</td>
         <td><span class="${STATE_CLASS[p.state] || ''}">${esc(p.state)}</span></td><td>${up ? (p.direction === 'outbound' ? 'us' : 'peer') : DASH}</td>
         <td class="mono">${up ? esc(p.local_addr) : DASH}</td><td>${up ? fmtTime(p.established_at_unix) : DASH}</td>
         <td>${up ? age(p.established_at_unix) : DASH}</td><td>${up ? (p.four_octet_asn ? 'yes' : 'no') : DASH}</td>
-        <td>${up ? esc((p.address_families || []).join(', ')) + addPath(p) + (p.extended_next_hop ? ' &middot; IPv4 with IPv6 next hops' : '') : DASH}</td>
-        <td>${up ? p.routes : DASH}</td>
+        <td>${up ? esc((p.address_families || []).join(', ')) + addPath(p) + (p.extended_next_hop ? ' &middot; IPv4 with IPv6 next hops' : '') + graceful : DASH}</td>
+        <td>${up ? p.routes + stale : DASH}</td>
         <td>${up ? (p.rejected_routes ? `<a class="bad" href="#refused">${p.rejected_routes}</a>` : 0) + (p.held_routes ? ` &middot; <span title="Routes rejected as RPKI or ASPA invalid, or for a next hop that cannot be reached, and kept aside, to be taken if that changes">${p.held_routes} held now</span>` : '') : DASH}</td><td>${detail}</td></tr>`;
     }).join(''));
     renderPeerChoices();
@@ -302,7 +308,7 @@ from <select id="lookup-peer"><option value="">any peer</option></select>
       <td class="mono">${s.next_hop ? esc(s.next_hop) : DASH}</td><td>${s.origin ? esc(s.origin) : DASH}</td><td>${s.med ?? DASH}</td><td>${s.local_pref ?? DASH}</td>
       <td class="mono">${commList(s.communities, true)}</td><td class="mono">${commList(s.large_communities)}</td>
       <td>${r.rpki ? `<span class="${RPKI_CLASS[r.rpki] || ''}">${esc(r.rpki)}</span>` : DASH}</td>
-      <td>${r.aspa ? `<span class="${RPKI_CLASS[r.aspa] || ''}">${esc(r.aspa)}</span>` : DASH}</td><td>${r.best ? '<span class="ok">best</span>' : ''}</td></tr>`; }).join(''));
+      <td>${r.aspa ? `<span class="${RPKI_CLASS[r.aspa] || ''}">${esc(r.aspa)}</span>` : DASH}</td><td>${r.best ? '<span class="ok">best</span>' : ''}${r.stale ? ` <span class="warn" title="Kept from a session of the peer's that ended (graceful restart), and not sent again since">stale</span>` : ''}</td></tr>`; }).join(''));
   }
 
   const RPKI_CLASS = { valid: 'ok', invalid: 'bad', 'not found': 'dim', unknown: 'dim' };