Metrics

The Prometheus series a node exports.

--metrics 127.0.0.1:9669 serves /metrics in the Prometheus text format on its own listener, separate from --pprof so a fleet node can be scraped without exposing a profiler. The flag is empty by default, which serves no listener. ranet3 metrics prints the same scrape over the control socket, with or without the listener.

It reports neighbor liveness and link cost, routes received per neighbor, routes selected and originated, established sessions per path, packets each peer chose not to send, inbound ESP packet and drop counters, and the sockets connected to 1 peer with those that failed and handed their peer back to the shared socket.

It also reports 3 subsystems that run inside the daemon. The route reconciler is reported as routes installed and skipped, when its last pass finished and whether that pass failed. Segment routing is reported as what this node did for peers (forwarded, delivered, dropped, answered, and for an egress segment egressed, refused and not ready) and what it did with its own traffic (steered, and dropped with a reason). cap.egress is reported as rules installed, connections translated, other translation found at the same hook, when its last pass finished and whether that pass failed, whether each family is ready, and the 2 prefix counts whose difference says this node is withholding an advertisement it cannot stand behind.

The border filter is reported as whether its chain reads back and the packets each of its drops counted, by reason: overlay_source in both families, and routing_header and sid_destination in ipv6. The counts are read from the chain at each pass, so they lag by up to 1 pass, and a rewrite of the chain starts them from 0, which a scrape reads as a counter reset.

A node without cap.table writes none of the reconciler's series, a node without cap.egress writes none of the egress series, and a node without cap.border writes none of the border's. Zeros there would read as a subsystem installing nothing. Segment routing writes its series on every node, zeros included.

Everything is read from live state at scrape time, so a scrape reflects the instant it happened rather than a sampled snapshot. A counter cannot carry the route table. ranet3 routes reads it, with each prefix's next hop, from the control socket.

Series

The peer label is the path of the peer's session, such as example/gateway:0@0. A series labeled by peer lasts as long as that session. ranet3_session_up names the same path in path.

ranet3_babel_neighbor_cost holds the amount cap.babel.cost.extra charges the neighbor, and no series reports that amount alone. It is configuration rather than a measurement, and the neighbor view shows it apart, in the Extra column of ranet3 neighbors and in the extra field of /v0/neighbors.

ranet3_egress_ready takes the place of ranet3_egress_device_ready, which is gone together with ranet3_segments_drained_total: an egress segment hands its inner packets to the exit, which writes them into the mesh tun, so no device of its own is left to report on. A query naming the old gauge reads no series, and one reading the new gauge sees it on every node with cap.egress, since readiness holds back both modes.

Transit counters

ranet3_transit_packets_total counts what the forwarding decision did with each packet a session delivered that no segment of this node took, and each read off the tun it dropped for a multicast destination or the limited broadcast, or at the mesh's edge, where it checks what the tun hands this node before the packet enters the mesh. decision is forwarded, delivered, dropped, answered or unanswered, and reason names why a packet was dropped and is empty for every other decision. The dropped series sum to the packets dropped. answered also counts each answer to a read off the tun that no route reaches or a hold holds, a read counted under no other decision.

unanswered counts a drop before the rules that keep an error from answering a packet, such as one about an ICMP error or to a group, and before the rate limits, so it counts a drop that would have drawn no answer with an address too. A drop from a source only this node's own default reaches counts only under its reason, before any address is looked for, and so does a drop whose answer is built and finds no route back. An ICMP error a rate limit held back counts in neither answered nor unanswered, and ranet3_icmp_withheld_total counts it, as path MTU says. Every series is written from startup, at 0 on a node that forwarded nothing. ranet3 status shows the same counts on its Transit line.

Tun reads

ranet3_tun_reads_truncated_total counts reads off the tun that lost the tail of a GSO frame, because the frame split into more packets than 1 read holds. Only Linux hands the tun GSO frames, so the counter stays at 0 elsewhere. There ranet3 sets the device's gso_max_segs to that read batch, 128 packets, whenever it creates the device or attaches to one, which has the kernel split a larger frame before it reaches the tun and keeps the counter at 0. A kernel that refuses the setting leaves a warning in the log at startup, and a sender with a small MSS then cuts reads short. Each such read loses the rest of its frame, which TCP sends again, and the reader goes on. A warning carries their count since the previous warning, at most once every 30 seconds.

ranet3_tun_frames_refused_total counts frames the tun handed over that could not be read as packets, and each such frame is lost alone while the reader goes on. On Linux that is a frame whose virtio-net header does not describe its packet, which a sender that writes the header itself, as through an AF_PACKET socket, can produce, or one the kernel could not copy out. A warning and a netstack.tun.refused event carry their count since the previous warning, at most once every 30 seconds. Any other error reading the tun means the device is gone, and the node then stops as the service describes.

Packets a peer did not send

ranet3_peer_send_dropped_total counts, per peer, every packet this node chose not to send. A peer holds packets from the read that brought them to the transmit that carries them within a budget of its own, which the design's limits give. A read that does not fit is dropped whole rather than waited for, so a peer whose transport falls behind never holds up another. The counter also takes the packets of a peer that was closing, of reservations its outbound SA refused and of batches that failed to seal, and the packets the delay control drops. ranet3_peer_send_delay_dropped_total is that last part on its own. Each peer's sender measures how long every data batch has waited since it was reserved, and once that delay has stayed above 5 milliseconds for 100 milliseconds it drops 1 packet at the head of the queue on the schedule CoDel (RFC 8289) sets, until a batch comes through below 5 milliseconds again or the queue empties. Babel's own packets wait in the same queue, but the delay control neither measures nor drops them. The sender records each start and end of that state as an event, netstack.codel.dropping and netstack.codel.drained, which the debug tree shows.

Delay drops say the queue toward that peer stands. This node cannot seal and send for that peer as fast as traffic for it arrives, because its CPU, its socket or the path falls behind. TCP senders are told to slow down, as a router's queue would tell them. The delay runs from the reservation, which makes it count the seal and the wait for the encryption workers every peer shares. On a node whose CPU is saturated, delay drops tend to show on every busy peer together, and they mean this node cannot seal and send as fast as traffic is offered, not that a path is slow. The rest of the counter, the total less the delay drops, counts the reads the budget refused, together with a closing peer's packets, reservations the outbound SA refused and batches that failed to seal. The last 2 leave a netstack send batch failed warning in the log and a closing peer ends its session, so the rest climbing on a live session without that warning is the budget refusing reads: a burst, a CPU that cannot keep the peer's sender running, or a transport that stopped taking packets for longer than the budget lasts. ranet3_peer_send_failed_total stays apart from all of these: those packets were sealed and the transport lost them, which is the link failing rather than this node choosing.

Path MTU

ranet3_packet_too_big_sent_total counts the answers this node sent to packets larger than their session carries, Packet Too Big over IPv6 and fragmentation needed over IPv4 with DF, read off the tun or carried to a waypoint, and ranet3_mss_clamped_total the TCP SYNs whose MSS it lowered to what their session carries. Both count for the node as a whole. ranet3_pmtu_probes_total counts the probes of the sessions' path MTU that left, by result: acked, or lost when no Ack came within the time a probe waits, and debugging says which probes count in neither. Lost probes climbing on a node whose sessions have settled say a path loses its session's largest packets now and then, and ranet3 debug pmtu names the session and shows its search. Path MTU describes the search itself.

ranet3_icmp_withheld_total counts the ICMP errors the rate limits held back, as the kernel's IcmpOutRateLimitHost and IcmpOutRateLimitGlobal do. limit="source" rising alone is a sender drawing past its own bucket, such as a flood of expired probes, and costs no other sender anything. limit="node" rising is answers past the node's cap, from many sources at once or from one sender's packets past their session, since Packet Too Big and fragmentation needed draw from the cap alone. Either way the cap holds back every sender's answers, Packet Too Big included.

Babel fragments

ranet3_babel_reassembled_total counts, per neighbor, the Babel packets its session sent in fragments that this node put back together and read, and ranet3_babel_reassembled_largest_bytes is the largest of them, a whole IPv6 packet with its headers. ranet3_babel_reassembly_dropped_total counts the datagrams dropped instead, by reason:

A neighbor flooding fragments of datagrams that never complete has the node hold under 280 KiB and allocate under 2 MiB a second. The node evicts and refuses only that neighbor's datagrams. A neighbor that sent no fragment has none of the 3 series, so the series name exactly the neighbors whose stacks cut their packets. The protocol deviations say which fragments the speaker takes and the bounds it holds each neighbor to. The first packet a session sent in fragments is recorded as a babel.neighbor.fragmented event as well, which the debug tree shows.