Platforms

What differs between Linux, macOS and other platforms: the tun, the route reconciler, the underlay socket, the sockets connected to 1 peer, moving hosts and path MTU.

The reconciler's job is the same everywhere and the facilities under it are not, so the configuration names what it wants and each backend reaches it the way its kernel allows. A backend that cannot reach a setting refuses it by name, at startup or on a reload. The sections below note where a backend ignores a setting or skips a route instead.

At a glance

The forwarding settings decide whether a node warns about forwarding without cap.border, and whether cap.egress advertises its prefixes. Transit needs neither, since the mesh forwards it between sessions itself on every platform. On macOS cap.border is refused at startup with egress: cap.border is an nftables chain, which darwin does not have, so this node cannot filter what enters the mesh from its other links, and a Mac that forwards gets the warning instead. The caller's credentials decide who may use the debug paths beyond the read paths.

The exit a build carries is chosen when it is built, and no setting names it. A Linux build carries the kernel exit, which writes what it carries into the tun and translates it with nftables on the way out of the host. Every other build carries none and refuses cap.egress at startup with egress: cap.egress asks this node to carry other nodes' traffic out of the mesh, and this build for darwin carries no exit, naming its own platform where this one names darwin. A node that accepted the block and carried nothing would advertise an exit and drop every flow that took it. A node on any platform still uses an exit.

The tun device

On Linux the tun is ranet3 unless link.tun names another device. The daemon creates ranet3 itself and never attaches to an existing device of that name, so a second instance on the same host needs a link.tun of its own. A link.tun that names an existing device attaches to it. The tun opens 1 queue per CPU the Go runtime uses, up to 256, and sets the device's gso_max_segs to its read batch.

On macOS the tun is a utun with 1 queue. link.tun takes utun for the next free unit or utunN for unit N, because the utun control creates nothing else. The daemon refuses any other name at startup. The reconciler works on the name the device got, such as utun6, rather than the one configured.

Linux

Linux has policy rules and 2^32 tables. Every node needs Linux 4.20 or later, because the network watch and the reconciler both open rtnetlink with NETLINK_GET_STRICT_CHK. With that option the kernel filters the reconciler's route dumps to its own table, so a host holding a full table in main costs it no more than its own routes. The reconciler derives its policy rules, as routing lists them, and marks each with FRA_PROTOCOL, which arrived in Linux 4.17, set to cap.table.proto, the same ownership marker the routes carry. A dump therefore reads back only this reconciler's rules, and a delete can never reach another writer's. systemd-networkd stamps RTPROT_STATIC on the rules it writes, so the 2 sets stay apart.

2 more behaviors are Linux only:

• At startup the reconciler warns when another routing protocol writes its table. The reconciler's install refuses a key another writer already holds, so it never takes another writer's route, while a writer that replaces a key can take the reconciler's.
• A deleted rule that carries cap.table.proto wakes the reconciler, and the next pass puts the rule back. systemd-networkd deletes such rules when it configures a link.

macOS

macOS has 1 forwarding table and lacks policy rules, so its reconciler installs none. It refuses any cap.table.id or cap.table.proto other than the defaults, 200 and 155. It takes cap.table.metric and ignores it, because a macOS route carries no metric. It installs nothing for an own prefix it originates outside the overlay, where Linux installs a throw route, so a lookup for one from a socket bound to no interface and no address of the utun ends as row D2 of where a lookup ends on macOS gives. With no cap.border on macOS it installs no hold of an overlay prefix either.

A route belongs to the reconciler when it leaves the utun with the shape ranet3 installs, a prefix route whose gateway is the interface itself. The reconciler adopts and withdraws such a route even when another process installed it, so give ranet3 a utun no other program writes. macOS cannot replace a route. A route another program holds under the same key stays, and the reconciler warns about it once.

Interface scope on macOS takes the place of the mark and the policy rules on Linux. Without link.underlay.bind it keeps the host's own traffic from being captured, and it keeps the ESP underlay to a literal endpoint address the trust document lists out of the tunnel carrying it, though not to the address an endpoint written as a hostname resolves to, as limits says. Where a lookup ends on macOS gives each route's scope and where each lookup ends. The reconciler installs these routes scoped to the tun, where no unbound socket can select them:

• a source-specific route
• a retracted route, which it holds as a reject route, unless the route counts as a default, whose hold it does not install at all
• any route that contains a literal endpoint address the trust document lists, but for a member of the zero-free cover
• without link.underlay.bind, any route that counts as a default

A source-specific route is installed only when its source prefix holds an address on the tun, because interface scope stands in for "from an address of ours". A destination holds 1 scoped route. A scoped route a pass still wants keeps its destination. Otherwise the pass installs the route with the most specific source, and skips and logs the rest once.

macOS has no rule to select a mark, so its underlay sockets carry none, and link.underlay.bind keeps them out of the mesh instead. It binds the underlay socket to the interface the host's default route leaves by. The socket follows the host's default to another interface 250 milliseconds after a change. It stays unbound until the host has a default route, and stays where it is while the host briefly has none.

With link.underlay.bind set, the reconciler installs a plain /0 as its zero-free cover, 8 unscoped routes per family that leave 0.0.0.0/8 and ::/8 to the host's own default, as routing describes. A Mac with bind set takes a mesh exit through the cover.

The daemon gives the socket's interface a default scoped to it only while an unscoped route out of another interface holds that family's zero address, such as a real 0.0.0.0/1 the mesh installs or another program's full-tunnel route. For a route of the mesh's own that holds the zero address it writes the default in the pass that installs the route, before the route goes in. With cap.table written it withdraws the default once no such route is left, without it the default stays until the socket moves onto another interface or the daemon stops, and with only the cover installed it writes nothing outside the tun. /var/run/ranet3/underlay.json records the scoped defaults it writes, and the next start with bind set withdraws the ones an earlier run recorded that still match the record, as sharing a host lists.

With link.underlay.bind set, a route that contains a literal endpoint address the trust document lists still stays scoped to the tun, unless it is a member of the cover. Unscoped, it would carry an unbound socket's ESP to that peer into the tunnel.

ranet3 down takes a scoped default the daemon wrote away with the capture that needed it, and ranet3 up writes it again once a capture needs it. A source-specific route and a retracted one stay scoped with or without link.underlay.bind. Interface scope is the only way macOS expresses a source. An unscoped reject route at a default would answer the whole host's traffic with an error.

Other platforms

On a platform other than Linux, macOS, iOS and Android:

• cap.table fails the start with kernel: unsupported on this platform.
• cap.egress, cap.border and link.underlay.bind are refused, cap.egress and cap.border in words that name the platform, and the underlay sockets carry no mark.
• An egress segment fails the start, since it needs cap.egress and cap.border, which are refused.
• The node has no network watch, so a moved host waits for each session's own liveness check.
• Every peer shares the 1 socket of its family, busy or not.
• The tun takes its MTU only when it opens, so a change to link.mtu or to the steering that moves the tun needs a restart. A reload applies one that leaves the tun where it was.
• The node assumes the host forwards both families.
• The node reads no caller credentials, so the debug tree answers only on its read paths.
• Datagrams leave without DF, as path MTU below describes.
• Answers leave from the kernel's choice of source, not the address the request reached.

iOS and Android build the underlay sockets, the sockets connected to 1 peer, the path MTU search, the forwarding read and the caller credentials of macOS and Linux, but none of the reconciler, the network watch, cap.egress or cap.border. An Android build, which Go makes with the linux build tag, and an iOS build, which it makes with the darwin one, refuse cap.egress and cap.border at startup in the same words, naming android or ios. cap.table fails the start there too, and a moved host waits for each session's own liveness check.

Moving hosts

A node keeps its sessions when its own addresses change, as a laptop's do when it moves from Wi-Fi to a dock or a DHCP lease renews, and when its peer's do. Every session follows its peer to the address and port its latest fresh authenticated request came from, and answers a request from the address it arrived on. Beyond those answers, a session this node dialed leaves the source address to the kernel. Its sends leave from whatever address the host holds at the time. A session a peer opened sends from the address the peer's latest request arrived on, on Linux and macOS alike, and a request that reaches another of this host's addresses moves it there. Once that address has gone and the kernel refuses a send from it, the session sends from the kernel's choice until a later request moves it to the address that one arrived on. On other platforms but iOS and Android the kernel always chooses.

The session falls back to the kernel's choice whenever the kernel refuses a send from the address with ENETUNREACH, EHOSTUNREACH, EINVAL, ENODEV or EADDRNOTAVAIL. Once a send from the kernel's choice has gone after EHOSTUNREACH, a fresh request to the address within 1 minute changes nothing, and the session keeps sending from the kernel's choice. The first one after that minute moves the session back to the address, and a send refused with EHOSTUNREACH again falls back again, at most once every minute. After any of the other refusals, the next request that reaches the address moves the session back at once. Where the session's traffic from the other address still reaches the peer, the session works, and it stays there until the peer's next request, such as a rekey. Each answer to a request that reaches the address is sent from the address itself first, so answers leave from it as soon as it routes again.

A session that cannot send at all, because the host has no route to the peer, holds while its liveness check retransmits on the usual schedule, and ends once the check has gone unanswered through its whole budget of 62 seconds. Its dialer then dials again. The first send that fails is logged as a warning, and the failures after it at debug until a send goes out again.

A node also watches the host's network, so it does not wait for a liveness deadline to find a moved path. On Linux it follows the link, address and route notifications of rtnetlink, keeping only defaults in main. On macOS it follows the routing socket. 250 milliseconds after a notification it reads, for each family, the host's own default route, its interface and gateway, and the global unicast addresses of the interfaces that are up. The mesh's own device is left out of both, so the reconciler's writes to its table and to the tun never count. When the reading differs from the last one acted on, every session is asked to prove its path at once, as ranet3 debug probe asks it, every dialer that is waiting out its reconnect delay dials again, and a network.changed event records what moved. A dialer that finds its session healthy stands down as it does after any wake. A change the host shows nowhere, such as a captive portal that lets nothing through, needs ranet3 redial --all instead.

Sockets connected to 1 peer

On Linux and macOS the transport listens on 1 UDP socket per family, bound to the wildcard address on link.port, and every peer starts on it. A busy peer then gets a socket of its own, connected to it, whose sends keep 1 cached route. On Linux that cached route holds the socket's datagrams to 1 transmit queue, as the kernel's queue selection in dev.c chooses it.

The transport counts the bytes each address pair carries, sent and received together. A pair is this host's address and the peer's address and port. On Linux and macOS the address of a session a peer opened is the one the peer's request arrived on, and a session this node dialed, or one whose sends fell back from that address, leaves it to the route. A pair that carries 1 MiB within 1 sweep of 30 seconds gets a socket on link.port, the listener's port, bound to its address or to the one the route picks then, and connected to the peer. The peer sees the same address and port as before. The pair's sends leave on that socket. The kernel hands the socket the peer's datagrams to that address ahead of the listener, and a reader of its own takes them to the same sessions. Before Linux 5.4, once 2 such sockets share 1 address, the kernel hands each datagram to that address from a peer with a socket of its own to 1 of them by a hash, so a peer's datagrams can arrive on another peer's socket. Every reader takes them to the same sessions, so none is lost. A node holds at most 256 such sockets. A pair past the cap stays on the shared socket.

On Linux the socket of a session a peer opened also leaves by the interface the peer's request arrived on, as the shared socket's replies do. An IPv4 socket binds to that interface with SO_BINDTOIFINDEX, which Linux has from 5.1 and allows a process without CAP_NET_RAW from 5.7. Where the kernel refuses the bind, such a session stays on the shared socket. An IPv6 socket names the interface with IPV6_UNICAST_IF, which like the reply's own choice prefers that interface only among the routes of the lowest metric. A link-local address binds on that interface alone. On macOS a link-local peer's socket binds on the interface the peer's zone names.

A socket connected to 1 peer closes:

• when it carried nothing for a whole sweep.
• when the last session sending on it ends, or moves to another address.
• when the host's network changes, as moving hosts describes. A busy pair connects again at its next send, from the address its pair names then.
• on macOS, when the underlay socket moves onto another interface.
• when a read on it fails other than for an ICMP error, or a send fails other than once for an ICMP error, as below, or for a refusal of its segments, as path MTU describes. Its datagrams then leave the shared socket, and the pair waits for the next sweep before it connects again.

The kernel hands a connected socket the ICMP errors its datagrams draw, such as a port unreachable from a peer that restarted or a smaller path MTU from a hop on the way. The next read or send on the socket reports such an error. A read goes on past it. A send that reports one is written once more on the socket, which stays when that succeeds or is refused for size, and closes when it fails again. A datagram past the path MTU is refused on that socket too. It then goes to the shared socket, whose refusal reports the path to the session's search, and the socket stays.

A connected socket keeps the source address it bound. On Linux an IPv4 send from an address the host lost is refused, and the socket closes at once. An IPv6 send from such an address still leaves, so only the network change closes the socket. Naming the source on each IPv6 send instead would cost the socket the cached route that keeps its transmit queue.

Each such socket follows the path MTU discovery mode of its family's shared socket. While a session of its family is under the floor, it lets the host fragment its datagrams as the shared socket does, as path MTU below describes. A socket the host does not let change its mode closes as failed.

On Linux each socket connected to 1 peer reads 4 messages of 64 KiB at once, which is 256 KiB of buffers per socket and 64 MiB at the cap, and asks the kernel for 4 MiB of socket buffer each way. Before Linux 6.13 the kernel matches each datagram to an address on the port against every socket bound to that address, up to 256, before it reaches the listener. From 6.13, on a kernel built without CONFIG_BASE_SMALL, a datagram from a peer with a socket of its own finds that socket at once, and the others still pass every socket bound to their address. On macOS each socket reads 1 datagram of 64 KiB at a time.

Sharing the port takes SO_REUSEPORT on Linux. The listener binds without it and sets it once bound. A socket connected to 1 peer sets it before it binds 1 address on the port. Linux lets a socket bind a port another socket holds only when both set SO_REUSEPORT and both belong to the same effective user ID. A second ranet3 on the port is refused, and so is any process that binds the port without SO_REUSEPORT. A process running as the daemon's own user, which is root under the service, can still bind the port when it sets SO_REUSEPORT before its bind. On the wildcard address it takes a share of the datagrams the listener would get. On 1 of the host's addresses it takes every datagram to that address from a peer without a socket of its own.

On macOS the listener binds with neither SO_REUSEADDR nor SO_REUSEPORT. A socket connected to 1 peer sets SO_REUSEADDR before it binds 1 address on the listener's port beside it. No socket sets SO_REUSEPORT.

Path MTU

On Linux and macOS each session searches its path for the largest packet it carries, as path MTU describes. The platforms differ in what the host tells that search.

On Linux every datagram leaves with IP_PMTUDISC_DO or IPV6_PMTUDISC_DO while no session of its family fragments. The host refuses a datagram past the path MTU its route holds, including a smaller one learned from an ICMP answer. The session then reads that MTU and lowers its budget to the largest packet it leaves room for, as path MTU describes, and the datagrams within it still go.

A batch of ESP for 1 peer goes out as segmented messages where it can. Linux can refuse a segmented message where it takes single datagrams. This happens on a route through an IPsec policy, on a kernel before 6.14 when a segment exceeds the path MTU, and on a socket that fragments for a session under the floor. ranet3 then sends those datagrams again, 1 per message. That peer gets single datagrams for 10 minutes, the time Linux keeps a path MTU it learned. Every other peer keeps segmenting. On a socket that fragments, a message refused with EMSGSIZE is first sent once more segmented, because a socket connected to 1 peer holds the EMSGSIZE of a Packet Too Big for its next send alone. Only a second refusal stops the segments. A socket connected to 1 peer does the same and stays open. The stop holds for that peer on the shared socket and on its own alike.

On macOS every datagram leaves with IP_DONTFRAG or IPV6_DONTFRAG, but for those the host fragments for a session under the floor, as below. A send refused for its size reads the MTU of the interface it sends through, and the session's search takes that as its route's MTU. A socket that link.underlay.bind holds to an interface sends through it. Otherwise a session the peer opened sends through the interface holding the address it sends from, the one the peer's latest request reached until a send from it falls back, as moving hosts describes. Any other session, one whose send fell back included, sends through the interface its route leaves by. The search reads the same MTU when it starts, and the probes alone find a narrower hop beyond it.

A session on a path under the floor fragments on either platform as the design's limits describe. While one does, Linux lets the host fragment every datagram of that family, and macOS does the same for IPv4. On macOS an IPv6 peer under the floor has only its own datagrams fragmented. On any other platform but iOS and Android datagrams leave without DF, nothing is searched and every session carries link.mtu, which leaves a narrower path to the host's and the routers' fragmenting.

The control surface

The control surface is an HTTP handler that serves JSON over any listener. The control package binds it to a Unix socket. On Linux and macOS, and on iOS and Android, it also reads each caller's credentials from that socket. Elsewhere it reads none, and the debug tree answers a caller it cannot name only on its read paths.