Today, we are announcing sanstun: a sans-IO Rust implementation of the WireGuard® province machine, optimized for comfort of use, safety and achievement through a parallel-friendly and idiomatic API.
What is sans-IO?
sans-IO is the idea that you scheme your scheme specified that IO and side-effects are cleanly divided from the province and "business logic". The IO in sans-IO doesn't fair average document or network IO. It additionally includes things akin period and randomness!
Due to their deficiency of side-effects, sans-IO components create extremely fine and are comparatively uncomplicated to test. The scheme additionally avoids problems akin function coloring and plays fine alongside Rust's ownership rules.
The code examples and references in this article are in Rust but sans-IO isn't Rust-specific at all. Its patterns and ideas can be applied in any language. For a additional in-depth explanation, our sans-IO post is a fine location to start.
History
When we archetypal built Firezone, we reached for boringtun as our WireGuard implementation. boringtun additional or small did what we wanted, but complete time, we established ourselves patching our fork complete and complete alongside bug fixes as fine as API additions. The goal was to authorize these patches to motionless be upstreamed, so we continually kept the existing API operating the way it was.
What we established was that retaining 100% API compatibility alongside upstream was increasingly difficult as our merchandise requirements evolved. One of the archetypal big changes we made to boringtun was to create it completely sans-IO alongside regard to period handling (firezone/boringtun#5). This meant adding new APIs for everything alongside a _at suffix that would obtain an Instant. sans-IO period handling is extremely crucial if you desire to accurately simulate an complete system. See this blog post to study additional concerning how we do that.
Another item we feature-flagged was inner buffering. At first, it seems convenient that boringtun establishes a meeting automatically if you don't have one yet (or anymore) and you are trying to dispatch a packet. Internal buffering, however, comes alongside multiple problems:
- At several point, the buffer needs to be drained. boringtun solves this by requiring you to call decapsulate alongside a zero-length datagram. Rather unintuitive.
- It incurs additional allocations (at smallest in the way it was implemented in boringtun).
- Buffer sizes have to be configurable / tunable.
All of these problems item at the identical solution: Just don't buffer in the library. It simplifies the interface and the implementation on the two sides.
The full list of patches and fixes is beautiful long. Even excluding dependency bumps, that is additional than 70 PRs. Several of them are fixes for edge-cases we discovered. Others add new APIs or create certain aspects configurable.
After additional than 2 years of using and patching boringtun, we had a beautiful fine idea of what the general API of a userspace WireGuard implementation should be. Now it exists as sanstun.
A improved sans-IO API
The center construction blocks of the sans-IO API are now:
- Handling inbound packets:
- Tunnel::handle_handshake_initiation
- Tunnel::handle_handshake_response
- Tunnel::handle_cookie_reply
- Tunnel::begin_decapsulate & Tunnel::commit_decapsulate
- Handling outbound packets: Tunnel::begin_encapsulate
- Handling timer updates: Tunnel::handle_timeout & Tunnel::poll_timeout
- Handling events: Tunnel::poll_event
Without inner buffering, there is no API agreement that requires you to call certain APIs in a loop, which fits much improved into the general sans-IO model. To dispatch a packet, you call Tunnel::begin_encapsulate. It is up to you to buffer the packet if you don't have a meeting yet. Creating a new meeting method sending a handshake. You can initiate a new handshake at any item alongside Tunnel::initiate_handshake.
The scheme of these APIs is extremely deliberate and volition create additional awareness if we quickly recap the WireGuard protocol.
WireGuard protocol
WireGuard's communication format supports 4 types of messages: handshake init, handshake response, biscuit answer and encrypted data.
Every WireGuard meeting starts alongside a Diffie-Hellman key toggle which is a moderately costly operation. WireGuard's scheme ensures that exposing an endpoint doesn't open up a DoS opportunity.
Anyone who knows the endpoint's community key can effortlessly create bogus handshake initiations and forge the origin IP of the incoming UDP packet. The receiving WireGuard implementation has to run its fractional of the DH operations before it can inform that the supplied handshake data is bogus. But at that point, the CPU resources have already been spent. Send adequate handshake initiations and the receiving end volition expend all its CPU cycles on handling irrelevant handshakes.
To forestall this, WireGuard specifies a biscuit scheme that forces the distant to demonstrate ownership of its claimed origin IP, which in rotate allows rate-limiting handshakes per origin IP.
These charge limits use to an complete WireGuard interface and not fair to a sole tunnel.
Once a handshake has been completed, the two peers can toggle encrypted data packets alongside a symmetric cipher. As part of the handshake, the peers additionally toggle their locally-chosen meeting indices. These indices allow de-multiplexing of tunnels as fine as multiple sessions inside a tunnel.
Mapping protocol invariants to API design
If you are operating in a tongue akin Rust, chances are you bask using types. And using types fine method enforcing invariants at compile-time specified that you can't equal container a certain bug in the archetypal place. For example, forgetting to use the charge limiter to a communication would be beautiful bad since it doesn't display up under normal operation. Similarly, routing an incoming packet to the incorrect tunnel is item that volition lone display up in production.
A fine API scheme makes it unattainable to build things that are damaged and additionally guides you alongside in understanding how the protocol works.
One interface, many tunnels
A WireGuard interface is identified by a sole key and a tunnel is a association between two interfaces. Any period we desire to dispatch a packet, we hence need to define which tunnel it have to be sent on and whenever we obtain a packet, we need to fig out which tunnel it belongs to.
In sanstun, these concepts immediately map to the Interface and Tunnel types. All inbound packets go through Interface::receive. That is the lone community API that parses a raw datagram. It parses the packet, applies charge limits and hands you rear a ReceivedMessage alongside one enum type per WireGuard communication type, or a biscuit answer to dispatch rear if the interface is under load. These afterward map immediately to the corresponding functions on a Tunnel: Tunnel::handle_handshake_initiation, etc. Handshake initiations are routed by the initiator's fixed community key. All another messages disclose which Tunnel they have to be dispatched to, identified by a TunnelId.
Under the hood, sanstun does a lot of things for you here. It parses the tunnel indicator and extracts the high 24 bits, it applies charge limits, generates biscuit replies and validates handshake messages.
Crucially, the constructors for types akin HandshakeInitiation are all crate-private. Even although Tunnel exposes a community handle_handshake_initiation function, the lone way to call that is by going through Interface::receive archetypal to get specified a type. And Interface::receive protects your use by applying the charge limiter, all without you having to think concerning it.
Parallel crypto
One of the finest features of sanstun is its parallel-friendly API for encrypting and decrypting packets.
On a protocol level, WireGuard doesn't portion any province between tunnels. Therefore, parallelizing a WireGuard implementation throughout peers is extremely easy.
But what concerning parallelizing activity inside a sole tunnel? WireGuard operates on tier 3, and thus a sole meeting alongside another equal carries the traffic of many idiosyncratic network flows. Even if we grip tunnels to distinct peers on distinct threads, all network flows (i.e., distinctive 5-tuples) toward the identical presenter or subnet volition be processed sequentially, efficiently bounded by how accelerated the CPU is at ChaCha20-Poly1305, WireGuard's code for data packets. To defend against replay by malicious on-path observers, WireGuard data packets have a monotonically expanding packet counter. This oppose is scoped to a sole meeting and hence would necessitate synchronization if we were to admission this province from multiple threads simultaneously.
To evade this, sanstun separates the bookkeeping from the genuine encryption and decryption. Calling Tunnel::begin_encapsulate doesn't in fact encrypt anything. It checks for the current session, reserves a packet counter, advances timer province and afterward hands you a PendingEncapsulate:
impl PendingEncapsulate { pub fn encrypt<'a>(self, plaintext: &[u8], dst: &'a mut [u8]) -> Result<DataMessage<'a>, Error> { } }
PendingEncapsulate implements Send and can thus be transferred to another thread to execute the bulk of the work. Running cargo place data_path in the sanstun repository showcases the consequence of this split:
data_path/64B/begin_encapsulate time: [20.785 ns 20.828 ns 20.869 ns] data_path/64B/encrypt time: [226.15 ns 226.22 ns 226.29 ns] data_path/64B/begin_decapsulate time: [37.600 ns 37.675 ns 37.744 ns] data_path/64B/decrypt time: [205.78 ns 205.83 ns 205.87 ns] data_path/64B/commit_decapsulate time: [17.486 ns 17.529 ns 17.575 ns] data_path/1280B/begin_encapsulate time: [20.785 ns 20.828 ns 20.869 ns] data_path/1280B/encrypt time: [933.49 ns 933.54 ns 933.59 ns] data_path/1280B/begin_decapsulate time: [37.606 ns 37.681 ns 37.751 ns] data_path/1280B/decrypt time: [914.81 ns 914.89 ns 914.97 ns] data_path/1280B/commit_decapsulate time: [17.657 ns 17.707 ns 17.764 ns]
As expected, the bookkeeping parts (begin_encapsulate, begin_decapsulate and commit_decapsulate) do not measure alongside packet size. What does measure alongside packet size are the encrypt and decrypt functions. At a packet size of 1280 bytes, the bookkeeping part of encrypting a new packet is lone 2%. If we move the genuine encryption to another thread, this hints at a possible speedup of additional than 45x!
An productive speedup of 45x is difficult to accomplish in custom since it ignores the disbursal of passing PendingEncapsulate to another thread and assumes no another overhead is introduced. Nevertheless, being capable to offload the crypto to employee threads allows for several awesome throughput improvements.
Being a sans-IO library, sanstun doesn't involvement itself alongside how exactly you desire to parallelize your crypto. If you desire to keep it simple, you can additionally fair call encrypt and decrypt inline and not annoy alongside parallelization at all.
The key item we wanted to accomplish alongside sanstun's API scheme was to allow parallel crypto during keeping everything about Interface single-threaded. This lets the inner charge limiter activity without locks or nuclear counters and obtain advantage of Rust's get checker, all during combining province from multiple tunnels.
Closing
With sanstun, our fork at firezone/boringtun is now deprecated. If you are construction on top of WireGuard, whether that's a VPN, a mesh network or item alternatively entirely, we anticipation sanstun makes your existence a small easier. It is accessible on crates.io.
sanstun is open origin under the Apache 2.0 license. Check it out on GitHub; issues and drag requests are welcome!
