Stalwart v1.0: The Fastest Mail Server on the Planet

Five years ago, on 3 October 2021, the first commit landed in a small repository called mail-parser. It was an e-mail parsing archive written in Rust, and nothing more. Over the following five years it grew into a complete communication and collaboration server alongside JMAP, IMAP, POP3, SMTP, CalDAV, CardDAV and WebDAV, a spam filter, clustering, and assistance for fractional a dozen retention backends. Today, following five years of uninterrupted development, we are arrogant to province Stalwart v1.0, the archetypal stable release.

Version 1.0 was never meant to be another characteristic release. It had two goals. The archetypal was performance: making all part of the server as accelerated as it can be, and proving it alongside numbers. The second was stability of the data itself: finalising the repository schema so that, from now on, upgrading Stalwart method installing a new type and nothing more, without the disruptive migrations several before releases required. Along the way a few new features established their way in too, including full-text hunt inner PDF and agency documents.

Let’s stroll through it.

Most of the activity in this publish happened under a profiler. For months we measured anywhere Stalwart spends its CPU cycles, its recollection and its repository circular trips, rewrote any stood out, and measured again. The publish contains near to 200 commits touching additional than 1,300 files, and the huge bulk of them be to create item faster or lighter.

A communication server spends a amazing portion of its period looking for particular bytes: the end of a header line, a MIME boundary, an equals sign in a quoted-printable body, the area that ends an IMAP tag. Done one byte at a time, that is a lot of iteration iterations for a 10 MB message. Two techniques let a CPU do this activity many bytes at a time:

• SIMD (Single Instruction, Multiple Data) uses the broad vector registers of contemporary processors (SSE2, SSSE3 and AVX2 on x86-64, NEON on ARM) to difference or change 16 or 32 bytes alongside a sole instruction.
• SWAR (SIMD Within A Register) applies the identical idea alongside average 64-bit integers: a few carefully chosen arithmetic and bitwise operations test all eight bytes of a term at once. It needs no particular instructions, so it plant on all CPU.

Stalwart now uses both, everyplace it matters. At the base of the stack sit two libraries that attain 1.0 together alongside the server:

• mail-parser 1.0, the extremely archive anywhere Stalwart began, has been rewritten from scratch. SIMD kernels find MIME boundaries and header fields, SWAR handles header names, the parsed communication lives in a fistful of flat vectors, and bodies are lone decoded whenever person really says them. The CPU-specific kernel is picked at runtime, and all one is fuzzed against a plain scalar reference. Against another parsers it is 3 times faster than the famous Rust mailparse crate, 9 times faster than Dovecot’s own parser, and 15 to 31 times faster than libEtPan!, VMime and GMime on a complete parse.
• encodify is a new encoding archive alongside SIMD kernels for Base64, Base32 and quoted-printable, the encodings that transport nearly all attachment and non-ASCII header in e-mail. Decoding a MIME-wrapped Base64 attachment is 22 times faster than alongside the norm Rust base64 crate on an AMD EPYC server, and quoted-printable decoding is 9 times faster than alongside the quoted_printable crate. It now handles all Base64, quoted-printable and IMAP UTF-7 conversion in the server.

Above them, the protocol tier got the identical treatment. The IMAP command parser is now table-driven and scans entire runs of bytes alternatively of strolling a province device per character. The SMTP command parser, the JSON parser rearward JMAP, the spam display tokenizers, the HTML-to-text converter, the variable-length entire figure codec and the log serializers all use SWAR scanning. Even the expression language, which administrators use for routing and filtering decisions, now compiles to a compact bytecode alongside specialised instructions for average comparisons and evaluates them in a recycled recollection arena.

Fast parsing is wasted if all petition afterward waits on the database, so we spent fair as lengthy on how Stalwart talks to its storage. Identifiers are now assigned inner the identical compose transaction that uses them, removing entire circular trips from all communication delivery. Read-modify-write updates, specified as adding a keyword or appending to a hunt index, run as merge operations inner the backend. Stored objects are compressed alongside zstd using dictionaries trained specifically on e-mail, calendar and communication data.

Each backend was additionally tuned individually:

• RocksDB: all benevolent of data now lives in a pillar family configured for how it is accessed, alongside bloom filters for item lookups, ample blocks for scans and compaction tuned for data that churns. Hot counters, specified as queue quotas, are dispersed throughout up to 256 slots to evade compose conflicts.
• PostgreSQL and MySQL: pre-generated and cached prepared statements, chunked repeat alongside automatic retry, and small circular trips per write.
• FoundationDB: says that a transaction volition need are prefetched in parallel windows, and ample values are assembled in chunks.
• SQLite: leaf size, cache, recollection mapping and WAL settings tuned for a communication workload.

On top of the retention tier sits a redesigned cache subsystem. Each account’s communication indicator is held in recollection in a compact packed form, and IMAP sorting, threading, emblem fetches, searches on communication properties and resynchronisation are now answered from that cache alongside few or no repository reads. Very ample accounts get their own cache tier alongside a recollection budget, and cache snapshots can be exchanged out to local disk, Redis or the blob shop alternatively of being rebuilt.

The built-in full-text hunt motor was rewritten from the dirt up. The former scheme stored one bitmap per term and had no notion of term positions. The new motor stores postings in blocks of 65,536 documents and keeps term positions for all message. That makes genuine expression searches possible, and lets a hunt inner one mailbox peruse lone the blocks that mailbox touches. It indexes term stems and prefixes, compresses its data alongside zstd, and indexes new communication through a backdrop queue alongside concurrency control, retries and rebuilds. The PostgreSQL and MySQL hunt backends were reworked as well. On a mailbox of 250,000 messages, a full-text hunt of all communication build now returns in under 4 milliseconds.

Optimising code without measuring it under realistic burden is guesswork, so we built a dedicated benchmarking and stress-testing lab to go alongside this release. It runs on Microsoft Azure, throughout additional than 30 regions, alongside up to 18 complete test labs operating in parallel and multiple hundred vCPUs occupied at once.

The test catalogue covers 35 kinds of test, dispersed complete additional than 550 idiosyncratic test cells, and they cover:

• Every important protocol: IMAP, JMAP (including shove complete EventSource and WebSocket), SMTP inbound, submission and relay, CalDAV, CardDAV and WebDAV.
• PostgreSQL, FoundationDB and RocksDB as data stores, Redis as the in-memory store, S3 and the filesystem as blob stores, and a three-node collection under blended load.
• Large datasets: 1.6 myriad messages and 750,000 calendar and communication items throughout 500 accounts, idiosyncratic mailboxes of 1, 2 and 5 myriad messages, 100,000 accounts on a sole server, communication queues of 10 myriad messages, and DMARC and TLS study floods of a myriad reports.
• Storms and abuse: 20,000 new connections per second, bulk logins, thousands of IDLE and shove subscribers per account, antagonistic and malformed messages, slow-loris clients and oversized literals.
• Six-hour soak tests that appearance for recollection growth, cache drift and dilatory leaks, including one that pushed complete 10 myriad messages through the MTA.

Runs were profiled throughout. CPU period was sampled alongside perf and turned into fire graphs, off-CPU inspection showed anywhere threads waited on locks and I/O, the heap was profiled alongside jemalloc and heaptrack, and a particular instrumented build counted all retention circular trip, per command, so we could see exactly which requests hit the repository additional than they should. Each fix was afterward measured against the former build on matched machines before it was accepted.

The archetypal complete circular of this sequence turned up 64 problems, which were fixed and retested. The second circular re-ran everything and fixed 33 more. Some examples of what the lab caught:

• Reopening a mailbox alongside 5 myriad messages went from 61 ms to 0.21 ms.
• A JMAP query on an document alongside 5 myriad messages went from 3.3 seconds to 12 ms.
• Peak inbound communication throughput on RocksDB nearly doubled, from 9,690 to 18,394 messages per second.
• A recollection matter anywhere the procedure held up to 9.6 GB during the genuine heap was under 0.4 GB was established and fixed. In six-hour soak tests recollection now stays flat.

This lab is not a one-off attempt for the release. It is now part of how Stalwart is tested internally: all forthcoming publish goes through the identical benchmark and stress-test rounds before it ships, so achievement gains remain gained.

Internal numbers lone inform you whether Stalwart got faster than it was. We additionally wanted to cognize how it compares to what most of the earth runs. So we put Stalwart v1.0 caput to caput alongside the most extensively deployed open-source communication stack: Postfix 3.11 for SMTP and Dovecot 2.4 for IMAP.

Both ran on identical Azure servers (8 AMD EPYC Zen 5 vCPUs, 32 GiB of RAM, local NVMe storage), loaded alongside the identical 1,000 mailboxes affirmative three extremely ample ones of up to 250,000 messages, and received exactly the identical work: receiving mail, delivering it and serving it to IMAP clients. Encryption, password hashing and full-text hunt were configured the identical way on both, and the two systems were measured alternately on the identical hardware.

Stalwart was faster in 35 of the 38 header comparisons.

Some of the results:

• IMAP: Stalwart is faster than Dovecot in 14 of 15 mundane client operations, by 1.4 to 19 times. A login alongside a complete TLS handshake takes 5.1 ms alternatively of 28 ms, fetching a 1 MB communication is 6 times faster, threading the INBOX 5 times faster, and moving or deleting 100 messages 18 to 19 times faster.
• SMTP: Stalwart accepts a communication 3.6 to 12 times faster than Postfix, and sustains 5,445 incoming messages per second against Postfix’s 723.
• Delivery: a communication is apparent to an IMAP client 16 times sooner, and a client waiting in IDLE is notified in 3.3 ms alternatively of fractional a second.
• Load: Stalwart serves 5.4 times additional concurrent IMAP sessions per second and relays 3.8 times additional communication before it starts to dilatory down.

The gap grows alongside the size of the mailbox. On a 250,000-message mailbox, a full-text hunt of all communication build takes 3.75 ms in Stalwart and 136 ms in Dovecot, and moving or deleting 25,000 messages finishes in concerning a tenth of a second alternatively of 24 to 34 seconds: additional than 200 times faster.

The three tests Dovecot won are all the identical operation: marking all communication in a mailbox as peruse at once.

Only Postfix and Dovecot were tested, but they are not fair any two products. They are the motor under nearly all famous self-hosted communication server: all open-source dockerized communication server alongside additional than 10,000 stars on GitHub, including docker-mailserver and mailcow, runs Postfix and Dovecot under the hood, as does Mail-in-a-Box. Beating that blend on nearly all benchmark method beating the stack rearward most of the self-hosted communication on the internet. We are now comfortable saying it plainly: Stalwart is the fastest communication server on the planet.

Don’t obtain our term for it. Every script, configuration, test circumstance and raw outcome is in the benchmarks repository, together alongside a methodology and a frank catalog of caveats, specified as the differences in disk-sync behavior between the two systems. Anyone alongside an Azure document can provision the identical servers and run the entire suite. Benchmarks of Stalwart’s distinct retention backends and clustered setups are being prepared for the identical repository.

Performance was the headline, but v1.0 additionally brings a set of new features:

• Search inner documents: Stalwart now extracts and indexes the content of PDF, Microsoft Office (docx, xlsx, pptx), OpenDocument (odt, ods, odp), EPUB and RTF documents, the two in e-mail attachments and in document storage. Searching for a expression now finds the agreement attached to final year’s message, not fair the communication that mentions it. The extractor is written from scratch in Rust, alongside no external document libraries, and processes all document under strict size and complexity limits, so a malicious or malformed document cannot exhaust the server.
• Full-text hunt of document storage, affirmative an indicator queue alongside per-partition status, retries and rebuilds.
• Metadata: JMAP Object Metadata for e-mails, mailboxes, calendars, contacts, records and more, and IMAP METADATA (RFC 5464).
• Post-quantum DKIM2: composite ML-DSA-44 and Ed25519 signatures for DKIM2.
• OAuth and OpenID Connect: OAuth Client ID Metadata Documents, per-client scopes for mail, contacts, calendars and files, a catalog of connected applications that users can revoke, and spectators matching, opaque token validation and unscoped token policies for external character providers.
• Storage and operations: a built-in in-memory shop for single-node deployments, additional authentication methods for Azure Blob Storage, secondary LDAP servers for failover, a recollection prosperity for buffered data, SSRF safety for all outbound HTTP request, and configurable client IP sources and trusted proxy hops.
• AI models and an AI spam classifier in the Enterprise edition, alongside assistance for OpenAI-compatible, Anthropic, Gemini and Ollama models.

The complete catalog is in the changelog.

Finalising the repository schema has a cost that is paid once: Stalwart v1.0 uses a new repository layout, and existing data have to be migrated to it. A v1.0 server volition not open a shop created by an before version.

We cognize that several former upgrades were bumpy, and we took that seriously. This immigration was designed and tested to be as gentle as we could create it, and it is the final of its kind: the schema is now stable, and forthcoming releases volition upgrade in place. There are two ways to get to v1.0.

For deployments operating Stalwart v0.16, stalwart-migrate is the easiest route. It requires v0.16.27, so before 0.16 releases have to be updated to it first. The tool converts your settings to the v1.0 schema and copies all data into a new, bare v1.0 store. It lone says from the old store, so the first deployment stays intact and can be restarted at any item before the switch.

It additionally migrates incrementally. The archetypal copy runs during v0.16 keeps serving users, nevertheless lengthy it takes. Each afterward run copies lone what changed since the former one, and the lone downtime is the abbreviated final run, following v0.16 is stopped. Account and communication identifiers, IMAP UIDs and alter former are preserved, so IMAP and JMAP clients transport on without downloading item again. Users who sign in alongside OAuth volition need to sign in again, and the full-text indicator is rebuilt in the backdrop following the switch.

For anyone operating Stalwart v0.15 or earlier, or anyone who prefers to move users one at a time, the migration guide describes the second route. The migration proxy sits in forefront of the two the old and the new server and sends all person to whichever one holds their account, and Vandelay copies accounts throughout one by one, incrementally. Each document is switched by flipping one admission in a mapping table, alongside no downtime for that user, and the old server keeps a complete copy until you decide to quit it. The identical procedure additionally plant for moving from Dovecot and Postfix to Stalwart. The migration overview compares the two approaches.

The construction of your settings is mostly the same. Around a hundred keys were renamed to prosecute accordant naming conventions, a few objects (OIDC directories, AI models, Azure storage, forwarded headers) were restructured, and stalwart-migrate converts all of it automatically. Two changes need notice before you upgrade:

• Clusters need Redis. A collection coordinator now requires a Redis-based in-memory store (Redis, Redis Cluster or Redis Sentinel). Earlier versions could autumn rear to the data shop for this, but that was dilatory under burden and has been removed. Single-node deployments get a built-in local in-memory shop instead.
• OAuth now follows the open community client profile. Stalwart now completely enforces the OAuth Profile for Open Public Clients, the emerging norm that lets any communication client sign in to any communication server alongside OAuth without being registered in advance. Stalwart’s own authority server supports it out of the box. Many external character providers do not yet assistance everything the overview requires, specified as Dynamic Client Registration, and several clients do not yet petition the required scopes.

If you use an external character provider that is not conformant, Stalwart’s token checks can be relaxed through the compatibility settings on the OIDC directory, at your own risk, since all one weakens a safety check. These settings chiefly keep web clients specified as the WebUI working. What they cannot fix is a provider that lacks the overview itself, and that is going to matter soon: famous e-mail clients, including Apple Mail, are adopting the profile, and formerly they do, their users volition not be capable to authenticate against a non-conformant provider. If you depend on an external character provider, now is a fine period to ask them to assistance it.

A connected alter affects the WebUI. Instead of a fixed client id, it now identifies itself to the character provider alongside a Client ID Metadata Document, a URL that describes the client. Some character providers do not assistance this yet. With those, administrators can enroll a client for the WebUI alongside the provider and set its id in oauthClientId, and the WebUI signs in as before. The particulars are in client id of the WebUI.

If you run into a issue or have a inquiry during the migration, article it on support.stalw.art. We volition be answering questions there regular throughout the immigration period.

Five years ago this was one Rust archive for parsing e-mail. Today it is a complete communication and collaboration server, and alongside v1.0 it is faster than item alternatively we have measured it against. None of it would have happened without the community: everyone who submitted an issue, sent a patch, answered a inquiry on the forum, or trusted Stalwart alongside their mail. Thank you.

Download Stalwart v1.0 and let us cognize what you think.