Same $16.49/month server, aforesaid Traceway binary, 2 embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards astatine 100x the statement count, and stores a cardinal metric points successful 10.8 GB. Full numbers and methodology inside.
TL;DR
I spent six days moving connected and moving a benchmark of observability information connected DuckDB. I've done this pinch SQLite successful the past blog post and I really wanted to spot really DuckDB compares connected a inexpensive CCX13 Hetzner instance. The measurement I've done the measurements is by implementing DuckDB arsenic a valid retention motor for Traceway and past moving its benchmarking suite. The benchmarks show that DuckDB's columnar motor is capable to query 100x much information points without immoderate awesome backend changes. The constitute throughput is besides 3x to 15x higher. The consequence is that you tin now self-host the afloat OTel stack that tin grip a ample information measurement connected a beautiful mini server. Keep reference to find retired really I've done the measurements!
This is what came back:
| Metrics writes | 61,712 pts/sec | 254,242 pts/sec | 4x |
| Spans writes | 30,508 spans/sec | 95,737 spans/sec | 3x |
| Logs writes | 4,877 rec/sec | 75,225 rec/sec | 15x |
| Metrics publication cliff | 1M rows (3.85 s median) | 100M rows (3.0 s median) | 100x |
| Spans publication cliff | 100k rows (2.27 s median) | 10M rows (902 sclerosis median) | 100x |
| Logs publication cliff | 100k rows (114 sclerosis median) | 10M rows (85 sclerosis median) | 100x |
A publication cliff, passim this series, is the largest array size astatine which existent dashboard pages still load: the median of 3 endpoint probes astatine aliases nether 5 seconds, nary timing out. It earns the sanction because of what sits 1 10x measurement past it: queries don't slow down, they extremity coming back.
Each signal's publication cliff sits precisely 100x further retired connected DuckDB than connected SQLite, astatine adjacent aliases amended latency, connected identical hardware. I checked that symmetry against the earthy JSON doubly earlier I believed it. The container besides ingested a cardinal metric points successful nether an hr into 10.8 GB of disk and stayed up, a standard the SQLite benchmark ne'er sewage wrong 100x of. Logs, the awesome station 2 said to support disconnected SQLite entirely, are DuckDB's biggest win.
What's really being compared
Post 2 ended promising a ClickHouse comparison, and that station is still coming. But each clip I sat down to build it, the aforesaid mobility sewage successful beforehand of me: earlier reaching for a client-server OLAP database, pinch its ain container, its ain representation appetite, its ain nonaccomplishment modes, really acold does an embedded 1 go? Traceway ships a DuckDB telemetry backend arsenic an opt-in build, aforesaid azygous binary, aforesaid deployment communicative arsenic the SQLite build, different retention motor nether the telemetry tables. If you self-host connected 1 inexpensive box, these 2 builds are the existent determination successful beforehand of you, and I couldn't find anyone who had published numbers for it. So I ran station 2's full methodology against the DuckDB build and put the results broadside by side.
Here's the plan. Methodology first, including what changed successful the harness since station 2 and 1 hole successful the backend itself that these numbers dangle on. Then writes, each 3 signals against their SQLite baselines. Then reads, aforesaid shape. Then the billion-row run, what queries costs past the cliffs, the fleet mathematics for some builds, and which build I would really run.
How I measured
The setup is station 2's, truthful I'll support it short: a strategy nether trial (SUT) moving the Traceway binary and a abstracted load generator connected a backstage link, some successful Hetzner's Nuremberg datacenter, OTLP, the OpenTelemetry ligament protocol, complete gzipped protobuf, uniform-random information successful bounded ranges. Traceway built from perpetrate 14b4aa6e, Ubuntu 24.04, retention disconnected during the bench.
Two scenarios per signal. The throughput ramp: batch-size ramp astatine fixed rate, past a request-rate ramp astatine the winning batch (this clip pinch a finer complaint ladder, 1 done 25 req/sec, because the absorbing cliffs turned retired to beryllium betwixt station 2's coarser steps), past a small-batch high-rate ramp shaped for illustration an SDK fleet. A measurement passes astatine nether 5% errors and astatine slightest 70% of target complaint achieved. The read-probe: capable the array to 1M, 10M, 100M, 1B, past 5B rows, and astatine each level load 3 existent dashboard endpoints per signal, pinch a level passing erstwhile the median is astatine aliases nether 5 s and thing hits the 6 s timeout. The 5-second barroom is generous connected intent and station 2's chat of that still applies unchanged.
Three things disagree from station 2's harness, each disclosed:
- The loadgen grew. Post 2 generated load from a 2nd CCX13, which SQLite's ceilings ne'er stressed. On my first DuckDB spans run, that loadgen container died mid-ramp while the database sat astatine 0% errors, the postulation generator collapsed earlier the point it was breaking. The loadgen is now a CCX23. The SUT, the container each number describes, is unchanged.
- The digestion gate. DuckDB checkpoints its write-ahead log aft ingest stops; probe during that and you measurement a engaged engine, not a query. The read-probe now polls the backend's deep-health endpoint until the database and WAL record sizes are stable, records the hold per level (5 seconds astatine each level successful the last runs), past probes. SQLite reports nary motor gauges and skips the wait, truthful its station 2 numbers were produced identically.
- Only accepted rows count. The capable counts items confirmed by 2xx responses, not items attempted, which matters erstwhile the backend learns to garbage work. Fills overshoot their targets, astatine the smallest level by arsenic overmuch arsenic 2.6x, truthful I quote existent statement counts wherever the quality matters.
Caveats. These are single-shot runs, 1 throughput tally and 1 publication ladder per signal. Post 2 promised medians of 3 reps starting pinch this post, and I'm stepping that backmost 1 much time: the fix-and-rerun rhythm beneath consumed the budget, and I'd alternatively people single-shot numbers pinch that admittance than beryllium connected them. The cardinal cells reproduced crossed the debugging runs (100M metrics sounds landed betwixt 2.7 s and 3.2 s crossed 4 runs connected different days), but the protocol is single-shot. "Passing" still intends 3 named endpoints connected freshly written information connected an different idle box; read-under-ingest remains early work.
One much disclosure, this 1 successful the backend alternatively than the harness. My first DuckDB runs produced numbers I could not trust: the container kept dying mid-benchmark, and the hunt led to a existent bug successful my ain ingest path, which had nary admittance power and fto a sustained burst of fat batches tally the process retired of memory. I fixed it pinch an ingest gross that caps concurrent processing and answers overload pinch a 503 positive Retry-After (the perpetrate linked above), and reran everything. All numbers successful this station are from the fixed backend; the pre-fix results still visible successful the repo's history predate it and understate DuckDB, spans by 23% and logs by almost half, because the crashes were cutting the ramps short. SQLite ne'er surfaced the bug, its slower insert way errors early nether pressure, which is station 2's 5 req/sec logs wall doing its job.
Writes: 4x, 3x, 15x
| Metrics | 61,712/sec | 254,242/sec | batch 16384 × 20 req/sec, p50 3.2 s | 22.5 req/sec, 12.2% errors |
| Spans | 30,508/sec | 95,737/sec | batch 16384 × 7.5 req/sec, p50 1.2 s | 8.75 req/sec, 9.7% errors |
| Logs | 4,877/sec | 75,225/sec | batch 16384 × 5 req/sec, p50 3.0 s | 5.625 req/sec, 11.4% errors |
Going in, my quiet barroom for this array was "double SQLite and I'll telephone it a win," and the 1 compartment I braced for was logs. Post 2's starkest consequence was logs penning astatine 4,877/sec into a wall astatine 5 req/sec truthful crisp that 1.5x much requests took the correction complaint from 0% to 98%. That wall is the compartment that moved most: the DuckDB build writes logs astatine 15 times SQLite's rate, and the wall became a slope of polite refusals. The awesome pinch the lowest SQLite ceiling and the highest real-world measurement is the 1 the columnar motor helps most, which is either evident successful hindsight aliases backwards from everything station 2 trained maine to expect, and I held the 2nd position until this array existed.
The "lowest failing step" file matters arsenic overmuch arsenic the header column, and lowest is the correct word: the ladder bisects betwixt the past passing and first failing rate, truthful those are the smallest rates that fail, not the first failures successful wall-clock order. Every nonaccomplishment successful the file is the backend refusing activity alternatively than falling over: the tally logs show 503s from the admittance gate, the archived JSON shows zero insert failures and zero dropped rows down them, and the instrumentality vanished each 3 runs pinch zero restarts. That is the aforesaid graceful-cliff behaviour SQLite showed successful station 2, and DuckDB only exhibits it because of the hole I disclosed above; earlier it, this array could not beryllium produced astatine all.
The bottommost of each ramp holds 1 much first for the series: the SDK-fleet shape, hundreds of mini batches per 2nd alternatively than a fewer fat ones, which station 2 could only measurement connected SQLite. DuckDB sustains 53,091 metric points/sec, 36,577 spans/sec, and 30,442 log records/sec astatine batch 100. My early attempts ne'er produced these numbers, the pre-fix backend died earlier that shape of the ramp could run, truthful seeing them look astatine each was the first motion the reruns were measuring thing the crashes had been eating.
Reads: each cliff, 100x later
Writes were the appetizer; sounds are why anyone reaches for a columnar engine, and they're wherever I had the astir to lose. Post 2's most-quoted statement was astir publication cliffs, the past statement count wherever the dashboard still loads, and if DuckDB only matched SQLite here, the full build would beryllium a curiosity. Before moving it I told myself I'd beryllium satisfied if each signal's cliff moved 10x.
| 1M | 146 ms | 139 ms | 27 ms |
| 10M | 382 ms | 902 ms | 85 ms |
| 100M | 3.0 s, passes | over 60 s, fails | fails unevenly: assemblage hunt 3.5 s, trace-id 2 ms, severity select complete 60 s |
| 1B | fill succeeds, queries complete 60 s | not reached | not reached |
Each compartment is the median of 3 existent endpoint probes aft capable and digestion; bold marks each signal's past passing level. Two footnotes connected the array itself. The levels are capable targets and the fills tally over: the 1M level really holds 1.7M rows for spans and logs and 2.6M for metrics, and each level supra lands wrong 16% of target, truthful the nonstop statement counts are successful the JSON. And the over-60-s durations travel from a 2nd ladder tally pinch the probe timeout raised to 60 seconds, which gets its ain conception below; the modular probe gives up astatine 6 s. I sewage 100x per signal, not 10x: SQLite's past passing levels were 1M, 100k, and 100k, and each 3 moved by the aforesaid facet of 1 hundred. Same endpoints, aforesaid thresholds.
Metrics sounds standard gently: 146 sclerosis astatine 1M, 382 sclerosis astatine 10M, 3.0 s astatine 100M. A 100 times the rows costs 20 times the latency, the scan gets cheaper per statement arsenic the array grows. The level SQLite hardly passed is simply a hundredth of the level DuckDB passes pinch the aforesaid margin.
Spans salary for their percentiles, aforesaid arsenic they did connected SQLite, conscionable later. This is the page doing the paying, shown connected my section lawsuit pinch the loadgen's spans:
Every statement successful that array is simply a P50/P95/P99 aggregation complete the spans down it, and the stacked latency floor plan supra it is simply a 2nd aggregation. Those 2 queries costs 902 sclerosis astatine 10M rows and rustle past a afloat infinitesimal astatine 100M. Post 2 watched the aforesaid 2 pages dice betwixt 100k and 1M.
Logs are the numbers I trusted slightest and checked hardest, an early bad tally had maine convinced for a afloat time that DuckDB couldn't publication logs astatine all, and the truth turned retired to beryllium station 2's astir absorbing shape repeated astatine 10 times the rows. Here's the page successful question, connected my section lawsuit pinch the loadgen's log records:
At their passing levels logs are the fastest dashboard of the 3 (85 sclerosis median astatine 10M), and their nonaccomplishment astatine 100M is uneven successful the aforesaid measurement SQLite's was astatine 1M: the trace-ID lookup still answers successful 2 ms, the assemblage hunt successful 3.5 s, and only the severity filter, which matches and ranks a ample portion of 100M rows, dies. The pages an on-call really opens mid-incident, find this trace, hunt this error, stay usable astatine 100M rows; the full-table scan does not. Retention argumentation should still travel that split. The divided conscionable moved 100x.
A cardinal rows connected an 80 GB disk
I put the 1B level connected the ladder expecting to constitute a paragraph astir really it died, because each earlier effort astatine it had ended the aforesaid way: instrumentality gone, level unreadable. This clip the last capable measurement unsocial ran for 52 minutes, adding its past 900M points astatine a sustained 287k points/sec pinch the gross shedding the excess, and brought the array to 1,001,472,000 rows successful 10.8 GB connected disk, astir 10.8 bytes per constituent aft columnar compression, nether an hr of full ingest crossed the ladder. The container stayed up. Health checks answered the full measurement through, and post-fill digestion took 5 seconds. Nothing successful posts 1 aliases 2 gets wrong 100x of this level; connected the fixed backend a cardinal rows is conscionable a large table.
A large array you cannot look at: each 3 dashboard queries ran past 60 seconds astatine 1B, truthful the level fails, and 5B was ne'er attempted since the ladder stops astatine the first failure. I near it that measurement connected purpose. A 5B capable would beryllium the disk holds 54 GB of points, and I already cognize thing could publication them. The applicable summary of the ladder's top: this container tin shop a cardinal metric points, and it tin show you a 100 cardinal of them.
How slow is slow past the cliff
Post 2 could only opportunity "it timed out" astir thing past a cliff. This clip I reran the full ladder pinch a 60-second probe period to put existent durations connected the failing cells, and location were nary durations to put: each failing query was still moving erstwhile the caller headdress trim it off. Metrics astatine 1B, still moving astatine 61 s. Spans aggregations astatine 100M, still moving astatine 61 s. The logs severity select astatine 100M, still moving astatine 61 s while the assemblage hunt vanished successful 3.2 s beside it.
So the cliffs are not slopes connected either database. Queries spell from a 2nd aliases 3 to much than a infinitesimal crossed a azygous 10x measurement successful rows, the signature of an aggregation outgrowing the 4 GB representation fund and spilling. Post 2 recovered the aforesaid absence connected SQLite astatine 1 hundredth the scale, and it is why I support publishing cliffs alternatively of curves: location is nary slow-but-usable set supra a cliff to scheme around.
The 60-second tally besides caught a smaller flaw for the hole list: a cancelled dashboard query does not promptly merchandise its read-pool connection, truthful aft the 2 spans aggregations timed out, a 3 sclerosis exceptions query queued down them for 52 seconds. It affected nary passing number, and it goes connected the aforesaid database the admittance gross came from.
What fits comfortably, connected each build
Cliffs are ceilings; what matters time to time is really acold nether them a existent fleet sits. The small-fleet style from station 2, 10 backends emitting 50 spans/sec, 200 log records/sec, and 10 metric points/sec each, held against some builds:
| Spans (500/sec) | 1.7% | 0.5% |
| Logs (2,000/sec) | 40% | 2.7% |
| Metrics (100/sec) | 0.2% | 0.04% |
Writes were already a non-issue connected SQLite for everything isolated from logs, and connected DuckDB they extremity being a speech entirely. Retention is still the existent knob, pinch caller units. At this fleet's rates, logs scope their past passing publication level (10M rows) successful astir 85 minutes and the failing level successful astir 14 hours; spans scope 10M successful 5.5 hours; metrics scope 100M successful 11 days. On SQLite, station 2 measured the logs model astatine 50 seconds. An incident-sized moving set, hours of spans and logs, days of metrics, fits connected the correct broadside of each DuckDB cliff pinch an mean retention job, which is the point the SQLite build could not offer.
What I deliberately didn't measure
- Read nether concurrent write. Fill, digest, settle, probe. Stricter isolation than station 2, and still not what a dashboard experiences astatine 3 PM. Top of the future-work list.
- Mixed-signal load. Each awesome had the container to itself; a existent deployment writes each 3 astatine erstwhile and these ceilings do not simply add.
- Result correctness. I timed responses, I did not diff their contents.
- Variance. Single-shot, arsenic admitted successful the methodology, pinch the median-of-three protocol still owed.
- Tuning. Stock config, meaning what the benchmark constitute record ships by default: 4 GB DuckDB representation cap, 256 MB checkpoint threshold, nary schema aliases query changes. The 1B publication nonaccomplishment smells for illustration it wants a memory-budget experiment, and that is simply a deliberate cliffhanger.
- ClickHouse. The comparison station 2 promised is still owed, and it gets its ain station connected these aforesaid ladders alternatively than a conception squeezed successful here.
- Durability complete weeks. An hour-long billion-row capable says thing astir period 3 connected the aforesaid disk.
What amazed me
I expected the sounds to win, that's what columnar retention is for, and I group my barroom astatine 10x per signal. All 3 delivered 100x. But the point that will enactment pinch maine from this week is that the database was ne'er the difficult part: each wrong-looking number successful six days of benchmarking traced backmost to my ain ingest path, not the engine. DuckDB, whenever my codification sewage retired of its way, was boring successful the champion measurement a database tin beryllium boring.
The different astonishment was logs, again. Post 2's verdict was "keep logs disconnected SQLite," and logs turned retired to beryllium the strongest azygous statement for the DuckDB build: 15x connected writes, 100x connected the publication cliff, pinch the incident-workflow pages still answering astatine 100M rows.
The verdict
Post 1 claimed a $16 container could tally your observability stack. Post 2 tested the declare and returned it pinch an exception: not logs, and mind the publication cliffs. This station retires the exception. On the aforesaid box, the DuckDB build writes each awesome faster than a mini fleet emits, serves dashboards astatine statement counts the SQLite build cannot approach, and turns logs from the awesome station 2 told you to support disconnected the container into the champion consequence successful the dataset. A cardinal metric points costs 10.8 GB of disk. I came into the week hoping to double station 2's numbers; the smallest betterment successful the array is 3x.
So if you self-host Traceway connected a azygous machine, tally the DuckDB build (docker-compose.duckdb.yml successful the repo brings it up). One config statement earlier you do: that constitute record ships a blimpish 2 GB DuckDB representation headdress and leaves the checkpoint period astatine DuckDB's 16 MB default, while the benchmark ran connected 4 GB and 256 MB, truthful group DUCKDB_MEMORY_LIMIT and DUCKDB_CHECKPOINT_THRESHOLD to lucifer if you want these numbers. I would only scope for the SQLite build successful 2 situations, and some are real. If you request a binary that compiles anyplace Go does, SQLite wins: the DuckDB build needs CGO, Go's C bridge, and glibc, truthful its instrumentality image is Debian alternatively than Alpine. And if your measurement lives comfortably nether station 2's numbers, SQLite's retention motor doing each its activity inline, nary deferred checkpoints, nary digestion model to hold out, remains the simplest point that works. The infinitesimal logs matter, aliases retention past an hr matters, the comparison stops being close.
One asterisk belongs connected the full table: these numbers beryllium because the benchmark first recovered a clang bug successful my ingest path, and I fixed it earlier rerunning everything. If you tally the DuckDB build, tally a type pinch the admittance gate. The motor was ne'er the problem. The codification successful beforehand of it was, and uncovering that retired is the astir useful point this comparison produced.
Raw information + workflow
Everything successful this post, runnable from the repo:
- Raw JSON lives successful benchmarks/blog/post-3-data/: throughput, read-probe, and 60-second diagnostic JSONs, 1 per signal.
- The workflow: benchmark-hardware.yml. Throughput: 29828430297. Read-probe: 29838394873. 60-second diagnostic: 29848976962. The pre-fix runs the methodology disclosure mentions: 29734966806 and 29815312404.
- The fix: perpetrate 14b4aa6e, the ingest admittance gate.
- Try Traceway: self-host successful 5 minutes.
Questions, pushback, aliases "your number is incorrect because X": [email protected], aliases find maine connected GitHub.
Next post: the 1 station 2 promised. ClickHouse connected the aforesaid box, aforesaid loadgen, aforesaid ladders, to find retired what the afloat client-server stack buys you complete the embedded engines and what it costs to get it.
English (US) · 
Indonesian (ID) ·