A Server Lost Power at 00:32. We Found Out at 08:18

Resolved. This incident ran from 00:32 to 08:47 UTC connected 17 August 2026 and is closed. No customer information was lost, corrupted aliases exposed. If a deployment of yours grounded during that model it will not person retried connected its ain — trigger it erstwhile much and it will spell through. Everything beneath is the afloat relationship of what happened and what we changed.

On Sunday 17 August 2026, 1 of our retention servers mislaid powerfulness astatine 00:32 UTC. It did not reboot. It did not crash. It simply stopped, and it stayed disconnected until an technologist pressed the powerfulness fastener by manus astatine 08:22 UTC.

For 8 hours and 15 minutes, S3-compatible entity storage, the instrumentality registry, serverless deployments and static-site deployments were unavailable. Running applications kept serving. No information was lost.

Our position page detected the nonaccomplishment and opened an incident astatine 00:35 UTC — 3 minutes in. It was correct, it was public, and it stayed up for the full outage.

Nobody saw it. We recovered retired astatine 08:18 UTC, erstwhile an technologist opened the position page for an unrelated logic and asked why truthful galore services were red.

That spread — betwixt a strategy knowing and a personification knowing — is the existent incident. Everything other is detail.

What Our Customers Saw

The nonaccomplishment had 1 origin and 4 symptoms, which is why the position page looked worse than a azygous dormant instrumentality should look.

• Object storage returned relationship refused, not slow responses. S3 was afloat unavailable alternatively than degraded.
• The instrumentality registry went down pinch it, because registry blobs are stored successful entity storage.
• Serverless and static-site deployments failed, because they push and propulsion images done that registry. Failed deployments stayed grounded — they do not retry connected their own, and needed a manual re-trigger afterwards.
• Existing workloads were unaffected. Applications kept serving their existent type throughout.

One item caught respective customers out, and it is worthy stating plainly: changing only an situation adaptable connected a serverless instrumentality still contacts the registry. Knative resolves an image tag to a digest connected each caller revision, truthful moreover a deployment that reuses an image you already pushed will neglect if the registry is down. If you assumed an env-var-only alteration was safe during the outage, that presumption was reasonable and wrong.

Why One Machine Took Down the Whole Service

This is the portion that matters, and it is not the portion we expected to write.

Customer entity information is stored pinch erasure coding — each entity is divided into 4 information chunks and 2 parity chunks, six pieces successful total, and immoderate 4 of them tin rebuild the object. On insubstantial that survives the nonaccomplishment of 2 pieces.

The problem is where those six pieces are allowed to live.

Our retention cluster spreads chunks crossed individual disks, without requiring them to beryllium connected different machines. Each retention server holds galore disks. So erstwhile 1 server went away, it took an mean of 1 and a half of each object's six chunks pinch it — and for a ample number of objects it took 2 aliases three.

• Lose 1 chunk of six: the entity still serves.
• Lose two: beneath the minimum needed to service safely — sounds stop.
• Lose three: beneath the 4 required to reconstruct astatine each — unreadable until the disks travel back.

That is why the outage was full alternatively than partial, and why it could not self-heal while the instrumentality was off. The information was ne'er lost. It was simply unreachable until the hardware returned.

The evident hole is to require the six chunks to onshore connected six different machines. We could not do that connected the day, because a four-plus-two strategy needs six nonaccomplishment domains and we were moving 4 retention machines. This was ne'er 1 configuration emblem distant from correct. It was a capacity determination that had been softly deferred down a default.

Additional retention capacity was already planned, and host-level redundancy for entity retention is the alteration we are prioritising retired of this incident. We are not putting a day connected it here; we would alternatively denote it erstwhile it is successful spot than perpetrate to a week and revise it.

A second, smaller problem made it worse. Alongside the bulk data, entity retention keeps a mini group of soul bookkeeping records, stored arsenic 2 copies alternatively than erasure-coded. Those copies were besides allowed to stock a instrumentality — and for a fistful of records, including the 1 the retention gateways publication astatine startup, some copies were connected the grounded server. That azygous mini grounds is the quality betwixt degraded reads and connection refused. Hundreds of terabytes of customer information behaved amended than the fewer kilobytes of configuration describing it.

This half needs nary caller hardware. Moving those bookkeeping records to 3 copies crossed 3 abstracted machines is imaginable connected the servers we already run, and it is the alteration that converts a full outage into a slow one. It is first successful the queue.

The Near-Miss

There is simply a 2nd point that did not happen, and we would alternatively show you astir it than not.

When a retention instrumentality disappears, the cluster starts rebuilding its missing copies onto the machines that remain. That is usually the correct behaviour. In this lawsuit it meant reconstructing astir 115 TiB onto 3 servers pinch 140 TiB of free abstraction betwixt them.

That trajectory ends astatine astir 97% full, past the period wherever the cluster stops accepting writes wholly — turning a publication outage into a full one, by our ain automation, without immoderate further hardware failure. At the observed rebuild complaint it was astir 5 days away, truthful we had days alternatively than hours. But the guidance of recreation was wrong, and we had not suppressed the rebuild.

The structural instruction is uncomfortable and simple: at 71% utilization, a four-node cluster cannot sorb the imperishable nonaccomplishment of a node. Surviving that needs either substantially much free abstraction aliases much machines.

What Caused the Power Loss

We do not know, and we are not going to dress otherwise.

Every software-visible origin is ruled out. There was nary kernel panic, nary clang log, nary representation errors of immoderate kind, and nary thermal events. All 10 drives passed their wellness checks pinch zero errors. The instrumentality had been moving continuously for 103 days without a azygous kernel informing earlier it stopped.

It was besides not a reboot. The strategy log ends mid-sentence and contains nary startup introduction astatine each until the manual power-on 8 hours later. The retention furniture independently confirmed an abrupt powerfulness trim alternatively than a shutdown.

Our hardware supplier reports nary powerfulness arena successful the information centre astatine that time, and nary monitoring information for the instrumentality itself. A afloat hardware inspection is disposable but requires astir six hours of downtime connected a retention node, which presently costs much than the reply is worthy while the instrumentality runs normally. The arena is connected the server record. If it happens a 2nd clip we book the inspection immediately, and the grounds from this occurrence will make that test considerably faster.

What We Changed the Same Day

Two things shipped wrong hours of recovery.

1. On-call now gets a telephone call

Before this incident, a level incident reached 1 person, by email. At 00:35 connected a Sunday morning, an email is not an alarm.

When an incident now appears connected our nationalist position page, it places a sound telephone to on-call, pinch an SMS alongside it. Three creation choices are worthy naming, because they are what make it trustworthy:

• It is deliberately extracurricular our normal notification system. That strategy applies preferences, digests, complaint limits and quiet periods — each correct for customer notifications, each incorrect for a pager. A page must not beryllium suppressible.
• The spoken connection is carried wrong the telephone request alternatively than fetched from a page connected our ain infrastructure. The outage being reported whitethorn good beryllium the point that would person served that page.
• It cannot beryllium triggered by history. Incidents written aft the fact, including the backfilled entries down this very postmortem, are ignored. A postmortem should not ringing a telephone astatine 3am.

2. A watchdog that tin move a dormant instrumentality backmost on

Our servers expose a guidance API that reports existent powerfulness authorities and tin rumor a power-button press. The eight-hour outage was, successful the end, a fastener that cipher was awake to press.

A watchdog now probes each server each infinitesimal and tin property that fastener unattended. Because the nonaccomplishment mode of getting this incorrect is terrible — sending a shutdown to a healthy retention node, causing the outage it exists to forestall — almost each of the activity went into making it garbage to act:

• It does not tally connected the infrastructure it watches. A watchdog wrong the point it monitors dies pinch it.
• Four independent signals must agree — web reachability, cluster health, reported powerfulness state, and workload wellness — earlier thing happens.
• Two aliases much machines failing astatine erstwhile is treated arsenic a monitoring aliases web fault, ne'er arsenic simultaneous hardware failure. It will not enactment connected a correlated outage.
• It will not conjecture astatine powerfulness state. If the hardware cannot corroborate a instrumentality is genuinely off, the watchdog raises an alert and stops.
• Several further interlocks — a confirmation window, a cooldown, and a veto from patient workloads — beryllium betwixt a grounded probe and immoderate action.

The determination logic is simply a azygous axenic usability pinch nary broadside effects, truthful each 1 of those rails is covered by tests alternatively than by hope.

It has already been useful successful an unglamorous way: the pursuing greeting it paged for a instrumentality that was wholly healthy, connected the 1 codification way that could occurrence earlier the confirmation model applied. A azygous dropped web packet was enough. That is fixed, and we would alternatively find it that measurement than the different way.

What We Have Not Fixed

We would alternatively people this database than a cleaner story.

• The retention layout is not fixed yet. Until information and bookkeeping copies are guaranteed to beryllium connected different machines, losing 1 instrumentality will still interrupt entity storage. The watchdog shortens that outage; it does not forestall it. The bookkeeping half needs nary caller hardware. The remainder needs capacity we person planned but not yet added.
• Capacity headroom is excessively thin to sorb the imperishable nonaccomplishment of a node. Same description , aforesaid timeline.
• On-call is 1 telephone number. A azygous recipient is simply a azygous constituent of nonaccomplishment successful the 1 strategy whose full intent is not having one.
• The guidelines origin of the powerfulness nonaccomplishment is unresolved, by our ain choice, pending recurrence.

What We Took From This

Detection and notification are different systems, and we had only built one. Our monitoring was accurate, accelerated and nationalist for 8 hours while everyone slept. Monitoring that cipher is woken by is simply a record, not an alarm.

Redundancy has a shape, and the style is the spot that matters. "Six copies" and "six copies that are allowed to stock a machine" are wholly different guarantees. We had configured the 2nd while believing we had the first.

The smallest constituent did the largest damage. Hundreds of terabytes of customer information degraded precisely arsenic designed. A fewer kilobytes of soul bookkeeping, stored pinch little attraction because it was small, is what took the work from slow to unreachable. If you tally thing similar, audit your smallest pools first.

Verify betterment wherever your customers really are. Partway through, we concisely believed the registry had recovered because it returned an authentication response. That consequence comes from a furniture that ne'er touches storage. Had we acted connected it, we would person told 3 customers to retry into a work that was still broken. The only valid trial is simply a petition that sounds existent data.

Where This Leaves You

No customer information was lost, corrupted aliases exposed. Backups remained intact throughout.

If you ran deployments during the model and they failed, they will not person retried connected their ain — trigger them erstwhile much and they will spell through.

The committedness we are making is circumstantial successful substance, not successful date: we are changing the retention layout truthful that losing a azygous instrumentality degrades the work alternatively of interrupting it, and adding the capacity that alteration requires. We will people a follow-up erstwhile it is done, alternatively than a schedule now.

We are sorry for the disruption, and peculiarly for the 8 hours during which our systems knew and we did not.