PyPI Blog: Incident File Hosting Errors

infrastructure transparency

Executive summary

For astir 2 weeks successful August 2026, some PyPI users deed intermittent 502 and 503 errors downloading files from files.pythonhosted.org, triggering failures during installation from PyPI. Thanks to our users filing reports successful the support tracker; one study successful peculiar narrowed the problem to a azygous cache node.

Two abstracted problems were uncovered. A canary deployment wrong Fastly's network had triggered a misconfiguration astatine 1 cache node, causing Fastly's routing furniture to return 502 responses for postulation reaching the affected cache node. Separately, I recovered and fixed respective bugs successful our ain Fastly configuration around root fallback and scope petition behavior. Those had been location for a while, and only surfaced while digging into these reports.

Both intermittent problems are now fixed, and downloads are backmost to afloat usability since August 28.

Background: How PyPI's record hosting cache useful 1

Most group tally pip instal (or your installer of choice) and it conscionable works: a petition goes out, the desired record comes back. Behind the scenes, files.pythonhosted.org is simply a Fastly CDN service in beforehand of 3 origins (aka 'backends').

When a record is uploaded, PyPI writes it to Amazon S3 first as the main, durable copy. A inheritance occupation syncs it to Backblaze B2, because Fastly and Backblaze person a zero-cost egress agreement: serving files retired of B2 done Fastly doesn't costs thing beyond retention fees.

On the scholar (installer) path, Fastly tries B2 first. If B2 doesn't answer, aliases answers pinch thing we don't expect, Fastly falls backmost to S3. PyPI uses the Amazon S3 Glacier Instant Retrieval retention class to equilibrium retention and retrieval costs. When Fastly calls S3 arsenic a fallback, this costs much than from B2, but will proceed to activity for consumers arsenic a stop-gap.

Once cached, Fastly nary longer has to cheque B2 aliases S3 - the record should ne'er change, and the Cache-Control header sets max-age=365000000, immutable, nationalist - astir 11.5 years.

A 3rd backend named Conveyor handles everything that isn't a package record request, like predictable URLs, plus a fistful of bequest redirects.

flowchart TD Client([Client request]) --> Edge{Fastly edge} Edge -->|package file| B2[(B2: egress-free cache)] Edge -->|everything else| Conveyor[Conveyor] B2 -->|200 aliases 206| Response([Response to client]) B2 -.->|404, timeout, aliases 5xx| Archive[(S3: origin, fallback)] Archive --> Response Conveyor --> Response

That fallback way only useful if the separator notices B2 has failed. One of the bugs I fixed was that it didn't ever announcement correctly, which added to the "normal" correction noise.

Timeline

August 15

Fastly's study places the commencement of the problem connected this Saturday, at a azygous Seattle-area cache node. This is corroborated by #11876.

August 17

The first 2 reports of persistent 502s from files.pythonhosted.org are opened, #11895/#11897.

August 18

#11908 adds a elaborate reproduction, showing 88 recorded 502s complete six hours, across 32 unrelated packages, including the mini .whl.metadata scope requests installers usage to publication PEP 658 metadata. With 3 unfastened web reports, I spell digging successful Datadog Logs for the corresponding PyPI Files errors and find thing of consequence. infra#237 merges, fixing the B2-to-archive failover for the lawsuit wherever B2 doesn't respond astatine all, rather than responding pinch an error.

August 19

#11925 isolates the problem to 1 Fastly cache node, cache-pae2080020: 502 for each petition routed to it, for complete 19 hours, confirmed by x-served-by headers connected the failing responses. Fastly's web operations removes a routing override sending a portion of our postulation to the affected constituent of presence. infra#238 merges, improving the logging configuration for the record hosting service. infra#239 merges, rejecting HTTP methods that person nary business hitting a record host.

August 20

Fastly observes betterment astatine the affected constituent of presence, and later confirms the elevated correction complaint has stopped.

August 21

infra#241 merges, exempting suffix and multi-range requests from segmented caching, after a greeting spike traced to a azygous customer sending logically invalid ranges.

August 24

infra#243 merges, after 2 surgery parallel downloaders generated 41,315 much of the aforesaid people of correction successful a azygous day.

August 28

Fastly patches the underlying canary configuration bug connected their side, and excludes each PSF traffic, including PyPI, from their canary cohort. infra#245 merges, fixing a URL-normalization ordering bug that fto a bad segmented-caching consequence get cached and served to each consequent petition for the aforesaid file.

Contributing factors

POP goes the canary 2

Fastly runs a canary cohort, a subset of their fleet moving caching and routing package up of a afloat rollout. PyPI's postulation had been portion of that cohort for a number of years, helping Fastly engineering validate changes.

A partial rollback during a canary deployment left the caching configuration connected 1 Seattle constituent of beingness (POP) reverted while the routing configuration successful beforehand of it was not. The mismatch caused that routing furniture to return 502s.

Fastly has since removed each PSF traffic, including PyPI, from the canary program. We'd for illustration to get backmost to participating eventually, once clearer controls and notifications beryllium astir this traffic. It's a reasonable measurement to thief Fastly validate infrastructure changes earlier they deed everyone, and it hasn't costs america overmuch earlier now.

Bugs astatine home

While that was going on, I recovered unrelated bugs successful our ain configuration that produced the aforesaid symptom: elevated 502s, and successful a fewer cases 501s that looked for illustration 502s from outside.

The archive fallback successful the sketch above only ran erstwhile B2 answered pinch an correction status. If B2 didn't reply astatine each (a timeout, a refused connection, a TLS failure), Fastly synthesized its ain 503 and skipped consecutive past the codification that would person tried the archive. Package files are immutable, so location was ne'er a "try the archive" way for that case until infra#237 added one.

Separately, we usage segmented caching to debar pulling a afloat gigabyte-sized instrumentality into cache only to service a Range petition for partial content. That characteristic has narrower support for scope syntax than HTTP does successful general: it can't reply a suffix scope (bytes=-1024, publication the past N bytes) or a petition pinch the commencement of the scope past the end, and returns a synthetic 501 for either. Some installers usage precisely this benignant of suffix range to publication instrumentality metadata without downloading the full file, so those sounds were failing outright until I started exempting them:

flowchart LR subgraph Before["Before infra#241 and infra#243"] guidance TB C1([Client: suffix aliases inverted range]) --> E1{Fastly edge} E1 --> S1[Segmented caching] S1 --> R1([501 Not Implemented]) end subgraph After["After"] guidance TB C2([Client: suffix aliases inverted range]) --> E2{Fastly edge} E2 -->|range style not supported| N2[Segmented caching skipped] N2 --> R2([Normal scope handling: 206 aliases 416]) end Before ~~~ After

A 501 for a malformed customer petition is the incorrect position class. I've opened a support summons pinch Fastly astir the segmented-caching scope handling. Once fixed, my handling codification tin astir apt beryllium reverted.

Another bug: an existing segmented caching exemption check ran earlier the petition URL was normalized, so a .metadata petition pinch a query drawstring still connected it didn't match, kept segmented caching enabled, and sewage a 501 backmost from the archive backend for what should person been a normal fetch:

flowchart LR subgraph Before["Before infra#245"] guidance TB C3([Client: <code>GET name.whl.metadata<b>?token=x</b></code>]) --> E3{Fastly edge} E3 -->|exemption checked earlier URL is normalized| S3[Segmented caching stays on] S3 --> B23[(B2)] B23 -->|501 for the 1MiB segment, cached astatine the edge| R3([Every later request: 501]) end subgraph After2["After"] guidance TB C4([Client: <code>GET name.whl.metadata<b>?token=x</b></code>]) --> E4{Fastly edge} E4 -->|URL normalized first| X4[Exemption matches: segmented caching off] X4 --> B24[(B2)] B24 --> R4([Normal response, cached correctly]) end Before ~~~ After2

infra#245 moved the exemption cheque aft URL normalization to adjacent that hole.

None of these were caller bugs. They'd been successful the configuration already, and it took existent postulation connected ample files pinch scope requests to trigger investigation and resolution.

5xx measurement complete time

Fastly's ain real-time analytics for the record hosting service show B2 errors (blue) climbing from August 15 onward while the S3 archive backend (pink) stays level astatine zero, because the fallback that should person been routing failures location wasn't firing yet:

Our ain Datadog metric shows a fuller story, from earlier the incident to past the extremity of it:

Note the log scale. The baseline was already noisy earlier immoderate of this started, thousands to tens of thousands of 5xx responses, which is why the summation starting August 15 is easy to miss. The August 21 spike is the 1 that isn't: a azygous customer sending logically invalid ranges, pushing the count to adjacent to a million.

The cliff instantly aft it is infra#241. Traffic past that constituent sits 2 to 3 orders of magnitude beneath the pre-incident baseline, tens to hundreds of errors alternatively than thousands. Some of what we'd been treating arsenic inheritance sound was this bug moving the full time.

Going forward

The Python Software Foundation is hiring an infrastructure engineer to adhd to the engineering unit of four. Part of their remit will beryllium PyPI, which should thief america drawback conditions for illustration this earlier and forestall alternatively than react. The continued financial support from our organization - individuals and companies alike - makes that possible.

A batch of the postulation hitting files.pythonhosted.org is CI jobs installing the aforesaid limitations they installed past run, and a ample stock of that tin beryllium traced backmost to GitHub Actions runs. If you're not already caching those downloads, it's worthy turning on: setup-python's pip, pipenv, and poesy caching is opt-in via the cache input, disconnected by default, and setup-uv's caching defaults to connected for astir GitHub-hosted runner events, but is worthy checking. An unchanged dependency that's cached doesn't touch america connected the adjacent tally astatine all, which intends less requests for america to serve, and 1 little point that tin break your build if we aliases Fastly person a unsmooth day.

Thanks

Thanks to everyone who took the clip to record an rumor and seizure the details instead of conscionable moving astir the problem.

If you tally into file-hosting issues successful the future, pypi/support is still the correct place, and the much item you tin see (especially x-served-by headers and timestamps), the faster we tin enactment connected it.

This activity would not beryllium imaginable without generous donations, please consider supporting the PSF to support this benignant of infrastructure running. Thanks to Alpha-Omega, which sponsors my role.