We got admin access to Baseten's production GitHub in 25 minutes

We were astir to spot Baseten pinch our ain and our customers’ data. So to beryllium safe, we ran Strix to guarantee they were unafraid first. About 25 minutes later, it had a unrecorded GitHub token pinch repository-level admin authorities connected soul Baseten repos.

We build Strix, an autonomous hacking agent, which of people intends we request (cheap and fast) inference. We were exploring our options, and Baseten is 1 of the evident choices. It's a awesome product, they're weighted astatine $13 billion, and a batch of superior companies dangle connected them.

But... we're a information company. Before we springiness a 3rd statement our data, models, aliases code, we scan them. We would overmuch alternatively find a problem and thief get it fixed earlier we commencement depending connected that work (we do this pinch almost each of our vendors and person a precocious complaint of uncovering superior issues).

So... we pointed Strix astatine *.baseten.co and fto it tally without credentials aliases root code.

It came backmost pinch an progressive GitHub individual entree token for basetenbot. That token had admin and push entree to Baseten's main merchandise repo, the GitOps repo that drives their clusters, and their Homebrew tap, positive read/write entree to different backstage repositories including circumstantial repos per customers.

The image build dated to March 2023, and the token still worked erstwhile we recovered it successful July 2026.

But earlier we get into the details, let's springiness immoderate kudos to Baseten's information team. They confirmed the rumor arsenic critical, locked down the registry project, and rotated the token by the adjacent afternoon. They were master and very speedy to woody pinch it (which is often not the lawsuit successful these situations).

How Strix really recovered it

Strix starts the measurement immoderate bully pentest does: recon. Many times the astir terrible vulnerability connected your infra whitethorn beryllium connected a work connected a subdomain you forgot astir (which is why we ever urge achromatic container testing alongside testing pinch codification access)! So Strix enumerated hosts, looked done certificate logs, mapped the afloat surface. Eventually, it recovered a Harbor registry astatine gcp-us-east4-zlw.registry.baseten.co.

Harbor stores instrumentality images and groups repositories into projects. One of these projects was public. Without immoderate token aliases auth, Strix could database repositories, get anonymous propulsion tokens, and download the existent image manifests and blobs. That included an image called baseten/baseten-app.

At this point, it would beryllium easy to study an exposed registry and move on. But companies do intentionally people images, and Strix ne'er wants to study a mendacious positive. The important mobility is what effect the images being exposed have.

Strix decided to propulsion an image and spot what was inside. Here's that portion of the tally arsenic it appears successful Strix (yes, the agent's ain penning sounds for illustration AI, for evident reasons):

The first promising deed was a brace of AWS keys successful baseten/baseten-app. Strix tried a read-only sts:GetCallerIdentity call, which tells you what relationship a credential belongs to. The consequence was InvalidClientTokenId.

That cardinal was dead, truthful Strix kept looking.

Then, a token that really worked

It pulled the layers, ran TruffleHog (shout retired to our open-source information friends!), and inspected the image config directly. And location it was: a classical GitHub individual entree token, sitting successful history[].created_by.

I americium not a Docker runtime expert, but luckily Strix is (thanks to it having beautiful overmuch each of quality knowledge astatine its disposal). So it knew that that section records really a build measurement was created. In this case, it contained a RUN bid pinch the worth of GITHUB_TOKEN expanded straight into it.

Strix utilized the token for a read-only GET /user petition to GitHub and… VOILÀ. 200, pinch the relationship sanction basetenbot.

The token successful the Docker build history, followed by GitHub identifying it arsenic basetenbot. The credential is redacted. Open image for afloat size.

Notice wherever the token was found. As I learned, a Docker image has filesystem layers, but it besides has a config containing accusation astir the image and its build history. That config is downloadable on pinch the image. Cleaning up a credential record doesn't thief if the build history still contains different transcript of the token.

And this 1 still worked much than 3 years later.

Okay, what tin basetenbot do?

A unrecorded token is interesting, but evidently the permissions matter. This token could person 0 permissions and frankincense 0 impact. So Strix checked the relationship and its statement membership. GitHub returned X-OAuth-Scopes: repo, and the relationship belonged to basetenlabs.

GitHub returned repo scope for basetenbot and listed basetenlabs arsenic its organization. Open image for afloat size.

Then it checked the individual repository permissions, again utilizing read-only requests:

This is an insane magnitude of entree to time off successful a publically downloadable image.

basetenlabs/baseten is the product. Someone pinch this token had admin and push permissions connected the main root codification repository for an conclusion platform. They could tamper pinch the codification different companies trust connected to tally their models. We were considering sending our ain codification and models to this company, which is precisely why we do these checks successful the first place.

basetenlabs/flux-cd is arguably moreover scarier. Flux is GitOps: the repository contains the desired authorities of the clusters, and Flux applies that authorities to the infrastructure. Admin entree present creates a way from a leaked build token to changes successful accumulation infrastructure.

basetenlabs/homebrew-tap is really their CLI gets onto developer machines. Tampering pinch the distribution transmission could move this into a proviso concatenation onslaught against group installing Baseten's tooling.

And past location was basetenlabs/fde. A listing of that backstage repo showed a top-level customers/ directory, pinch subdirectory aft subdirectory named aft Baseten customers.

At that point, we had capable to study and beryllium assured this was not a mendacious positive. We didn't clone the customer repo, push anything, aliases alteration immoderate configuration. We stopped location and wrote the disclosure email immediately.

How does a token extremity up there?

The build history was timestamped. The measurement containing the token ran connected March 3, 2023. This was an aged build credential that still had each of that entree erstwhile we tested it successful July 2026.

The underlying correction is beautiful familiar. A build needed to fetch backstage limitations from GitHub, truthful personification passed a token successful arsenic a build argument. The applicable shape looked for illustration this:

I tin spot really personification ends up penning this. You request a backstage dependency, you walk successful the token, Git authenticates, and the build works. But Docker tin grounds that build statement successful the image's metadata and history. In this case, it recorded the existent token value. Docker explicitly warns astir this.

There is besides a 2nd problem pinch this pattern: git config --global writes the authenticated URL into Git's configuration file. Even if you alteration really the token gets into the build, you still request to debar redeeming it into the image.

The hole is to usage a BuildKit concealed mount and impermanent authentication that doesn't persist the credential. Then inspect some the image's layers and its history. And revoke the aged token! Changing the Dockerfile doesn't do thing astir an image that personification already downloaded.

What Strix did connected its own

Baseten has a responsive information squad and already uses AI information tooling. Still, this token from a 2023 build had admin entree to their merchandise and deployment repos erstwhile we recovered it.

It's easy to attraction connected the exertion and the root repositories, and hide astir an aged instrumentality image. Even if you scan the image's files, you still request to cheque its build history.

What I for illustration astir this scan is that Strix kept pursuing the finding. It recovered a registry, checked whether it could really propulsion an image, tested a credential and recovered it was dead, recovered different credential successful the build history, and checked what that 1 could access.

We hadn't told it to look for Harbor aliases fixed it immoderate hints astir a token. It worked done the full point autonomously successful astir 25 minutes.

This is why we're building Strix. AI-powered attacks person been getting ace scary successful the past fewer weeks, and we judge the only measurement to take sides yourself is to perpetually beryllium hacking yourself to find these issues (because location will ever beryllium issues) earlier the bad guys do.

Disclosure

Baseten handled this well. The timeline was:

• July 13, 11:10 PM: I reported the unrecorded basetenbot token, the nationalist Harbor project, and the repository permissions.
• July 14, morning: Baseten made the Harbor task private. I flagged that the token itself still worked.
• July 14, 4:34 PM: Anton from Baseten Security confirmed the rumor arsenic captious and said they had made the Harbor task backstage and rotated the token. He besides asked america to securely delete the images we'd pulled.
• July 14, 5:05 PM: We confirmed deletion and sent complete 2 lower-severity findings from the aforesaid scan.
• July 17: Baseten closed retired the remaining findings.
• September: We fto Baseten cognize we planned to disclose the uncovering publically and sent them a draught of this post.

They besides sent america immoderate T-shirts and sweatshirts arsenic a thank-you for uncovering this captious bug.

Go cheque your aged images

If you tally containers and usage GitHub, this is worthy checking successful your ain infrastructure:

1. See what personification tin propulsion without logging in, including aged tags and projects you haven't thought astir successful a while.
2. Read the build history pinch docker history --no-trunc, aliases inspect the config blob's history[].created_by fields. Check the layers too.
3. Get secrets retired of build arguments. Use concealed mounts, and make judge the commands consuming those secrets don't constitute them backmost into the image.
4. Check what your build tokens tin really do. Fetching a dependency needs publication entree to that dependency. Giving that token admin connected your merchandise and deployment repos makes a leak overmuch worse. Limit the permissions and springiness it an expiry.

And tally thing for illustration Strix against your ain systems. This full scan started because we wanted to usage an conclusion provider. We gave it a domain and sewage backmost a captious vulnerability that Baseten could enactment connected the adjacent morning.

AI attackers tin travel these aforesaid paths. If an supplier tin find a unrecorded admin token successful an aged image successful 25 minutes, you want yours to find it first.