Tailscale didn't stop the Hugging Face intrusion

By now, youve apt heard astir the AI supplier that escaped a information information and attacked Hugging Face, an LLM marketplace. The supplier decided Hugging Face mightiness person the answers to its benchmark, truthful it stole them conscionable to cheat connected the exam. That's a funny motive but a scary outcome.

Hugging Face published a detailed reconstruction of the intrusion. It covers astir 17,600 recovered actions complete 4 and a half days, including sandbox escapes, codification execution, unreality credentials, improvised command-and-control systems, and eventually, the usage of Tailscale to dispersed passim their organization.

But Tailscale is simply a zero trust network! The full constituent of zero spot is to forestall attackers from moving laterally crossed your company. What went wrong?

Where Tailscale appeared

No “vulnerabilities” successful Tailscale were recovered aliases exploited, and that mightiness make it moreover much uncomfortable for us. Tailscale turns up everyplace successful AI infrastructure: virtually each AI institution uses Tailscale. So it's not astonishing we were successful an AI incident study eventually.

But, we're a information tool. Their intrusion is our intrusion, and it's our occupation to return it seriously. As successful immoderate post-mortem, we request to bespeak connected what happened and what we could person done to forestall it.

By the clip the supplier escaped its sandbox and recovered Tailscale, it had already gained codification execution privileges wrong a accumulation worker, reached guidelines entree connected a Kubernetes node, and publication a accumulation concealed shop containing 136 keys. In a sense, it was crippled complete earlier we moreover arrived connected the scene.

But isn't that the problem? Let's backmost up a level aliases two. Why were long-lived concealed keys accessible astatine each to anyone?

Because long-lived credentials are the standard

They shouldn't be, though. And we've now seen a awesome illustration of why. In the aged world wherever astir intrusions were done by humans astatine quality speed, credential leak mitigations were treated arsenic a nice-to-have. A large credential store, wherever you tin publication 136 keys astatine once, was a to-do point location successful a information team's low-priority list.

Now, successful a world of rogue AI agents, the large credential vault is the prize. It's not okay anymore.

I only cognize of 2 main solutions to long-lived credentials.

First, you tin configure a vault that only issues short-lived creds based connected long-lived creds that you insert erstwhile and that it ne'er gives back. HashiCorp Vault tin do this, for example; they telephone it dynamic credentials. Unfortunately, move credentials are a batch of activity to group up and maintain. When information requires work, group don't do it.

The 2nd action is simply a credential-injecting proxy. If customer C wants to entree server S, don't springiness C credentials for S; instead, person it nonstop its requests via a hardened proxy P that does thing but person a request, insert a credential, and walk it through. Credential injection is gaining fame now that modern mesh-based networking removes astir of its latency. A fewer months ago, we acquired Border0 ("Tailscale PAM"), which makes an elegant and rather fancy credential-injecting proxy, tightly integrated pinch Tailscale. Border0, afloat deployed, would person prevented reference those 136 keys and logged each effort to usage one. (The aforesaid credential injection powers Aperture Connectors.) But, this worldly is excessively recent. Most of our customers haven't moreover looked astatine it yet.

A prize 3rd option: Tailscale itself. In Tailscale, your node cardinal tin beryllium machine bound utilizing a TPM to forestall export, and if you're an endeavor you should decidedly do that. Alas, we had to move TPM retention disconnected by default connected Linux and Windows, because of HSM issues we recovered connected immoderate users' hardware. Anyway, successful today's case, the onslaught is what granted entree to Tailscale, truthful we arrived excessively late.

That long-lived auth cardinal didn’t request to exist

Neither move credentials nor credential-injecting proxies were successful place, truthful 136 keys leaked. What's the adjacent statement of defense?

One of those 136 credentials was a reusable Tailscale auth key, utilized to create caller Tailscale CI (continuous integration, utilized for automated testing) nodes successful their tailnet. The supplier copied that cardinal into a bid of outer sandboxes and utilized it, complete respective days, to enroll a full of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale personality tag granting each the entree a CI node would get.

We couldn't prevention those 136 keys. But a reusable Tailscale cardinal didn’t request to beryllium among them.

We built workload personality federation for cases for illustration this. It relies connected your unreality supplier to connection "ambient authority" successful the shape of short-lived creds generated from a moving VM's (or container's) unreality identity. The CI occupation tin inquire its unreality level for a signed OIDC token. Tailscale verifies the token, past grants entree pinch the tags and scopes assigned to that workload. What's neat is this tin hap automatically erstwhile enabled: rotation up the CI node, Tailscale gets the identity, assigns the correct tags. There's nary credential to leak and, erstwhile configured carefully, nary measurement to usage this credential anyplace but successful CI.

That intends you can't return a CI credential, tally it connected a node elsewhere, and move it into a paved way retired of the trial cluster into the remainder of the network.

Alas, not capable group usage workload personality federation. We should person done much to make judge customers knew it's the easier and safer way and had a clear measurement to adopt it. We tin amended the docs, and adhd much nudges successful the UI.

No customer logs doesn’t mean nary evidence

In an effort to hide its tracks, the supplier ran Tailscale pinch --no-logs-no-support, which suppresses reporting from that client.

That's an action designed for users who are concerned astir sending telemetry metadata to Tailscale. Even if we didn't connection it, it would beryllium easy to modify the root codification to region the telemetry.

But stopping the logs doesn’t make the relationship invisible. If you alteration Tailscale network travel logs, they study postulation from both ends of each connection, arsenic good arsenic from subnet routers and exit nodes. This is subtle but important: a compromised node mightiness not nonstop travel logs, but each node it connects to does. And past your SIEM, configured pinch care, tin raise an contiguous reddish alert if the 2 ends don't match.

Flow logs tin thief discovery erstwhile they watercourse into a cautiously configured SIEM. But that's a batch of work. Flow logs request to beryllium enabled, and you request to person the correct unrecorded discovery rules successful spot truthful they’re useful successful existent time, not conscionable for forensics later. We’re looking astatine really to make travel logs easier to discover, configure, adopt, and service arsenic alert triggers. I want america to make travel logs truthful easy to usage that they thief moreover if you don't person a information squad to watch them.

If you want nonstop power beyond conscionable logging, you tin besides alteration Tailnet Lock. This gives you nonstop visibility and strict, programmable admittance power for each azygous caller node. For example, pinch immoderate work, you could programme your signing node to cheque that "CI" tags ever person a peculiar IP reside scope aliases different side-channel impervious of validity.

Make the safe way the easy path

Network information is hard. It has ever been hard. In the caller world of rogue AI agents, it's not conscionable hard, but essential. And that's a problem because galore orgs simply don't person web information expertise.

So astatine Tailscale, we return it personally. People expect our merchandise to forestall these sorts of lateral activity attacks, by default, truthful they don't person to. Even if they person nary thought what a lateral activity onslaught is.

If this incident has you looking a small nervously astatine your ain infrastructure, commencement by looking astatine the reusable Tailscale auth keys your workloads tin read. For unreality and CI successful particular, switch them pinch workload personality federation wherever you can. Get free of those long-lived auth keys.

(Auth keys still person bully uses, particularly for one-time provisioning and environments without a level identity. When you request one, for illustration one-off keys; usage OAuth clients to support the auth cardinal expiry periods short; usage constrictive tags; audit the permissions granted to those keys successful your ACLs.)

Turn connected web travel logs and nonstop them to the devices your information squad already uses.

Use unafraid node authorities retention connected managed fleets, wherever you person power complete your TPMs. Use device posture to isolate and restrict nodes wherever you don't.

I cognize we haven’t made these safer choices evident enough. That’s connected us. We'll amended our docs, adhd nudges to the UI, do our champion to move these connected by default, pass you erstwhile you're doing thing dangerous, and propose amended alternatives.

This is our very Canadian apology: sorry you stepped connected our toes. The onslaught didn’t utilization Tailscale, and Tailscale didn’t origin the compromise. But, we didn't extremity it. Next time, we will.

If you tally Tailscale and want to excavation deeper, get successful touch pinch our support and solutions engineering teams. We tin thief you harden your settings and thief you find the unsmooth edges earlier the adjacent AI supplier does.