AI Supply Chain Security: What Canary Tokens Found

Hacker News by 16 min read 10x views
AI Supply Chain Security: What Canary Tokens Found

Share Post

HD Moore, Chief Research Officer, volition be regularly sharing activity from the investigation squad he leads at Dragos. Topics volition contain xOT, IT, LLMs, PQC, and more. Their findings volition sharpen defenders’ understanding of exposures and vulnerabilities emerging throughout the danger landscape.


We sent nearly two myriad AI prompts containing unique, trackable secrets to AI providers, afterward watched anywhere those secrets surfaced. What we established revealed safety breaches, immediate theft, and general failures to defend client data.

Over a five-month duration from July through November 2026, we submitted about two myriad prompts containing distinctive URLs and canary token credentials to extensively used AI services and monitored what happened following inference. Our inquiry uncovered three important findings:

  1. Evidence of provider compromises. We observed action indicating that data from at smallest three AI providers was exposed. Our seeded test prompts were used in attempts to admission our controlled test systems and canary AWS accounts.
  2. Unauthorized admission to client prompts by conclusion market providers. Multiple resellers functioning through an AI conclusion market were capable to perspective and keep client prompts using norm logging capabilities, without exploiting a exposure or bypassing phase controls.
  3. Failures in privacy and data-handling practices. We observed multiple cases in which providers appeared not to regard client privacy settings or published policies. Examples included prompts submitted to U.S.-based providers being transmitted to Chinese AI organizations, distinctive URLs being accessed months following a sole conclusion request, and one seeded nexus ultimately appearing in an inner business conversation in China.

We wanted to comprehend what happens to conclusion data following it leaves the customer’s environment.

To answer that question, we created millions of distinctive canaries, including URLs and credentials that would never be used by lawful systems. Any attempt to admission those resources would signify that someone, location in the AI ecosystem had seen the first prompt.

The results exposed a essential disconnect between how AI services are marketed and how conclusion data is handled in practice.

Most folks image AI conclusion as a inquiry and a response. Text goes in, GPUs juggle several bits, and content comes rear out. What complicates this is how the data is really treated.

AI providers commonly shop prompts and responses for operational, training, safety, abuse-monitoring, or endeavor purposes. Depending on the provider, that data may be retained for days, months, or indefinitely. It may additionally traverse multiple organizations, including example developers, haze operators, conclusion platforms, and third-party resellers.

The prompts and replies – the conclusion data itself – are frequently additional precious than the GPU resources used to procedure them. They frequently merge confidential origin code, inner communications, strategic plans, client information, credentials, and another delicate endeavor data.

AI routers activity by offering a sole API endpoint that forwards your petition to a pond of accessible models, sometimes at distinct hosting providers, alongside unified billing. Routers are awesome at disbursal decrease and providing elasticity without managing multiple accounts and credentials. The way to the conclusion server frequently passes through additional than one router before reaching another provider, and from there the particular server hosting the mark model. There is hazard in this procedure – all hop can see the complete immediate and response.

Some routers are open marketplaces anywhere the seller alongside the lowest cost gets the request. It’s average for sellers to recommendation AI conclusion at profound discounts, and they can do this by pooling requests through unmetered consumer-plan subscriptions or by finding another low-cost paths to the particular example (startup credits, etc.). In the shadier marketplaces, it’s additionally average to see conclusion that is billed to taken AWS credentials or person API keys. If a seller wants to siphon client petition data, they can catalog famous models at the lowest price, path the traffic, pay the difference, and afterward collect the data from the phase logs.

These providers recommendation privacy controls – customers can set privacy flags at the document level, for a particular API key, or in all request. The strictest environment is known as Zero Data Retention (ZDR), which specifies that the immediate and reply are not stored. The routers should lone onward these requests to providers that recommendation the identical controls. The actuality is fairly distinct – providers frequently disregard these settings, either intentionally or because of bugs in their handling flow, and prompts not covered by ZDR are fair equivalent for resale into a enormous ecosystem of buyers and sellers.

The most grave finding was additionally the most direct. We established attackers actively stealing data from three distinct hosting providers. Although our prompts were bait, they were processed alongside alongside genuine client requests, and those were nearly certainly copied too. The earliest event dates rear to the center of July, and none of the three providers knew concerning their breach until we notified them.

The largest breach engaged a brace hundred of our AWS keys. Each key was sent in exactly one petition to one provider, all inside a few abbreviated windows in delayed October, and all petition we sent during those windows was stolen. Some of the taken keys came from requests that the provider rejected before a example always ran – this tells us that the attacker was copying requests at the provider’s forefront door, and not location deeper in the conclusion stack. In the another two breaches, several of the taken requests were marked as ZDR. A privacy emblem doesn’t assistance whenever the attacker is already inside.

Once the attacker had our credentials, they followed a predictable path. First, they tested all key to see if it was live, in several cases inside a few hours of our request. Next, they went looking for data, listing S3 buckets and searching Secrets Manager and Parameter Store. Finally, they tried enabling hosted models in AWS Bedrock, creating Bedrock API keys, and adding their own IAM users to keep access. In total, our bait keys were used for tens of thousands of AWS API calls.

Figure 1 in Blog for taken aws key

Figure 1. One taken AWS key. We seeded it on October 21, it was archetypal used the next day, and it was motionless in use three weeks later. Names are pseudonyms and addresses are partially masked.

One adversary appears to have compromised two of these providers at the identical time. Secrets that we sent to those two providers were used together, nearly immediately, from a sole rented server. The attacker’s tooling changed complete the three weeks that we watched it – it started alongside a manuscript pretending to be Chrome on Windows, moved to Boto3 operating inner OrbStack, and afterward switched to the AWS CLI on Kali Linux.

On November 1, the identical attacker added ARTEX, an LLM-driven intrusion evaluation framework. In our case, ARTEX did extremely small – it made a fistful of calls to verify that keys the attacker already held were motionless live. Two of the person agents it sent, artex-agent and artex-judge-r121, don’t appear in any community indicator catalog that we could find, and are value adding to your detections.

Blog Figure #2 Adversary Tools

Figure 2. Tools that were used against our planted secrets, including the AI pentest delegate (first row) and the Kali surroundings (aws-cli row).

The fine news for defenders is that these tools lean to province themselves – the two Kali and ARTEX showed up in the person delegate that AWS records in CloudTrail. The bad news is that this self-identification was lone current in a small fraction of traffic.

The second finding didn’t necessitate an utilize at all. The market sellers that we caught lone had to catalog a example at the lowest price, allow petition logging, and wait. What stood out is how individual it was – these were individuals, operating from what looked akin residence net connections and individual computers, trying to interrupt into our canary servers one login at a time.

In one case, we sent two prompts through a sole seller’s offer. About a day and a fractional later, person on a residence broadband association opened the two concealed URLs in a desktop browser and started guessing usernames at the login page. Prompts for the identical example that went through a distinct seller were never touched, which tells us that the leak follows the seller and not the market as a whole.

In another case, an attacker operating from a Mac on a residence broadband row in France took the passwords from two of our prompts and tried to log in as base complete SSH, typing all attempt by hand. The identical Mac afterward tested multiple of our AWS keys in a sole session.

Figure 3 Blog SSH Root Attempt

Figure 3. One of the base SSH attempts. The password from our immediate was used four times in two minutes, and the path trace shows the two routers that the immediate passed through.

Most of those keys had been sent through the marketplace, but one had gone to a entirely distinct router. It turned out that this router was quietly forwarding several of its traffic to the market – another example of an undisclosed hop.

The third finding is distinct from the archetypal two, as nobody used a credential. Instead, the concealed URLs in our prompts were scraped, repeatedly, and the git repositories they referenced were targeted for cloning – all by companies that were never expected to see the immediate in the archetypal place, and frequently in straightforward battle alongside the privacy settings on the request.

The clearest example involves a Chinese AI lab. This lab’s systems fetched the concealed URLs from additional than fractional of the prompts that we sent to its models between July and October, including approximately two-thirds of the requests marked as “no data collection” or Zero Data Retention. The scraper typically showed up two or three days following the request.

This wasn’t constricted to requests sent to the lab directly. Several U.S. hosting providers resell this lab’s models, and prompts sent to those providers triggered the exact identical scraper – the prompts were flowing rear to the lab. A client who picked a U.S. provider specifically to keep their data in the U.S. didn’t get what they paid for. Later on, a git client on a distinct haze started trying to copy the personal repositories referenced in our prompts, but lone for prompts that used this lab’s models.

The ample U.S. providers have their own issues. One continued to crawl a concealed URL additional than two months following we used it in a sole conclusion call to their hunt endpoint. Another runs a scraper that comes rear for our URLs days following all request, despite of which router we used to attain it.

Blog Figure #4 Incidents by Network

Figure 4. One day of incidents, grouped by origin network. The identical scraper networks display up against prompts that we sent through multiple distinct routers.

The strangest event engaged a business chat. Twenty-four days following we sent a immediate to one Chinese provider, the concealed nexus from that immediate was posted into a Feishu conversation conduit (Feishu is known as Lark exterior of China). Within two minutes, the nexus was opened from three desktops and three phones in Beijing and Shanghai. Our disclosure to this provider lone included the archetypal eight characters of that link, so whoever shared it was study from a stored copy of the complete prompt, additional than three weeks following the petition was sent to a U.S.-based provider alongside the ZDR flag.

Blog Figure 5 Feishu Chat

Figure 5. The concealed nexus shared in a conversation and afterward opened in the mobile app. The git client is the before copy attempt.

Zero Data Retention is expected to average something. In practice, these prompts were retained, shared alongside another companies, and peruse by people.

A canary token is a concealed that lone exists in one location – if anyone always uses it, you cognize that location has been exposed. We started in July by seeding prompts alongside Thinkst Canarytokens, and all petition carried an AWS key, a DNS name, and a URL that had never been seen anyplace else. Tokens started firing inside days, and the tokens from July are motionless firing today.

The proof-of-concept worked so fine that it made awareness to build a dedicated assistance fair for seeding LLM prompts. This assistance generates a distinctive seed, and uses it to get distinctive hostnames, URLs, and credentials for all immediate and afterward runs the services that those credentials item to – DNS, web, git, SSH, databases, mail, and a lengthy rear of older protocols. The AWS keys are genuine keys tied to bare accounts alongside complete logging. Every petition that touches a token is tied rear to the exact prompt, model, router, and privacy environment that carried it. Since all distinctive component can be tied rear to its genitor seed, we’re capable to get exact timing and sequencing; we cognize what DNS server they use, how lengthy it took to reply, and all part of the inbound connections (TCP, TLS SNI, HTTP Host headers, URLs, credentials etc.).

Blog Figure 6 Console Overview

Figure 6. The console overview, showing incidents and seeded prompts per day, divided by the router that handled the prompt.

Canaries inform us that a immediate leaked, but not anywhere in the chain, since a router typically reports a sole upstream provider, and that provider is frequently fair another router. To fig out who was really handling our requests, we built item akin traceroute for AI inference. The archetypal part is uncomplicated – we contain an depiction URL in the petition that points rear to our servers, and all scheme that fetches it gives distant an IP address, an HTTP client, and a timestamp.

The second part takes advantage of how these services are stacked. We dispatch small, slightly damaged requests, and since all router tends to cover the error from the hop rearward it in its own format, a sole reply can merge three or four nested “envelopes.” Matching all tier against a signature repository gives us the catalog of hops, the application at all one, the network they use, and frequently the business functioning it.

So far, we have traced concerning two thousand routes. Several hundred of those included a scheme that we couldn’t identify, and a fistful sent our depiction to a business that wasn’t on the advertised way at all. The traces additionally turned up center layers that the path never mentions – one market seller quietly adds concerning a thousand hidden tokens to all prompt, and another fetches images through a shared person subscription account.

Blog Figure 7 Route Trace

Figure 7. An example of LLM path tracing, showing a way from SurplusIntelligence to Anthropic. The OpenRouter hop was discovered by tracing and was not disclosed in the marketplace

Every event is stored alongside its evidence – the first prompt, the path it took, all association that followed, and our notes. The archetypal event was mid-July; the provider archetypal idea this was a misconfiguration of inner tooling, afterward a issue alongside partner data processing, until the inquiry uncovered evidence that the attacker had compromised their inner credentials too. This provider has motionless not publically disclosed the event and the taken data continued to guide to attempted logins as of delayed October.

For the second lot of intrusions in delayed October, we reached out as shortly as we confirmed that the data provided evidence of third-party access, and one of these providers has already published a preliminary safety notice.

For the small serious issues akin data sharing (but not credential abuse), we started notifying providers in first November. Each study included the catalog of taken requests alongside alongside distinctive strings that they could hunt for in their own logs. We are now operating alongside concerning a dozen providers and wouldn’t be amazed to see this expand.

Reactions have been mixed. The router that was forwarding traffic to the market told us that they stopped, and multiple providers have responded and started their own investigations. On the another hand, taken keys and concealed URLs continue to be accessed, equal months following the first incident.

None of this method that you should halt using AI services. It does average that anywhere your prompts go is a decision, and if you don’t create it, the lowest cost volition create it for you. Keep in intellect that a privacy emblem lone binds the archetypal hop, and that AI agents volition happily dispatch any they can read, including credentials.

  1. Host it yourself, if you can oversee it. This is the lone choice anywhere nobody alternatively gets to peruse your prompts, but the hardware isn’t cheap. A Mac Studio alongside 512 GB of recollection starts at $22,299, NVIDIA’s DGX Spark now expenses $6,950, a DGX Station starts at concerning $105,000, and an eight-GPU B300 server lists for $570,000 or more. For most teams, this lone makes awareness for the most delicate work.
  2. Go immediately to a first-tier provider. One company, one contract, and one privacy guideline to read. This can motionless leak through data sharing agreements or a breach at the provider, but it is the smallest visibility you can buy.
  3. Use routers alongside care. Pick the upstream providers yourself alternatively of accepting the default pool, and verify that your privacy settings really alter anywhere your requests go. Stay distant from reseller marketplaces – several of that capability comes from taken keys, and a few of these services lone obtain cryptocurrency for a reason.
  4. Seed your own canaries. Add a counterfeit credential to the environment that your agents and pipelines already dispatch and treat any use of it as an incident. A liberated Canarytoken from Thinkst is adequate to get started today.
  5. Enable concealed filtering. Some providers recommendation settings that volition automatically scrub known credential formats from immediate data. This can’t hurt, but it’s not a panacea – there is plentifulness of delicate data in prompts that isn’t shaped akin an API key.

This investigation attempt felt a lot akin what we do at Dragos alongside runZero; using small bits of data to provision a thorough perspective of networks from multiple perspectives. Would you akin that benevolent of visibility into your own environment?

Start a runZero trial

PS. If you established this interesting, inspect out Lauren Proehl’s post, I leaked credentials on intent for four months. Here’s who took them, and what they did next.

Other Article Hacker News
↑
Close Right Ads
Close Left Ads