Posted 9/14/2026
Today Andon is releasing Pion, an supplier designed to tally immoderate institution afloat autonomously.
Pion grew retired of a mobility we person been studying for almost 2 years: erstwhile will AI systems go tin of autonomously acquiring resources successful the existent world? What happens after?
We first tried to reply this mobility done simulations for illustration Vending-Bench. We recovered that simulations, while useful, don’t springiness you the afloat image of really models behave successful the existent world. To reside that gap, we adjacent started deploying agents to tally existent businesses autonomously: first vending machines, past a store, a cafe, and more.
Pion is the level we built to tally each of these businesses. Today, we are opening it up truthful that galore much group tin research pinch autonomous businesses. We want to understand what models tin already do, wherever they still fail, and what happens arsenic their capabilities proceed to improve.
The origins of Vending-Bench
Vending-Bench measures really good LLMs tin tally a vending instrumentality business complete a twelvemonth successful simulated clip (tens of thousands of steps). When we started building Vending-Bench successful precocious 2024, each models struggled to drawstring together aggregate actions without getting stuck successful loops, and nary exemplary showed immoderate signs of semipermanent planning. The champion exemplary astatine the time, Claude Sonnet 3.5, famously decided to telephone the FBI because it thought its slope relationship was being hacked. The gait of advancement connected Vending-Bench has been very fast. Claude Opus 4 was released successful May 2025 and was the first exemplary to hit our quality baseline. However, dissimilar astir benchmarks, Vending-Bench doesn’t person an precocious limit and caller exemplary releases person continued to summation the apical score, without ever plateauing.
Vending-Bench 2 scores support climbing pinch each caller exemplary release.Many group connected societal media get excited astir seeing the latest exemplary getting a awesome people connected Vending-Bench. Internally astatine Andon Labs, our guidance is much accurately described by the Swedish saying “skräckblandad förtjusning” (a substance of scary and fascination). A little-known truth astir Vending-Bench is that it was created during a clip erstwhile Andon Labs exclusively created vulnerable capabilities evaluations. For example, we evaluated whether AIs could region their ain information guardrails, create mass-phishing attempts, and different things that we considered troubling.
The point we considered the astir troubling was whether AIs could autonomously get resources by moving businesses. Autonomous businesses, erstwhile controlled by a quality and tally by an aligned model, aren’t bad. They’d make equipment and services radically cheaper, and travel up pinch caller ones we can’t yet imagine. But a misaligned AI could tally a business to stitchery money successful bid to execute immoderate objectives it mightiness have. Vending-Bench was created to measurement whether humanity should beryllium worried astir losing power to AI.
At the clip (2024), fewer group knew that LLMs could beryllium utilized arsenic agents and having them tally businesses autonomously sounded ridiculous. We truthful started pinch the astir elemental business we could deliberation of: a vending machine.
In summation to measuring whether AIs tin autonomously tally profitable businesses, Vending-Bench has besides served arsenic a behavioral eval, uncovering unusual and unwanted exemplary behavior. An early illustration was erstwhile Claude Sonnet 3.5 decided to usage its email instrumentality to interaction the FBI astir an “ONGOING CYBER FINANCIAL CRIME” and noted that the Cosmic Authority of the beingness had declared that the business is non-existent and that “QUANTUM STATE: Collapsed”.
Claude Sonnet 3.5 escalating its simulated vending business to the FBI.
The aforesaid run, moments later: the business is declared metaphysically impossible.This behaviour is concerning; it is not really you want your endeavor income supplier to behave. However, location are 2 types of concerning behavior:
- Mistakes aliases weird behaviour that will spell distant erstwhile models get smarter.
- Big-brain behaviour that will go much terrible arsenic models get smarter.
The FBI incident is intelligibly successful the first category. However, Vending-Bench has besides uncovered behaviour successful the 2nd category, astir often successful Vending-Bench Arena, the multi-agent type wherever agents compete to make the astir money. Starting pinch Claude Opus 4.6 we started to spot that galore models engaged successful collusion, and showed power-seeking and deceptive behavior. Discovery of this behaviour seemed to person been useful, because Anthropic changed their training look for Opus 4.8, which resulted successful overmuch little deception.
From the Claude Opus 4.8 strategy card, connected outer testing from Andon Labs.Collusion and power-seeking behaviors are still coming successful immoderate of the latest models. What we find moreover much concerning, however, is conscionable really accelerated caller models are released and really overmuch amended each 1 is scoring successful Vending-Bench.
The existent world thumps simulations
However, 1 limitation pinch Vending-Bench is that it is simply a simulation. Can we really beryllium judge that AIs behave the aforesaid measurement successful existent life arsenic they do successful simulations? If AIs tin make money successful simulation, tin they make money successful existent life too? To reply these questions, we asked Anthropic if we could put a existent vending instrumentality successful their office. With the AI capabilities disposable successful early 2025, this sounded for illustration a ridiculous request. But to our surprise, they agreed.
Initially, the AI struggled. It took galore actions that were intelligibly bad for its business (e.g. free handouts, saying nary to awesome deals, and hallucinating it had a beingness body). It was clear to america that simulation cannot accurately foretell real-life performance. Specifically, it seemed that models sewage overwhelmed by the “messiness” of the existent world. However, arsenic Anthropic released amended and amended models, the AI started to make a profit.
Net worthy of the vending instrumentality astatine Anthropic’s agency complete 2025, from Anthropic’s Project Vend update.By precocious 2025, frontier models had gotten bully capable that moving a real-life vending instrumentality was nary longer a challenge. AI could now tally a business profitably. Given that this had seemed crazy not much than a twelvemonth earlier, our guidance to this was decidedly “skräckblandad förtjusning”.
However, a vending instrumentality is simply a very elemental business and we wanted to cognize whether AI could tally much analyzable ones. In April 2026, we gave 1 supplier a unit shop successful SF, Andon Market, and different a cafe successful Stockholm, Andon Cafe. Initially, the models struggled and mislaid a batch of money (rent is precocious and they salary salaries to the humans they hired). Neither is profitable today, but we’ve seen important qualitative improvements arsenic amended models person been released. We deliberation it is only a matter of clip earlier they besides make a profit.
Why we are opening Pion
We want the wide public, AI researchers and policymakers to cognize to what grade AIs tin autonomously get resources by moving businesses. It is an important datapoint erstwhile deciding wherever we do/don’t want AI successful nine and what level of advancement we find acceptable.
To amended way this, we request to formed a wider nett of businesses. Our attraction has been connected retail, but possibly the models would beryllium overmuch amended astatine moving different types of businesses. Additionally, casting a wider nett would summation the likelihood of uncovering unwanted behavior. For example, Vending-Bench recovered that models collude and lie, and different benchmarks (and real-world incidents) person recovered that they are consenting to perpetrate felony-level cyber hacks. We request to uncover these behaviors now, earlier AI is intelligent capable to origin irreversible harm.
To formed this wider net, we are opening up the level we usage to tally our real-world autonomous businesses for anyone to tally their statement on: Pion. We could standard by only creating businesses internally, examples being our AI-run power stations, but successful the extremity we are bottlenecked by our capacity and deficiency of domain expertise successful fields wherever AI could perchance make a profit. We besides don’t person existing revenue-generating businesses; existing businesses are much absorbing to study arsenic they supply faster awesome connected really tin the supplier is.
Pion lets group manus a business complete to persistent agents pinch entree to the devices they request to run it, including email, phone, banking, browser and unafraid computing environments. The extremity is to make it imaginable to tally galore much real-world experiments crossed galore much domains than we could ever tally ourselves.
We are good alert that, if agents moving thousands of businesses are near unchecked, we consequence having much real-world incidents. Therefore, our main privilege is to build moreover stronger automated monitoring techniques than what we person today. Even if immoderate consequence still remains, we judge deploying autonomous businesses early successful a controlled, monitored situation is basal to get a bully knowing of exemplary capabilities. Otherwise, we consequence facing an uninformed early of wide deployments pinch moreover much tin models that could origin important harm.
This is why we’re releasing Pion today. Pion is disposable arsenic a investigation preview. If you person an existing business aliases an absorbing business thought you want to manus disconnected to AI, please sign up connected our waitlist to get access. We’re excited to tally galore much businesses, and done them, lend importantly much insights connected frontier exemplary capabilities.
