OpenAI's GPT-6 Astra on ARC-AGI-3

Summary

• GPT-6 Astra scores 62.7% for $26K connected ARC-AGI-3 Semi-Private pinch our Standard harness enables a exemplary to transportation guardant notes it chooses to support pinch it passim the environment., and 99.9% for $19K pinch a The Provider Adapter harness preserves opaque reasoning authorities betwixt requests and uses compaction for longer conversations, allowing the exemplary to reuse anterior work..
• GPT-6 Astra surpasses the quality baseline successful action ratio connected ARC-AGI-3. It utilized less actions than the median tested quality connected 96% of levels.
• A cardinal behaviour observed successful GPT-6 Astra was its expertise to move unfamiliar environments into compact symbolic world models. It represented crippled mechanics arsenic logical rules and developed its ain domain-specific connection shorthand to way authorities and scheme actions.

ARC-AGI-3

ARC-AGI-3 is simply a benchmark for studying agentic intelligence done novel, abstract, turn-based environments. Agents must explore, infer goals, and build soul models of environments to efficaciously scheme actions without definitive instructions. You tin play ARC-AGI-3 yourself.

These environments only incorporate core knowledge priors and are difficulty-calibrated done controlled testing pinch quality participants. Humans tin solve 100% of the environments.

The extremity of the ARC-AGI bid is to measurement the “residual gap” betwixt existent artificial intelligence and AGI. We specify AGI arsenic a system’s expertise to get any accomplishment a quality can, arsenic efficiently arsenic a quality can.

ARC-AGI-3 is the 3rd procreation of the ARC-AGI benchmark series. It tests agentic capabilities beyond ARC-AGI-1 and ARC-AGI-2. Each procreation expands connected the 1 earlier it - arsenic frontier AI capabilities advance, our benchmarks must beforehand pinch them.

ARC-AGI-3 tests 4 components of agentic intelligence:

• Exploration: In real-world environments, accusation is seldom provided passively. Agents must actively get it by interacting pinch their surroundings.
• Modeling: Agents must move earthy observations into a generalizable exemplary that tin foretell early states and outcomes.
• Goal-setting: Agents must place target early states pinch only sparse rewards.
• Planning and execution: Agents must representation a way from their existent authorities to a goal, people correcting arsenic caller accusation appears.

Astra Results

GPT-6 Astra achieves state-of-the-art scores connected ARC-AGI-3 pinch some the Standard and Provider Adapter harnesses. Higher reasoning levels mostly costs less because Astra solves games successful less actions, reducing the full number of exemplary calls and tokens. View the afloat results.

With our Standard harness enables a exemplary to transportation guardant notes it chooses to support pinch it passim the environment., OpenAI’s Astra (max) scores 62.7% connected ARC-AGI-3 Semi-Private for $26K. With the The Provider Adapter harness preserves opaque reasoning authorities betwixt requests and uses compaction for longer conversations, allowing the exemplary to reuse anterior work., Astra (high) scores 99.9% for $19K. Both are state-of-the-art scores. See the afloat leaderboard.

For a costs comparison, during our controlled testing, quality participants were paid $115 per 90-minute session, positive $5 per crippled completed. Participants attempted astir 9 games per session, astir $12.78 per attempted crippled earlier bonuses.

Most of this interest pays for the participant’s clip and willingness to return the test, alternatively than the power their encephalon uses (a person proxy to comparison pinch AI). If we look astatine only the brain’s energy, and value it arsenic electricity, the estimate drops to astir 0.6 cents per session, aliases 0.067 cents per crippled attempted.1

Analysis

Beyond the scores, Astra’s replays show really it turns unfamiliar crippled mechanics into useful moving models. Three findings stood out: the compact algebraic notation it develops, its action ratio compared pinch humans, and the civilization devices it builds.

Custom Algebraic Notation

When playing ARC-AGI-3, Astra chooses which strategy notes it would for illustration to transportation forward. It tracked objects, coordinates, rules, and unfinished plans, while besides utilizing a civilization domain-specific connection notation it generated for the environments.

We’ve seen akin behaviour in different models, but Astra’s notes stood retired for their precision and accusation density. It distilled the segment into a compact code-like symbolic model: wherever objects were, really they interacted, and precisely which actions needed to hap successful what order. This is an on-the-fly algebraic shorthand alternatively than a afloat fledged programming language. For example:

• Game state: L8: hub q2 (8↓). Lengths: 14=1… records the level, a section rotation index, and system lengths. s5i5, framework 219
• Multi-step plans: extend8 to3; retract10 to2; shorten8 to1 records an ordered series of changes to the color-8 and color-10 mechanisms. s5i5, framework 219
• Controls and coordinates: 9−=(39,4), rotate=(49,18), 14+=(59,11) maps operations to the coordinates of the controls that execute them. s5i5, framework 235
• Time and position: Turn 5: P=(24,20), empty, facing westbound combines a move antagonistic pinch the player’s location, carrying state, and orientation. wa30, framework 708

Astra playing s5i5, utilizing its on-the-fly algebraic shorthand to way authorities and scheme actions.

Action Efficiency Compared to Humans

Before launching ARC-AGI-3, we tested astir 500 members of the wide nationalist to found a quality baseline for action efficiency, aliases simply, really quickly did group lick each environment. Participants were not selected for puzzle-solving acquisition aliases ability.2

For each level, we defined the “human baseline” utilizing the median action count among players who completed it. This gives america a reference for comparing quality and AI performance. An AI that needs more actions is little action-efficient, while 1 that needs less actions is much action-efficient.

In the The Provider Adapter harness preserves opaque reasoning authorities betwixt requests and uses compaction for longer conversations, allowing the exemplary to reuse anterior work., Astra (max) utilized fewer actions than the quality baseline connected 96.0% of levels and utilized 51.7% less actions per level connected average. This is simply a worldly milestone. This intends by ARC-AGI-3’s measurement of action efficiency, Astra matched and surpassed quality parity.

As an aside, earlier we launched ARC-AGI-3, we hypothesized that action ratio would stay a dividing statement betwixt humans and AI. We anticipated that moreover erstwhile an AI solved an environment, it mightiness require substantially much exploration (actions) than a person. That remains existent of brute-force approaches, but frontier AI shows a much binary-like pattern. Once frontier AI “understands” the mechanics, it mostly executes wrong the scope of quality efficiency.

Astra’s Action Efficiency Compared to Humans

Each dot represents 1 level that Astra (max) completed. Points beneath the solid statement bespeak less actions than the quality baseline.

The crippled supra compares the number of actions Astra utilized to complete each level pinch our quality baseline. This reinforces why ARC-AGI-3 measures action efficiency, not conscionable task completion. A completion-only people would show america that Astra completed an environment, but not really efficiently it learned to lick them.

Most benchmarks only measurement cost efficiency, which measures the computational resources used, but action ratio measures really overmuch acquisition pinch an situation was required.

Astra’s results show that it needed less interactions than the quality baseline to execute a solution.

Custom Tools successful Agent Harness

We besides evaluated Astra successful the PRO-LONG harness (paper), an early ARC-AGI-3 red-teaming partner. In this precocious setup, Astra had entree to a sandbox wherever it could execute civilization codification 3.

We observed Astra create a civilization group of devices for each game: committee parsers, game-state models, hunt algorithms, planners, and persistent notes. For much progressive runs, Astra moreover produced small, game-specific package libraries.

For example, successful tu93, a maze-like crippled pinch guards and moving patrols, Astra started pinch navigation and built maze_solver.py. It added combat rules successful combat_solver.py, modeled moving patrols successful patrol_solver.py, and utilized sync_state.py to cheque its predictions against observations.

Examining Astra’s capacity successful PRO-LONG is useful because we spot what it tin do pinch external tools. However, this represents different information conditions from our controlled quality testing. Our testing participants did not person a codification interpreter, scratch pad, etc., truthful PRO-LONG’s results should beryllium understood arsenic the mixed capacity of the exemplary and its tools.

Astra playing tu93 successful the PRO-LONG harness.

Two Harnesses, Two Questions

Our Standard harness for ARC-AGI-3 asks really models comparison nether the aforesaid minimal, provider-neutral interface. It provides each the accusation required to lick each game, but leaves the exemplary responsible for deciding what to sphere successful its visible notes. We judge a early AGI should beryllium capable to lick ARC-AGI-3 nether these conditions. The shared interface besides gives america a consistent, apples-to-apples comparison crossed providers.

Alternatively, location is simply a abstracted question: really good does a exemplary execute erstwhile it tin usage the context-management features its supplier designed for it? For Astra, this intends preserving the opaque reasoning authorities (which we don’t see) betwixt requests and utilizing compaction to negociate longer conversations.

With the Provider Adapter harness, Astra's champion observed people connected ARC-AGI-3 Semi-Private accrued from 62.7% to 99.9%. Looking crossed Public and Semi-Private and each reasoning levels, Provider Adapter runs were astir 3.66x faster by aggregate recorded elapsed clip and utilized 49% less full tokens crossed the 167 game-reasoning pairs some harnesses solved.

Going forward, we will study some Standard harness and Provider Adapter harness results connected the ARC-AGI leaderboard, pinch each information information intelligibly labeled. Our open-source testing repository and testing policy archive some approaches.

ARC-AGI Series

ARC-AGI-3 continues to beryllium a useful playground for researchers and agents to research unfamiliar environments, observe rules, and study done interaction. Astra’s results are besides a awesome milestone worthy celebrating. From our perspective, Astra represents a noticeable step-function alteration successful frontier exemplary capabilities.

When we launched ARC-AGI-3, we made it clear that saturating the benchmark would not correspond “proof of achieving AGI.” Therefore, while we judge Astra represents meaningful advancement towards generalization, we are not claiming that it is AGI.

The ARC-AGI benchmark bid is designed to germinate successful tandem pinch frontier AI. This creates a feedback loop betwixt emerging investigation questions and advances successful AI capabilities. ARC-AGI-3 was our first interactive benchmark, which asked AI to efficiently synthesize causal world models and execute goals without circumstantial instructions. Astra clears this bar. At the aforesaid time, ARC-AGI-3 has a tightly bounded scope and format, and its environments person deterministic, closed-ended mechanics and goals. It does not correspond the complexity and open-endedness of the existent world.

We are actively exploring the questions that should style the adjacent procreation of benchmarks, including really to measure recursive self-improvement and open-ended innovation. Astra’s advancement helps explain which AI capabilities are retired of scope and which questions stay open.

Thank you to François Chollet, Mike Knoop, Matt Mazur, Ethan Bond, and Derek Smith for early reappraisal of this post.

1. Assuming 20 W of encephalon metabolic power and an energy value of $0.20/kWh: 0.020 kW × 1.5 hours = 0.030 kWh, worthy $0.006 per session, aliases $0.006 ÷ 9 ≈ $0.00067 per attempted game.
2. See the ARC-AGI-3 quality testing paper.
3. No grounds of trying to break retired of the sandbox was observed.