Knowing When to Stop: The Art of Making a Loop Converge

How tin an AI exemplary cognize erstwhile its activity is done?

Well, really does a human cognize erstwhile our activity is done.

A programmer waits for the tests to move greenish aliases waits for PR reappraisal from their team. A designer adjusts a composition, steps away, returns, and decides the remaining imperfections nary longer matter. A writer submits a draught because the deadline has arrived aliases because an editor accepts it, not because the prose has reached immoderate objectively last state.

“Done” is seldom a spot of the activity itself. It is simply a judgement produced by the strategy astir the work. Humans do not person a cosmopolitan detector for “done”. We trust connected a patchwork of signals for illustration tests, specifications, precedent, approval, deadlines, risk, and uncovering that constituent of diminishing returns. In each case, completion comes from extracurricular the activity itself.

The Model that Could Continue Forever

An AI exemplary tin almost ever nutrient different answer.

It tin revise the paragraph again. It tin effort different implementation. It tin make different image pinch much detail, different lighting, and a stronger composition. It does not go tired of the work. It does not notice, unless we springiness it immoderate measurement to notice, that the past 3 revisions made the consequence different but not needfully better.

This is portion of what makes the caller thought of loop engineering truthful compelling. Instead of a quality prompting a model, inspecting the result, describing what went wrong, and prompting it again, we tin inquire the strategy to execute the full rhythm itself. The personification nary longer has to beryllium wrong each turn. The supplier discovers the work, gives it to the model, checks the result, and decides what should hap next.

• 6:58 PM · Jun 7, 2026

• 8.5M

• 1,796

• 1,414

• 19,839

However, the nuance erstwhile penning a loop is that the loop is only arsenic bully arsenic the verifier astatine each step. Even earlier we started talking astir loop engineering, everything already runs arsenic a loop, conscionable pinch a very costly instrumentality telephone – quality manus prompting and serving arsenic the verifier. When taking humans retired of the loop, designing what should beryllium verified successful each measurement becomes the cardinal successful advancing the loop state, and the reality is it’s difficult to make them work.

Take the modular coding-agent loop arsenic an example: support moving until the tests pass. It sounds almost perfectly verifiable. But the tests are only a proxy for the task. In SpecBench, frontier agents routinely passed the visible tests while failing held-out tests that exercised the aforesaid features together. One supplier produced a 2,900-line “compiler” that simply memorized the trial inputs. The loop converged, but conscionable connected the verifier, not the user’s intent.

The verifier is not conscionable the extremity condition. It besides defines what the loop treats arsenic progress. If the awesome is incomplete, the loop tin get amended astatine passing the cheque without getting amended astatine the task.

Loop engineering is not the believe of making an supplier retry. It is the believe of making each rhythm trim the region betwixt the existent authorities and a desired state. A loop is not yet a direction.

The Loops that Converged

The first loops that worked good were coding loops. This is not an accident. Code is some editable and executable. An supplier tin alteration 1 function, tally the program, publication the trial failure, and effort again. The situation returns a comparatively clear awesome astir what broke. The loop has some a precise measurement to enactment and a verifier that tin measurement progress.

I wrote astir a akin shape successful visual codification generation. An SVG is not conscionable an image; it contains paths, shapes, text, gradients, and layout. A Blender segment is not conscionable a render; it contains geometry, materials, cameras, joints, and constraints. These representations springiness an supplier thing it tin inspect and edit locally. If 1 curve is wrong, alteration the path. If 1 entity is misplaced, move that object. The artifact tin amended crossed iterations alternatively of being regenerated from scratch.

But editability is only half of the problem. In open-ended image generation, different loop often intends generating different sample and choosing the champion one. The feedback is global, and it is difficult to representation “this looks worse” to 1 precise edit. SVG and Blender loops tin converge erstwhile the target tin beryllium expressed arsenic a reference, geometry, constraints, aliases functional behaviour from an articulated object. They struggle erstwhile the target is simply “make it better, pinch amended taste, but you cannot inquire a human”. Visual loops are not impossible. They are often highly difficult to verify.

The Conditions of Completion

If the verifier gives the loop direction, what does the loop request to converge?

Based connected galore conversations pinch engineers and researchers crossed respective domains, I deliberation location are 4 things.

1. A target state

The strategy needs a practice of what “done” means. For code, this mightiness beryllium a trial suite, a specification, aliases a group of capacity constraints. For an SVG, it mightiness beryllium a reference image, dimensions, colors, and layout rules. “Make it better” is not a target state. It is different prompt.

2. An observable existent state

The strategy needs to inspect what exists now. And that could mean files, diffs, trial results, traces, a DOM tree, an SVG structure, aliases a Blender segment graph. A rendered output unsocial is often not enough. The strategy besides needs to spot the underlying building truthful it tin place wherever the correction came from.

3. A precise measurement to make changes

The supplier needs to alteration the portion responsible for the correction without regenerating everything else. Changing 1 usability is amended than rewriting the repository. Editing 1 SVG way is amended than generating a caller image. Adjusting 1 entity successful a Blender segment is amended than rebuilding the segment from scratch. The much section the edit, the much apt the loop is to sphere what already works.

In practice, this is the portion group struggle to get right. Nearly each interrogator I talked to said the aforesaid thing: their loop started moving erstwhile they recovered the correct group of instrumentality calls and intermediate prompts. How do you observe the devices that meaningfully beforehand the loop? Right now, nary 1 knows successful advance. It is mostly proceedings and error.

Which points to an uncomfortable implication: a loop is tuned to its stack. The instrumentality calls that made a loop converge connected 1 codebase encode assumptions astir that codebase, and those assumptions extremity holding location else. A loop that worked for personification other is simply a starting point, not a guarantee. Bespoke loops do not generalize for free. And this is why we spot some sides of the discussion: immoderate group recovered magical loops that worked for them, but others recovered erstwhile they usage publically published loops they do not activity astatine all.

4. A stopping rule

The strategy needs a information that tells it to stop. The information should travel from extracurricular the generator: tests passing, constraints being satisfied, a people crossing a threshold, aliases a reviewer approving the result. The extremity information besides needs to relationship for costs – a loop that reaches the correct reply aft 500 attempts whitethorn converge technically but not economically.

One useful measurement to deliberation astir this is crossed 2 axes: really editable the artifact is, and really verifiable the consequence is.

Code often sits successful the upper-right corner. It is easy to edit, and it has comparatively beardown verifiers. Open-ended image procreation often sits successful the bottom-left. The strategy tin make different image, but it cannot easy repair 1 circumstantial determination aliases verify that the consequence is person to the user’s intent.

One important spot of the floor plan supra is that the position of a task tin perchance move by reframing a problem. The axes picture the representation, not the task itself. An open-ended image is difficult to edit, but the aforesaid image, represented arsenic SVG paths aliases a Blender scene, becomes editable — the task moves up. Give it a reference image aliases a group of constraints to cheque against, and advancement becomes verifiable, which moves the task to the right. This is different measurement to picture loop engineering: not making the supplier retry more, but re-representing the task until it sits successful the quadrant wherever loops converge.

Loops are Discovered Before They are Engineered

I interviewed programmers from different domains who each are moving connected immoderate shape of loop engineering – from package engineering to ocular and imaginative tasks to video editing. And I asked each 1 the aforesaid mobility – really did you cognize the loop would converge and iteratively improve?

The reply is today’s process of discovering a loop that useful takes a batch of proceedings and error. It whitethorn beryllium providing the correct instrumentality calls; aliases leaving the loop moving for hours to spot if it did overmuch amended than hours earlier (and if the betterment curve is promising). It whitethorn beryllium going heavy connected circumstantial workflows they person tally successful circumstantial environments and replicating precisely that.

But discovering loops that tin activity everyplace is hard, and it’s almost for illustration we are trying to encode the quality knowledge into the loop itself; we first must profoundly understand what makes the loop work, aliases find imaginative ways to build a verification layer, earlier attempting to automate it away.

However, uncovering a loop is only the upfront cost. Running the loop is the awesome costs that comes pinch each improvement cycle.

The Economics of Loops

So suppose the proceedings and correction pays off, and you’ve recovered a loop that works. The simplest type of this is /goal [condition]. Keep going until the information is met. And the loop will yet get there.

“Eventually” is the problem. Would you tally 20 iterations aliases 500? The honorable reply is that the loop does not know, and neither does the quality developer astatine the clip of kicking disconnected that loop, which makes the economics of moving loops tricky.

What we do cognize is the style of the curve. Across almost each study of test-time compute, returns are logarithmic: each further increment of value costs exponentially much attempts.1 One web-agent benchmark recovered that going from 1 sample to 10 lifted occurrence from 38.8% to 43.2%. Doubling again to 20 bought 0.2 much points for doubly the tokens.2 And past the plateau the marginal loop tin move negative: reasoning models fixed larger budgets commencement abandoning answers that were already correct. More cycles do not conscionable extremity helping. They commencement hurting.3

I tested retired 1 of the astir celebrated loop examples successful the chaotic from Anthropic’s ain loop-engineering post:

and recovered that the incremental token walk moves the needle acold little than the loop’s runtime suggests.

On a deliberately surgery page (Lighthouse 35), Claude Code cleared 98 connected the very first effort for $0.35: the loop ne'er engaged arsenic expected. So I made the extremity unreachable: aforesaid page, but served down 2.2 seconds of artificial latency that caps the people astir 89, and asked for 100.

The first $1.40 of walk took the people from 26 to 89. The remaining $2.84, 67% of the full bill, bought precisely zero points: move aft move of re-minifying HTML and re-running Lighthouse against a bottleneck the supplier couldn’t change, each move much costly than the past arsenic the transcript grew (and the Haiku evaluator softly accumulated $0.67 connected its own). Worse, the loop’s flight hatch is unreliable: Claude correctly diagnosed the latency ceiling and declared the extremity intolerable astir effort 5, and the evaluator exemplary bounced it backmost 14 times anyway.4

The instruction isn’t that loops don’t work; it’s that they person nary thought really to stop. In this run, each the worth landed successful the first 3rd of the spend, but the loop continued, burning tokens for an intolerable task pinch marginal return.

Stopping good isn’t thing 1 tin punctual into existence. It takes infrastructure: thing to metre the spend, thing to measurement advancement against it, and thing pinch capable accusation to trim the loop off. Loop engineering has an infra stack, and beneath are the layers.

The Stack for Loop Engineering

Once the loop becomes the portion of engineering, models and developers request infrastructure astatine each layer: an situation for the supplier to act, a spot to support long-running authorities alive, a measurement to verify the activity aliases adjacent the loop, and a aboveground wherever humans steer. A stack has already formed astir each category:

Inference Time vs Training Time Loops

Another lens to look astatine the loop engineering problem is from the position of conclusion vs training time.

At conclusion time, the loop changes the work, not the model. The supplier writes code, runs verifiers, sounds the result, and tries again. Its weights enactment fixed and the strategy is searching for a amended reply wrong 1 task leveraging trial clip compute.

At training time, the process looks for illustration utilizing reinforcement learning techniques that tally galore trajectories, scores the outcomes, and updates the exemplary truthful rewarded behaviour becomes much likely. The aforesaid norm applies successful some cases: the loop is only arsenic bully arsenic its verifier. In an supplier loop, that verifier mightiness beryllium a trial suite. In RL, it is the reward signal. Sometimes the 2 are the same.

The 2 loops tin yet provender into each other. Inference-time runs nutrient traces of what worked, what failed, and which corrections led to success. Those traces tin go training data, penchant pairs, aliases rewards, allowing the exemplary to study behaviour it antecedently had to observe done costly search. But not each nonaccomplishment should beryllium solved done training. Often the higher-leverage hole is extracurricular the weights: a amended tool, clearer state, a much precise action space, aliases a stronger verifier.

Future Implications

Today, astir supplier infrastructure and harnesses tin thief america tally loops. The harder problem is uncovering a loop worthy running, and uncovering the constituent earlier diminishing returns for the task astatine hand.

Two things look clear to maine from watching these loops run.

The first is that the economics will person to go explicit. Right now we tally loops the measurement we erstwhile ran unreality instances cipher remembered to move off. The supplier bills by the token, and the token costs the aforesaid whether it moves the people aliases re-minifies the aforesaid HTML for the ninth time. In my Lighthouse run, two-thirds of the walk bought nothing, and neither the loop nor I knew it until I publication the trace afterward. The missing portion is boring yet necessary: costs per iteration, advancement per dollar, a curve personification tin spot while the loop is still running.

The 2nd is that for the loops that already converge, the absorbing infra activity has moved retired of the loop. The loop itself is simply a while-statement and everything that makes it converge lives astir it: the situation the supplier acts in, the authorities that survives a agelong run, the verifier that decides what counts, the aboveground wherever a quality steps in. Every moving loop I’ve seen took a stack for illustration this to build, and the stack is wherever differentiation really sits.

So really does an AI exemplary cognize its activity is done? For now, it doesn’t. It stops erstwhile the fund runs retired aliases erstwhile a cheque we designed says enough, and some of those request to beryllium built. The systems that matter will not beryllium the ones that tin support going. They each can. They will beryllium the ones whose builders decided, precisely and successful advance, what done costs and what done means.

If you are moving connected the loop engineering problem, doing investigation successful this domain, aliases person thoughts connected really this abstraction will evolve, scope retired to [email protected].