People are trusting their agents to do much activity without supervision, but really do you determine erstwhile to spot them?
Some deliberation the reply depends connected really bully the exemplary is – i.e. erstwhile models get better, you tin spot them to do more. But trusting your agents conscionable because the models sewage smarter is for illustration skipping your seatbelt because you sewage a nicer car.
The existent reply has thing to do pinch the model, and everything to do pinch the task. You request to build a intelligence exemplary for erstwhile to delegate and really much, which you tin past adapt.
Here’s a elemental guideline to maximizing supplier autonomy, truthful you tin vessel accelerated without breaking things.
You request to inquire yourself 2 questions first:
To tally safely connected their own, agents request contiguous feedback erstwhile they’re wrong. This is imaginable for astir codification pinch deterministic checks, for illustration portion and integration tests. But subjective tasks, for illustration renaming a parameter for clarity, are harder without quality sensation and judgment.
Just for illustration successful accepted package engineering, if you want to spot an supplier tally without supervision, you request a guaranteed Ctrl+Z for worst lawsuit scenarios. That’s why StampHog, our PR support agent, routes thing that contains deny-list keywords to a human.
Together, these 2 factors1 constituent you to 1 of 4 levels for immoderate task:
Level 0: Agent arsenic assistant. For tasks that are difficult to cheque and difficult to undo. Necessary erstwhile dealing pinch tricky and delicate code.
Level 1: Human-in-the-loop. For tasks that are difficult to cheque and easy to undo. Usually erstwhile you request subjective evaluation.
Level 2: Agent delegation. For tasks that are easy to cheque and difficult to undo. The default ceiling for astir dev activity today.
Level 3: Self-driving mode. For tasks that are easy to cheque and easy to undo. Everything is accelerating towards this.
You tin representation these 4 levels to a very elemental determination tree, which you tin use to immoderate task:
This helps you make decisions astir really overmuch to delegate to your agents, but you still tin technologist your pipeline to alteration higher autonomy for immoderate circumstantial task.
When the activity is hard to check + costly to undo
This is the lowest level of supplier autonomy. Think asking ChatGPT for advice, aliases auto-complete successful Cursor, conscionable for illustration the bully ol’ days of 2024.
But conscionable because it’s old-fashioned doesn’t mean it’s bad; this mode is perfect erstwhile dealing pinch tricky problems successful delicate codification surfaces.
For example, erstwhile Dylan updated our characteristic emblem motor to support generic spot targeting past year, he had to migrate an presumption that was indirectly baked into each characteristic emblem astatine PostHog.
This would person been difficult for agents to cheque deterministically since it wasn’t imaginable to grep for. The update besides had a immense blast radius since it touched unrecorded customer flags, API consequence shapes, and reason-scoring functions.
Break the task down. Small pieces make it evident wherever delegation is safe aliases not. Dylan handed disconnected little captious work, for illustration propagating the caller targeting logic crossed our JavaScript, PHP, Ruby, and Flutter SDKs, to agents while doing the riskier halfway migration by hand.
When the activity is hard to check + cheap to undo
This level of supplier autonomy is communal for tasks that request subjective information since it’s difficult to thatch agents sensation and judgement (for now, astatine least).
Human-in-the-loop tasks are considered inexpensive to undo since the codification stays successful draught mode and won’t get merged until it’s verified by a human. An undo conscionable intends kicking disconnected different iteration.
This codification readability refactor by Thomas was mostly done by hand, but it's a bully illustration of thing an supplier wouldn't cognize really to grade. It was conscionable a fewer lines that made it easier for humans to understand – adding comments, grouping actions, swapping strings for an enum – and didn't present immoderate breaking changes.
Use LLM-as-judge. This is really astir group are building their agentic codification reappraisal systems. As models proceed to improve, much tasks that require quality judgement tin beryllium checked pinch LLMs.
Define a scoped, measurable goal. Success metrics aliases contracts tin activity arsenic a proxy for a subjective evaluation. For example, you could instruct a strategy to research pinch landing page transcript until a version reaches 3% conversion.
Write civilization skills. This helps agents nutrient activity that reliably fits your standards, conventions, and sensation pinch little steering required. For example, galore developers constitute civilization codification reappraisal skills to travel team-specific standards.
When the activity is easy to check + costly to undo
This is the level that astir developer tasks are today. An supplier writes codification that tin beryllium tested deterministically, but the last enactment of merging it is gated down a last information check.
When Robbie rewrote our SQL parser successful Rust from scratch, he hardly publication the codification since he had a instrumentality oracle to cheque the work. But since the parser touches each query astatine PostHog, the agent’s activity was gated down aggregate information checks: protector mode successful production, past a staged cutover.
Enforce policies and guardrails pinch code. Most group default to gating Level 2 tasks down a quality (a.k.a., themselves), but this wont turns you into your ain bottleneck. Instead, encode arsenic galore of your guardrails straight into your pipeline pinch policies for illustration dry-running by default, scoping credentials, and putting changes down feature flags.
When the activity is easy to check + cheap to undo
There aren’t very galore tasks successful this class yet – conscionable smaller ones for illustration dependency bumps, lint fixes, adding trial sum to existing code. But the class is increasing quickly, particularly pinch long-running agents, goal-driven loops, and much analyzable orchestration.
We’re going all-in astatine PostHog connected making self-driving mode a reality for builders. Last month, for example, we launched Scouts – agents that tally connected a schedule, analyse signals from merchandise data, and draught a PR based connected what they find.
Train domain-specific models. Much of the adjacent activity of tooling will beryllium astir improving verification tasks that are difficult for LLMs today. One evident way is purpose-trained models that cognize what “good” looks for illustration successful a circumstantial domain, which is why we’re training our ain AI models astatine PostHog.
Build expert-level discourse banks. A deficiency of supplier autonomy is often conscionable owed to a discourse deficit. We learned firsthand from engineering our discourse layer for the PostHog Wizard that closing this spread pinch system and caller knowledge is 1 of the highest-leverage things you tin build for reliable agents.
Design clear signals for scouts. The bottleneck for long-running agents is going to beryllium whether they cognize erstwhile there’s activity worthy doing, and really to separate valid signals from random noise.
Words by Jina Yoon, who writes these newsletters astatine Level 0.
Technical Customer Success Manager - Americas
Technical Customer Success Manager - EMEA
BDR Team Lead
Revenue Ops Manager (sales focused)
Agentic Autonomy Levels – Addy Osmani
Repositioning PostHog – Charles Cook
How Cerebras Built Its Enterprise Knowledge Base – Isaac Tai, Daniel Kim, Mike Gao
What is simply a discourse warehouse? – Lizzie Epton
The caller rules of discourse engineering for Claude 5 models – Thariq Shihipar
English (US) · 
Indonesian (ID) ·