How well do agents use test/verification techniques?

We antecedently noted that, while it's easier than ever to deed a peculiar value barroom by having coding agents usage effective trial techniques, package value seems to beryllium getting worse, indicating that immoderate defaults developers are utilizing whitethorn not activity very well. Here, we trial if elemental instructions to agents to usage peculiar techniques aliases libraries amended implementation correctness, arsenic a benignant of trial to spot really effective agents are erstwhile guided by personification pinch nary expertise successful testing who's possibly heard that you should use definite techniques aliases usage definite libraries.

We'll re-use the Zstd implementation eval discussed successful this comparison of agentic programming connection effectiveness and, instead, comparison different testing techniques and testing libraries erstwhile agents are fixed a punctual to instrumentality Zstd pinch different addendums, specified arsenic "Use test-driven development", "Use Lean 4", "Use QuickCheck", "Use property-based testing", etc. I besides ran immoderate different evals, specified arsenic connected the IMAP RFC, which are concisely discussed.

All implementations were successful Rust. The 26 punctual conditions tested were ACL2, Alloy, "Audit and fuzz risky areas", "Audit first", Creusot, Default (no further instructions), Differential testing, Fuzzing, Hegel, Insta, Judgement (agents asked to usage the champion technique), Kani, Lean 4, "Make nary mistakes", Metamorphic testing, Mutation testing, Property-based testing, Proptest, QuickCheck, rstest, Rust built-in trial framework, SMT solvers (with Z3, cvc5, and Yices, each available), Spin, TDD, TLA+, and Verus. Additional, 4 skills were tested: Hegel pinch the charismatic Hegel skill, the ECC Rust trial skill (ECC is simply a postulation of skills pinch 250k GitHub stars and 38k forks), the Trail of Bits spot trial skill, and a trial accomplishment I wrote (I'm a luddite who uses prompts alternatively of skills and person nary consciousness for really to constitute a bully skill). Other than my skill, the skills were chosen because those were the apical skills codex turned up erstwhile asked to find applicable skills.

Predictions

I pre-registered immoderate guesses connected really conditions will do:

• TDD will underperform (55% confidence)
  • I really added TDD specifically because I thought it would underperform
  • My assurance is debased present because I don't cognize what agents will do erstwhile instructed to do TDD; possibly agents won't do TDD and will do thing that doesn't underperform (or possibly I'm incorrect astir TDD underperformance)
• Formal methods will not overperform (52% confidence)
  • My thought present is that general methods are effective and useful (more truthful now than ever), bully trial methods are besides effective and useful and, connected elemental problems, general methods shouldn't outperform if utilized astatine a akin level of competence
  • As pinch the above, but moreover much so, my assurance is debased present because I don't cognize what agents will do erstwhile instructed to do anything, and general methods person been much hyped than effective trial techniques for agentic coding, truthful it's wholly plausible that labs person trained agents pinch RL environments pinch synthetic information which trains them to beryllium very effective pinch general methods without having trained agents to beryllium effective pinch bully trial techniques (which I would expect to beryllium easier to do, but not done because of really comparatively untrendy effective trial techniques are)
• Make nary mistakes will not outperform nary instructions (95% confidence)
  • It's a joke, and 1 that a batch of group person tried. If it worked, surely group would've noticed?
• The ECC trial accomplishment (with 250k stars and 38k forks) will not outperform (65% confidence)
  • It's somewhat large and doesn't person immoderate accusation I'd expect to beryllium useful. It instructs agents to usage TDD; to the grade that it gets agents to usage TDD, I'd expect this to make things worse (and it's much directive than the TDD information and possibly much apt to succeed, though for each I cognize that makes it little apt to succeed); the remainder of the accusation doesn't look useful and has immoderate cost
  • All of my accomplishment predictions are debased assurance because I don't thin to usage skills and don't cognize really to really measure them. I'm reasoning of this like, "how effective would it beryllium if I passed the matter successful arsenic a punctual and had this point floating astir successful the LLM's discourse window?"
• Hegel's accomplishment will not outperform (65% confidence)
  • It's very large (the SKILL.md positive the linked Rust reference are complete 20k tokens) and sounds much for illustration a tutorial than supplier instructions
• The Trail of Bits trial accomplishment will not outperform (55% confidence)
  • It has what looks for illustration it mightiness beryllium useful information, but it's besides reasonably big

Overall results

Below, we person a very messy chart which shows the results for the conditions tested (codex pinch GPT-5.6 Sol, pinch mean and xhigh efforts). When looking astatine data, I thin to for illustration overmuch denser and messier graphs than astir people, such arsenic the first chart here. Because astir group find these kinds of graphs unreadably messy, I thin to divided accusation retired into a bid of graphs, each of which shows little information, erstwhile presenting accusation to others. For reasons discussed elow, I'm not going to do this present and americium conscionable going to coming this highly messy chart wherever we person costs connected the x axis and the fraction of runs that passed 100% of the (hidden) tests connected the y axis, mean of 80 runs from each information and effort (mousing complete items shows bootstrap covariance, 50% uncertainty, and there's immoderate effort astatine making for illustration things akin colors, e.g., blue-ish for general methods, green-ish for property-based testing, etc.):

One point we tin spot is that thing really wildly outperforms. However, Default (no further instructions) does good supra average. Looking astatine xhigh, connected average, the fuzzing and PBT-related conditions did a small amended than general methods connected average, pinch the business being a batch much mixed astatine medium. The testing-related skills codex recommended we effort underperformed, though our speedy civilization accomplishment did good (a awesome quality is that our accomplishment is designed to nudge distant from their default behaviour towards much productive behaviors whereas the different skills look much for illustration tutorials). TDD didn't do well, arsenic predicted (one accomplishment besides suggested that agents utilized TDD, and that accomplishment besides fared poorly successful the cases wherever agents attempted to travel the instruction).

If we really look astatine what agents did, it quickly becomes evident that, successful general, agents don't cognize really to usage these devices aliases techniques very well. As we noted here, and arsenic everybody I've talked to has besides noted, agents are really bad astatine testing and don't look to understand really to trial reasonably "by default". For example, here's a remark by Gary Bernhardt:

AI agents' attack to testing, much aliases less:

1. Take the pathological cases dreamed up by personification objecting to mocks 15 years ago, without ever having really utilized mocks. Naive dreams of excessive mocking.

2. Make those pathologies the backbone of your testing strategy.

It turns out, if you inquire agents to usage a peculiar trial method aliases trial library, this attack doesn't alteration arsenic overmuch arsenic you'd hope. We'll look astatine what happened successful cases successful much detail, but astatine a precocious level, pinch trial techniques, agents thin to either conscionable constitute the tests they would usually write, but wrong a model for a different type of trial technique, aliases they'll usage a method superficially but not really do the things that get the worth retired of the technique. For the astir part, erstwhile a method was named, they did what Gary described, but pinch respect to that method (for example, for general methods, they mostly proved irrelevant properties and pinch property-based testing, agents would thin heavy connected wholly random inputs and heavy deed invalid/rejection cases aliases find a trivial spot to cheque and tally low-value random cases against the trivial property). Results weren't materially different connected the IMAP RFC (where I tried 40 runs of each condition) aliases different random RFCs (where I tried a fewer individual runs). In general, sloppy of the type of problem, whether it's immoderate benignant of spot manipulation problem for illustration Zstd, a protocol for illustration IMAP, aliases thing else, agents did not usage general methods aliases trial libraries aliases techniques successful an effective way.

On xhigh, agents were mostly capable to get the tests they wrote to pass, but they wrote mediocre tests (e.g., they'd taxable 4 identical bitstreams into a trial of a characteristic that uses 4 bitstreams and miss immoderate bug that would hap because they transposed bitstreams). And arsenic we noted antecedently connected the Zstd eval pinch respect to languages, moving astatine a little effort level successful a naive loop gets worse results (agents do moreover much of this and stall retired pinch little correctness).

I'm funny why AI labs haven't created RL envs to get agents to study really to trial good since package not moving reasonably seems important for coding supplier take and it besides seems for illustration the benignant of point that's amenable to RL. As we antecedently saw, agents person gotten rather bully astatine bounded runtime optimization problems, which makes consciousness because that's precisely the benignant of point you cheaply create a ton of RL envs to train on. Maybe this is 1 of those things that's harder than it seems erstwhile you effort it, but creating RL envs for effective testing and trial techniques seems for illustration it's successful the aforesaid people of problem. Perhaps the limiting facet is conscionable that knowledge of effective trial techniques isn't very widespread, truthful nary one's thought to effort it and group are getting agents to trial inefficiently (for example, by doing modular portion testing), aliases possibly this problem is overmuch harder to package up than runtime optimization for immoderate reason? It's imaginable this will beryllium a moot constituent soon if agents get truthful bully that they tin mostly constitute correct codification without testing aliases verification, but astatine slightest for the authorities of publically disposable agents from inception until now (September 2026), it seems for illustration agents having immoderate thought really to trial without being guided by a testing master would've substantially accrued agentic coding effectiveness.

Below we'll look astatine really agents did things for each condition, ordered from worst correctness to best, but I would be aware anyone against drafting immoderate benignant of beardown conclusions from the ordering.

A batch of the failures present look analogous to the failures we saw erstwhile we looked astatine the effect of programming connection connected token usage and correctness, successful that the failures are often idiosyncratic. For example, pinch programming languages, we saw that agents had a reasonably precocious complaint of getting the semantics of byte conversion incorrect successful Clojure but not Java, moreover though agents "should" (and astir apt benignant of do) cognize that they tin get Java byte conversion semantics by converting pinch unchecked-byte alternatively of byte.

Although group person each sorts of manus wave-y high-level explanations for why immoderate languages are amended for agents than others, erstwhile we look astatine what agents really do and what the nonaccomplishment modes are, nary of the explanations I've heard for why someone's pet language, whether it's Ruby aliases Ocaml aliases J, are really existent (with the objection of comments astir Rust's representation safety). Instead, we spot a bunch of idiosyncratic failures that hap for unclear reasons. With languages, because we tin observe a mean relationship betwixt connection fame and capacity (both little costs and higher correctness), it seems reasonable to conjecture that the logic is because location was much training information (possibly synthetic information and not conscionable human-written code) for much celebrated languages. Here, location isn't a clear pattern, different than that agents are mostly not very effective astatine applying trial aliases verification techniques erstwhile each they person is the sanction of a room aliases method (we'll talk what useful amended afterwards). If you don't want to publication astir what happened successful each condition, click present to skip to the past item.

Verus

Verus uses an SMT solver and various types of reasoning to beryllium that the codification matches specifications.

Although Verus tin beryllium that codification matches specifications, agents didn't do that. Instead, they made proofs astir various absurd properties relating to Zstd. I've not utilized a instrumentality for illustration Verus myself, truthful I can't speak to what an master aliases moreover a beginner personification would usually do, but from reference the tutorial, I find it a spot overseas that agents didn't effort to usage Verus to verify immoderate of the existent codification and only utilized it to do absurd reasoning, arsenic it seems designed to make it easy to beryllium properties astir the existent code.

Additionally, if we look astatine the properties proved, location were mostly fewer properties proved and the properties that were proved were uninteresting. For example, agents would beryllium things for illustration "given a valid cursor/index/distance, the resulting cognition remains successful bounds", which isn't bad to prove, but wasn't really a root of bugs. Also, agents would often constitute vacuous proofs that were efficaciously A => A. An existent Verus impervious of this shape was:

requires 0 < a <= window, 0 < b <= window, 0 < c <= window, ensures 0 < c <= window, 0 < a <= window, 0 < b <= window,

In cases wherever agents really proved something, they mostly proved thing comparatively elemental and avoided proving properties astir the parts that were apt to person a bug (for example, agents often grounded to reverse the bitstream bid for encode and decode and would constitute tests that grounded to observe this because the tests were palindromic; possibly immoderate benignant of impervious of reversal present mightiness get agents to "think" astir this successful a different way).

It doesn't look that agents were getting worth retired of Verus erstwhile conscionable provided pinch Verus and the Verus docs.

If we look astatine the result, the aggregate xhigh Verus results are good (slightly little correctness than average, but overmuch cheaper). The mean results had mean costs and the lowest percent of correct runs arsenic good arsenic the lowest mean number of correct tests. Because agents didn't really get worth retired of Verus, what they really did for correctness was mostly conscionable accepted tests (built-in Rust #[test] functions pinch portion tests). When going from mean to xhigh, agents walk overmuch much effort connected accepted testing and only a spot much effort connected utilizing Verus, which allowed the xhigh consequence to beryllium ok.

Looking astatine the existent tests, for 1 of the 2 features which agents utilizing Verus did overmuch worse connected (the four watercourse jump table), Verus agents wrote a trial for this successful 89 retired of 160 cases, coincidentally the nonstop aforesaid number arsenic Default agents, but Verus agents were overmuch much apt to constitute bad tests. They were much apt to encode incorrect results successful the tests arsenic good arsenic make easy to walk tests that don't screen the abstraction well, specified arsenic making each 4 streams identical. This benignant of point is what I meant erstwhile I said that the failures were idiosyncratic. There's thing astir Verus that needfully makes 1 constitute mediocre tests erstwhile not utilizing Verus and we wouldn't, successful general, expect a quality who's utilized Verus to constitute bad portion tests, successful the aforesaid measurement that we wouldn't expect a quality utilizing Clojure to make much byte conversion mistakes, but this happened present for immoderate logic (possibly a coincidence).

I don't cognize if folks wrong AI labs tin get entree to amended accusation connected why things happened, but present connected the extracurricular it's mostly rather difficult to show why thing for illustration this happened (even erstwhile we formed a plausible presumption for the connection issue, it required moving galore samples of galore languages, and papers we looked astatine which studied the aforesaid point didn't observe the connection fame / agentic effectiveness relationship because they either looked astatine excessively fewer languages to beryllium capable to logic astir specified a anemic relationship aliases they looked astatine problems that were excessively mini and excessively trivial).

Alloy

Alloy is often called a bounded exemplary checker. This is possibly not rather correct pinch Alloy 6 since that introduces immoderate other features, but this is measurement extracurricular of my area of expertise. My knowing is that, pinch Alloy, you usually beryllium properties astir your exemplary (as opposed to proving that your codification works).

Alloy sewage the 2nd worst correctness people and, unusually, scored mostly poorly connected some mean and xhigh. Although it isn't shown (because it doesn't look to adhd anything), successful general, results were highly correlated betwixt max and xhigh, which were rather different from mean results.

As we saw pinch Verus, agents utilizing Alloy beautiful overmuch relied connected modular Rust #[test] for correctness and mostly faffed astir pinch Alloy. Once again, utilizing a general instrumentality poorly did not thief pinch correctness.

There were individual cases of Alloy usage that were adjacent to uncovering an rumor aliases risk, but moreover then, only a mini number. In 1 case, Alloy recovered a counterexample which past caused the supplier to instrumentality the Rust type pinch a mitigation for the imaginable bug. Unfortunately, the counterexample relied connected an 8-bit overflow that couldn't hap successful believe because the existent implementation utilized 64-bit usize pinch nary anticipation of overflow fixed the inputs, truthful it conscionable made the codification much analyzable without preventing an existent bug.

In different case, the Alloy specification was incorrect and a related trial failed. After the trial failed, the supplier fixed the Alloy specification. Had the specification been correct, possibly the supplier would've written the correct codification without the failure. There were immoderate cases wherever it's imaginable the bully type of this happened, but it's not clear if an existent imaginable bug was prevented.

Alloy agents did exemplary things that were much intimately related to the Zstd algorithm than Verus agents (which mostly checked things for illustration arithmetic), but it was still the incorrect modeling.

Differential testing

Differential testing is simply a method wherever you springiness the aforesaid inputs to aggregate implementations and past comparison results to find issues. In principle, this seems for illustration a reasonable point to effort pinch LLMs arsenic we often get different results from different rolls of the dice, and as we noted here, having an supplier iterate much connected an implementation (which mightiness beryllium only portion of the full thing, possibly moreover only portion of a function) often useful worse than having the supplier restart from scratch.

But this gave america the 3rd worst results. In this case, we had somewhat supra mean results connected xhigh and acold beneath mean results connected medium. None of the agents created 2 afloat implementations to compare. Out of 160 runs, 135 did thing you mightiness telephone differential testing, but for illustration the different conditions we've seen, these were mostly trivial and efficaciously useless. And, successful the cases wherever differential testing might've caught a bug, alternatively of implementing things successful independent ways, agents conscionable did the aforesaid point doubly and encoded the aforesaid bug successful some versions.

I sometimes show agents to do things independently and get them to motorboat pinch abstracted contexts, but this was not done efficaciously for differential and agents would mostly conscionable constitute the aforesaid point twice.

Hegel Skill

It makes consciousness to talk really the charismatic Hegel accomplishment changes Hegel behavior, but successful reverse correctness order, Hegel Skill appears supra Hegel because the consequence was worse connected correctness. See the Hegel conception beneath for chat of this skill.

Lean 4

Lean 4 tin possibly beryllium described arsenic an interactive theorem prover.

Although I didn't pre-register a conjecture astir Lean, if I had pre-registered guesses connected which general devices would do well, I would've put Lean connected the database of things I'd expect to do good because it's comparatively hot/trendy and truthful seems comparatively apt to person bully capacity owed to synthetic information from RL envs.

The Lean agents did beryllium properties, for illustration the Verus condition, agents mostly did arithmetic proofs that didn't deed the bug-prone aliases consequence aboveground areas.

Like the different general conditions, Lean agents relied heavy connected modular Rust tests. As pinch the general conditions truthful far, doing a fewer proofs of things that don't matter didn't thief pinch correctness.

QuickCheck

QuickCheck is a property-based testing library, astir apt the champion known specified room for a agelong time, though Hypothesis mightiness presently clasp that crown.

Unfortunately, agents were astir arsenic effective astatine utilizing property-based testing arsenic they were astatine utilizing the general devices we've seen truthful far. When utilizing QuickCheck, agents mostly wrote very elemental "smoke tests" that didn't cheque much. They besides utilized random inputs, which, erstwhile afloat randomized, are beautiful mediocre for testing thing for illustration Zstd (because they conscionable spell down 1 of a fewer failure/rejection codification paths).

Also, comparatively fewer properties were checked. Although each agents utilized QuickCheck, 63 retired of the 160 runs only checked a azygous property. Agents erstwhile again mostly relied connected accepted testing, though they technically did usage QuickCheck. For immoderate reason, agents really wrote much accepted tests than nether the Default information aliases astir different conditions, but did less test-fix iterations (which resulted successful this information coming successful pinch beneath mean cost).

TDD

TDD underperformed present arsenic good arsenic successful the IMAP RFC eval.

The TDD punctual seemed to origin ample changes to supplier behavior. Agents produced doubly arsenic galore tests, and worked successful a overmuch much iterative test-code-test-code-etc. workflow, though a TDD advocator would astir apt opportunity that agents didn't really usage TDD. There were only a fewer instances of agents doing immoderate benignant of fine-grained iterative TDD.

Overall, agents wrote much tests up front; for example, agents had 1 aliases much failing tests successful 67 of 160 cases earlier doing important (non-stub) implementation, vs. 0 of 160 for the Default condition.

For wide trial classes, TDD had much tests of each kind. There were much small, trivial tests and location were besides much integration and end-to-end tests. Any benignant of evident high-level "agents did excessively overmuch aliases excessively small of X" doesn't look to fresh the data. If we look astatine circumstantial failures and really they were missed by tests, we tin observe that the TDD information had a number of these. For example, Zstd uses thing called a jump array erstwhile location are 4 Huffman streams.

TDD agents were much apt to neglect the eval trial for this though they wrote much tests that screen the wide case. For immoderate reason, TDD agents were much apt to constitute tests that don't screen difficult cases (e.g., making each 4 streams identical and past besides making them trivial, for illustration we saw pinch Verus). This is different lawsuit wherever I'd beryllium funny what benignant of visibility group astatine AI labs person since it's not evident from the extracurricular why priming agents pinch TDD made them constitute worse tests and worse implementations.

If we only had TDD and a fewer trial conditions to spell on, a presumption mightiness beryllium that TDD'd codification often seems to person a batch of mini tests that aren't very good, truthful possibly priming agents pinch TDD causes them to constitute much of these sorts of ineffective tests. But it's not clear why we should spot the aforesaid shape pinch Verus. Maybe we could show whether aliases not this is existent for TDD if there's a shared logic for the Verus (or other) behaviour by re-running the research connected an unfastened exemplary and inspecting what's really going connected wrong the exemplary astatine immoderate level?

Two of the skills besides caused agents to tally successful a much iterative approach, possibly connected the mentation that executing much often would springiness amended results, and some of those skills besides underperformed. In general, crossed each conditions, agents were capable to get the tests they wrote to walk connected xhigh and max (not shown, but max had somewhat amended correctness than xhigh astatine substantially amended cost). Getting their ain tests to walk much iteratively tended to get agents to constitute much incorrect tests that would enforce incorrect behavior.

Yossi Kreinin had this thought for why TDD mightiness consequence successful worse tests:

fwiw, I deliberation if you constitute the tests earlier the code, it's harder to trial the harder cases since you cognize little astir what is going to beryllium hard, and moreover if you do random testing which I don't deliberation "tdd" is associated with, you are little apt to steer the distribution successful the guidance wherever the bugs are. if you wrote the codification aliases astatine slightest tin look astatine it, you cognize what seems trivially correct and what mightiness aliases mightiness not activity since it's not easy to understand what it does. successful different words, tdd steers you towards achromatic container testing which for analyzable machinery seems to maine to beryllium little effective than achromatic container testing; beautiful judge this is really it useful pinch people, little judge astir agents

Was my conjecture that TDD would underperform correct? Strictly connected the result, the reply is yes. On my reasoning (not explicitly pre-registered successful writing, but I do cognize what I was thinking), I deliberation it's not clear. My reasoning was thing like, arsenic we've recently discussed successful a assortment of contexts, getting agents to really do thing for illustration the correct point and not conscionable overfit is simply a cardinal portion of achieving bully capacity aliases correctness pinch agents. Speaking to the methodology successful wide and not really this instruction changed supplier behavior, TDD seems primed to origin overfitting.

Agents did constitute worse tests and sometimes utilized a comparatively costly and ineffective iterative workflow, but I don't cognize that the nonaccomplishment mode I'd expect from a quality utilizing TDD and past directing agents to instrumentality was the existent problem here, and that problem was wherever my intuition came from. I would complaint the reasoning present arsenic possibly and possibly not successful the correct vicinity; I deliberation much evals and investigation would beryllium basal to determine this and I would conjecture that the consequence of further information would beryllium that my original reasoning is wrong.

Spin

Spin is simply a exemplary checker.

Now we're getting into the scope wherever results weren't acold from average. Spin did moderately worse than mean connected some mean and xhigh, astatine beneath mean cost. As we saw pinch the different general tools, usage of Spin was mostly ineffective. In this lawsuit specifically, utilizing Spin to exemplary a definite people of behaviour had nary relationship to passing aliases failing the hidden tests covering that behavior. Usage of Spin was superficial and not productive.

Hegel

Hegel is simply a property-based testing room based connected Hypothesis.

As we mightiness expect by now, agents didn't usage Hegel effectively. To the grade they utilized it, they utilized it superficially, and they mostly utilized it aft heavy relying connected mean testing. Since conscionable saying that agents didn't really meaningfully do the point is repetitive, I'll make these sections short and only item peculiar curiosities.

The existent workflow agents utilized was generally

1. Read the RFC and API/contract
2. Implement Zstd
3. Run normal tests
4. Read Hegel docs
5. Use Hegel to constitute 1-4 elemental property-based tests
6. Continue utilizing normal built-in Rust tests

As noted above, the Hegel accomplishment didn't amended correctness. Correctness was worse (though it was adjacent capable that this could've been random). What was much striking was that costs was overmuch higher (26% higher connected mean and 41% connected xhigh), for reasons which look causal.

The accomplishment caused agents to make much tests. The further tests were mostly checks that malformed inputs don't origin a panic and round-trip tests. The erstwhile is thing that agents were already inclined to do an excessive magnitude of for each of the property-based and fuzzing conditions, truthful further effort location wasn't useful. The second doesn't look for illustration an inherently bad thought (I moreover often explicitly instruct agents to create round-trip tests and they look to beryllium useful to cheque circumstantial properties), but it wasn't done successful immoderate of the astir bug-prone areas. Without further instruction, agents were inclined to create round-trip tests for comparatively trivial properties that were already apt to beryllium correct.

As for the cost, location are aggregate reasons for the cost. One is that the accomplishment is reasonably ample (34k characters for the skill, which besides loads a 45k Rust-specific reference, which ends up being much than 20k tokens). This was loaded astatine the commencement of the tally and was re-read connected galore consequent actions. This resulted successful an mean further dollar costs of 16% for mean and 18% for xhigh (by earthy tokens, the mean summation was 900k connected mean and 1.8M connected xhigh; though the cache deed complaint connected these was very high, 99.85% aft the first read, they were re-read capable that this was still a important fraction of full cost).

A multiplicative costs (this multiplier is included successful the erstwhile numbers) is that the accomplishment besides specified a system group of operations that origin a batch much activity to get done. This activity didn't summation correctness, truthful this accrued costs without a concomitant benefit.

One point to statement is that the accomplishment was "only" utilized successful 157 retired of 160 cases. As is mostly the lawsuit erstwhile utilizing LLMs, the actions and results are random. If you person a accomplishment disposable that you deliberation an supplier should usage for a peculiar task, it whitethorn aliases whitethorn not usage it depending connected factors that look opaque to group extracurricular of AI labs.

ToB skill

In this case, only 108 retired of 160 runs really opened the skill to publication it. The accomplishment suggests utilizing proptest successful Rust, but the accomplishment suggests support is required to adhd a dependency and these were each single-turn autonomous runs, truthful this wasn't done.

As pinch the different spot trial cases seen truthful far, spot testing was rudimentary and not done successful a adjuvant way.

Rstest

Rstest is simply a fixture-based trial library.

Agents efficaciously didn't usage rstest. They technically did usage it, but they beautiful overmuch conscionable wrote modular portion tests wrong rstest and didn't usage rstest arsenic intended, defeating the intent of rstest. While this is arguably existent astatine immoderate precocious level for techniques seen truthful far, agents were astatine slightest superficially utilizing immoderate of the different techniques (such arsenic penning immoderate low-value spot tests pinch Hegel), but present agents didn't usage the point that makes Rstest Rstest (the analogous behaviour for the property-based testing libraries would beryllium if they conscionable wrote non-property-based portion tests pinch them).

Rust test

This is referring to the modular Rust built-in trial model that agents utilized successful the Default information and besides very heavy relied connected successful the different conditions.

Explicitly asking agents to usage the built-in trial model resulted successful much tests (double normal connected medium, 25% much connected xhigh), but this didn't consequence successful amended correctness. When agents sewage things wrong, it was often because they didn't trial important behaviour aliases implemented incorrect trial behavior. Adding much tests didn't materially summation sum of risky behaviors aliases trim the fraction of runs pinch tests that encoded incorrect behavior.

Yossi Kreinin added:

i deliberation the fixed input/output style of testing encourages this successful machines and humans alike. if you make inputs you request to past person codification that classifies output arsenic correct aliases incorrect, and while this codification itself mightiness beryllium buggy, it astatine slightest makes you deliberation astir what correct intends and really to show if thing is correct much easy than moving the codification and assuming its output is the correct answer. pinch fixed outputs you are rather apt to conscionable encode the output of the codification and person yourself that it makes sense

Creusot

Creusot sits successful the aforesaid abstraction arsenic Verus.

As we've seen pinch the different general conditions, Creusot was not utilized effectively.

Mutation testing

Mutation testing involves modifying the codification to find really effective tests are and past adding tests to get bully coverage. Although mutation testing is simply a modular programming term, agents mostly didn't really do mutation testing and alternatively did normal testing pinch immoderate mini magnitude of mutating things successful a measurement that isn't really mutation testing, akin to really the TDD instruction modified behaviour but didn't get agents to do TDD.

There were a fewer cases wherever mutation testing occurred, but only a mini amount, and that was rare.

Judgement

This information asked agents to adaptively usage testing methods arsenic due based connected their judgement. Given what we've seen truthful far, unsurprisingly, agents mostly utilized modular Rust portion tests. A fewer agents did immoderate constricted fuzzing. Agents had entree to different trial and general libraries but didn't usage them.

Fuzzing

Fuzzing involves randomizing trial inputs successful immoderate way.

Agents relied heavy connected sending random bytes in, which mostly resulted successful going down the aforesaid codification paths (invalid input). Agents besides tried sending successful random variations of valid inputs, which mostly besides conscionable many times exercised input rejection paths.

On the uncommon juncture that agents generated random system inputs (10 retired of 160 cases), this recovered existent bugs half the time, immoderate of which were non-trivial cases. Using fuzzing a spot efficaciously successful 5 retired of 160 cases isn't precisely good, but this was 1 of the much effective uses of a method that we've seen truthful far. This besides seems to bespeak that agents could beryllium trained to do this amended and besides that that tin beryllium directed to do this amended without changes successful training. They do, successful immoderate sense, cognize really to do this; they conscionable don't usually really do it without being pushed into doing it.

Insta

Insta is simply a room for snapshot testing (sometimes called aureate testing), wherever you comparison results to a "snapshot" aliases "golden file" of correct results. Speaking generally, a snapshot is usually immoderate benignant of serialized data, e.g., it could beryllium a JSON entity of a information structure, a log of CLI output, etc.

As you mightiness expect, snapshot testing was hardly utilized and agents mostly relied connected accepted tests.

SMT

Agents were instructed to usage an SMT solver, pinch Z3, cvc5, and Yices installed.

Agents mostly utilized the SMT solver arsenic a benignant of scratchpad to compute things for illustration FSE authorities ranges, header arithmetic, etc. Even erstwhile agents modeled something, they would mostly not exemplary the correct point to debar a communal mistake.

For example, there's a computation that should've been byte1 + (byte2 << 8) + 0x7F00. Many agents implemented byte1 + (byte2 << 8) | 0x7F00 instead. Agents utilized SMT solvers to beryllium properties relating to this computation, but past still wrote the incorrect code, making SMT usage seemingly nary amended than Default (no instructions).

TLA+

TLA+ is simply a connection and instrumentality for modeling behaviors.

We're into the group of supra mean results (but still worse than Default) but, arsenic noted above, I wouldn't return the existent ordering excessively seriously. Though this isn't needfully significant, TLA+ did people a spot supra mean connected mean and much supra mean connected xhigh.

159/160 agents created immoderate benignant of TLA+ model, mostly a state-machine exemplary of Zstd. For peculiar coverage, 30 modeled Huffman/FSE/entropy (areas that often had bugs). As pinch the different general cases, TLA+ modeling happened comparatively precocious successful the travel (after a batch of modular tests and implementation). Agents sometimes recovered and fixed errors successful the TLA+ model, but I didn't find an lawsuit of a TLA+ rumor resulting successful an existent alteration successful the Rust code.

Although location was immoderate existent looking TLA+ modelling happening, if this improved correctness, it did truthful successful a mini measurement that was difficult to observe. In general, runs that had much blase TLA+ modeling did not person amended correctness.

Metamorphic testing

With metamorphic testing, we cheque that related inputs nutrient outputs pinch the expected relationship. For example, you could cheque that, for a benignant function, changing the bid of unequal inputs doesn't alteration the bid of the outputs, aliases for addition, adding a worth to an input adds the worth to the output modulo overflow.

As we've seen for the different conditions, Metamorphic testing wasn't done very usefully pinch respect to correctness. Some really reasonable properties were checked (e.g., inserting a skippable framework astatine a framework bound shouldn't alteration the output, ineligible artifact repartitioning shouldn't alteration outputs, etc.), but these didn't deed the areas that agents sewage incorrect comparatively often truthful checking these properties didn't help. In general, agents seemed to beryllium fans of the aged joke:

A policeman sees a drunk man searching for thing nether a streetlight and asks what the drunk has lost. He says he mislaid his keys and they some look nether the streetlight together. After a fewer minutes the policeman asks if he is judge he mislaid them here, and the drunk replies, no, and that he mislaid them successful the park. The policeman asks why he is searching here, and the drunk replies, "this is wherever the ray is".

Curiously, metamorphic testing was utilized little connected xhigh than connected medium.

ECC

The ECC Rust trial skill did ok, but mostly because ample parts of the accomplishment were ignored. Agents mostly opened and publication the accomplishment (153/160 publication it) and this seemed to origin them to make much tests. Not only did agents make much tests successful this condition, if we look astatine erstwhile agents publication the accomplishment (earlier vs. lateer vs. never), there's an exposure-based gradient successful really galore tests were added.

Although ECC scored almost arsenic good arsenic Default, based connected really agents did erstwhile much exposed to the skill, I would conjecture that this is random. The earlier an supplier looked astatine the skill, the much its behaviour was impacted and the worse the correctness result.

ECC seemed to do good successful position of earthy people because the 7 agents that didn't publication the accomplishment did unusually good and sewage a 100% correct result, and past the 9 agents that looked astatine ECC precocious and were only hardly influenced besides did good and had 100% correctness. This besides explains the different ECC consequence that mean had the aforesaid people arsenic xhigh (all but 1 of these runs wherever agents didn't really look astatine the accomplishment happened connected medium). While it's existent that location whitethorn beryllium immoderate benignant bias successful erstwhile the accomplishment gets invoked aliases not, the wide shape would bespeak that ECC is not effective unless you deliberation ECC acts arsenic a bully luck charm that improves results, but only erstwhile the accomplishment isn't really used, which is much apt to hap astatine little effort levels.

Of course, agents shouldn't beryllium influenced by a accomplishment they didn't look astatine and we should people this based connected the cases wherever the accomplishment was used. If we look astatine the cases wherever the accomplishment really influenced agents, ECC scores beneath mean (between Rust built-in model and Creusot), pinch a very akin nonaccomplishment mode to Rust built-in model of having a ample number of mini and not meaningful tests. The accomplishment tells agents to usage red-green TDD. The supplier behaviour astir apt isn't what a TDD practitioner would telephone TDD, but agents do constitute a mini trial earlier implementing functionality, which results successful a ample number of tests. As noted above, this isn't an effective measurement for agents to develop, truthful the consequence is worse than nary instruction and nary skill.

BTW, as we noted erstwhile we tried retired Caveman mode, there's rather a spot of variance and group are often misled into reasoning a accomplishment is useful by a fewer mini runs. In this case, we tried 160 runs of a skill, a reasonably ample number, much than immoderate reasonable personification would do. And yet, superficially, if we conscionable look astatine the score, ECC seems ok.

We would request a overmuch larger number of runs to mean retired the sound inherent erstwhile utilizing an LLM. We tin do what we did here, and inspect the results and usage our quality brains a small bit, but I seldom spot this done erstwhile group are talking astir nationalist LLM benchmarks, whether it's for skills aliases thing other (I did effort having LLMs analyse the results but, arsenic usual, moreover pinch existent nationalist SOTA models, the study was mediocre and afloat of basal reasoning errors). Instead, I mostly spot group walk astir the top-line number, moreover erstwhile it's not meaningful for boring statistical reasons or, worse yet, the benchmark is fatally flawed, arsenic we saw pinch Senior SWE-Bench.

Default

Default gave the supplier nary trial aliases verification instructions.

Given what we've seen truthful far, it's not astonishing that Default scored supra average. Agents mostly did things that were not useful erstwhile asked to usage peculiar libraries aliases usage peculiar trial techniques. It stands to logic that not telling agents to do things that will make them do useless activity does amended than telling them to do things that will make them do useless work.

Audit

Audit asked agents to audit the codification aft implementation. 152/160 agents really did this and 151 agents claimed find an rumor and past made a alteration arsenic a consequence of the audit. Agents mostly picked reasonable areas to audit, but usually didn't do an independent audit pinch a caller discourse (which I will often inquire agents to do) and often conscionable made the aforesaid correction successful the audit that they had already made.

42 utilized an independent agent, but these runs really scored worse (it's imaginable this isn't causal and agents decided to rotation disconnected an independent audit because they were successful a worse aliases harder situation). Audit ended up pinch the champion correctness connected xhigh, but beneath mean correctness connected medium, and each of this auditing substantially accrued cost, particularly connected xhigh. On average, Audit did astir arsenic good arsenic Default and it's not clear if it's really amended connected xhigh and worse connected medium. That would beryllium plausible, but I don't deliberation we person capable grounds to tell.

Em Chu had the pursuing comment:

The results present are accordant pinch my experience. Auditing codification is wherever astir of my tokens spell astatine the infinitesimal because I find it rather useful. I ever springiness the 2 instructions though:

• Don't spawn subagents; publication and understand the code/diff yourself
• Don't execute immoderate of the code

because I find the LLM to beryllium importantly dumber if you fto it do either of those (though of people I haven't measured...). It really doesn't publication aliases logic astir codification by default, moreover if I'm ne'er making a alteration large capable to transcend its discourse window.

I besides usually see immoderate BS for illustration "be adversarial" "consider each imaginable combinations of features" "consider the full input space" but I'm little judge that helps astatine all.

It would beryllium absorbing to effort that, but arsenic I've noted successful my caller posts, I'm trying to spell into little item successful posts, truthful possibly that will beryllium a taxable for different post.

Audit and fuzz risky areas

For Zstd, erstwhile this instruction was followed, it caused agents to attraction heavy connected FSE, Huffman, spot readers, and state. These were areas where, successful general, agents often missed issues, truthful agents were correct to deliberation that these areas were risky. The areas that were targeted for fuzzing were amended choices than the plain Fuzzing condition.

On mean effort, agents mostly ignored the instruction and didn't do it, but they did travel instructions connected xhigh. While this information didn't execute poorly, it didn't look to do amended than nary instructions.

When looking astatine what agents really did, 1 rumor was that agents often conscionable generated a bunch of random inputs which were mostly invalid and wouldn't trial immoderate absorbing condition.

When a quality tester generates randomized tests, they'll mostly effort to target the randomization successful a measurement that generates "interesting" inputs and agents grounded to do that. Agents besides didn't cheque outputs very efficaciously and, successful galore cases, only looked for crashes. Fuzzing is often associated pinch only checking for crashes and not checking for properties, truthful this is possibly not excessively surprising, but it's astir apt not what a quality would want if they were testing a Zstd implementation.

Make nary mistakes

Although this technically scored higher than Default, behaviour didn't look to beryllium meaningfully different and the scores are rather close; I would conjecture that this is owed to random variation. At each level astatine which I looked astatine the results, they were indistinguishable from random draws of Default.

Kani

Kani is simply a Rust model checking library.

In position of "actually utilizing a general method connected the codification that will execute", Kani had the champion sum successful that Kani was really utilized connected the Zstd code. However, that only happened occasionally and astir usage was superficial.

There was 1 lawsuit wherever existent Kani usage caught a non-trivial bug that caused Rust codification to change. 1 retired of 160 isn't amazing, but it does bespeak that agents tin stumble into utilizing Kani reasonably sometimes (which, I would guess, intends that, if utilized successful an RL env, models could study really to usage Kani much effectively).

Kani had noticeably higher costs than different conditions. This seemed to beryllium because reference Kani output many times was expensive, which resulted successful a precocious input token cost.

ACL2

ACL2 is simply a theorem prover. One point to statement astir the consequence present is that, successful galore cases, ACL2 OOM'd (192 GiB limit). OOM results weren't counted, which biases the results successful immoderate opaque way.

Although ACL2 scored higher than Default, I deliberation it would beryllium astonishing if this was causal and significant. As we saw pinch almost each of the different general methods, ACL2 was mostly utilized to beryllium things that didn't importantly effect correctness, truthful it's not clear why this would amended correctness.

With galore different conditions, we wouldn't expect Default aliases the seemingly balanced Make nary mistakes to beryllium astatine the apical unless different conditions had severely degraded performance.

Proptest

Proptest is simply a property-based testing library.

Just arsenic we saw pinch the different randomized testing, astir tests weren't very interesting, and a too-heavy reliance connected randomness caused mediocre coverage.

Despite mostly mediocre usage of property-based testing, proptest's shrinking (finding a simpler input that causes a trial failure) did sometimes supply immoderate value, which is amended than the small to nary worth we saw successful astir different cases.

Property-based testing

As pinch the different technique-based approaches, agents had a instrumentality pinch each options installed. Every supplier chose to usage proptest, truthful this efficaciously became a 2nd proptest condition.

As pinch the proptest condition, tests were mostly not very bully but they did sometimes find bugs and shrinking seemed to make immoderate wins.

I find it mildly absorbing that this 2nd "accidental" proptest limb besides scored good supra average, conscionable for illustration proptest.

Skill

Here, Skill refers to the accomplishment I wrote to trial having a elemental accomplishment (as opposed to the large/complex skills that were what I recovered erstwhile I asked an supplier to find applicable testing skills).

Maybe I should usage skills, but I mostly don't and alternatively trust connected prompting and seeing what happened and past prompting immoderate more. As a result, I person nary intuition for what makes a bully accomplishment since I don't person immoderate believe astatine it, but Max Bittker suggested that it would beryllium absorbing to spot the consequence pinch a trial accomplishment that attempts to encode immoderate accusation I person successful my caput astir testing. On seeing the consequence of this, he had an "I told you so" reaction.

I didn't constitute a pre-registered conjecture astir this, but the conjecture successful my caput was that this wouldn't activity well. From my effort astatine conveying this to humans successful 2015, which I would opportunity beautiful overmuch failed, I don't deliberation I'm bully astatine explicitly laying retired really personification should trial successful writing. I've sat down pinch group and showed them what to do, which has mostly converted them for life and turned them into way supra average bug finders, but being capable to convey thing by showing personification is simply a different (and easier) accomplishment than conveying it by penning down really to do it.

In this case, the accomplishment was:

• Think astir areas apt to person subtle bugs earlier implementing; for each, authorities apt mistakes and plausible replacement interpretations, past travel up pinch a cheque wherever the results disagree (prefer asymmetric / bound examples connected some sides of the boundary)
• After implementing, for precocious consequence areas, independently re-derive the consequence without discourse connected accumulation codification and comparison (fresh context, do not re-use helper functions)
• When feasible, usage property-based testing aliases randomized inputs to effort to research the space, minimizing effort connected no-panic aliases no-crash randomization
• When randomizing, thin towards inputs that will research absorbing authorities and codification paths (don't conscionable naively randomize inputs that each autumn into the aforesaid correction paths); this whitethorn require system random inputs
• If you're unsure astir details, usage independent reasoning to cheque what's correct (fresh context, do not re-use helper functions)

This sewage the highest score, but didn't activity arsenic intended. It didn't really do the caller discourse point almost ever, truthful it was pointless to person that successful location and we don't cognize if that's thing that's effective that needs to beryllium refined to unit agents to do it much often aliases if it's thing that should beryllium removed (while it's technically imaginable it's happening astatine the optimal frequency, I highly uncertainty it).

We noted successful "Audit and fuzz risky" that agents seemed to cognize really to place risky areas. This was existent present arsenic well, but this didn't needfully mean that agents did the correct thing. For example, agents identified bitstreams being reversed for encoding vs. decoding successful Zstd arsenic being risky, but agents didn't do amended connected tests that exercised this. If we look astatine circumstantial examples, for mean tally #35, an supplier identified this arsenic risky, did independent derivations and an audit, but still failed. It had a applicable test, but the input was palindromic, truthful reversing the bid gave the aforesaid result, allowing for a failing implementation that had this backwards.

Another issue, if we tin telephone it that, is that each of the fuzzing / property-based testing was done "by hand". Given that agents look good astatine utilizing proptest and that proptest has immoderate useful machinery to thin on, this accomplishment could astir apt trivially beryllium improved by instructing agents to usage proptest. The instructions to agents that were intended to minimize the modular nonaccomplishment mode of generating galore useless "too random" tests directionally worked and a larger fraction of agents generated somewhat meaningful tests, but the tests were still worse than I'd expect a quality to constitute (or an supplier pinch progressive quality guidance). Without iterating connected this, I'm not judge what generic guidance would beryllium bully (as opposed to spending a fewer minutes looking astatine the building of Zstd and giving Zstd-specific guidance, which is 1 benignant of point that's worked good for maine connected different problems).

As a first draught for a accomplishment to iterate on, I don't deliberation this is horrible, but I don't deliberation it's really fresh to usage either. I could spot an improved type of this moving if it were tried pinch galore much examples to make judge location isn't overfitting to RFC-like problems, bit-manipulation-intensive problems, etc., but, since I don't usually make skills and haven't ever tried to iterate connected one, this fails to seizure what I aliases different quality would do if really driving an agent.

Since I'm utilized to prompting and past looking astatine the consequence (not needfully the code, but astatine slightest what agents opportunity they did and immoderate benignant of agentic summary of what happened, and parts of existent results for immoderate kinds of experimental work) and past re-prompting based connected that, I'm not utilized to front-loading information, which is simply a reasonably different problem than reacting to information. From erstwhile fuzzing work, I've seen nonaccomplishment modes that agents often autumn into and the accomplishment was intended to forestall those nonaccomplishment modes, but it's easier to do this if you cheque backmost successful moreover occasionally than to do it afloat up front, and the up beforehand instructions weren't capable to extremity the modular nonaccomplishment modes, though they did mitigate them somewhat.

As we noted above, I didn't effort to break down the information successful a nice, easy to look astatine way. I didn't do that because, erstwhile we look astatine what agents really did, it seems for illustration they were mostly beautiful ineffective and I don't deliberation it's peculiarly absorbing to spot really good "agents utilizing Verus badly" do compared to "agents utilizing QuickCheck badly". One point that I find a spot absorbing is that, erstwhile asked to place areas that are risky aliases prone to subtle bugs, agents were capable to do that.

But, successful general, sloppy of the room aliases method suggested, agents grounded to usage the technique. As previously discussed, conscionable asking agents to "test" aliases many times asking them to trial much results successful mediocre testing. It turns retired that asking them to usage trial techniques (some of which I've personally recovered to beryllium highly effective) besides results successful mediocre testing. The speedy and soiled accomplishment I wrote seems for illustration it could amended things a bit, but would request much than the 2 minutes I spent connected it to beryllium really useful. Yossi Kreinin made the remark that the authorities of package testing is atrocious, truthful we should expect mediocre results if agents autumn backmost to their training, truthful to speak, which is what we saw.

For immoderate reason, agents seemed to beryllium somewhat amended astatine utilizing proptest, though the level of testing was good beneath what I'd expect retired of a reasonable quality who's publication the proptest manual and is fixed immoderate guidance connected really to test. I'd beryllium funny if agents that are fixed much guidance are much effective pinch proptest than pinch different libraries, but that's a taxable for different station arsenic I've been trying to get posts retired successful half an hr and we're approaching 9000 words here, which is beyond a reasonable magnitude to effort to type successful half an hour.

How do you get agents to constitute bully tests?

My acquisition has been, if you guideline agents to group up a reasonable trial and triage structure, getting agents to adhd to that efficaciously without a immense magnitude of supervision useful ok-ish. Because of my inheritance (bias), the benignant of testing I thin to thin connected is immoderate shape of randomized testing / fuzzing / property-based testing.

I talked to Jamie Brandon astir this, and he's recovered the aforesaid pinch snapshot testing. He mentioned that, connected 1 project, erstwhile he asked agents (using a assortment of models) to do snapshot testing, they would opportunity that they were doing it and past conscionable wouldn't do it (they would constitute a portion trial and past opportunity they wrote a snapshot test). On a different project, he was capable to get them to constitute reasonable end-to-end tests pinch mocked IO, but only aft moving the tests into a abstracted crate and putting instructions successful AGENTS.md to support tests successful the crate and not modify the nationalist interface.

At slightest to date, I've been leaning much heavy connected getting agents to constitute the trial codification than Jamie (my inclination has been to type to agents successful a CLI; astatine slightest for now, he favors penning codification by manus a batch much than I do), but it doesn't look to matter really you do it arsenic agelong arsenic you group up immoderate benignant of reasonable structure.

Similar to this earlier problem we looked at, it seems for illustration doing thing remotely reasonable works. If you "talk to" an supplier and springiness it ray instructions for illustration we did for the Zstd aliases IMAP evals, the supplier will do mediocre work. But if you look astatine what it does and type a fewer much sentences, you tin often get it to a bully spot beautiful quickly (or that's what my acquisition has been connected different problems, anyway).

I'd beryllium funny if this tin beryllium efficaciously packaged up into skills, aliases if AI labs are going to commencement training models to get amended astatine testing aliases general methods, aliases if immoderate they're doing that doesn't straight amended those things will still amended those indirectly capable that agents will constitute decent tests without overmuch supervision aliases structure.

Prediction accuracy

• TDD underperforms (55% confidence)
  • True
• Formal methods do not outperform (52% confidence)
  • True, but not for the logic I expected. Agents grounded to usage them remotely effectively, truthful of people they couldn't outperform
• Make nary mistakes doesn't outperform nary instructions (95% confidence)
  • True; outperformed astir conditions because a no-op is amended than getting agents to do ineffective things
• ECC accomplishment will not outperform
  • True; I deliberation I would've had much assurance successful this if I utilized skills more, since astir of the accomplishment matter seems for illustration it won't do overmuch of anything, and the matter that seems for illustration it will do thing looks counterproductive
• Hegel accomplishment will not outperform
  • True; different 1 wherever I would've had higher assurance if I'd utilized skills more, since this didn't activity good for the logic I guessed; I conscionable didn't person immoderate assurance successful my emotion astir this
• ToB accomplishment will not outperform
  • True

Guesses I didn't register, but I could show I held implicitly because I was amazed erstwhile I saw the result:

• Lean will do comparatively good among general methods
  • False
• My accomplishment will beryllium mediocre to bad
  • False, moreover though Max Bittker correctly guessed what would happen, told maine this successful advance, and named a logic that's accordant pinch what happened

Skills

At various times, I've felt for illustration I person a bad/antiquated/ineffective workflow because I perceive group are doing thing and I've been excessively lazy to effort it out. I've felt this measurement astir skills for a while since I don't really usage skills. Instead, I support a ample scratchpad of things that I sometimes copy+paste successful arsenic prompts, which benignant of feels for illustration the balanced of commenting retired blocks of codification to prevention them alternatively of utilizing type control.

But past I saw this talk by Thorsten Ball, wherever he mentions he doesn't trust heavy connected skills, and I talked to a mates group who look comparatively effective pinch LLMs who besides don't really usage skills, and it made maine wonderment if I'm not missing retired connected much.

Then I tried this experiment, wherever my emotion was that the skills I looked astatine weren't going to thief and are astir apt really going to hurt, pinch debased assurance since I don't cognize thing astir skills. The skills did beautiful overmuch what I thought they would do, truthful it turns retired the intuition I person from conscionable seeing really agents respond to things and moving a bunch of small experiments seems to clasp up good for skills. I besides looked astatine a number of different skills that allegedly amended testing which I didn't see successful this research that looked for illustration they would person the aforesaid nonaccomplishment modes arsenic the skills we tested.

I besides ran 2 experiments (details not discussed here, possibly successful different 10k connection station different time) connected immoderate different skills that are "official" skills that companies person to support their product. One is from a large AI laboratory and the different is from a "small" fewer cardinal dollar company, but successful some cases, the skills made results worse, conscionable for illustration we saw here. Funnily enough, aft these experiments, I'm really much bullish connected skills for individual usage than I was earlier since the nonaccomplishment modes look predictable and truthful fixable without a immense magnitude of costly experimentation. Creating a publically released accomplishment that's intended to beryllium really good, activity good crossed different models and harnesses, etc., seems for illustration it mightiness beryllium difficult (claude and codex look to "want" different styles of prompting, truthful of people that should beryllium existent for skills arsenic well), but conscionable addressing the issues that origin a batch of skills to beryllium worse than nary accomplishment for individual usage seems rather doable?

Naive thoughts connected accomplishment writing

I don't cognize capable astir skills to opportunity really to constitute a bully skill, but pinch each the skills we looked astatine successful this station (except for the 1 I wrote successful a infinitesimal aliases two) and the skills from these different 2 experiments, the skills seemed written for illustration they're quality tutorial instructions, successful that the extremity of the accomplishment seems to beryllium to explicate really to do something. My naive thought arsenic personification who's written each of 1 accomplishment is, I'd conjecture that this isn't optimal erstwhile moving pinch a exemplary that should already person immoderate knowledge of the taxable (which was the lawsuit present and successful the different experiments arsenic well). The exemplary is already going to person immoderate benignant of default behaviour distribution, truthful I consciousness for illustration the much earthy point to do is to springiness statements that will modify that behavior, not constitute instructions that would let a quality aliases non-knowledgeable supplier to do the behaviour astatine all.

One evident problem is that we get different default behaviors from different harnesses, models, and effort levels, but throwing a bunch of matter into a punctual aliases a accomplishment doesn't really alteration this; that's conscionable a longer measurement to push the supplier distant from its default, pinch a batch of matter that whitethorn do immoderate benignant of unintentonal pushing. As we saw here, overmuch of the matter conscionable gets ignored (and what gets ignored and erstwhile is of people harness, model, and effort dependent). For example, for the ECC skill, moreover erstwhile it was read, had astir instructions ignored, and though the TDD instructions were influential (which made results worse), the instructions to do TDD were still not really followed contempt them being laid retired clearly.

Since what benignant of prompting is effective changes capable betwixt exemplary releases and effort levels, for thing wide for illustration "testing codification well", it's not clear to maine really these skills are expected to activity crossed truthful galore models and efforts. Just going from GPT-5.5 to GPT-5.6 changed really I worked substantially because a number of things that worked reasonably reliably pinch GPT-5.5 either stopped moving aliases became overmuch little reliable (even though, overall, the level of capacity seems higher). In the aforesaid measurement that I don't punctual GPT-5.6 the aforesaid measurement I prompted GPT-5.5, I don't deliberation I'd want to usage the aforesaid skills.

I'm not judge who, different than personification astatine an AI lab, would really spell done the problem of moving evals connected skills to spot what's effective for each exemplary and effort level and past create a portfolio of skills that are differentiated by exemplary and effort, and I wouldn't expect AI labs to person skills that are optimized for their competitors' harnesses and models, truthful I don't cognize astir things for illustration generic "testing" skills (as noted above, rather a fewer testing skills that I looked astatine but didn't trial present looked for illustration they would person the nonstop aforesaid nonaccomplishment modes arsenic the pre-existing skills we tested), but I could spot having a fewer skills that activity for my ain usage cases pinch the circumstantial harness/model/efforts that I thin to scope for.

Just reasoning astir testing, while location are peculiar pitfalls that definite models autumn into astatine definite effort levels that I want to nudge them distant from, location isn't really a generic trial workflow that I want to springiness agents that's independent of the point being tested and the level of value I want from the point and the dimensions successful which I want quality, truthful I don't deliberation I'd want a generic trial accomplishment that lays retired a group of testing steps that agents should, successful general, do. Something for illustration this goes for a batch of task that I do, which I want done successful task-specific measurement and not a generic way. I could ideate immoderate benignant of accomplishment that asks maine questions and past emits the correct instructions to agents, but fixed really accelerated models are improving, if I'm making thing for individual use, I don't deliberation it makes consciousness to walk clip tweaking a accomplishment for illustration that until it's useful. If I was moving connected an agentic merchandise and wanted much group to usage it, that mightiness beryllium a different story, but the skills I've tried person had the aforesaid nonaccomplishment modes arsenic the skills we tested here, truthful it seems reasonably easy to make a accomplishment that turns retired to not beryllium that effective.

I could spot skills being generically useful for things for illustration school agents really to execute workflows aliases really to interact pinch APIs/interfaces, specified arsenic Sawyer Hood's accomplishment that helps agents thrust a web browser. Since I haven't tried that skill, I'm not endorsing it, but from reference done it, it seems for illustration the benignant of point that would activity good and prevention maine a batch of hassle erstwhile I'm trying to get an supplier to thrust a web browser. However, if you publication the existent accomplishment (and scripts), it has a very different style than the testing skills we tried here.

Thanks to Max Bittker, Yossi Kreinin, Em Chu, Dennis Snell, @panoramic.blue, and Jamie Brandon for comments/corrections/discussion.

P.S. I've had this statement connected my past fistful of posts indicating that I'm trying an research wherever I constitute up half-baked (barely fixed/audited/cleaned up) results arsenic quickly arsenic imaginable because agents fto you tally experiments truthful quickly that I different wouldn't constitute thing up astatine all. I really ran this research instantly aft I ran the programming connection token costs / correctness experiment, but I haven't had clip to constitute this up because I wanted to constitute up this creation of a regex motor pinch an expert and a autochthonal codification compiler, this experiment pinch moving a forked type of ripgrep that uses the autochthonal codification compiler connected codex's ripgrep queries, and a fewer different things I haven't had clip to constitute up; I've had a extremity to do each of these write-ups successful half an hour, but I'm still falling beautiful acold down successful position of experiments I've tally vs. what I've written up.

For a mates years, I was moving experiments for illustration this and conscionable telling a fewer friends astir funny results and past moving connected without really ever talking astir these things publicly. If you person opinions connected these quicker (and little quality) experiments and write-ups, fto maine cognize what you deliberation (X Bsky Mastodon)!

Appendix: immoderate responses

David R. MacIver said:

Regrettably, @danluu is right. The Hegel accomplishment sortof sucks correct now.

I deliberation the communal problem pinch a batch of supplier skills is that agents suck astatine penning supplier skills and besides everyone (including us) uses an supplier to constitute their skills.

A communal friend mentioned that MacIver (later?) group up a benchmark and confirmed the deficiency successful the Hegel Skill and is presumebly moving connected either improving the accomplishment aliases getting set Hegel truthful the accomplishment isn't necessary. If the only point that comes retired of this station is that Hegel gets an improved skill, I deliberation that would already beryllium beautiful awesome. As antecedently discussed, I deliberation measuring and benchmarking are underrated. I've had much than a fewer posts that person driven immoderate benignant of alteration conscionable by showing wherever there's a gap. It's bully that this is different 1 of those posts.

Appendix: supplier silliness

After asking an supplier to do a elemental lookup of thing for an analysis, it exec'd a perl process that ran for 2 hours and 20 minutes earlier I killed it (I really request to person thing that automatically catches things for illustration this, because it's reasonably common).

A subagent utilized perl to do a regex hunt complete a comparatively mini record (44kB, 1364 lines), but the look was degenerate pinch PCRE and did a combinatorially ample magnitude of work. I tried re-running this pinch the FRE regex motor we tried building successful a fewer minutes here and it vanished matching successful 0.7s (Rust regex was 0.6s).

You ne'er cognize what's going to hap erstwhile agents are disconnected doing things, but location are 3 reasons this ne'er should've happened successful the first place. First, the supplier shouldn't person invoked a regex motor that tin springiness you a combinatorial detonation for illustration this; there's nary logic not to usage a safer regex motor for this (such arsenic ripgrep astatine default settings). Second, the look was wrong; the existent regex returns a uselessly ample seizure and doesn't do what's intended erstwhile it succeeds. Third, why did the subagent (or the harness) not automatically termination this aft the subagent finished? Of people this shouldn't hap for everything a subagent runs, but agents often time off runaway processes for illustration this lying around.

I really person a process that goes astir cleaning up aft things agents time off lying astir (agents that, themselves, leak memory, impermanent build artifacts that devour space, etc.), but it wasn't looking for runaway perl processes. That's different 1 to add, but surely I'm not the only personification who's tally into this problem. I conjecture I could unfastened root my silly instrumentality for this, but it should go obsolete erstwhile the awesome harnesses hole this, truthful location doesn't look to beryllium immoderate bully logic for anyone to moreover prime up my point successful the first spot were I to unfastened root it.

Appendix: experimental details

In the liking of penning this quickly, I'm going to punt connected this (sorry!). The distribution crossed conditions wasn't fundamentally different than we saw present erstwhile we looked astatine really connection impacts correctness and token cost. Somehow, this station I wanted to constitute up quickly successful half an hr is almost 10k words, which is decidedly much than half an hr of penning (10k words successful half an hr would beryllium complete 300 words per minute).

One point I'll statement is, just for illustration pinch this station connected programming languages, location were a mates of things that looked for illustration really interesting/compelling results (at slightest from the standpoint of conscionable looking astatine the top-level chart and seeing if thing stands out), but connected looking much intimately astatine those, they were owed to an experimental correction caused by giving a short punctual to agents to group up the experiment. On fixing those errors, we sewage a overmuch much boring antagonistic result, isolated from for the portion wherever the skills codex suggested mightiness beryllium useful seemed to beryllium counterproductive.