Which tools do Claude, Codex and Cursor choose? We measured 17k runs to find out

Sep 04, 2026 04:20 AM - 1 hour ago 2

Research · September 3, 2026 · The Armature team

Disclaimer: Armature sells maturation services to dev tools. This study is portion of our broader activity connected really to power coding agents choices and get products picked.

As agents return complete much and much parts of the coding journey, location is 1 circumstantial portion everyone outsources to their agent, from vibe coders pinch nary package inheritance to elder engineers: selecting which work to instrumentality for a circumstantial request successful an existing codebase.

Let’s return the illustration of selecting a database:

A vibe coder builds a individual recreation app and realizes the app resets astatine each connection. They inquire Claude Code:

I request you to shop what I input successful the app location truthful that adjacent clip it’s still location erstwhile I reopen the app

Claude Code analyzes the codebase and answers 5 minutes later:

You request a database and Neon fits good because it has a free tier, is elemental to instal and won’t region your app for illustration Supabase does if you don’t usage it excessively often.

The personification accepts and the agents installs it. Done.
A elder technologist moving connected a accumulation app asks Cursor:

What is the champion database solution for this app, it should person predictable costs and beryllium afloat managed.

5 minutes later, the conclusion is the same, Neon is the proposal pinch clear reasons why competitors don’t fresh the need. The technologist approves and the supplier implements.

This is really an research we ran. Two sandboxes, different agents, different codebases, different personas and prompts, aforesaid conclusion. So we wondered: if we generalize the trial to different instrumentality categories, making discourse / codebases / personas alteration moreover more, will the consequence change?

It is an important mobility for developers trying to cognize if they tin spot their agent’s judgement connected what genuinely fits their needs. But it matters moreover much for vendors whose endurance will soon dangle connected getting picked by coding agents (last April, Vercel shared that “over 30% of deployments were initiated by coding agents, up 1000% from six months ago”).

That’s why we decided to tally the largest research ever to understand really coding agents deliberation astir tools, really they observe and prime them and which 1 ends up winning successful each category. We watched almost 17k sessions crossed different types of personas (e.g., vibe-coders, inferior engineers successful startups, elder developers astatine enterprises) pinch 1,163 punctual variations, 75 repositories and 3 coding agents (Claude Code, Codex, Cursor) really implementing the solutions alternatively of conscionable recommending one.

Today we are sharing everything: aggregated results and leaderboards per category, but besides each study and moreover the full traces pinch the personification prompts, reasoning traces and existent codification diffs applied by the agent.

You tin commencement exploring the results correct here, aliases proceed reference the article below

How did we tally each these experiments concretely?

Our sheet of repositories

We started by moving an study complete thousands of nationalist GitHub repositories from which we extracted statistic astir programming languages & frameworks, third-party services, deployment platform, squad sizes, and codebase age. Since Tech startups are much apt to person open-source repositories than ample enterprises, and stacks are apt very different we past unbiased our statistic based connected publically disposable information and reached our perfect sheet distribution.

We past staffed various coding agents to create real-world repositories to lucifer these nonstop requirements. Finally, we generated variants successful which we removed parts of the codebases and pinch them, full third-party work implementations truthful we could tally due unbiased experiments.

We landed connected 75 repositories, successful 10 languages, each utilizing clone institution names, clone git histories, clone API keys and existent lockfiles checked against package head registries for illustration npm.

Real-world tasks

Each research is simply a existent task to beryllium performed wrong a repository, asked by 1 of the pursuing 4 profiles:

Vibe-coder: only describes symptoms and perfect state, seldom the instrumentality class name
Junior engineer: usually mentions the desired authorities and the class name
Senior engineer: is much precise astir requirements and things to avoid
Engineer astatine a ample enterprise: specifications circumstantial constraints, compliance, procurement, etc.

Prompts are mostly elemental and nonstop and somewhat tailored to each research (taking into relationship connected the repository and the persona) but successful 20-25% of the cases we tested adding circumstantial mentions to the prompts for illustration costs aliases usage measurement to trial their effect connected the last output.

We ended up pinch 1,163 variations for illustration this one: “Now I request that each invoice that we make gets sent to the user’s email reside pinch a bully message, find the champion solution and instrumentality it”.

Runner

Each research is tally successful a dedicated ephemeral sandbox. We verified that the prime of the sandbox didn’t effect the conclusions but conscionable to beryllium safe we decided to rotate betwixt 3 different sandbox providers (namely E2B, Blaxel and Daytona).

A “simulated human” successful the loop

Since real-world conversations are seldom conscionable 1 punctual and an supplier moving continuously connected its extremity pinch nary interruption, we decided to usage a “simulated human” successful the loop. We achieved this utilizing an orchestrator, played by Gemini 3.7 Flash. This allowed america to play much realistic scenarios wherever the supplier would beryllium first asked to analyse the codebase and urge the champion solution. At this shape the simulated quality would ever spell pinch the apical 1 solution aliases inquire the coding supplier to take the champion 1 and instrumentality it. But we noticed that asking astatine the opening to instrumentality without returning immoderate mobility would bias the supplier towards building everything in-house arsenic it was not capable to inquire authorization to prime a circumstantial third-party solution. Adding this “human” successful the loop reduced the leaders & unreality platform-native solutions power towards a much realistic picture.

For illustration successful the entity retention experiment, Cloudflare R2 started winning successful sessions successful which the supplier would ever usage Amazon S3 before.

Our judge

Another lawsuit of Gemini 3.7 Flash was utilized to analyse the sessions. Its domiciled is twofold:

Assess if a convention is valid regarding a database of criterias, e.g., the prime wasn’t biased by a repository that already “pre-chose” the provider; a solution was really chosen (for observability it would cull OpenTelemetry unsocial if not coupled pinch a platform).
Identify each subordinate that was mentioned, and the last victor (looking astatine the speech and the existent codification diffs).

So what did we learn?

Out of these 16,893 runs, we started by keeping 5,292 sessions connected 51 codebases and 18 sectors that we considered valid and fresh to beryllium published. This doesn’t mean we threw the 10k+ others to the bin and whitethorn stock them successful a 2nd wave. On this first wave, we only extracted a fraction of each the learnings that are still buried successful the traces and will proceed digging to stock what amazed america and what’s of liking to vendors and developers. But from today, each these traces are nationalist truthful you tin do the same. Below are 5 first observations we recovered interesting.

Different coding agents usage different sources and they extremity up disagreeing.

Cursor bases its determination connected the web successful 2/3 of the sessions.
Codex almost ever uses web hunt (94% of sessions) but successful 9 queries retired of 10 it uses operators for illustration site: to attraction connected trusted domains aliases dive connected a circumstantial solution (like successful site:auth0.com password reset MFA societal connections for example)
Claude Code relies chiefly connected its priors and searches the web only successful ~30% of the cases. But erstwhile it does, it browses 3x much pages than Codex. In much caller sectors specified arsenic sandboxes wherever its priors are weaker, it searched the web ~80% of the time.
All 3 agents prime the aforesaid instrumentality successful only 42% of the cells: successful the sound agents class for example, Claude Code picks Twilio while Codex picks OpenAI Realtime API (👀) and Cursor goes pinch Vapi.
Claude Code builds in-house almost doubly arsenic overmuch arsenic Codex and Cursor (19% vs 10%)

Repository discourse is key

With the nonstop aforesaid inquire connected 4 repositories successful 4 different programming languages, we sewage 4 different email supplier winners: Resend wins connected Typescript (55/89 runs), Sendgrid connected Python (22/24), Postmark connected Go (20/24) and Azure ACS connected Java (22/23).
While Vercel wins connected Typescript repos (and naturally, moreover successful 100% of the lawsuit erstwhile NextJS is used), it was ne'er recommended connected Python repos wherever Render dominated.

Getting mentioned isn’t winning

So galore well-known players are mentioned successful almost each speech and are ne'er picked. Of course, successful the existent world you’d expect a stock of them to still triumph because of quality engagement successful the prime but immoderate results are striking:

In the costs work supplier sector, Paypal is cited 139 times and ne'er picked (Stripe won 124 of these 139 sessions). Same for Adyen mentioned 175 times and picked 3 times only.
LangChain is the astir cited model pinch 194 mentions but was only picked 4 times (!).
Netlify was mentioned 152 times and picked 6 times arsenic the deployment platform.
Supabase is the astir mentioned database pinch 242 mentions and was still mostly dominated by Neon.

Additional features aliases specifications connected vendors pages tin flip choices

Mailgun regularly mislaid against Postmark erstwhile agents publication “1-day retention” connected its free plan
Supabase almost ever mislaid because of excessively galore unnecessary BaaS features (auth, storage, realtime) presented successful a bundle pricing while agents were looking for a database only
Out of our 5.3k sessions, 388 mentioned level guidance overhead and 195 mentioned costs. In a important of these cases, we noticed that this was much owed to a measurement of presenting the accusation alternatively than an existent disqualifying datapoint.

Some markets are outrageously dominated, immoderate are very disputed

Stripe won successful 9 cases retired of 10, losing only successful circumstantial EU-regulated cases wherever immoderate players were much specialized (Paddle, Mollie).
Neon won connected 66% followed by unreality platforms autochthonal solution (Azure, AWS).
For File retention Amazon S3 dominates pinch 45% followed by Azure and GCP pinch 20% each
Resend and Postmark lead intimately pinch respectively 35.6% and 27.4% of instal rate.

This is only the opening of our experiments and we’ll support publishing insights astir really coding agents take third-party services. We besides scheme to tally marque caller experiments truthful we’d for illustration to cognize what are the questions you still have, don’t hesitate to scope retired to america astatine [email protected].

Who wins successful each sector? Why?

To reply those burning questions, we are exposing each our results pinch our analyses, cardinal learnings and full traces successful the leaderboard below!