Grep beats LSP? Why coding agents ignore your fancier tools

Why would a coding supplier disregard a retrieval interface that returns much precise results?

I explored this mobility successful a mini study comparing lexical hunt pinch grep against LSP-backed semantic navigation. I expected semantic navigation to trim sound and prevention tokens. Instead, agents often stayed pinch grep. When I forced them to usage the semantic way first, task occurrence sometimes fell.

This is simply a mobility of LLM-friendliness. A instrumentality is not friends to a exemplary simply because its results are precise. It must return capable discourse for the adjacent measurement and coming that discourse successful an interface and output style the exemplary tin usage directly. Familiarity whitethorn besides matter: the exemplary whitethorn person learned akin action paths during training. The interface properties tin beryllium evaluated directly. Training support is simply a presumption accordant pinch these results, not thing this study proves.

The consequence is not a wide statement against LSP. The protocol includes capabilities acold beyond codification navigation, and this study tested only a mini subset. Instead, the results constituent to a broader engineering problem: a exemplary does not usage devices successful isolation. It uses them done a harness that defines the disposable actions, their names, their inputs, and the discourse returned to the model.

In this post, I picture really codification retrieval affected some code-finding and editing tasks, why grep had an advantage successful immoderate conditions, and what this intends for supplier platforms.

A exemplary and its acquainted instrumentality loop enactment arsenic 1 capacity surface.

Comparing 2 codification retrieval interfaces

I compared 2 ways for an supplier to retrieve codification context. grep performs lexical search: it finds matching text. The tested LSP-backed devices execute semantic navigation done references, definitions, and archive symbols, allowing them to separate a existent usability telephone from the aforesaid connection successful a comment.

The aviator covered 3 Claude models, respective Python and TypeScript repositories, and aggregate task types. I measured token usage only erstwhile some approaches completed the task successfully. This controls for a communal information error: a grounded tally tin look businesslike simply because it stopped early.

On elemental code-location tasks, each 3 models chose the semantic instrumentality only 0% to 6% of the clip erstwhile some devices were available. Forcing a semantic-first way reduced occurrence from 100% to 89% successful that arm.

Reference-completeness tasks produced a different result. When asked to find each caller, the models chose semantic navigation 45% to 57% of the time. The LSP-backed way reached 1.00 precision, compared pinch 0.76 for grep, by removing mendacious matches. However, callback stayed adjacent 0.66 successful some arms. Semantic navigation did not find much existent calls. The remaining limit came from really thoroughly the supplier worked, not from retrieval precision. For the stronger models, the precision summation besides came pinch higher token usage alternatively than a saving.

The exemplary doesn't blindly for illustration grep — it routes by task

Share of semantic (LSP) instrumentality calls erstwhile some grep and LSP are disposable and the supplier chooses freely.

Legend: Opus 4.8 (blue), Sonnet 4.6 (magenta), Haiku 4.5 (green).

Same models, aforesaid free prime — the routing flips pinch the task. On localization and rename the supplier almost ever reaches for grep; connected reference-shaped activity it reaches for the LSP astir half the time, unprompted. The action distribution is task-shaped, not a unsighted habit.

The codebase was besides important. On a cleanable TypeScript repository, LSP-backed navigation produced nary F1 summation and utilized 16% much tokens. On a noisy TypeScript repository, it improved F1 by 0.246 and utilized 12% less tokens. The useful predictor was lexical noise, not whether the connection had beardown fixed types.

Codebase sound determines the worth of semantic navigation

Accuracy summation from semantic retrieval connected reference-completeness (ΔF1 = LSP − grep). Bar colour encodes really noisy grep is connected that repo; prec = grep’s precision there.

Legend: bluish intends grep is cleanable here; magenta intends grep is noisy here.

Two repositories successful the aforesaid language, other verdicts. On cleanable remeda the LSP adds thing — grep already resolves each reference correctly, truthful semantic retrieval is axenic overhead. On noisy hono it adds +0.246 F1. The predictor is really severely grep's precision degrades connected that codebase, not whether the connection is statically typed.

These results are conditional alternatively than categorical. The agents did not simply “always usage grep.” Their routing changed pinch the task, and the worth of LSP-backed navigation changed pinch the repository.

The tested LSP-backed devices initially returned only a location: a record path, line, and column. The supplier past had to unfastened the record to inspect the code. grep, by contrast, usually returned the matching statement immediately: src/auth.ts:42: return validateToken(token).

I changed the semantic-navigation consequence to see root matter successful a akin shape. The semantic backend and the group of references stayed the same; only the accusation returned to the exemplary changed. Pass@1 connected the rename tasks roseate from 0.67 to 0.83, while follow-up record sounds fell from 15.2 to 3.2 per episode.

Returning root discourse improves semantic navigation

Multi-file rename, Opus 4.8, pyright pinch a pre-warmed index. Same semantic backend successful some LSP arms — only the output shape differs.

Legend: grep (blue), LSP — locations only (magenta), LSP + inline discourse (green).

Returning locations forces the supplier to spell publication each site; returning the statement inline does not. Attaching ±2 lines of root to each reference trim follow-up record sounds 15.2 → 3.2 — beneath grep's ain 4.3 — and lifted pass@1 from 0.67 to 0.83. The retrieval backend ne'er changed; only the style of what came back.

This consequence illustrates a rule that Anthropic besides emphasizes successful Writing effective devices for agents: devices are interfaces for non-deterministic agents, truthful the discourse they return is portion of the design. A semantically correct instrumentality tin still create a mediocre supplier workflow if each consequence requires respective other actions to interpret.

The output alteration does not beryllium that post-training information caused the improvement. It whitethorn besides person helped simply because each consequence contained much useful information. However, the consequence is accordant pinch a broader hypothesis: models study actual action patterns, not “tool use” successful the abstract. A acquainted loop—prompt, instrumentality call, readable result, adjacent action—can beryllium portion of the capacity observed successful practice.

Why lexical hunt had an advantage

Interface familiarity is only portion of the explanation. Lexical hunt besides had a existent structural advantage for immoderate tasks.

A semantic reference is only 1 benignant of matter match. A rename whitethorn besides request to update comments, docstrings, configuration, aliases strings. find_references will not return those by design, while grep can.

semantic references ⊂ textual occurrences

For text-wide edits, grep tin beryllium the amended retrieval instrumentality moreover for a exemplary pinch cleanable training connected LSP-backed navigation.

This gives america 2 explanations for the observed behavior:

1. Structure: immoderate tasks request textual completeness, which the tested semantic-navigation methods do not provide.
2. Distribution: the exemplary whitethorn person much believe pinch acquainted devices and consequence shapes.

The first mentation follows straight from what the devices retrieve. The 2nd is simply a presumption accordant pinch the routing and output-format results, but this study did not manipulate training information and truthful cannot beryllium it.

Structure explains erstwhile grep is better. Distribution explains why acquainted paths still win.

The harness is portion of the system

Here, I usage harness to mean the runtime astir a model: the instructions placed successful context, the devices made available, their input schemas, the style of their results and errors, and the loop that decides what the exemplary sees next.

This surrounding strategy tin materially alteration behavior. Anthropic’s activity connected effective harnesses for long-running agents shows the aforesaid thought astatine a longer clip scale: the exemplary unsocial is not capable to make reliable advancement crossed sessions. Environment setup, advancement artifacts, and verification routines impact what the supplier tin accomplish.

The aforesaid rule applies wrong a azygous instrumentality loop. When post-training includes supplier trajectories, the harness defines the prompts, instrumentality calls, results, and betterment paths successful those examples. A exemplary trained done repeated usage of read, grep, edit, and bash whitethorn study policies that dangle connected those interfaces. Moving the aforesaid exemplary into a different instrumentality furniture tin truthful alteration its effective capability.

agent capacity = exemplary × harness

This is why benchmark results for a exemplary do not ever transportation unchanged to a different runtime. Supporting the aforesaid exemplary is not needfully the aforesaid arsenic reproducing the aforesaid agent. Tool selection, signatures, output formats, and correction behaviour tin each impact the argumentation the exemplary follows.

Preserving autochthonal runtimes pinch ACP

This is 1 logic my squad and I built AgentConnect astir autochthonal coding-agent runtimes. AgentConnect does not spot Claude aliases Codex models wrong a shared, generic instrumentality loop. It runs runtimes specified arsenic Claude Code and Codex connected the user’s ain machine, wherever each runtime keeps its autochthonal devices and normal prompt-to-tool workflow.

AgentConnect communicates pinch these runtimes done the unfastened Agent Client Protocol (ACP). ACP standardizes the bound betwixt a customer and a coding agent, including convention setup, prompts, streaming updates, tool-call updates, cancellation, and support requests. It does not require each runtime to expose the aforesaid soul tools.

One unfastened boundary. Each supplier stays connected its location turf.

This separates 2 concerns. At the extracurricular boundary, a communal protocol lets AgentConnect supply squad channels, triggers, schedules, convention history, collaboration, and power crossed aggregate agents. Inside that boundary, Claude Code tin proceed to activity for illustration Claude Code, and Codex tin proceed to activity for illustration Codex.

The extremity is not vendor lock-in. ACP provides an unfastened bound crossed runtimes. Our creation rule is to sphere the instrumentality aboveground each exemplary already uses well, past adhd coordination astir it.

These findings do not mean that teams should debar LSP, MCP, aliases caller supplier skills. The study recovered a clear precision summation from LSP-backed navigation successful noisy code, and a mini response-format alteration removed astir follow-up reads. The applicable instruction is to measure a caller retrieval interface arsenic portion of the afloat supplier loop.

My proposal is to commencement pinch the autochthonal instrumentality surface, past use the pursuing checks erstwhile adding a caller capability:

1. Test existent tasks astatine adjacent accuracy. Do not observe little token usage if occurrence besides fell.
2. Measure whether the supplier calls it. Availability is not adoption.
3. Return capable discourse for the adjacent decision. A consequence for illustration path:line:content whitethorn activity amended than a bare location object.
4. Keep a autochthonal fallback. Semantic and lexical hunt lick different problems.
5. Route by the task and the codebase. A noisy repository whitethorn use from semantic navigation. A text-wide hunt whitethorn still request grep.
6. Reinforce the caller trajectory erstwhile it matters. A punctual tin present a tool, but it whitethorn not create a reliable argumentation for utilizing it.

As Anthropic notes successful Building effective agents, successful supplier systems often trust connected simple, composable patterns. More devices do not automatically nutrient a much tin agent; devices must beryllium distinct, understandable, and useful wrong the model’s workflow.

Conclusion

The study shows why “better retrieval” cannot beryllium evaluated extracurricular the afloat supplier system. An interface tin beryllium much precise and still usage much tokens. It tin return correct locations and still create unnecessary reads. A mini alteration successful output style tin make the aforesaid semantic consequence overmuch easier for the exemplary to use.

For teams building supplier platforms, the accusation is straightforward: measure the exemplary and harness together. Preserve the interfaces that already support reliable behavior, and trial changes against existent tasks earlier assuming that a much blase abstraction will help.

For the afloat experimental setup, task definitions, and results, spot Does a Language Server Save Tokens for Coding Agents?.

This is the merchandise rule down AgentConnect: usage an unfastened protocol to link agents, while keeping each exemplary together pinch its autochthonal runtime and instrumentality loop.