Where AI agents get stuck on your site

The adjacent wide frontier of AI is agentic:

• Google introduced agentic tasks successful Search.
• The web gets much visits from bots than humans now.
• Salesforce recovered 20% of income coming from agents marks a signpost.
• 60% of companies usage agents unrecorded successful production, and 3 retired of 4 companies put successful AI agents.

To fig retired really fresh B2B sites are for agentic visitors, I teamed up pinch David Kaufman, laminitis of Siteline, and I analyzed really agents scan websites and wherever they get stuck. The answer: astir sites are agent-ready, but location is 1 captious breaking point.

Agents don’t publication websites for illustration humans. They person a task, hunt the web, fetch pages, extract facts, and mention the sources they used. A page tin seduce a quality and still neglect an supplier if the facts are difficult to find (opacity), difficult to fetch (machine-readability), aliases difficult to mention (access friction). 

AI agents move websites from showrooms into barcodes.

How we looked astatine supplier behavior:

• The supplier had to find the charismatic tract itself. We did not supply starting links, eg to homepages.
• We gave agents 3 buyer-related tasks for 100 B2B products: find pricing/features, integrations, and security/compliance. We ran each task 5 times to measurement the effect of the probabilistic quality of LLMs.
• We weren’t comparing whether aliases not the accusation existed location connected the web; instead, we measured whether the supplier could reliably reply from the vendor’s ain site. 

View embedded content

1. Pricing breaks first-party sites

The infinitesimal a imaginable looks astatine pricing, they extremity browsing and commencement comparing. High purchaser intent, bottommost of the funnel. That makes pricing the hardest and astir important trial of whether a vendor tract tin service agents directly.

Pricing besides sits successful a triangle of 3 “wants” that bully pricing pages request to satisfy:

• Companies want to power pricing disclosure.
• Buyers want accelerated comparison.
• Agents request clear, fetchable, citable facts.

When AI agents effort to retrieve pricing, they get stuck overmuch much than for information aliases integrations.

• Pricing/features: 79% first-party reply rate, 84% first-party citation share.
• Integrations: 93% and 99%.
• Security: 92% and 99%.
• Pricing/features produced 77% of each third-party citations.

If you wonderment whether that’s because immoderate B2B companies don’t publically show pricing, you’re only half right.

2. Hidden pricing is only portion of it

Hiding prices forces agents to look elsewhere, but published prices do not afloat lick the problem. Among pricing punctual runs wherever the vendor did not disclose a existent price, 45% cited astatine slightest 1 third-party source. The different 55% stayed connected first-party citations, usually by saying the vendor required interaction income aliases did not people a actual price.

Even erstwhile the vendor showed a numeric nationalist price, agents still cited astatine slightest 1 third-party root successful 18% of runs, suggesting value tin beryllium connected the page but still beryllium difficult for the supplier to extract, trust, aliases mention cleanly.

You tin effort to hide your pricing, but you amended make judge nary 1 other knows and writes astir it. Once it’s “out there”, it’s excessively late. If you person analyzable pricing methodology, the champion measurement is to explicate it intelligibly and make it accessible to agents.

Some pricing pages are visible to humans but not reliable capable for agents to parse and cite. You can’t ever spot your eyes.

3. Agents neglect for 3 reasons

Agents neglect to retrieve pricing from a marque for 3 reasons: opacity, machine-readability, and entree friction.

Example of an supplier struggling to retrieve Zendesk’s pricing and pivoting to third-party sources.

• Pricing opacity simply intends the marque doesn’t publically disclose the price, aliases it’s vaguely packaged. Opacity explains elevated fallback, meaning agents person to trust connected 3rd parties for information.
• Machine-readability describes the business erstwhile prices exist, but agents still do not confidently extract them. Machine-readability explains fallback contempt disclosed pricing. Machine-readability fails erstwhile the value is difficult to extract because of page structure, JavaScript, calculators, toggles, screenshots, PDFs, aliases ambiguous tables.
• Access friction is what astir group expect to beryllium the problem pinch agents. The supplier hits fetch failures, complaint limits, blocking, aliases unreachable pages, making supplier runs much costly.

Access errors were not the main logic agents near first-party sources, but erstwhile they happened, they were severe. They appeared successful only 7% of each runs. In pricing runs, entree errors pushed third-party fallback to 77%, compared to 17% without entree errors.

The effect of errors connected supplier tally costs (tokens, web searches, fetches, retrieves, time) is important erstwhile comparing the 90th pinch the 10th percentile successful our study:

• Cost: 4.4x
• Token: 4.7x
• Time: 2.0x

Brands don’t salary that measure directly, but it is simply a useful proxy for friction. The harder your tract is to retrieve, the much activity an supplier has to do earlier it tin reply from your page. If your pricing page is blocked, slow, difficult to fetch, aliases difficult to parse, the supplier has 2 choices: walk much activity connected your site… aliases get the reply location else.

4. The fallback web is messy

Fallback occurs erstwhile agents person to trust connected third-party sources alternatively than first-party sources arsenic a consequence of the 3 nonaccomplishment modes. This is the biggest consequence because third-party accusation is spotty and beyond your control.

Agents do not autumn backmost to 1 cleanable root category. They reconstruct pricing from a mixed web of explainers, directories, app stores, partner pages, and low-trust aggregators.

Key stats from the 580 pricing third-party citations:

• 52% were editorial (blogs, media articles, comparison guides, explainers, and different article-style pages).
• 46% fell into the directory category (review, procurement, and software-listing sites specified arsenic G2, Capterra, Vendr, Tekpon, and akin domains).
• 2% from broader ecosystem pages (app stores, marketplaces, partner pages, and integration directories tied to different platform).

The examples show the consequence of missing pricing transparency and supplier stumbling blocks connected your site.

Example journey:

Here, nonaccomplishment mode indicates the logic the supplier grounded to get on-site / first-party pricing information.

5. How to make your tract agent-proof

An agent-proof pricing page is really you support the supplier quoting you alternatively of a directory for illustration Vendr. The fixes representation to the 3 nonaccomplishment modes.

Disclose the truth (opacity)

• Publish existent prices successful matter for each self-serve tier. If a tier is genuinely custom, opportunity what drives the number alternatively of “contact sales.”
• Keep scheme names, prices, limits, and features connected 1 canonical pricing URL, and constituent each different mention backmost to it.
• Mark bequest plans intelligibly truthful third-party contented can’t support old tiers alive.

Make the truth extractable (machine-readability)

• Put prices successful crawlable HTML. Many supplier fetches ne'er tally JavaScript, truthful a value rendered client-side is invisible. In testing, prices successful server HTML sewage publication successful nether a second; a JavaScript-only value sewage missed.
• Add schema.org Product and Offer markup pinch value and priceCurrency. This azygous lever moved a page from 73 to 93 successful the readiness test.
• Explain usage-based pricing successful text, not a calculator-only widget.

Let the supplier successful (access friction)

• Allow AI crawlers successful robots.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Check you aren’t allowing Googlebot while blocking them.
• Don’t artifact server-side AI fetches connected pricing pages. Access errors deed only 7% of runs, but they push fallback from 17% to 77% erstwhile they do.
• Keep the value early successful the DOM and the page light. A 1 MB pricing page taxes each supplier and pushes inexpensive runs to way astir you.

Fix opacity and machine-readability first; they thrust astir of the fallback. Then tally the query yourself, “Find each pricing and features for [product],” and measurement it pinch the accomplishment below.

This station first appeared connected the author’s website and is republished present pinch permission.