Common Crawl Published A Manual For Being Visible To AI, I Automated It

Aug 10, 2026 07:00 PM - 1 hour ago 1

Every month, a non-profit called Common Crawl fetches complete 2 cardinal webpages and gives the transcript distant to anyone who wants it.

That free archive is wherever astir AI training information starts.

When Mozilla audited the LLMs released betwixt 2019 and 2023, 64% had trained connected Common Crawl information successful immoderate form, and GPT-3 was astir 60% filtered Common Crawl by training weight.

If a model already knows your marque without searching the web, likelihood are this is portion of wherever it learned.

The crawler down it is called CCBot, and whether it tin publication your tract decides whether you beryllium successful that archive astatine all.

This is really it looks successful your logs.

CCBot/2.0 (https://commoncrawl.org/faq/)

The July 2026 crawl holds 2.14 cardinal of those pages.

Three of its records travel from bbc.com. CCBot asked for permission, publication the refusal, and left.

As acold arsenic the training information pipeline is concerned, 1 of the biggest news sites connected world doesn’t exist.

The BBC robots.txt blocks 13 of the 14 AI crawlers the checker tracks, and different apical news sites person been blocking training crawlers connected intent since 2023. BuzzStream measured it astatine 75% of the apical U.S. and UK publishers.

The mobility that interests maine is really galore sites are missing from that crawl without ever deciding to be.

Chris Green posted HTTP Archive numbers connected LinkedIn recently. Roughly 492,000 sites sanction CCBot successful robots.txt, and astir 95% of those mentions beryllium to artifact it.

His unfastened mobility stuck pinch me.

How galore of that half cardinal meant it?

Since July 1 2025, Cloudflare blocks AI crawlers by default connected each caller domain it serves.

Its managed robots.txt reached 3.8 cardinal much domains that ne'er wrote one. Ad plugins adhd blocklists. In 2026, a CCBot artifact is arsenic apt to beryllium a level default arsenic a decision.

Common Crawl itself wrote a manual for auditing precisely this. But it’s each manual work. So I automated it.

The Common Crawl Visibility Checker is free, needs nary signup, and answers 1 domain astatine a time, consecutive from Common Crawl’s ain archives.

Where Common Crawl Sits In The AI Pipeline

The crawling has tally monthly since 2008, and the snapshots are the earthy worldly for C4, RefinedWeb, RedPajama, Dolma, FineWeb and astir of the different large training sets.

Labs seldom touch the earthy crawl. They return a filtered derivative, truthful your page’s way into a exemplary runs from CCBot done Common Crawl into 1 of those datasets, and each measurement keeps only what passed the 1 earlier it.

The pipeline is still moving, too. FineWeb added six caller 2025 crawls past July, and NVIDIA hosts its Nemotron training group wrong Common Crawl’s ain bucket.

I covered the pre-training life of your contented successful the Three Lives of schema markup.

The short type is that CCBot is the 1 crawler whose visits compound.

GPTBot fetches your page for OpenAI alone. A CCBot sojourn ends up pinch everyone who’ll ever train connected the unfastened web.

The Audit Common Crawl Expects You To Run By Hand

In June, Common Crawl published thing interesting.

The AI Visibility Audit is an 18-page guideline telling tract owners really to cheque their opinionated successful the dataset.

It’s a bully guideline pinch 1 catch.

Every cheque is manual.

You curl your homepage pinch CCBot’s personification supplier and comparison the consequence to a browser’s.

Next comes the scale API, wherever you eyeball your seizure counts, and their web graph, wherever you look up your rank. Then you compile a scorecard per domain, by hand. lol!

I publication the checklist and past automated it. Conveniently, Common Crawl opened its information to browser devices successful May, which makes this the uncommon instrumentality the information supplier really wants built.

I deliberation they cognize precisely what they’re doing. Fair play to them.

What The Checker Shows

Type a domain and the test comes backmost successful 4 panels.

Captures Per Crawl

How galore of your pages each of the past 12 monthly crawls took, arsenic a trend.

My tract went from 2 captured pages successful past August’s crawl to 55 successful July’s, which is the astir flattering floor plan anyone has ever drawn of it.

A large domain gets an estimate instead. Type wikipedia.org and you’ll spot astir 3.7 million, labelled arsenic an estimate, because the instrumentality sounds the scale building for immense sites alternatively than counting each record.

Robots.txt History

Common Crawl stores the robots.txt it publication earlier each crawl. The checker checks those stored copies backwards period by period and diffs the AI crawler rules, truthful a artifact gets a commencement date.

Run bbc.com and each period sounds the same. CCBot blocked, 13 of 14 tokens, Disallow slash.

Whose Block It Looks Like

When a artifact matches a known template, the instrumentality says so. Cloudflare’s managed robots.txt has a unique licence comment. Squarespace’s default has its ain unsocial signature.

And erstwhile 8 aliases much AI tokens spell down successful 1 sweep, that’s usually a plugin aliases a copied list.

A Live Probe

History says what happened, truthful this 1 checks what’s existent correct now. It fetches your homepage arsenic CCBot/2.0 adjacent to a normal browser and a power bot, which catches the artifact robots.txt ne'er shows, a firewall challenging the personification supplier astatine the edge.

Those 4 are the diagnosis. The remainder of the checker is what to do pinch it.

Every cheque ends successful a sheet of hole cards matched to what it found.

A Cloudflare template artifact gets the dashboard way and a warning.

Handwritten blocks get the two-line robots.txt stanza pinch a transcript button.

When it’s an separator situation instead, the paper will show the firewall vendor and links CCBot’s published IP ranges.

Under the cards, it will show the day of the adjacent crawl, because a hole made coming shows up successful adjacent month’s data, and knowing erstwhile to look is important.

Sitemap Coverage

The sitemap sheet is the characteristic I wanted for myself. The checker fetches your sitemap and diffs it against the captured URLs, which turns the header number into a activity queue. Mine sounds 42 of 97 pages successful the July crawl, and the different 55 download arsenic a CSV.

Tracking A Single Page

You tin besides paste a afloat page URL alternatively of a domain and get that azygous page’s history crossed the year. Doing that taught maine 2 things astir my ain tract wrong a minute.

My ChatGPT sources teardown was captured successful only 1 of the past 12 crawls. And the transcript Common Crawl holds isn’t the cleanable URL astatine all.

CCBot recovered the page done a newsletter nexus and stored it pinch the ConvertKit search parameters still attached. I stopped utilizing Kit a agelong clip ago.

The crawler doesn’t browse your tract the measurement you imagine. It follows immoderate links the web really contains.

Stored Copy Check

The past portion is simply a stored transcript check. The instrumentality pulls the existent bytes CCBot archived for your homepage, extracts the text, and compares it against your unrecorded HTML.

Both sides are publication earlier JavaScript runs, which is the point, because CCBot ne'er runs it.

The Invisible Blocks

Two kinds of artifact ne'er show up erstwhile you cheque a robots.txt by hand.

1. CDN default. A domain created connected Cloudflare aft July 2025 blocks AI crawlers unless personification turned that off, and the managed robots.txt reached millions of sites whose owners ne'er touched a file.

2. An separator rule. A WAF challenging CCBot while robots.txt sounds clean. Common Crawl’s ain guideline says to cheque your CDN dashboard earlier you touch thing connected your server, and they’re right. It’s besides the 1 spot nary classical SEO crawler looks.

What Actually Moves Inclusion

If your consequence shows unfastened but hardly captured, blocking was ne'er your problem.

Crawl privilege was.

Common Crawl samples each tract by really good connected the domain is, truthful a fistful of things matter.

Fix the CDN first. If the unrecorded probe shows a challenge, nary robots.txt edit will thief you.
Let CCBot in. Absence of a norm already intends yes. Mine allows it explicitly, and quotes The Matrix while doing it. (Check it out)
Add a sitemap successful robots.txt. CCBot follows the sitemap protocol.
Render connected the server. CCBot doesn’t execute JavaScript, truthful contented that only exists aft hydration leaves an quiet ammunition successful the training data. Size stopped being an excuse successful March 2025, erstwhile the stored limit went from 1 MiB to 5 MiB per page.
Earn links from well-connected sites. Crawl privilege follows harmonic centrality, which you tin look up successful Metehan Yeşilyurt’s web chart tool. (Don’t perceive to the “SEO is dead” naysayers. Keep buying, sorry, I meant building, those links.)

What This Tool Can’t Tell You

Being successful Common Crawl doesn’t put you successful a model.

Every training group filters the crawl, and the frontier labs stopped disclosing their mixes astir 2023.

Thin sum proves thing connected its ain either, because Common Crawl is simply a sample of the web, not a transcript of it.

That’s why the checker looks astatine captures, robots history, and the unrecorded probe together earlier it gives you a verdict.

When the 3 disagree, it says so.

Run It

Check your domain.

Then cheque a client’s. If the robots.txt history shows a artifact that appeared the aforesaid period the tract changed CDNs, you’ll cognize precisely what happened, and you’ll astir apt beryllium the first personification to show them.

The access checker covers the remainder of the AI crawler picture, GPTBot and ClaudeBot and friends, live.

More Resources: