How AI labs choose the websites that enter their training corpora

Founder of Epovest Aug 7, 2026

Ask an AI assistant a question about your market and part of the answer comes from memory: what the model absorbed during training. That memory was built from a corpus, the corpus was built from the web, and your site either made it through the selection or did not. The stakes are concrete: when your coverage is thin, the model answers from whatever its corpus does contain, including your namesake's record.

The selection is not editorial. Nobody at an AI lab reads websites and decides yours deserves a seat. It is a pipeline: a crawl, a stack of filters, a deduplication pass, a set of contracts, a sampling policy. And it is unusually well documented, because the labs publish how their corpora are built even when they keep the corpora themselves private. This guide walks that pipeline stage by stage: what each one keeps, what it drops, and what that implies for the way you publish.

Training is one of two doors into an answer. The other is live reading: pages fetched at the moment the question is asked, under different rules and on a much faster clock. We measured how differently the two doors behave. This guide is about the memory door.

One shared starting point: the public crawl

Most documented training corpora do not begin with a lab's own crawl. They begin with Common Crawl, a nonprofit that has crawled the web since 2008 and publishes a fresh snapshot of several billion pages roughly every month. C4, the corpus behind Google's T5 models, is one filtered Common Crawl snapshot. The web portion described in OpenAI's 2020 paper "Language Models are Few-Shot Learners" is filtered Common Crawl. So are RefinedWeb (the corpus behind Falcon), Dolma (Allen Institute for AI) and FineWeb (Hugging Face), which filters 96 snapshots collected between 2013 and 2024.

Three properties of that starting point are worth knowing:

Common Crawl's bot is named CCBot, and it honors robots.txt. Admitting it is the widest single door you control; blocking it removes you from the upstream of most open corpora.
URLs are picked by how the web points at them. Common Crawl ranks domains with harmonic centrality, a link-graph measure: domains that reference pages link to get crawled wider and deeper. A site nothing links to sits at the thin edge of the crawl, so every reference you earn also buys crawl depth.
The crawl fetches HTML and does not execute scripts. Text that only appears after JavaScript runs does not exist for the corpus. Server-rendered HTML is the price of entry.

The labs' own crawlers, and the switches you hold

On top of the shared crawl, the labs run named crawlers of their own. Each is documented, and each responds to its own line in robots.txt:

Operator	User-agent	What it feeds
OpenAI	GPTBot	Training corpora
OpenAI	OAI-SearchBot, ChatGPT-User	Search index and live browsing, not training
Anthropic	ClaudeBot	Training corpora
Anthropic	Claude-SearchBot, Claude-User	Search and user-requested fetches
Google	Googlebot, governed by the Google-Extended token	Google-Extended controls AI training use; the crawling itself is Googlebot
Apple	Applebot, with the Applebot-Extended token	The token controls AI training use
Meta	meta-externalagent	Training corpora
Common Crawl	CCBot	The public snapshots most open corpora filter

Two details matter more than the list. First, training and answering are separate switches: blocking GPTBot keeps your future pages out of training corpora and does not touch what OAI-SearchBot can read at answer time, and the reverse. A site can stay out of training yet remain fully readable live, or the opposite. The decision is per use, not per company. Second, robots.txt is declarative: the crawlers above document that they honor it, and bot management at the CDN is the enforcement layer for everything else.

Those switches now live in a shrinking commons. "Consent in Crisis" (Data Provenance Initiative, 2024) measured that in a single year, robots.txt restrictions came to cover about a quarter of the tokens from the domains most heavily represented in common training corpora. Cloudflare shipped one-click AI-crawler blocking in 2024, then made blocking the default for its new customers in July 2025. Every publisher that closes raises the relative weight of every page that stays readable: checking what your own stack blocks by default is no longer optional hygiene.

What the filters drop first

A raw crawl is mostly noise, and the first stages that thin it are mechanical.

Main-content extraction comes first. Pipelines such as RefinedWeb and FineWeb run an extractor (typically trafilatura) that keeps the main text block of a page and discards navigation, headers, footers, sidebars, cookie banners and lists of links. Whatever lives in those zones never reaches any later judgment. A page whose substance sits in its main body survives extraction; a page that is mostly furniture around a thin paragraph leaves nothing behind.

Then come line and document heuristics. C4, published with Google's T5 work in 2019, dropped every line not ending in terminal punctuation, every page with fewer than three sentences, and every page containing "lorem ipsum" or a curly brace. The Gopher rules (DeepMind, 2021), reused by FineWeb, drop documents shorter than 50 words or longer than 100,000, documents whose mean word length falls outside 3 to 10 characters, documents where more than 90% of lines start with a bullet, and documents missing the most common function words. A language-identification step routes each document, and a corpus keeps a defined set of languages.

None of this measures meaning. It measures whether a page contains actual prose: full sentences, punctuation, paragraphs with substance. Text assembled as fragments around keywords fails these tests by construction.

The quality classifiers: what "a good page" is trained to mean

The pages that survive the mechanics are then scored by a classifier, and the labs disclose what their classifiers were trained to recognize:

OpenAI's 2020 paper filtered the crawl with a classifier trained on WebText: pages that Reddit users had linked to with a minimum karma, as the positive class.
The LLaMA paper (Meta, 2023) kept pages resembling those used as references by Wikipedia articles.
FineWeb-Edu (2024) scores pages from 0 to 5 for educational value, with annotations produced by a large language model, and keeps roughly the top tenth of an already filtered corpus.

Different labs, one pattern: the positive class is always some version of "pages humans considered worth citing". A quality classifier is a trained proxy for reference-worthiness. Pages that explain, define, date and source their claims score the way their training examples do; pages assembled for search engines sit on the wrong side of a boundary drawn through examples of both. This is where your off-site record loops back into corpus selection: being linked from the kinds of pages classifiers treat as ground truth is exactly what corroborations build.

Deduplication rewards the original

Every documented pipeline deduplicates, exactly (identical documents) and fuzzily (near-identical ones, typically via MinHash). The RefinedWeb paper reports that deduplication alone removes about half of what the filters had already accepted; filtering and deduplication together discard the large majority of the raw crawl.

When several pages carry the same text, one survives. For a publisher the consequence is plain: syndicated, scraped and templated copy does not accumulate. A description reproduced on twenty portals enters the corpus once, and not necessarily as your page. Text that exists nowhere else is the only text that is reliably yours in the corpus.

The contract door: licensed content

Since 2023 a second door exists that skips the pipeline entirely: licensing. Publicly announced agreements include Associated Press with OpenAI (July 2023), Axel Springer (December 2023), Reddit with Google (February 2024), Le Monde and Prisa Media (March 2024), the Financial Times (April 2024), News Corp, Reddit and Stack Overflow with OpenAI (May 2024). Licensed archives arrive complete, clean and attributed, without passing a single filter.

That door belongs to large archives, and its effect reaches everyone: professionally edited text now enters corpora in bulk, so the boundary the classifiers draw for the open web is calibrated against increasingly edited material. The open web that stays in is the open web that reads like it was edited.

Being in is not the end: weight, repetition, cutoff

A corpus is not read uniformly during training. OpenAI's 2020 paper details its sampling weights: the most curated sets were seen more than three times over training while the filtered crawl was sampled less than once. Corpora also span years of snapshots (FineWeb filters eleven years of them), so a page that has existed at a stable URL across many monthly crawls simply occurs more often than a page published last quarter. And every model has a cutoff: what you publish after it waits for the next corpus. That lag is one more reason the live-reading door matters as much as the memory door: the pages an engine fetches at answer time carry what your record has not yet taught the model.

What this changes about how you publish

Each habit below is one stage of the pipeline, turned into practice:

Serve your text as server-rendered HTML. The crawls do not execute your scripts.
Decide robots.txt per crawler and per use, training versus live answering, and audit what your CDN or firewall blocks by default: since 2025, unreadable is the default state for a growing share of the web.
Write prose. Full sentences, terminal punctuation, substantial paragraphs: the heuristic filters count exactly that, line by line.
Put the substance in the main content block. Extraction discards menus, footers and sidebars before any quality judgment happens.
Be the original. Deduplication keeps one copy of any given text; pages that exist nowhere else are the only ones reliably credited to you.
Earn references. The quality classifiers were trained on pages humans linked to; mentions won on the pages assistants and corpora already trust are the off-site half of selection.
Publish early and keep URLs stable. Presence across snapshots compounds; a young or thin record leaves the model answering from someone else's.

What the corpus retained about you is observable from the outside: it is what the assistants say when they answer from memory. Tracking measures exactly that, question by question, engine by engine, and our guide to GEO situates the memory door inside the wider discipline.