What is Chunking?
Chunking is splitting a document into smaller passages before indexing, so a retrieval system can match a query against a passage instead of a whole page.
Published
Definition
Chunking is the process of splitting a document into smaller passages before it is embedded and indexed, so a retrieval system can match a query against an individual passage rather than an entire page.
The practical consequence is the part most people miss: the chunk, not the page, is the unit of retrieval.
Why It Matters for AI Visibility
Your page is never retrieved whole. It is cut into pieces, and each piece has to make sense alone. A paragraph that opens with “This means that…” loses its subject the moment it is separated from the one above it, and a chunk without a subject rarely matches a query.
Chunk boundaries usually follow structure — headings, paragraph breaks, list items. That is why heading hierarchy matters to retrieval far more than its visual role suggests: it decides where the cuts land.
Example
Fails when isolated:
It also handles this automatically for subdomains, which is why the second case behaves differently.
Survives when isolated:
The llms.txt file is not inherited by subdomains. Each subdomain needs its own file at its own root.
Same fact. Only one of them is retrievable.