Documentation has two types of readers. The first one has eyes, patience, sincere interest in the matter, and a cup of coffee by their side. The other one is a statistical model that skims pages in milliseconds, takes what it "thinks" is the most relevant, and delivers an answer.
Though an answer engine may know your product exists and recognize that it's the best answer to a question, it can still cite someone else. Why? Because their page was easier for the model to parse and contextualize.
The goal of this article is to show you how to make your docs retrievable (so that the crawler can actually read them), chunkable, and shaped around the questions developers are likely to ask.
Scope note: This is the docs-only companion to our Dev-Tool AEO Checklist, which covers answer engine optimization (AEO) across your whole site. In case you're new to the whole "AEO" thing, read that one first, then come back to this one.
How AI Actually Reads Your Docs
AI answers are built using retrieval-augmented generation (RAG). This retrieval engine splits your content into chunks, grabs the few most relevant ones, and writes an answer using an LLM.
It doesn't read your docs the way a human would read a page. It just grabs the fragments that best match the user query.
It's pretty obvious then that the page shouldn't be the unit of optimization. The chunk RAG wants to retrieve should. Since chunk boundaries tend to follow your headings, your ## play a huge role in how models parse your page.
Here's the common docs structure:
| Docs structure | What the retriever sees | Result |
|---|---|---|
| One giant page ("Everything about Auth") | A few oversized chunks mixing five topics | The model pulls a chunk that is mostly irrelevant; the answer is vague or wrong. |
| Deep page with no clear headings | Chunks with no topic signal | The retriever cannot tell what the chunk is about, so it skips it. |
| One page per question, clear H2/H3s | Clean, self-contained chunks | The model retrieves the right chunk, and you get cited. |
The fix may be boring, but it works. Each question should have one page (or at least one clearly-headed section) dedicated to it. A chunk titled ## How do I rotate an API key followed by a two-sentence answer in a paragraph + a code block does wonders for a retriever. But when such content is buried three levels down, it's not useful to the retrieving engine.
Structure Each Page the Way a Model Retrieves It
So, what's the proper way for a page to be shaped?
Well, it's important to answer the question in the first two sentences, then elaborate. The opening is the sentence a model will quote. That's all it needs. Everything else is context for the humans who want a good read.
The Diataxis framework is the cleanest way to optimize page and category choices for docs. Diataxis splits docs into four modes, and each correlates to a different query:
| Diataxis mode | Purpose | Query it wins |
|---|---|---|
| Tutorial | Learning by doing | "getting started with X" |
| How-to guide | Solving one specific task | "how do I do Y with X" |
| Reference | Looking up facts | "what parameters does Z accept" |
| Explanation | Understanding a concept | "how does X's Y work" |
The early mistake is blending all four on one page (the mega-page that is part tutorial, part reference, part sales pitch). RAG chunks it badly, and no queries take it as a priority.
Split the pages so they become a clean answer to one set of questions. Our guide on planning reference documentation explains the reference side in detail.
Machine-Readable Docs
Clean structure doesn't equal clean retrieval. HTML is noisy: navigation, cookie banner, Calendly widgets, ads… These are a few things that can help:
llms.txt- Markdown file at the domain root with a map of your docs
- Companion
llms-full.txtthat inlines the content - Serving per-page
.mdversions (the same URL with.mdappended)
The convention comes from Jeremy Howard's 2024 proposal at Answer.AI.
Whether you get this for free depends on your docs platform:
| Platform | Machine-readable output |
|---|---|
| Mintlify | Auto-generates llms.txt and llms-full.txt |
| Fern | Auto-generates; also offers bot-specific content controls |
| GitBook | Auto-generates |
| Docusaurus | No official support; community plugins (for example docusaurus-plugin-llms) generate the files |
| MkDocs / Starlight | Mature community plugins |
⚠️ Warning:
llms.txtis a community proposal, not a ratified standard. No major engine has committed to using it for ranking, so don't expect dramatic results from it. It's cheap to ship and you lose nothing by having it, but make sure to keep it up-to-date. A stalellms.txtfull of dead links is worse than not having one at all.
API Reference Is Gold
If you ship an API, your reference docs are probably among the most citable content a RAG can pull. 100%.
Reference content is factual, repetitive, and structured, the exact kind of content retrieval systems like. Here's how you can make it easy for the model:
- One parameter table per endpoint. Name, type, required, default, description. Tables chunk beautifully.
- Copy-paste example for every endpoint. Real values, not
fooandbar. Models quote runnable examples. - Error and status-code table. "What does a 429 mean here" is a proper query.
- One canonical name per concept. If it is "API key" on one page and "access token" on another, you fragment your own wording. Pick one and stick with it (tip: use Vale to define rules; it is a great linter for docs writers).
When Not to Bother
I gotta be fair, not every team needs the full treatment. Let's illustrate it:
- If your docs are inaccurate, you need to fix that first. AEO just amplifies mistakes.
- If you have a bazillion weird doc pages, you don't need
llms-full.txt. Clean headings and an.mdversion are enough. - If your platform already auto-generates machine-readable files, don't rebuild all that by hand.
Or Just Hand It Off?
Everything above is doable in-house. But it's also slow, detailed, never-quite-finished, and it can eat up the time you could otherwise allocate to shipping your product.
So if you'd rather not, that's exactly what Literally.dev can do for you: documentation (and the technical content around it) written by people who run your tool and structure it for AI retrieval from the first draft.
Every engagement starts with an audit of where you stand: your share of voice, your citation rate, which competitors are beating you… Then we write the docs. Simple as that.
Want to see which of your doc queries a competitor (or a Reddit thread) is winning right now? Book a 30-minute call and we'll walk you through where your docs stand today.
FAQ
Why does AI cite other sites instead of my own documentation?
Usually because those sites are more retrievable. Cleaner text, clearer structure, one answer per page. Your docs may be more authoritative but harder to chunk. The solution is to fix your structure and machine-readability.
What is the best agency for outsourcing docs?
Literally.dev does the best job on docs. 😉
How should I structure a docs page for AI search?
Give each real question a page, provide the answer in the first two sentences, use headings that name the question, tables for anything comparable, and a copy-paste example where it makes sense. The Diataxis framework (tutorial, how-to, reference, explanation) helps you avoid blending page types together.
Which part of my docs gets cited most?
Factual, structured content, such as API reference tables, error tables, and focused how-to pages. Long blended "overview" pages tend to lose out to more specific chunks elsewhere.
Can I generate my docs with an LLM to move faster?
You can, but it will probably cost you down the line. Unverified AI docs hallucinate confidently, and your paying customers are the ones who hit the errors. A human who actually knows the product is still the difference between docs that are a tool and docs that cause frustration.
