How AI Finds Your Documentation: llms.txt, Sitemaps, and Source Pages Explained
How do AI systems find and choose documentation to cite? Learn the differences between llms.txt, sitemap.xml, and source pages and what they mean for your knowledge base visibility in AI answers.
I’m a big fan of outsourcing chores. If a robot can do it, I’m usually the first in line 🙋
And that’s why I bought a robot lawnmower; I’m way too lazy to spend my Saturdays pushing a heavy machine around the garden. But last weekend, my "convenience" decided to throw a tantrum.
It got stuck in a state where its lights started flashing like a Christmas tree, and the thing just wouldn't turn off. Each time I turned it off, it turned back on 😮💨
Naturally, I didn't want to hunt down the paper manual I shoved somewhere, probably under a pile of boxes. So, I asked an AI.
It confidently told me to "hold the power button for 30 seconds" Sounds reasonable, right? Except the thing didn't listen, and no matter how long I pressed it for (I was bending down for minutes at one point), it would turn back on. Following those instructions almost led to a very expensive paperweight.
The AI just made it up because it got lost in a sea of similar-looking documentation.
This is the reality for a lot of our customers' users. We’re moving from a world of "Google-fu" to "AI-fu," and the stakes have changed. If your knowledge base isn't set up correctly, these AI systems aren't just going to miss your content, they're going to hallucinate a version of it that makes your support team's life a nightmare.
Become an expert in all things Knowledge Base with our monthly newsletter. No spam, just expert content, delivered.
There’s a massive misconception floating around that if your docs are "crawlable," you’ve won. If Google can see it, ChatGPT can use it, right? Not exactly.
Making documentation discoverable is just the first step. Making it usable and, more importantly, trustworthy for a reasoning system is a whole different ball game. AI visibility isn't a single switch you flip. It's a combination of discovery, intent, and authority signals that tell a machine whether your content is worth repeating.
Let's dig into this a little more.
The Sitemap Is Not The Territory
Think of your sitemap.xml as a basic map of a city. It tells the AI crawlers that "Street A" and "Building B" exist. It’s a discovery mechanism, and it's been the gold standard for search engines for decades. If you don't have one, you're basically invisible, just a ghost in the machine 👻
But here’s the catch: sitemaps were designed for indexing, not for reasoning.
A search engine wants to know which pages match a keyword. A reasoning system (like an LLM) wants to know which pages provide the definitive answer to a complex problem. A sitemap doesn’t tell an AI which page is the "canonical" answer and which one is an outdated legacy version from 2019.
River Sloane
It doesn’t provide context or explain that "this page is the master guide for our API." It just says, "Here is a URL."
When an AI tries to synthesize an answer, it needs more than a list of addresses. It needs to know which buildings are actually worth entering. If you have five different versions of a "Setup Guide" in your sitemap, the AI is essentially flipping a coin.
Without more signals, it might choose the one written for a version of your software that hasn't existed since jelly shoes were in fashion.
Giving AI A Helpful Nudge With llms.txt
Lately, everyone’s talking about llms.txt. It’s the new "cool kid" on the block, a simple text file that basically acts as an intent signal. We've recently started supporting it.
It’s your way of whispering to an AI agent, "Hey, these are the pages you should actually care about, and here’s how they're structured."
"You can’t put a fancy 'AI-friendly' label on top of a dumpster fire of a knowledge base and expect it to work."
It’s a great tool for indicating which pages are documentation and how an AI should approach them. You can use it to point to a summary file or a specific directory that holds your most important logic.
But let's be real: it’s not a formal standard yet, and different AI systems interpret it differently. Some might treat it as gospel; others might ignore it entirely.
Most importantly, llms.txt isn't a silver bullet or some magical growth hack.
River Sloane
You can’t put a fancy "AI-friendly" label on top of a dumpster fire of a knowledge base and expect it to work. If your content is outdated, unclear, or messy, an llms.txt file is just a map to a mess 🚮
It helps with the "how" of approaching your content, but it can't fix the "what" if the content itself is broken.
The Power Of A Source Page
If discovery is the map and intent is the nudge, authority is the destination.
This is where source pages or canonical documentation hubs come in. A source page is a high-level, human-readable entry point that defines a topical boundary. Think of it as the "Home Base" for a specific feature or product. We have a feature to generate your own source page here.
AI systems absolutely love stability and clear hierarchies. When you have a strong internal linking structure that points back to a central "hub" for a specific topic, you're sending a massive authority signal. You're telling the AI, "Everything under this umbrella is the truth." This is how AI systems decide what to trust and reuse when they have conflicting information.
"When an AI sees a page that is heavily linked to by other authoritative pages on your site, it marks that content as 'high-confidence'."
Documentation often fails to show up in AI answers because it’s too fragmented. If your answer is buried three layers deep in a "Miscellaneous" folder with no links from your main pages, the AI is going to assume it’s not important.
River Sloane
It favors well-linked, clearly scoped content because that’s how it determines what to trust. When an AI sees a page that is heavily linked to by other authoritative pages on your site, it marks that content as "high-confidence."
If your docs are a flat list of 500 unlinked articles, the AI sees 500 guesses, not one certain answer.
So What Should You Actually Use?
It's understandable to feel a bit overwhelmed with all these files and standards flying around. If you're wondering where to start, think of it as a hierarchy of needs for your robots.
First and foremost, don't ignore the basics.
You absolutely need a clean sitemap and a logical internal structure. These are the foundations of discovery. If an AI can’t find your pages, nothing else matters. You should also prioritize building out those source pages. They are the single most effective way to communicate authority to both humans and machines.
When it comes to the newer stuff like llms.txt, I’d say it’s a "nice to have" once your house is in order.
It’s worth implementing if you have a massive, complex knowledge base where you really need to guide an AI's attention to specific "primary" files. But if you’re still struggling with broken links and outdated articles, your time is better spent cleaning those up first.
A robot can ignore a text file, but it’s much harder for it to ignore a well-linked, highly authoritative hub of information 🏗️
Why Most Knowledge Bases Fail
Some perfectly public, indexed knowledge bases never show up in AI-generated summaries. It usually boils down to the fact that the documentation was built for a search bar, not for a conversation.
River Sloane
When we look at why these sites fail, it's almost always a combination of these factors:
- Fragmentation: The answers are split across ten different pages instead of one comprehensive guide. If a user has to click "Next Page" five times to finish a setup, the AI might only read the first page and guess the rest
- No Entry Point: There’s no "Source Page" to anchor the topic. Without a central hub, the AI doesn't know where the "official" version of a topic begins and ends
- Internal Linking: The site structure is a flat list rather than a meaningful hierarchy. If every page is treated with equal importance, the AI has no way to distinguish a minor tip from a critical safety warning
- Hidden Context: Important answers are buried in tables or poorly labeled images that the AI can't parse effectively. If your "Error Codes" are just an unlabelled list, the AI won't know they're related to your robot lawnmower's Christmas-light tantrum
AI systems are remarkably good at spotting patterns. They reward well-organized documentation because, ironically, what's easy for a human to navigate is also easier for a machine to interpret. When your structure is logical, the AI doesn't have to guess; it just knows 🧠
Build For Humans And The AI Will Follow
At the end of the day, you don't need "AI hacks" or complex technical workarounds. You don't need to spend weeks optimizing for a specific version of GPT. You need a knowledge base that's built with intention.
Focus on creating authoritative entry points, keep your URLs stable, and make sure your internal linking actually makes sense.
When you prioritize a clean, hierarchical structure, you're doing the heavy lifting for the AI. You're giving it the context it needs to distinguish between a "blade guard" and a "reset button" without it having to hallucinate the details.
When your documentation is easy for a person to read and navigate, it becomes infinitely more reliable for an AI to fetch and summarize. We don't need to overcomplicate it.
The best way to make your docs "AI-ready" is simply to make them excellent for the people who actually use your product every day. Just build something great for your users, and the robots will figure it out along the way 🤖