Гайди
What Is llms.txt, and Should Your Site Have One?
llms.txt is a proposed Markdown index for helping AI tools find useful content. It is not robots.txt, not a licensing mechanism, and not a proven ranking shortcut.
Запусти свіжий аудит DomainLens і використовуй звіт як список пріоритетів.
The honest status of llms.txt
The llms.txt proposal suggests publishing a Markdown file, usually at /llms.txt, that summarizes a site and links to the most useful machine-readable resources. It can reduce discovery work for an AI tool that deliberately looks for the file.
It is not a web standard with universal crawler support, and publishing it does not create an obligation for an AI provider to read it. Treat it as an inexpensive navigation aid for documentation-heavy sites, not as evidence that content will be cited, indexed, trained on, or ranked.
| File or signal | Primary job | Can it enforce access? |
|---|---|---|
| llms.txt | Curate useful pages for compatible AI tools | No |
| robots.txt | Request crawler access restrictions by user agent | Only for compliant crawlers |
| XML sitemap | List canonical URLs for search discovery | No |
| Canonical | Indicate a preferred duplicate URL to search engines | No; it is a hint |
| Terms/licensing | State legal permissions and obligations | Not replaced by llms.txt |
A decision rule: publish it only if you can curate it
| Situation | Recommendation | Why |
|---|---|---|
| Versioned product or API documentation | Useful experiment | A short map can direct agents to current authoritative docs |
| Large knowledge base with clear canonical guides | Potentially useful | Curation can hide navigation noise and obsolete pages |
| Small brochure website | Usually skip | Normal crawlable HTML already explains the site |
| Site expects ranking or traffic gains | Do not use for that reason | No established search-ranking benefit |
| No owner for updates | Skip | A stale map can recommend removed or superseded material |
Write a small, opinionated file
Start with the site or product name as an H1, add a short blockquote describing the audience and scope, then group the best canonical resources under H2 headings. Link to stable Markdown versions when you maintain them, otherwise link to accessible HTML. Optional links should be genuinely secondary.
- Use canonical absolute URLs and one clear description per link.
- Prefer current source material over every page on the site.
- Exclude login pages, internal search, legal boilerplate, tags, parameters, and thin archives.
- Keep the file human-reviewable; a generated dump of the sitemap adds little value.
# Example Analytics API
> Documentation for developers integrating the current v2 API.
## Start here
- [Authentication](https://example.com/docs/auth): Required headers and token lifecycle.
- [API reference](https://example.com/docs/api): Current endpoints and schemas.
- [Errors](https://example.com/docs/errors): Status codes and retry behaviour.
## Optional
- [v1 migration](https://example.com/docs/migrate-v1): Changes for legacy clients.Do not confuse discovery, permission, and freshness
If you want to restrict a named crawler, use the controls that crawler documents and verify requests in server logs. llms.txt does not override robots.txt, authentication, rate limits, copyright, or your terms. Conversely, allowing a crawler in robots.txt does not guarantee that it will use links in llms.txt.
A file can also become wrong silently. Add it to the same release check as navigation and sitemaps: every linked URL should return 200, resolve to itself as canonical, and describe the current product version. Remove redirects and retired documentation rather than relying on the consumer to interpret them.
How to validate an llms.txt experiment
DomainLens can verify whether the file is reachable and can audit the linked pages for status, canonical, indexability, and content signals. Review access rules with the robots.txt best-practices guide . Neither check can prove that a particular AI vendor consumes the proposal or that the file affects model training.
- 1Serve /llms.txt as UTF-8 plain text or Markdown without authentication or a redirect chain.
- 2Fetch it from outside your network and confirm HTTP 200 plus the intended content type.
- 3Check every linked resource and its canonical target.
- 4Review access logs for user agents that request the file; do not infer adoption from the file existing.
- 5Track citations, referral traffic, support quality, or agent success against a baseline. Stop maintaining it if there is no measurable benefit.