Does LLMs.txt Help AI Visibility? What Google, Perplexity, and Server Logs Actually Show
You don’t need llms.txt for Google Search. It may have limited use for Perplexity, Claude, and developer documentation, but there’s little evidence that it improves AI visibility. If it’s easy to maintain, you can add one, but focus first on making your site easy to crawl, keeping content clear and useful, and configuring robots.txt correctly.
Key highlights
- Google says Search does not use llms.txt, and no major AI platform has confirmed support.
- 97% of llms.txt files received zero requests in a 137,000-domain log analysis.
- Sites with the llms.txt file are cited about as often as sites without it.
- Crawl access and answer-first content move AI visibility far more than any extra file.
What is llms.txt?
LLMs.txt is a proposed Markdown file placed at a site’s root that lists key pages with short descriptions, so AI tools can find important content without parsing menus, scripts, and ads.
Does Google use llms.txt?
Google Search does not use llms.txt. In its own to optimize for GenAI features, Google said that no special AI markup file helps in improving AI visibility.
In June 2025, Google’s Search Advocate John Mueller stated on Bluesky that no AI system currently uses llms.txt. He had previously compared the file to the obsolete keywords meta tag, arguing that AI systems can verify a website’s actual content rather than rely on self-reported descriptions.
In a separate June 2026 episode of Google’s Search Off the Record podcast, Mueller added that such files cannot help AI systems decide which websites to surface because they cannot be trusted to differentiate between competing sites. However, an automated agent already visiting a website might find the file useful for navigation.
Does llms.txt help SEO or AI citations?
LLMs.txt does not improve rankings or AI citations. Citation studies show almost no gap between sites with and without the file, and server-log data explains why: the bots that produce citations rarely request it.
According to Trakkr’s analysis of 37,894 AI-cited domains, sites with llms.txt averaged 6.8 citations, compared with 6.7 for sites without it. The difference was not statistically significant among the top 5,000 domains, although the full-sample analysis found a statistically significant but very small effect.
Separately, SE Ranking’s analysis of nearly 300,000 domains found no meaningful relationship between llms.txt presence and AI citation frequency.
Together, these studies provide little evidence that implementing the llms.txt file improves AI visibility. Content quality, website authority, and accessibility deserve greater attention than treating llms.txt as a citation shortcut.
What do server logs show about llms.txt?
Server logs show that AI crawlers rarely fetch llms.txt, even on websites that publish it. Ahrefs’ analysis of 137,000 domains found that 97% of published files received zero requests, while a separate 12-week test showed that OpenAI and Claude crawlers fetched robots.txt thousands of times more often than llms.txt.
Key findings:
- 97% of published files received zero requests: Ahrefs’ May 2026 analysis found that 28% of 137,000 domains had an llms.txt file, but 97% of those files received no requests during the study period.
- AI retrieval bots rarely request the file: Bots associated with ChatGPT and Perplexity accounted for approximately 1% of requests.
- Most requests came from other tools: Audit and scanning tools accounted for 12% of requests, coding agents 10%, training crawlers 5%, and assistants 2%.
- AI bots don’t actively look for missing files: The study recorded no requests for llms.txt on domains where the file did not exist.
- Crawler behavior favors robots.txt: A separate 12-week test across 83 websites found:
- OpenAI’s crawler fetched llms.txt 7 times versus 3,990 fetches of robots.txt.
- ClaudeBot fetched llms.txt 9 times versus 3,120 fetches of robots.txt.
AI crawlers routinely check robots.txt, but rarely request llms.txt. However, these studies measure file requests, not whether bots use the information after fetching it.
Do Perplexity and Claude use llms.txt?
Perplexity and Claude both publish llms.txt for their own developer documentation, but neither has confirmed using it to choose sources for answers.
Perplexity’s file at docs.perplexity.ai lists documentation pages as Markdown links, with an “Optional” section for extras. Anthropic hosts llms.txt and llms-full.txt for its own docs. In both cases, the reader is a tool or coding agent that needs the right API page quickly.
Log data points in the same direction. PerplexityBot fetched llms.txt zero times across the 83 test sites, against 775 robots.txt fetches. In Ahrefs’ dataset, Slackbot fetched llms.txt more often than PerplexityBot did, and Claude Code and GPTBot were the top individual bots.
A short 30-day log review showed PerplexityBot and ClaudeBot fetching /llms.txt, so it’s worth checking your own logs before drawing firm conclusions.
LLMs.txt vs robots.txt: what is the difference?
Robots.txt controls which crawlers may access a site, and major crawlers follow it. LLMs.txt only suggests pages for AI tools to read, and no platform has committed to follow it.
| robots.txt | llms.txt | |
| Purpose | Allow or block crawler access | Point AI tools to key pages |
| Format | Plain-text directives | Markdown with links |
| Support | Followed by major crawlers | No major AI platform has confirmed support |
| Blocks bots | Yes, for compliant bots | No |
| Effect on AI visibility | Direct: blocked bots cannot cite the page | No measured effect |
That difference sets the priority order. A robots.txt rule blocking GPTBot, ClaudeBot, or PerplexityBot can remove a site from AI answers entirely, while no llms.txt file can offset it.
Should I add an llms.txt file to my website?
Add llms.txt when the site has developer documentation, an API, or a help center that coding agents may read. For ecommerce, local, and service sites, treat it as a low-priority extra: it costs little and shows no downside, but it shows no measurable gain either.
| Site type | Add llms.txt? | Why |
| API or developer docs | Yes | Coding agents fetch these files |
| SaaS with a help center | Optional | Low cost, possible agent benefit |
| Ecommerce, local or service site | Not a priority | No measured citation lift |
| Site blocking AI bots in robots.txt | Fix access first | Blocked bots cannot cite the site |
The work that does move AI visibility is less novel and more reliable:
- Confirm AI crawlers can reach key pages through robots.txt and the firewall.
- Check server logs for GPTBot, ClaudeBot, and PerplexityBot requests.
- Write answer-first sections with clear headings and verified facts.
- Add schema markup and keep key pages current.
Where llms.txt actually belongs
LLMs.txt is a documentation convenience, not an AI visibility lever. Treating it as a ranking tactic pulls time away from crawl access, content structure, and log monitoring, which are the things AI systems demonstrably respond to.
For a clearer picture of how AI crawlers interact with your website, get a free AI visibility audit from our AI SEO experts. We help you identify and fix crawling, access, and content issues first, then decide whether implementing llms.txt makes sense for your site.
FAQs
Is llms.txt an official web standard?
No. llms.txt is a community proposal, not a standard approved by any standards body or adopted by a major AI platform. robots.txt and sitemaps went through formal adoption; llms.txt has not.
Can an llms.txt file hurt SEO?
No, having or not having an `llms.txt` file does not hurt your SEO or AI visibility. Just keep its links updated to avoid pointing AI tools to outdated or removed pages.
Should llms.txt be noindexed?
Yes, you can noindex llms.txt to prevent it from appearing in search results instead of your actual website pages. It’s a simple precaution that keeps the file out of human-facing search results.
What is the difference between llms.txt and llms-full.txt?
LLMs.txt is an index of links with short descriptions. LLMs-full.txt bundles the complete text of those pages into one file. Documentation sites use the full version; most business sites don’t need to.
