Free tool

llms.txt Checker & AI Bot Access Test

Enter a domain to see whether it publishes an llms.txt, and whether its robots.txt lets AI crawlers like GPTBot, ClaudeBot and PerplexityBot read it. You get copy-paste fixes and a generated llms.txt template.

Free, instant, no signup. Checks robots.txt, llms.txt, sitemap.xml and JSON-LD.

The AI crawlers this tool checks

Each one reads your robots.txt separately.

User-agentCompanyWhat it does
GPTBotOpenAICrawls pages to train and ground OpenAI models
OAI-SearchBotOpenAIIndexes pages so they can appear in ChatGPT Search
ChatGPT-UserOpenAIFetches pages live when a ChatGPT user asks about them
ClaudeBotAnthropicCrawls pages for Claude
PerplexityBotPerplexityIndexes pages so Perplexity can cite them
Google-ExtendedGoogleControls use of your content in Gemini and AI grounding

FAQ

llms.txt and AI crawler questions

What is llms.txt?

llms.txt is a proposed convention: a Markdown-formatted text file at the root of your site (yourdomain.com/llms.txt) that gives large language models a concise summary of what your site is about and links to its most useful pages. Think of it as a curated, human-written map for AI agents.

How do I add llms.txt to my website?

Create a plain-text file named llms.txt with an H1 title, a one-paragraph summary in a blockquote, and sections of Markdown links to your key pages. Upload it to the root of your site so it is served at /llms.txt. This checker generates a starting template from your homepage that you can edit and publish.

Does llms.txt improve my visibility in ChatGPT or Gemini?

No AI vendor has committed to using llms.txt as a ranking signal, so treat it as cheap, low-risk hygiene rather than a guaranteed win. What reliably matters is that AI crawlers are allowed to read your site, that your pages answer buyer questions clearly, and that trusted third-party sources mention you.

Which AI bots should I allow in robots.txt?

If you want to appear in AI answers, allow GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Gemini). OAI-SearchBot and PerplexityBot matter most for being cited in live AI search results.

What happens if I block GPTBot?

Blocking GPTBot stops OpenAI from crawling your pages for model training. ChatGPT Search uses a separate crawler, OAI-SearchBot, so you can block one and allow the other. This tool shows the status of each bot separately.