Skip to content
Check my product
Check my product

robots.txt

robots.txt is a small text file at the top of a website, like yourwebsite.com/robots.txt, that tells crawlers which pages they may visit.

Each group in the file names a crawler (User-agent) and says what it may not fetch (Disallow). User-agent: * means every crawler. A line saying Disallow: / under it asks every crawler to stay out of the whole site.

It’s a request, not a lock. Anthropic says its crawlers follow it. In December 2025 OpenAI removed the line saying its live fetcher, ChatGPT-User, follows it, on the view that a fetch a person asks for isn’t automatic crawling (PPC Land).

Most software companies want their public pages open and their app, sign-in and API paths kept out. Check yours if you want AI tools to name you, because sites sometimes block AI crawlers without meaning to. Cloudflare, for example, offers a “managed robots.txt” that adds lines blocking AI training bots (Cloudflare). Our guide explains what to look for.

Our own robots.txt lets every crawler read every public page.

  • Updated
  • Written by [Author name]

Sources

Every number here links to where it came from. We checked each source on 6 Nov 2026.

See what AI tells your buyers.

Type your product's name or website. In about a minute you'll see what ChatGPT, Gemini, Perplexity and Google AI Overviews say when a buyer asks for a tool like yours.

Free. See who AI recommends instead of you, in about a minute.