AI crawler

An AI crawler is a program an AI company uses to visit web pages and read them. Some collect pages to train AI models, some build a search index, and some fetch a page live when someone asks a question.

Also called AI bot

Each crawler gives a name when it visits. OpenAI’s are GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic’s are ClaudeBot, Claude-SearchBot and Claude-User (PPC Land). You can let each one in, or ask it to stay out, in your robots.txt file.

They’re busy. In one month in late 2024, Vercel counted 569 million visits from GPTBot and 370 million from Claude’s crawler on its network, against 4.5 billion from Googlebot (Vercel).

The catch is that most of them don’t run JavaScript. Vercel and MERJ found that “none of the major AI crawlers currently render JavaScript,” apart from Google’s and Apple’s (Vercel). If your site is a JavaScript app that fills in its words in the browser, these crawlers may see an empty pricing page. Our guide shows how to check.

Updated 6 Nov 2026
Written by [Author name]

Sources

Every number here links to where it came from. We checked each source on 6 Nov 2026.

PPC Land Anthropic's three crawlers and OpenAI's ChatGPT-User change
ppc.land Checked 6 Nov 2026
Vercel The rise of the AI crawler
vercel.com Checked 6 Nov 2026
Measured with MERJ on Vercel's network, late 2024.

See what AI tells your buyers.

Type your product's name or website. In about a minute you'll see what ChatGPT, Gemini, Perplexity and Google AI Overviews say when a buyer asks for a tool like yours.