How to let AI tools read your site: pricing pages, docs and JavaScript apps

Which pages matter most?

The plain-fact ones: what your product does, what it costs, what it works with, and how it compares. In practice that’s your home page, pricing, features, integrations, docs or help center, and any comparison pages. Your app itself, behind the sign-in, doesn’t need to be read by anyone’s crawler.

Can AI tools read the words on those pages?

Many software sites are built as JavaScript apps: the server sends a nearly empty page, and a script fills in the words in the visitor’s browser. People see the full page. Most AI crawlers don’t.

Vercel and MERJ measured this and found that “none of the major AI crawlers currently render JavaScript.” That covers the crawlers from OpenAI, Anthropic, Perplexity and Meta. Only Google’s crawler (which Gemini also uses) and Apple’s ran it (Vercel). That study is from December 2024. A 2026 review of the evidence found nothing to say it has changed, and warns against loading pricing tables or FAQs only in the browser (TripleDart). One article describes a site where a rendering fault meant GPTBot, ClaudeBot and PerplexityBot got pages with no words at all (Passionfruit).

To check a page yourself:

1. Open the page in a browser on a computer.
2. Open the page source: right-click and choose “View page source”, or press Ctrl+U (Option+Cmd+U on a Mac).
3. Search the source (Ctrl+F or Cmd+F) for a sentence you can see on the page, like the price of your cheapest plan.
4. If that sentence isn’t in the source, AI crawlers probably can’t read it either.

Pay special attention to pricing tables, plan grids, FAQs that open on click, and docs that load their text from another service. The fix is usually to render those pages on the server or build them ahead of time, a job for whoever builds your site.

Broken links matter too: in the same study, ChatGPT’s crawler hit missing pages on 34.8% of its visits, against 8.2% for Google’s (Vercel). When you move your docs or rename a plan, redirect the old address.

Which AI bots read websites?

AI companies send programs called crawlers, or bots, to fetch web pages. They come in three kinds, and the difference matters:

• Search bots build the index an AI tool searches when it answers a question.
• Live-fetch bots visit a page at the moment someone’s question needs it.
• Training bots collect pages that may be used to train future AI models.

The first two decide whether an AI tool can find and quote you today. These are the main ones, by the names they use when they visit:

Two things surprise people. First, Google’s AI Overviews and AI Mode use the normal Googlebot. Blocking Google-Extended doesn’t take you out of AI Overviews. It only covers AI training and some of Google’s other systems (Google Search Central).

Second, in December 2025 OpenAI removed the line saying ChatGPT-User follows robots.txt. Its view is that a fetch a person asks for isn’t automatic crawling (PPC Land). You probably don’t want to block it anyway: it’s the bot that reads your pricing page when a buyer asks ChatGPT about you.

How do I check my robots.txt?

robots.txt is a small text file that tells bots which pages they may visit. To see yours, type your website’s address followed by /robots.txt into your browser, like yourproduct.com/robots.txt. If your docs or help center live at another address, like docs.yourproduct.com, check the robots.txt there too.

Look for two things:

• Disallow: / under User-agent: *. That asks every bot to stay out of your whole site. If you find it, check it isn’t left over from a test site.
• Lines that name AI bots, like User-agent: GPTBot followed by Disallow: /. That blocks that one bot.

For most software companies, a simple file works: let every bot read the public site, and keep them out of the app and sign-in pages, which hold nothing a buyer needs.

User-agent: *
Allow: /
Disallow: /app
Disallow: /login
Disallow: /api

Sitemap: https://yourproduct.com/sitemap.xml

If you’d rather your pages weren’t used to train AI models, you can block the training bots and still let the search and live-fetch bots in:

# Keep these pages out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

# Everyone else, including AI search and live-fetch bots
User-agent: *
Allow: /
Disallow: /app

Which should you pick? For a small software company we’d let them all in: you want AI tools to know your product’s name and what it does. The cost is that your public pages may be used for training. Our own robots.txt lets every bot in.

Is Cloudflare blocking AI tools from your site?

Cloudflare is a service that sits in front of a website to make it faster and safer. If your site uses it, check its settings, because its rules for AI bots have changed several times:

• July 2024. A one-click option to block AI bots, which more than 1 million customers turned on (Cloudflare).
• July 2025. A “managed robots.txt”, on every plan including Free, which adds lines to your robots.txt that block AI training bots (Cloudflare). Many reports said new sites now block AI crawlers by default (Medium). Cloudflare’s own post describes the controls as opt-in, so look at your settings rather than guess.
• From 15 Sep 2026. For new sites and sites on the Free plan, Cloudflare blocks “Agent” and “Training” bots on pages that show ads, and allows “Search” bots (chudi.dev; Let’s Data Science). These are second-hand reports, and most software sites don’t show ads. Check anyway.

An “Agent” block matters most: it covers the live fetch when a buyer asks ChatGPT or Claude about you. If your site uses Cloudflare, log in and check:

1. Security > Bots > AI Scrapers and Crawlers. Make sure Search and Agent bots are allowed. The setting is per site, so check your marketing site and your docs site.
2. Managed robots.txt. Make sure it isn’t adding blocks you don’t want.
3. Any “block AI bots” switch. Make sure it’s set the way you want.
4. Your robots.txt again, in your browser, to see what bots actually get.

Do I need an llms.txt file?

No. An llms.txt file is a plain summary of your site written for AI tools, and there’s little sign that AI tools read it. In Ahrefs’ server logs for 137,000 sites, 97% of llms.txt files got no requests at all in May 2026. Google says the file won’t help or hurt your search results (PPC Land).

It costs little to add one, and we have one. Just don’t expect it to change who gets named.

Does schema markup help?

Schema markup, also called structured data, is code that describes your product in a standard format: its name, what kind of software it is, its plans. Google says “there’s also no special schema.org structured data that you need to add” for its AI features (Google Search Central).

In the one controlled test we found, Ahrefs compared 1,885 pages that added schema with 4,000 similar pages that didn’t, and found no clear lift in how often AI tools quoted them (Ahrefs). Keep it accurate, but don’t pay extra for it as a way to get named.

A five-minute checklist

1. View the source of your pricing page and search it for your cheapest plan’s price.
2. Do the same for your home page, one features page and one docs page.
3. Open yourproduct.com/robots.txt, and your docs site’s. Make sure neither blocks AI search and live-fetch bots.
4. If you use Cloudflare, check Security > Bots > AI Scrapers and Crawlers.
5. Ask whoever builds your site to fix anything that fails, and to redirect any old links that now show “page not found”.

That’s one of the jobs outshone does each week: we check that AI tools can read your pricing page, docs and comparison pages, and when something stops them, we draft the fix for you to approve. See the jobs we do in AI visibility.