Skip to content

Playbooks

Can AI crawlers read your site?

Godfrey, lessClick

AI crawlers can only quote what they can read. Two checks tell you whether they can read your site: turn JavaScript off and see what is left on your key pages, and check that your robots.txt does not block the crawlers AI assistants use. Anything that disappears without JavaScript should move into the page’s HTML.

Why does JavaScript matter?

JavaScript matters because some crawlers read the HTML a server sends and do not run the scripts that build the rest of the page. If your product description, pricing or answers to common questions only appear after JavaScript runs, those crawlers may see an empty page.

Google’s crawler runs JavaScript, though sometimes later than it reads the HTML. Many AI crawlers are less patient. The safe rule is simple: every word that matters should be in the HTML before any script runs.

Vercel’s study of AI crawler traffic, published in December 2024, found that the major AI crawlers it measured, including those run by OpenAI, Anthropic and Perplexity, fetch pages without running their JavaScript. Gemini, which uses Google’s own crawling, was the exception. Crawler behaviour can change, which is why the safe rule above does not depend on it.

How do you run the JavaScript-off check?

The check takes five minutes per page in any desktop browser:

  1. Open the page in Chrome or Edge.
  2. Open developer tools, then the command menu (Ctrl+Shift+P, or Cmd+Shift+P on a Mac).
  3. Type “Disable JavaScript” and press Enter.
  4. Reload the page and read what is left.
  5. Turn JavaScript back on when you are done.

Check your homepage, pricing page, main product pages and any comparison pages. Look for missing text, empty sections, prices that vanish, tabs or accordions whose contents are gone, and reviews loaded by a widget.

What should move into static HTML?

Move anything a buyer or an assistant would need to describe your product:

  • What the product is and who it is for.
  • Prices and plan names.
  • Answers to common questions. Use the HTML <details> element for accordions: the text stays in the page and opens without a script.
  • Comparison tables.
  • Customer quotes and case studies, if they are loaded by a third-party widget.

Animations, chat widgets and calculators can stay in JavaScript, as long as the facts they show are also in the text.

Which crawlers should robots.txt allow?

Robots.txt is the file at the root of your site that tells crawlers what they may read. A setting added years ago, or a security tool, can block AI crawlers without anyone noticing.

These are the user agents most relevant to being named in AI answers:

User agent Run by What it is for
OAI-SearchBot OpenAI Showing sites in ChatGPT search results
ChatGPT-User OpenAI Fetching a page for a user’s action in ChatGPT, not automatic crawling
GPTBot OpenAI Collecting content that may be used to train OpenAI’s models
Claude-SearchBot Anthropic Indexing pages to improve Claude’s search results
Claude-User Anthropic Fetching a page when a Claude user asks a question
ClaudeBot Anthropic Collecting content that may be used to train Anthropic’s models
PerplexityBot Perplexity Showing and linking sites in Perplexity search results
Perplexity-User Perplexity Fetching a page for a user’s action in Perplexity
Google-Extended Google A robots.txt token that controls whether content is used to train Gemini models; it does not affect Google Search

Each row follows the vendor’s own documentation: OpenAI, Anthropic, Perplexity and Google. Crawler names change, so check those pages before editing your own robots.txt. Be careful which ones you block: OpenAI’s documentation says a site that blocks OAI-SearchBot is not shown in ChatGPT’s search answers.

The simplest robots.txt for a marketing site allows everything:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If you want to block training but allow search, add a separate group for the training crawler. Blocking a search crawler means that assistant cannot read your pages when a buyer asks.

How else can a crawler be blocked?

Robots.txt is not the only gate. A firewall or bot-protection service can turn crawlers away before they ever read robots.txt. If your site sits behind one, check its settings for AI crawlers and verified bots.

lessClick’s own site is built to pass these checks: every word is in static HTML, the questions use <details>, and robots.txt allows every crawler.