AI crawler checker

Reads a site's robots.txt and shows which AI crawlers it allows or blocks, from GPTBot and ClaudeBot to Google-Extended.

Put Ken on the rest of your outbound

Inboxes, warmup, leads and copy in one place. Ken runs these checks on every inbox it manages.

Start free - 1,000 credits, no card

What the AI crawler checker does

Enter a domain and the checker reads its robots.txt, then reports which AI crawlers it allows or blocks. It covers OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, Common Crawl's CCBot, Meta-ExternalAgent, Bytespider and others.

Training, search and user fetches

AI companies run separate bots for separate jobs, and blocking one doesn't block the others. GPTBot and ClaudeBot collect training data. OAI-SearchBot and Claude-SearchBot index pages for AI search answers. ChatGPT-User and Claude-User fetch a page when a person asks the assistant to read it.

Google-Extended and Applebot-Extended aren't crawlers. They're names you use in robots.txt to say whether content that Googlebot or Applebot fetched may be used for AI models. Blocking Google-Extended doesn't remove you from Google Search.

How to read and change the result

Each bot shows as allowed, blocked or partly blocked, with the robots.txt group that decided it. Under RFC 9309 a crawler follows the group that names it and falls back to the * group, and when rules conflict the longest matching path wins.

To block a bot, add a group with User-agent and its name, then Disallow: /. To allow one bot while blocking the rest, give it its own group with Allow: /. robots.txt is a request, not a lock, so a bot that ignores it needs a firewall or CDN rule.

Why it matters for cold email

When a prospect asks an AI assistant about your company after reading your email, the answer depends on what that assistant could read. If robots.txt blocks the search and user-fetch bots, the assistant may know little about you or lean on other sites.

A CDN or CMS setting can add AI crawler rules to robots.txt without anyone on your team deciding to. Check what yours says.

Questions

How do I block GPTBot?

Add a group to robots.txt with the line User-agent: GPTBot followed by Disallow: /. That stops OpenAI from collecting your pages for training. To stay out of ChatGPT search results as well, add the same group for OAI-SearchBot.

Does blocking Google-Extended affect my Google rankings?

No. Google says Google-Extended has no effect on inclusion or ranking in Google Search. It controls whether your content is used for Gemini models and grounding. AI Overviews in Search follow Googlebot and your snippet settings instead.

Which AI crawlers should I allow?

It depends on what you want. Allowing search and user-fetch bots lets AI assistants read and cite your pages when people ask about you. Training bots are a separate decision, and you can block them while allowing the others.

Do AI crawlers respect robots.txt?

The documented training and search crawlers from OpenAI, Anthropic, Google, Apple and Perplexity are stated to follow it. User-triggered fetchers, which act for a person asking about a page, don't always. robots.txt is voluntary under RFC 9309, and some scrapers ignore it.

What is the difference between ClaudeBot and Claude-User?

ClaudeBot collects public pages that may be used to train Anthropic's models. Claude-User fetches a page when a Claude user asks about it. Claude-SearchBot indexes pages for search inside Claude. Each one follows its own robots.txt group.