Free tool
More people now ask ChatGPT, Claude or Perplexity to recommend a local business. Enter a website to see which AI crawlers its robots.txt lets in, which it keeps out, and the rule that decides.
Each AI company runs more than one crawler, and a site can treat them differently. For a local business, the first two kinds are the ones that bring customers.
OAI-SearchBot, Claude-SearchBot, PerplexityBot
They read sites so the assistant can show and link to them when someone asks a question, such as which plumber nearby is open on Sundays. Blocking them takes the site out of those answers.
ChatGPT-User, Claude-User, Perplexity-User
They visit one page because a person asked the assistant to look at it. Blocking them means the assistant can't open the page when asked.
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider
They gather text used to train AI models. Blocking them is a reasonable choice and doesn't remove the site from the same companies' answers. Google-Extended, for example, has no effect on Google Search or its AI Overviews.
A crawler named in a rule of its own follows only that rule. Otherwise it follows the rule for all robots, written User-agent: *. Where several lines match, the longest one decides, and a tie goes to allow.
Blocked means the homepage is off limits. Only some pages means the site is closed by default with a few pages let back in. Rules that close an admin or checkout area are normal and still count as allowed.
Open the site's robots.txt file and remove the lines that name that crawler, or the Disallow: / line under it. On WordPress, an SEO plugin may manage this file for you.
Some hosts and security services, Cloudflare among them, can add these rules for you or block AI crawlers at their firewall. A firewall block doesn't show in robots.txt, so check the service's bot settings too.
Being readable is the first step. Assistants also need clear facts to repeat: what the business does, where, and its hours. See the AI Readiness checks.
Seven growth scores and a prioritized fix-list, in about a minute. Your first audit needs no account.
Get your free audit