FREE TOOL
Free AI crawler checker: is your robots.txt blocking the engines?
A free Searchmaxxed tool: your robots.txt, read against every AI crawler that matters.
THE TOOL
What this tool does
Fifteen AI crawlers read against your live robots.txt, with an allow or block decision for each one. Most blocks are accidental, inherited from a template or a plugin, and cost visibility in the assistants without saving meaningful bandwidth.
You enter a domain. We fetch the live file and resolve every known AI user-agent against your rules. You get the exact line behind each block and what that crawler feeds.
WHAT YOU LEAVE WITH
You leave knowing which of fifteen AI crawlers you allow, which you block, and the exact line in your robots.txt responsible for each block.
CRAWLER ACCESS
READY WHEN YOU ARE
Your evidence will appear here after the live run finishes.
The panel keeps the primary finding, supporting evidence and next action together so the result is easy to use.
THE SHORT ANSWER
An AI crawler checker reads your robots.txt and reports which AI crawlers you allow and which you block. It matters because blocking the wrong one removes your site from live AI answers. Blocking GPTBot only affects model training, while blocking OAI-SearchBot or ChatGPT-User removes you from what ChatGPT can find and fetch when a user is waiting.
WHY
Why this tool exists.
A lot of robots.txt files still block AI crawlers, set deliberately in 2024 or inherited from a developer who did it for you. The cost is a site the engines your buyers now ask cannot quote. This check reads your live robots.txt and reports the decision you have made for each crawler, with what that decision costs you.
A robots.txt tester tells you whether a rule matches. This tells you which answer engine you locked out. It also tells you whether that engine trains models or fetches pages at answer time, and what changes when you open it.
Which AI crawlers does this check?
Fifteen user agents across three roles. Answer-time fetchers pull your page while somebody waits for a reply. ChatGPT-User, Claude-User and Perplexity-User do that. Index crawlers build the set an answer draws on. Googlebot, OAI-SearchBot, PerplexityBot, ClaudeBot, meta-externalagent and Amazonbot do that. Permission tokens control corpus use. GPTBot, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider and CCBot do that.
Google-Extended and Applebot-Extended are permission tokens rather than crawlers. Blocking Google-Extended does not block Googlebot, and the two decisions have completely different consequences. That distinction is where most robots.txt files go wrong.
How does the robots.txt check work?
The tool fetches your live robots.txt. It resolves each crawler against your rules under RFC 9309. That covers user-agent groups and the most specific matching group. It also covers longest-match precedence, wildcards and end anchors.
The result reports a decision per crawler, quotes the exact directive responsible for every block, and counts how many live-answer crawlers are shut out. A missing robots.txt allows everything, which the tool reports as a finding rather than an error.
Why does blocking a crawler cost more than it saves?
Many sites blocked AI crawlers in 2024 to keep their content out of training data. That call was defensible. Most of those blocks were broad. They now also shut out the crawlers that decide whether you appear in an answer a buyer reads right now.
Blocking a training crawler protects a corpus. Blocking an answer-time fetcher removes you from the recommendation. Those are separate choices and they deserve separate decisions.
Who should check their robots.txt?
Anyone who did not write the file, which is most site owners. Robots.txt arrives from developers, migrations, platform defaults and security plugins. Nobody reviews it until something goes missing.
It takes seconds and costs nothing, and the outcome is a deliberate decision per engine rather than an accident.
Does allowing a crawler guarantee you get quoted?
Access comes first. An open robots.txt makes you eligible for a read. A quote still depends on whether the page answers the question and whether the engine trusts you.
WHAT YOU GET
You leave knowing which of fifteen AI crawlers you allow, which you block, and the exact line in your robots.txt responsible for each block.
CHECK MY ROBOTS.TXTHOW TO USE IT
Enter your domain.
We fetch your live robots.txt.
Every known AI user-agent is resolved against your rules.
You get an allow or block decision per crawler, and what each one controls.
EXPLORE OUR OTHER TOOLS
Eleven more tools, all free to run.
Each Searchmaxxed tool answers a different question against live data. Nothing here needs a trial, a credit card or an account.
See where you are invisible
Free AI visibility checker: see where you are invisible
A free Searchmaxxed tool: one domain, the buying question your market asks, and who three engines name instead of you.
RUN THE CHECK
Who Google ranks, and who its AI cites
Free AI overview checker: who Google ranks, and who its AI cites
A free Searchmaxxed tool: the same search, two result sets, side by side.
COMPARE THE TWO
Can an answer engine use this page?
Free AEO checker: can an answer engine use this page?
A free Searchmaxxed tool: one URL, the structural findings, and the fixes in order.
CHECK THE PAGE
Find your real position
Free keyword rank checker: find your real position
A free Searchmaxxed tool: one keyword, one domain, the live position and what sits above it.
CHECK THE RANKING
100 ideas with volume and difficulty
Free keyword research tool with volume and difficulty
A free Searchmaxxed tool: one seed keyword, 100 real ideas, grouped by what the searcher wants.
GET KEYWORD IDEAS
See who links to any domain
Free backlink checker: see who links to any domain
A free Searchmaxxed tool: one domain, its real link profile, and what the links are worth.
CHECK THE BACKLINKS
Core Web Vitals and crawlability
Free website speed test: Core Web Vitals and crawlability
A free Searchmaxxed tool: one page, its real performance, and what a crawler sees while it waits.
TEST THE PAGE
Do you appear in the map pack?
Free local rank checker: do you appear in the map pack?
A free Searchmaxxed tool: the live local pack, the competitors in it, and your profile gaps.
CHECK THE LOCAL PACK
Built from your live site
Free llms.txt generator, built from your live site
A free Searchmaxxed tool: your real URLs, grouped, described and ready to publish.
GENERATE THE FILE
Markup an answer engine can use
Free schema markup generator for answer engines
A free Searchmaxxed tool: entity, service and FAQ markup in one block, ready to paste.
GENERATE THE MARKUP
The month it pays for itself
Free SEO forecast: the month it pays for itself
A free Searchmaxxed tool: twelve months, three scenarios, and every assumption on the table.
RUN THE FORECAST
QUESTIONS
What people ask about this tool.
Which crawlers does the check cover?
+
Fifteen agents in total. The check covers GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI, plus ClaudeBot, anthropic-ai and Claude-User for Anthropic. It also covers Google-Extended, Applebot-Extended, PerplexityBot, Perplexity-User, Bytespider, CCBot, meta-externalagent and Amazonbot.
What is the difference between a training crawler and an answer-time fetcher?
+
A training crawler collects pages to train a model. An answer-time fetcher retrieves your page while a user is waiting for an answer. Blocking the second one removes you from live answers, which is usually the expensive mistake.
Should I allow every crawler?
+
That is a commercial decision, not a technical one. The check gives you the current state per crawler so the decision is deliberate.
Does allowing a crawler guarantee I get quoted?
+
No. Access is the precondition. Being quoted still depends on whether the page answers the question and can be trusted.
Win the searches this tool just showed you are losing.
We build the pages, the structured data and the source coverage that change what this tool reports, then hold the position while it compounds. Book a call and we will open your own data live and show you what your market is worth.
LET'S TALKWin the searches this tool just showed you are losing
Build the agentic website. Then run the weekly search system that turns it into qualified demand and revenue.