AI crawlers list: GPTBot, ClaudeBot, PerplexityBot and more (2026)

AI crawlers are the bots that companies like OpenAI and Anthropic send to read your website, and they do three jobs. Training crawlers such as GPTBot and ClaudeBot collect pages for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index that ChatGPT, Claude and Perplexity search before they write an answer. User fetchers like ChatGPT-User read a page live when someone asks about it. Each bot has its own robots.txt token, and most vendors publish the IPs their bots crawl from, so you can check a visit is real.

For a B2B company, search crawlers and user fetchers matter most, because they decide whether an AI answer can read and name you. That's where shortlists get made now. Here at Searchmaxxed, we make your company the answer in Google and in the AI engines your buyers ask, from ChatGPT to AI Overviews. Letting the right crawlers in comes first.

Which AI crawlers should you know in 2026?

You need to know about 27 crawlers and tokens from 11 companies, led by OpenAI, Anthropic, Perplexity and Google. Each row gives the token, the job its operator describes and where it publishes IP addresses, from that operator's crawler page on 1 November 2026.

Sources: the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft Bing, Meta, Amazon, Mistral, DuckDuckGo and Common Crawl.

Two of those names never visit. Google-Extended and Applebot-Extended are permission tokens. You name them in robots.txt, but Googlebot or Applebot does the crawling, and the token controls how their AI models may use the content. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

Bytespider, linked to ByteDance, turns up in plenty of robots.txt files, yet no operator page above documents it. Treat it as unverified.

What is GPTBot?

GPTBot is OpenAI's crawler for training its foundation models. Blocking it tells OpenAI not to train on your content, but it has no effect on whether ChatGPT search can show your pages. OAI-SearchBot does that job.

OpenAI runs four named bots and says each robots.txt setting "is independent of the others", so you can allow OAI-SearchBot and disallow GPTBot. If you allow both, OpenAI may reuse a single crawl for both jobs, and a robots.txt change takes about 24 hours to reach ChatGPT search.

GPTBot and OAI-SearchBot may also add a robots.txt marker to their user agent when they fetch that file. It shows even in logs without paths.

Should you block GPTBot? That's a business call about training, and our guide to whether you should block AI crawlers works through it by company type. For most B2B sites, leave it open.

What is the ChatGPT user agent?

ChatGPT visits websites as ChatGPT-User, and its full user agent string is `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`. Seeing it means a person asked ChatGPT about your page.

OpenAI says ChatGPT-User doesn't crawl "in an automatic fashion", and because a person started the request, "robots.txt rules may not apply". It doesn't decide search inclusion either. OAI-SearchBot does, and its string reads `...Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`.

ChatGPT's agent trips people up.

When ChatGPT's agent browses for a user, its user agent hides that, as Simon Willison found when he logged a visit in August 2025. It looked like plain Chrome. Instead, OpenAI's allowlisting guide says its Cloud browser signs each request with Web Bot Auth, the HTTP Message Signatures standard (RFC 9421). Look for three headers: Signature, Signature-Input and Signature-Agent: "https://chatgpt.com". Cloudflare, Akamai, HUMAN and Vercel check those signatures for you, and on Cloudflare, OpenAI points to bot detection ID 129220581 for your allow rule.

Expect to see it a lot. Cloudflare reported that in July 2025, ChatGPT-User made nearly three quarters of all user-action requests from AI bots across its network. Training still dominated overall, at nearly 80% of AI bot crawling.

What is PerplexityBot?

PerplexityBot is Perplexity's search crawler. It finds and links the websites in Perplexity's answers, and Perplexity says it "is not used to crawl content for AI foundation models". Its user agent reads Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot).

Perplexity runs one more bot. Perplexity-User fetches a page when a user's question needs it, and Perplexity says that fetcher "generally ignores robots.txt rules". To control it, use your firewall.

Perplexity's crawler page gives one firewall tip worth copying for every vendor: match each user agent against its own IP list, never a shared one. In Cloudflare:

(http.user_agent contains "PerplexityBot" and ip.src in {<PerplexityBot IP ranges>})
or (http.user_agent contains "Perplexity-User" and ip.src in {<Perplexity-User IP ranges>})

What are ClaudeBot, Claude-SearchBot and Claude-User?

They're Anthropic's three bots. ClaudeBot collects content that could train Claude, Claude-SearchBot indexes pages for Claude's search results, and Claude-User fetches a page when someone asks Claude about it. Anthropic says blocking either of the last two "may reduce your site's visibility" in Claude's answers.

Three differences from OpenAI matter:

• Anthropic publishes one IP file, claude.com/crawling/bots.json, for all three bots.
• It doesn't publish full user agent strings, so match on the token (ClaudeBot, Claude-SearchBot, Claude-User) inside the string.
• It honours the non-standard Crawl-delay rule, which helps if ClaudeBot hits a fragile server hard.

Anthropic wants robots.txt on every subdomain and warns that IP blocks may fail as an opt-out, since a blocked bot can't read the file.

Which Google, Apple, Microsoft and Meta bots feed AI answers?

Mostly their classic search crawlers. Googlebot feeds AI Overviews and AI Mode, Bingbot feeds Copilot and Applebot feeds Siri, then most add AI-specific tokens and fetchers on top.

Google. Googlebot's rules cover "all Google Search features", so no separate AI Overviews bot exists to block. Google-Extended governs Gemini training and grounding. Google-Agent is newer: Google uses it when "agents hosted on Google infrastructure" browse and act for a user, and it ignores robots.txt.

Apple. Applebot crawls for Spotlight, Siri and Safari, and Apple says the data "may also be used to help train Apple foundation models". Disallow Applebot-Extended to opt out of training. If your robots.txt never mentions Applebot, Apple follows your Googlebot rules instead.

Microsoft. Bing documents no separate Copilot crawler. Its Webmaster Guidelines use meta tags instead. NOARCHIVE "prevents content from being used in Copilot responses", and NOCACHE limits Copilot to your URL, title and snippet.

Meta. Meta-WebIndexer helps Meta AI "cite and link to your content", while Meta-ExternalAgent collects content for "training foundation AI models" or indexing. Meta-ExternalFetcher fetches links at a user's request, and Meta says it "may bypass robots.txt rules".

Amazon. Amazonbot may train Amazon's AI models. Amzn-SearchBot powers search in products like Alexa, isn't for training and follows your rules for other search bots when you don't name it.

What are the exact AI crawler user agent strings?

Use these published strings to filter logs or write firewall rules, but match on the bot's name, because version numbers change. Where Google and Amazon show W.X.Y.Z, expect the current Chrome version.

Anthropic's bots carry their token in the string, though Anthropic doesn't publish the full text. The two permission tokens have none.

To pull every AI bot out of a raw access log in one pass, this pattern catches all the names above:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|Google-CloudVertexBot|GoogleOther|Applebot|bingbot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Amzn-SearchBot|Amzn-User|MistralAI|DuckAssistBot|CCBot|Bytespider" access.log

How do you tell a real AI crawler from a fake one?

Check where the request came from. Anyone can type "GPTBot" into a user agent. Operators give you three ways to check: a published IP list, a reverse DNS lookup, or a cryptographic signature. Common Crawl says it already sees crawlers "falsely identifying themselves as CCBot".

Reverse DNS takes two commands. For Googlebot, the hostname must end in googlebot.com, google.com or googleusercontent.com, and the forward lookup must return the same IP. Google's own example:

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Apple's hostnames end in applebot.apple.com, and Common Crawl's in crawl.commoncrawl.org.

How often do the AI crawler IP lists change?

Often enough that a hard-coded allowlist goes stale within weeks. On 1 November 2026 we downloaded 16 published IP files from nine vendors and read the date each file carries. Seven were under two weeks old.

Source: each vendor's JSON file, downloaded by Searchmaxxed on 1 November 2026. A recent date can mean the vendor regenerates the file on a schedule, not that the ranges changed.

ChatGPT-User's file lists 235 prefixes, six times OAI-SearchBot's 39, so a rule copied from one OpenAI bot won't cover another. The oldest dates from January 2024. Pull the files on a schedule, daily if your firewall allows it, and let a script rewrite your allow rules each time.

Do big B2B and cybersecurity companies block AI crawlers?

Rarely, and none of the 40 we checked blocked the AI search bots. On 1 November 2026 we read the robots.txt files of 40 large vendors, half B2B SaaS and half cybersecurity, and tested each AI bot.

Source: Searchmaxxed's read of 40 public robots.txt files on 1 November 2026, using RFC 9309 group matching. Firewall and CDN rules aren't visible in robots.txt, so a site can still refuse a bot the file allows.

Twelve of the 40 named at least one AI bot, and the other 28 left AI bots to their general User-agent: * rules.

One site caused most of the blocks. Its file disallows everything for User-agent: *, then opens the site to a named list of 28 bots, including Claude-SearchBot and Claude-User. ClaudeBot isn't on the list. Perhaps that's deliberate, but it shows how an allowlist behaves: it shuts out every new AI crawler until someone adds its name. Vendors keep adding bots, so someone has to own that list.

Which AI crawlers should a B2B company allow?

Allow every search crawler and user fetcher, plus Googlebot, Bingbot and Applebot, because those bots let ChatGPT, Claude, Perplexity, Copilot and AI Overviews quote you. Training crawlers are a separate call, and our guide to blocking AI crawlers covers when blocking them makes sense.

B2B shortlists depend on it. In G2's March 2026 survey of 1,076 B2B software buyers, 69% chose a different vendor than they'd planned because of an AI chatbot's guidance. A blocked site sits that out.

Here's our own setup. The live robots.txt on searchmaxxed.com groups bots by job and adds a Content-Signal line to every group:

# AI search and answer crawlers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Content-Signal is a robots.txt line, introduced by Cloudflare, that states how bots may use your content once they've fetched it. Its three values are search, ai-input (feeding content into AI answers, such as grounding) and ai-train. Not every crawler honours it, Cloudflare admits. So treat it as a statement of intent beside your Allow and Disallow rules, never a replacement. The same goes for llms.txt, another proposal that our guide to what llms.txt is and whether it helps covers in full.

Named groups make later changes safe. If we ever decide to block training, we edit one group and the search bots stay untouched.

How do you check which AI crawlers can reach your site?

Read your server or CDN logs. GA4 can't see crawlers, because they don't run its JavaScript tag. Work through these five checks:

1. Read your live robots.txt on every subdomain, docs and help included.
2. Search 30 days of logs with the pattern above, and count requests by bot and by site section.
3. Verify the IPs against each vendor's JSON file, so impostors don't skew the count.
4. Read the status codes. A run of 403 or 429 responses to OAI-SearchBot or PerplexityBot means your firewall is refusing it.
5. Check your CDN's bot settings, because a "block AI bots" toggle can override a perfect robots.txt.

Crawler access is step one. Our guide on how to show up in AI search covers the rest: answer-first pages, trusted outside sources and measuring each engine.

Is Searchmaxxed right for you?

We're a strong fit if you're a law firm, clinic or premium builder that's losing ground in Google or missing from AI answers. You want your company named when buyers ask ChatGPT, Gemini, Perplexity or Google who to choose.

Crawler access is where our technical SEO work starts. We fix crawl and indexing problems, site structure, redirects, schema markup and rendering, so Google and the AI engines can read every page that sells. Then we go after the searches and AI answers that bring buyers, and put your name on the pages those answers trust. You talk to the people doing the work, with no account manager in between. And we track which searches and AI answers turn into revenue, through your CRM as well as GA4.

One client came to us with a website that wasn't working. Three months later he was on page one for one of the most competitive searches in his industry. Now he has more high-intent leads than he can handle, and we're building the systems to handle them.

Searchmaxxed's work has generated more than $15 million in revenue through search for its clients.

“I can't believe this worked.” That's what we hear. Clients tell us they finally found a growth partner in search that keeps its word and gets things done.

We're the wrong call in two cases. If all you need is a one-off robots.txt fix, your own developer can do it this afternoon with the tables above. And if you need leads this month just to survive, search takes longer than that.

What should you do next?

Open your robots.txt today and check it against the first table, then confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot got 200 responses in last month's logs.

Want us to look with you? Get your free search analysis. We'll benchmark you against your top competitors across Google, ChatGPT and the other AI engines. We'll show you where they're named instead of you, and the few moves that will matter most.

Frequently asked questions