# Log file analysis for SEO: a practical guide for large sites (2026)

> Log file analysis for SEO shows every URL Googlebot and AI crawlers request. Get the logs, verify bots, find crawl waste and orphans, then ship the fixes.

Canonical: https://www.searchmaxxed.com/research/log-file-analysis-seo
Last updated: 2026-10-27

HOW-TO GUIDE

Log file analysis for SEO shows every URL Googlebot and AI crawlers request. Get the logs, verify bots, find crawl waste and orphans, then ship the fixes.

BH

Brenden House

Founder of Searchmaxxed. We build agentic SEO and GEO workflows that help businesses rank on Google and get named in ChatGPT, then turn that search into more customers and revenue.

PUBLISHED 27 OCTOBER 2026 UPDATED 27 OCTOBER 2026 19 MIN READ

Log file analysis for SEO means reading your server's access logs to see every request Googlebot, Bingbot and AI crawlers make on your site. It shows which URLs they fetch, how often, and what status code your server sends back. Nothing else records crawling as it happened. On a large site it answers what a crawler tool can't: where Googlebot spends its time and which money pages it ignores. It also finds URLs Google still visits that your own links never point to.

I'm Brenden House. I run Searchmaxxed, an SEO and AI SEO agency, and log analysis is part of our technical SEO for B2B, SaaS, manufacturing and public-sector sites. This guide is our method, from getting the logs to shipping the fixes, and I read every source below on 27 October 2026 (AEST).

## In summary

- A server log records each request: IP address, timestamp, URL, status code and user-agent. GA4 can't see crawlers, and Search Console's Crawl Stats report covers Google only, as a summary.
- Google says crawl budget matters most for sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. Smaller sites still use logs for migrations and AI crawler checks.
- Verify bots before you count them, with reverse DNS for Googlebot and the published IP lists for OpenAI, Anthropic and Perplexity.
- Look for four things: crawl waste, orphan URLs, bad status codes and blocked AI crawlers.
- Findings are cheap. A fix with an owner, checked in next month's logs, is what changes your crawl and eventually your rankings.

## What is log file analysis in SEO?

Log file analysis in SEO, also called server log analysis, means filtering your server's request records down to crawlers and reading what they did. Raw text in. Answers about crawling and indexing out.

Five terms come up all the time:

- **Log file (access log):**  the text file your web server, load balancer or CDN writes, with one line for every request it receives.
- **User-agent:**  the name a bot calls itself.
- **Crawl budget:**  the URLs Google can and wants to crawl on your site, which [Google's crawl budget guide](https://developers.google.com/crawling/docs/crawl-budget) says crawl capacity and crawl demand decide.
- **Crawl waste:**  crawler requests spent on URLs that will never rank or earn, such as filter combinations, internal search pages and redirect hops.
- **Orphan page:**  a URL bots still request even though no internal link points to it.

Every other SEO dataset models crawling. Logs record it.

## What does a server log line contain?

Each line holds the requester's IP address, the time, the URL, the status code, the response size and the user-agent it claims. Here's one Googlebot request in the "combined" format Apache and Nginx use by default:

```
66.249.66.1 - - [27/Oct/2026:09:14:03 +1000] "GET /integrations/salesforce HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
```

| Field | In the example | What it tells you |
| --- | --- | --- |
| IP address | 66.249.66.1 | Who sent the request; use it to verify the bot |
| Timestamp | 27/Oct/2026:09:14:03 +1000 | When; gaps between hits show crawl frequency |
| Method and URL | GET /integrations/salesforce | What the bot asked for, parameters included |
| Status code | 200 | What your server answered |
| Bytes | 48213 | Response size; spot heavy pages |
| Referrer | - | Usually empty for bots |
| User-agent |...Googlebot/2.1... | Which bot it claims to be; this one is Googlebot Smartphone |

Ask for two extra fields: the hostname, which keeps subdomains apart, and the response time, which shows where crawlers wait on you. Speed matters. Google's guide says that when a site slows down or returns `5xx` errors, "the limit goes down and Google crawls less."

## Does your site need log file analysis?

Yes, if your site has tens of thousands of URLs, a migration coming, or pages stuck as "Discovered, currently not indexed". Google's crawl budget guide names three kinds of site:

- Large sites (1 million+ unique pages) whose content changes about once a week
- Medium or larger sites (10,000+ unique pages) whose content changes daily
- Sites where Search Console puts a large share of URLs under "Discovered - currently not indexed"

Google calls those numbers "a rough estimate". B2B sites pass 10,000 URLs fast. Docs versions, integration pages, help-centre search, locale folders and product filters each create their own crawlable URLs.

Smaller sites still need logs at two moments: during a migration, and whenever you're deciding what to allow or block for AI crawlers. For a 300-page brochure site with neither on the horizon, Search Console is usually enough.

That's where our [technical SEO service](https://www.searchmaxxed.com/services/technical-seo) starts. We crawl the site the way engines do and check that crawl against Search Console and the logs, then rank each fault by revenue.

## Why isn't Search Console's Crawl Stats report enough?

Crawl Stats shows Google's crawlers only, as grouped totals with sample URLs, while logs show every request from every bot.

| What you want to know | Server logs | Search Console Crawl Stats | GA4 |
| --- | --- | --- | --- |
| Every URL Googlebot requested | Yes | Sample URLs only | No |
| Bingbot, GPTBot, ClaudeBot, PerplexityBot | Yes | No | No |
| Status code per request | Yes | Grouped by response type | No |
| Discovery versus refresh crawls | Work it out from first-seen dates | Yes, by crawl purpose | No |
| Host availability and robots.txt fetch problems | Partly | Yes | No |
| History | As long as you retain it | About 90 days | Not applicable |

Google's [Crawl Stats help page](https://support.google.com/webmasters/answer/9679690) says it plainly: "Example URLs are not comprehensive". And GA4 can't fill the gap, because, as [Conductor's guide](https://www.conductor.com/academy/log-file-analysis/) notes, "search engine crawlers don't execute the Google Analytics tracking code".

So use all three. Crawl Stats flags a host problem fast, the logs tell you which URLs it touched, and GA4 shows what humans did once they arrived.

## How do you get your log files?

Ask whoever runs your hosting, CDN or platform for raw access logs, which on enterprise sites usually means engineering or DevOps. The request moves faster when you name the source, fields, date range and format up front.

Logs live in three places:

1. **Web servers**  (Apache, Nginx, IIS) write access logs on each origin machine.
2. **Load balancers**  on AWS, Google Cloud or Azure can log each request before it reaches a server.
3. **CDNs**  (Cloudflare, Akamai, Fastly and others) log requests at the edge.

If your site sits behind a CDN, start there. A page served from the CDN's cache never reaches your origin, so origin logs miss part of what Googlebot fetched from you. Edge logs also show what your firewall or bot protection did to each crawler.

Ask early, because retention is short. Conductor's guide says logs "may only be kept for a short time, let's say 7 to 30 days", and suggests three months. If your platform only keeps a week, set up a daily export now and analyse later.

Here's the request I'd send to engineering:

> Please export raw access logs from the CDN, plus the origin if the CDN misses anything, for the last 30 days. Then export them daily from now on. We need timestamp with timezone, client IP, hostname, method, full URL with query string, status code, bytes, response time, referrer and user-agent. Filtering to requests whose user-agent contains Googlebot, bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot or Perplexity-User is fine. Send gzipped files or a BigQuery table.

That bot-only filter does two jobs. It cuts gigabytes of raw logs down to a size any laptop will open. It also keeps human IP addresses out of the export, and your privacy team may well count those as personal information.

## How do you verify Googlebot and AI crawlers in your logs?

Check the IP, not the name. Anyone can type "Googlebot" into a request header. Google's [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) gives two methods: a DNS check, or a match against its published IP ranges.

The DNS check takes four steps:

1. Run a reverse DNS lookup on the IP from your log, using the `host` command.
2. Confirm the hostname ends in `googlebot.com`, `google.com` or `googleusercontent.com`.
3. Run a forward lookup on that hostname.
4. Confirm the IPs match.

```
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
```

At scale, it's faster to match IPs against the JSON lists that Google, OpenAI, Anthropic and Perplexity each publish for their crawlers. I downloaded every list on 27 October 2026 (AEST) and counted the IP ranges in each:

| Crawler | Published IP list | IP ranges listed | List last updated (AEST) |
| --- | --- | --- | --- |
| Googlebot and Google's common crawlers | [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) | 170 IPv4, 147 IPv6 | 27 Oct 2026 |
| GPTBot (OpenAI) | [gptbot.json](https://openai.com/gptbot.json) | 18 IPv4 | 22 Sep 2026 |
| OAI-SearchBot (OpenAI) | [searchbot.json](https://openai.com/searchbot.json) | 39 IPv4 | 2 Jan 2026 |
| ChatGPT-User (OpenAI) | [chatgpt-user.json](https://openai.com/chatgpt-user.json) | 234 IPv4 | 21 Oct 2026 |
| Anthropic's crawlers | [bots.json](https://claude.com/crawling/bots.json) | 14 IPv4 | 20 Oct 2026 |
| PerplexityBot | [perplexitybot.json](https://www.perplexity.com/perplexitybot.json) | 8 IPv4 | 8 Feb 2025 |

Look at the dates. Google's list changed the same day I downloaded it, and OpenAI's ChatGPT-User list, with 234 ranges, changed six days before that. Old copies reject real bots. Fetch the lists fresh every run.

Anything that claims to be a bot but fails verification goes in its own bucket. Sometimes that bucket outgrows the real crawl, and then it tells you more about your bot protection than about your SEO.

## How do you find crawl waste in your logs?

Group verified Googlebot requests by site section and URL pattern, then set each group's share of crawl beside its share of clicks or pipeline. A section with 40% of the crawl and 1% of the clicks is waste.

On a combined-format log, three commands give you a first pass. Run them on verified Googlebot lines only:

```
# Requests by top-level folder
awk '{split($7,p,"/"); print "/" p[2]}' googlebot.log | sort | uniq -c | sort -rn | head -20

# Requests to URLs with parameters
awk '$7 ~ /\?/' googlebot.log | wc -l

# Requests by status code
awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn
```

The usual culprits on large sites, with Google's advice on each:

| Waste pattern | What it looks like in logs | What Google recommends |
| --- | --- | --- |
| Faceted filters and sort orders | Thousands of `?sort=`, `?color=` and `?page=` combinations | Consolidate duplicates; block "differently sorted versions of the same page" in robots.txt |
| Internal search results | `/search?q=` URLs with random queries | Block with robots.txt |
| Duplicate URLs | One page under tracking parameters, case or slash variations | "Consolidate duplicate content" |
| Soft 404s | Empty pages returning 200 | "soft 404 pages will continue to be crawled, and waste your budget" |
| Redirect chains | Two or more 301s before a 200 | "Avoid long redirect chains" |
| Noindexed pages | Pages Googlebot fetches, then drops | Google says it will "still request, but then drop the page" |

One trap. Google says robots.txt blocks won't shift crawling elsewhere "unless Google is already hitting your site's crawl capacity limit". So cutting waste helps when key pages are under-crawled. It does little for a site Google already crawls in full.

## How do you find orphan pages with log files?

Compare two URL lists: every URL verified Googlebot requested, and every URL your site crawler reached by following internal links alone. In the logs but missing from the crawl? Orphans. In the crawl but missing from the logs? Pages Googlebot ignores.

Add your XML sitemap as a third list and you get four groups worth acting on:

| In logs | In your crawl | In sitemap | What it usually means | Fix |
| --- | --- | --- | --- | --- |
| Yes | No | No | Old orphan: retired pages, campaign URLs, old docs versions | Redirect to the closest live page, or return 410 |
| Yes | No | Yes | Live page with no internal links | Link it from its hub and related pages |
| No | Yes | Yes | Linked and in the sitemap, but Googlebot skips it | Check depth, duplicates, speed and content quality |
| No | No | Yes | Sitemap entry nothing links to | Link it or take it out of the sitemap |

The third row hides the money. When a pricing, product or integration page goes 90 days without a verified Googlebot request, Google has little reason to think it changed. Our [technical SEO checklist](https://www.searchmaxxed.com/research/technical-seo-checklist) covers the internal-linking and sitemap checks that fix it.

## Which status codes should you check in your logs?

Check what share of crawler requests on pages you want indexed get anything but a 200, because redirects, 404s and 5xx errors each cost something.

| Status code | What it means for crawling | When to act |
| --- | --- | --- |
| 200 | Page served | When it's a soft 404, or a page you don't want crawled |
| 301 or 308 | Permanent redirect | When internal links still point at it, or it chains |
| 302 or 307 | Temporary redirect | When it's been "temporary" for months; make it a 301 |
| 304 | Not modified since the last crawl | Rarely; it saves you bandwidth |
| 404 or 410 | Gone | Google recommends them "for permanently removed pages"; act when live pages or internal links hit them |
| 429 or 503 | Rate limited or unavailable | Google warns against returning them "for more than two or three days" |
| 5xx | Server error | Always; Google crawls less when they rise |

Say a deploy breaks the product template. The 5xx spike shows in your logs within hours, URL by URL, while Crawl Stats shows it later and only as a total.

## What do AI crawlers do in your logs?

AI crawlers come in three groups: training crawlers, search crawlers and user-triggered fetchers. Logs show which ones reach your site, which pages they read and whether your firewall lets them in.

| Group | OpenAI | Anthropic | Perplexity | What blocking it does |
| --- | --- | --- | --- | --- |
| Training | GPTBot | ClaudeBot | None listed | Signals your content shouldn't train future models |
| Search indexing | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Can keep your pages out of that engine's answers |
| User-triggered | ChatGPT-User | Claude-User | Perplexity-User | Limits fetches a user's question triggers; robots.txt may not stop them |

Sources: [OpenAI's crawler page](https://developers.openai.com/api/docs/bots), [Anthropic's crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity's crawler page](https://docs.perplexity.ai/docs/resources/perplexity-crawlers).

The user-triggered group plays by different rules. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, while Perplexity says its user-triggered fetcher "generally ignores robots.txt rules" too. Each ChatGPT-User hit started with a person's question.

My three checks on every large site:

1. Do OAI-SearchBot, Claude-SearchBot and PerplexityBot get 200s on your product, pricing and comparison pages? A 403 or a CDN challenge there means your bot protection is turning away the search crawlers you want to reach those pages.
2. Which sections do the training crawlers hit hardest?
3. Is ChatGPT-User fetching the pages buyers would ask about, like pricing? If it never touches them, ask why.

Whether to block any of them is a business call, and our guide on [whether to block AI crawlers](https://www.searchmaxxed.com/research/should-you-block-ai-crawlers) weighs it bot by bot.

Logs also keep "AI traffic" claims honest. Here's what we found on an enterprise B2B client's account. A client's GA4 showed AI-assistant visits jump from 354 to 6,050 in a month. Almost none of it was real. About 5,700 came in three one-hour bursts: same 1920x1080 desktop Chrome, all new users, no referrer, about 20 seconds each, landing on two pages. Real AI traffic was 385 sessions against 354 the month before, roughly flat.

Logs tell you what crawled. Analytics tells you who arrived, once someone has checked it for bots.

## How often does Googlebot crawl a site?

There's no fixed schedule. Google crawls as much as your servers can take and as much as it thinks your content needs. So one URL can get hit hourly while another waits months.

Your logs give the real answer. Count verified Googlebot requests per URL over 30 days, then compare the gap between visits template by template. Hubs linked from everywhere get crawled most. Deep pages wait.

New URLs arrive as "discovery" crawls. Google's Crawl Stats help says a big batch of new content or a fresh sitemap should bring "a bump in discovery crawls". How fast can that happen on a new site? Here's one of ours. We launched a new site with 161 guides, each built for one search from the live results and People Also Ask. We submitted the sitemap and pinged IndexNow. No paid indexer. About 25 hours after Google's first crawl, 203 of 228 URLs were indexed. Indexed isn't ranked, though: 5,198 impressions in ten days and 5 clicks.

That's a site we own, not a client result. Fast crawling is step one. Ranking still comes down to the page.

## Which log file analysis tools should you use?

Use Screaming Frog's Log File Analyser for one-off audits, a cloud platform for ongoing monitoring, or BigQuery if your data team runs a warehouse. Choose by log volume, and by who will act on what you find.

This is my own selection, and I run Searchmaxxed, so I put it first. Searchmaxxed is a managed service rather than software, so its row is the one where a team does the analysis and ships the fixes.

| Option | Best for | Published price | Log volume |
| --- | --- | --- | --- |
| Searchmaxxed (managed technical SEO) | Large sites that want fixes shipped as well as found | Quoted per business | Whatever your CDN exports |
| [Screaming Frog Log File Analyser](https://www.screamingfrog.co.uk/log-file-analyser/) | One-off audits on a desktop | Free for 1,000 log events; £99 a year | Set by your hard drive |
| [JetOctopus](https://jetoctopus.com/pricing/) | Ongoing cloud monitoring with crawl, GSC and GA4 | From US$349 a month, billed annually | 2M log lines a month on Growth |
| [Botify](https://www.botify.com/pricing) | Enterprise platform with crawler and logs | Not published | Not published |
| Command line, BigQuery or your data warehouse | Teams with data engineers | Your own infrastructure | Unlimited |
| Search Console Crawl Stats | A free baseline for Google | Free | Google only, about 90 days |

Prices come from each vendor's own pricing page, read 27 October 2026 (AEST).

Tools find problems, but only shipped and verified fixes change what Googlebot does on your site next month and the month after.

## How do you turn log findings into fixes?

Write each finding as a ticket: the cause, the fix, an owner and the log metric that will prove it worked. Then rank the tickets by the revenue behind the affected pages, not by URL count.

| Finding | Likely cause | Fix | Check it in next month's logs |
| --- | --- | --- | --- |
| Parameter URLs take a large share of Googlebot requests | Filters and sorts linked as crawlable URLs | Canonicals, robots.txt rules for sort and filter patterns, fewer crawlable links | Parameter share of requests falls |
| Money pages not requested in 90 days | Deep in the site, few internal links | Link from hubs and navigation; sitemap with `lastmod` | Verified requests to those URLs appear |
| Redirect chains on old URLs | Several migrations stacked up | Point every old URL straight at its final page; update internal links | Fewer 301s; no two-hop chains |
| 5xx spikes on one template | Slow queries or failing deploys | Fix the template, add caching, alert on errors | 5xx share back near zero |
| Search crawlers get 403 from the CDN | Bot protection blocks verified bots | Allow verified IP ranges for the crawlers you want | 200s for OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Orphan URLs still crawled | Retired pages with no redirect | 301 to the closest live page, or 410 | Requests to the old URLs taper off |

A broken canonical on a money page outranks four hundred cosmetic warnings. Rank your list the same way.

### A worked example: a B2B SaaS site with 40,000 URLs

Picture a B2B SaaS company with about 40,000 URLs across product pages, 300 integration pages, versioned docs and a help centre. These are example figures to show the method, not a client.

Thirty days of CDN logs show 120,000 verified Googlebot requests. Run the folder count and 46% of them, about 55,200, turn out to hit help-centre search results and filtered integration listings. The 300 integration pages drive most demo requests. They get 4%, and 61 of them get nothing at all.

The status code count finds the old `/v1/` docs redirecting twice before reaching a page. Then the AI crawler check turns up something worse. GPTBot reads the docs thousands of times, while the CDN's bot rules hand OAI-SearchBot a 403 on `/pricing`.

The tickets, in order:

1. Let verified OAI-SearchBot through the CDN on product and pricing pages.
2. Link all 300 integration pages from a crawlable hub, and add them to the sitemap with `lastmod`.
3. Block internal search and sort parameters in robots.txt, and canonicalise filtered listings.
4. Point every `/v1/` URL straight at its current page.

Thirty days later, rerun the three commands. Did the search share fall? Did all 61 integration pages finally get crawled by Googlebot? If yes, move on to rankings and demo requests in Search Console and GA4; if not, the logs show which fix didn't land.

## How often should you run log file analysis?

Monthly on a large site. Run it daily through the first weeks of a migration and again after any big release or CDN change, when crawling shifts most. Conductor's guide calls it "an ongoing process".

| When | What to check |
| --- | --- |
| Every month | Crawl share by section, status codes, money pages not crawled |
| Migration, first 2 to 4 weeks | Googlebot hitting old URLs, redirect status, new URLs discovered |
| After a big release | 5xx and 404 spikes on changed templates |
| After a CDN or firewall change | Verified bots still getting 200s |
| Every quarter | AI crawler access and which sections they read |

## Where does Searchmaxxed fit?

Searchmaxxed is an SEO and AI SEO agency for enterprise, B2B SaaS, venture-backed, manufacturing and public-sector companies. We do technical SEO, content, link building, digital PR, GEO, AEO and LLM SEO. The aim is to rank in Google and be the name ChatGPT, Gemini, Perplexity and Google's AI answers give for the searches that bring pipeline.

Log analysis sits inside our technical SEO. We pull your CDN or server logs and verify the bots before joining the results with a full crawl, Search Console and GA4. We rank faults by revenue. Then we fix them on the site, and next month's logs show whether each fix landed. Our [enterprise SEO service](https://www.searchmaxxed.com/services/enterprise-seo) shows how that runs across large, multi-team sites.

Here's how we do it. We build on a website you own, our Agentic Website. AI agents pull your Search Console and GA4 data, read the Google results and check what AI answers say about you and your competitors. That feeds the Managed Search Loop, which shows which page, fix or link should bring in the most enquiries next. We build it, check the live page and measure what changed.

| Package | Who it suits | What it includes |
| --- | --- | --- |
| Growth System | A company growing in one or two markets | 30 page releases a month, with technical, schema and internal-link work on top; AI visibility tracking |
| Agentic Search Engineering | A competitive or national market | 50 page releases a month, comparison and industry pages, digital PR and link campaigns, AI answer tracking |
| Custom & Enterprise | Several brands, several markets or a large, complex site | Dedicated engineering, custom reporting and original first-party research |

Every managed package includes ongoing audits, a 90-day plan in weekly releases, the build work itself and a monthly decision memo. You own the site, repository, content and data from day one.

Searchmaxxed's work has generated more than A$10 million in revenue through search for its clients.

## Is Searchmaxxed right for you?

We fit when your site is big enough for crawling to be a real constraint and you want fixes built, not reported. Need one log report, with your engineers handling the rest? A tool will do.

| Searchmaxxed is a good fit if | Look elsewhere if |
| --- | --- |
| Your site has tens of thousands of URLs, docs, filters or several locales | Your site has a few hundred pages and no migration planned |
| You're planning a migration and want Googlebot watched through it | You want a one-off log report; Screaming Frog costs £99 a year |
| You want log findings turned into shipped fixes, ranked by revenue | You only want a dashboard to monitor yourself |
| You want Google and the AI search crawlers handled together | Your team can't share logs or Search Console access |
| You want direct access to the people doing the work | You're shopping for the cheapest retainer |

If you're in the right-hand column, start with Search Console's Crawl Stats report and the free version of Screaming Frog's Log File Analyser.

## Frequently asked questions

### What is a log file in SEO?

A log file is the text record your web server, load balancer or CDN writes, one line per request with IP, URL and status. SEO teams keep only the crawler lines.

### How long should you keep log files for SEO?

At least three months, or 12 if you want to compare seasons, track a migration from start to finish or spot slow trends. Many hosts and CDNs delete logs after 7 to 30 days, so set up a daily export to cloud storage before you need the data.

### Can Google Analytics show Googlebot visits?

No. Crawlers don't run GA4's JavaScript tag. To see Googlebot, use server or CDN logs, or Search Console's Crawl Stats report for a Google-only summary.

### Is log file analysis worth it for small websites?

Not as a monthly task. Google's crawl budget guidance targets sites with 10,000+ pages that change daily, or 1 million+ that change weekly. Small sites check logs around a migration.

### What's the difference between log file analysis and a site crawl?

A site crawl shows what a bot could find through your links, while log analysis shows what Googlebot and other bots requested. Put the two side by side and you find orphan pages, plus important pages Google skips.

### Do log files show whether ChatGPT or AI Overviews cite you?

No. Logs show which pages AI crawlers fetched, not which answers named you. ChatGPT-User hits are a clue, since a person's question triggered each one, but to see citations you have to test the answers themselves.

### Which user-agents should you filter for?

Googlebot and bingbot first. Then GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot and Perplexity-User. Verify each by IP first.

## What should you do next?

Send engineering the request above for 30 days of bot-filtered logs, then run the three counts: by folder, by parameter and by status code. Hold the results against your money pages. If you'd like us to run the analysis and ship the fixes on your site, [let's talk pricing](https://www.searchmaxxed.com/contact).

REFERENCES

1. [Google's crawl budget guide](https://developers.google.com/crawling/docs/crawl-budget)
2. [Crawl Stats help page](https://support.google.com/webmasters/answer/9679690)
3. [Conductor's guide](https://www.conductor.com/academy/log-file-analysis/)
4. [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
5. [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
6. [gptbot.json](https://openai.com/gptbot.json)
7. [searchbot.json](https://openai.com/searchbot.json)
8. [chatgpt-user.json](https://openai.com/chatgpt-user.json)
9. [bots.json](https://claude.com/crawling/bots.json)
10. [perplexitybot.json](https://www.perplexity.com/perplexitybot.json)
11. [OpenAI's crawler page](https://developers.openai.com/api/docs/bots)
12. [Anthropic's crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
13. [Perplexity's crawler page](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
14. [Screaming Frog Log File Analyser](https://www.screamingfrog.co.uk/log-file-analyser/)
15. [JetOctopus](https://jetoctopus.com/pricing/)
16. [Botify](https://www.botify.com/pricing)

[Get an AI visibility audit](https://www.searchmaxxed.com/contact)

See where buyers find someone else across Google and AI answers.

See where you are invisible →

## Let's make you the answer.

## Related

- [Research](https://www.searchmaxxed.com/research)
- [Searchmaxxed](https://www.searchmaxxed.com/)