Log file analysis for SEO: a practical guide for large sites (2026)

Log file analysis for SEO means reading your server's access logs to see every request Googlebot, Bingbot and AI crawlers make on your site. It shows which URLs they fetch, how often, and what status code your server sends back. Nothing else records crawling as it happened. On a large site it answers what a crawler tool can't: where Googlebot spends its time and which money pages it ignores. It also finds URLs Google still visits that your own links never point to.

I'm Brenden House. I run Searchmaxxed, an SEO and AI SEO agency, and log analysis is part of our technical SEO for B2B, SaaS, manufacturing and public-sector sites. This guide is our method, from getting the logs to shipping the fixes, and I read every source below on 27 October 2026 (AEST).

In summary

• A server log records each request: IP address, timestamp, URL, status code and user-agent. GA4 can't see crawlers, and Search Console's Crawl Stats report covers Google only, as a summary.
• Google says crawl budget matters most for sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. Smaller sites still use logs for migrations and AI crawler checks.
• Verify bots before you count them, with reverse DNS for Googlebot and the published IP lists for OpenAI, Anthropic and Perplexity.
• Look for four things: crawl waste, orphan URLs, bad status codes and blocked AI crawlers.
• Findings are cheap. A fix with an owner, checked in next month's logs, is what changes your crawl and eventually your rankings.

What is log file analysis in SEO?

Log file analysis in SEO, also called server log analysis, means filtering your server's request records down to crawlers and reading what they did. Raw text in. Answers about crawling and indexing out.

Five terms come up all the time:

• Log file (access log): the text file your web server, load balancer or CDN writes, with one line for every request it receives.
• User-agent: the name a bot calls itself.
• Crawl budget: the URLs Google can and wants to crawl on your site, which Google's crawl budget guide says crawl capacity and crawl demand decide.
• Crawl waste: crawler requests spent on URLs that will never rank or earn, such as filter combinations, internal search pages and redirect hops.
• Orphan page: a URL bots still request even though no internal link points to it.

Every other SEO dataset models crawling. Logs record it.

What does a server log line contain?

Each line holds the requester's IP address, the time, the URL, the status code, the response size and the user-agent it claims. Here's one Googlebot request in the "combined" format Apache and Nginx use by default:

66.249.66.1 - - [27/Oct/2026:09:14:03 +1000] "GET /integrations/salesforce HTTP/1.1" 200 48213 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Ask for two extra fields: the hostname, which keeps subdomains apart, and the response time, which shows where crawlers wait on you. Speed matters. Google's guide says that when a site slows down or returns 5xx errors, "the limit goes down and Google crawls less."

Does your site need log file analysis?

Yes, if your site has tens of thousands of URLs, a migration coming, or pages stuck as "Discovered, currently not indexed". Google's crawl budget guide names three kinds of site:

• Large sites (1 million+ unique pages) whose content changes about once a week
• Medium or larger sites (10,000+ unique pages) whose content changes daily
• Sites where Search Console puts a large share of URLs under "Discovered - currently not indexed"

Google calls those numbers "a rough estimate". B2B sites pass 10,000 URLs fast. Docs versions, integration pages, help-centre search, locale folders and product filters each create their own crawlable URLs.

Smaller sites still need logs at two moments: during a migration, and whenever you're deciding what to allow or block for AI crawlers. For a 300-page brochure site with neither on the horizon, Search Console is usually enough.

That's where our technical SEO service starts. We crawl the site the way engines do and check that crawl against Search Console and the logs, then rank each fault by revenue.

Why isn't Search Console's Crawl Stats report enough?

Crawl Stats shows Google's crawlers only, as grouped totals with sample URLs, while logs show every request from every bot.

Google's Crawl Stats help page says it plainly: "Example URLs are not comprehensive". And GA4 can't fill the gap, because, as Conductor's guide notes, "search engine crawlers don't execute the Google Analytics tracking code".

So use all three. Crawl Stats flags a host problem fast, the logs tell you which URLs it touched, and GA4 shows what humans did once they arrived.

How do you get your log files?

Ask whoever runs your hosting, CDN or platform for raw access logs, which on enterprise sites usually means engineering or DevOps. The request moves faster when you name the source, fields, date range and format up front.

Logs live in three places:

1. Web servers (Apache, Nginx, IIS) write access logs on each origin machine.
2. Load balancers on AWS, Google Cloud or Azure can log each request before it reaches a server.
3. CDNs (Cloudflare, Akamai, Fastly and others) log requests at the edge.

If your site sits behind a CDN, start there. A page served from the CDN's cache never reaches your origin, so origin logs miss part of what Googlebot fetched from you. Edge logs also show what your firewall or bot protection did to each crawler.

Ask early, because retention is short. Conductor's guide says logs "may only be kept for a short time, let's say 7 to 30 days", and suggests three months. If your platform only keeps a week, set up a daily export now and analyse later.

Here's the request I'd send to engineering:

> Please export raw access logs from the CDN, plus the origin if the CDN misses anything, for the last 30 days. Then export them daily from now on. We need timestamp with timezone, client IP, hostname, method, full URL with query string, status code, bytes, response time, referrer and user-agent. Filtering to requests whose user-agent contains Googlebot, bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot or Perplexity-User is fine. Send gzipped files or a BigQuery table.

That bot-only filter does two jobs. It cuts gigabytes of raw logs down to a size any laptop will open. It also keeps human IP addresses out of the export, and your privacy team may well count those as personal information.

How do you verify Googlebot and AI crawlers in your logs?

Check the IP, not the name. Anyone can type "Googlebot" into a request header. Google's verification guide gives two methods: a DNS check, or a match against its published IP ranges.

The DNS check takes four steps:

1. Run a reverse DNS lookup on the IP from your log, using the host command.
2. Confirm the hostname ends in googlebot.com, google.com or googleusercontent.com.
3. Run a forward lookup on that hostname.
4. Confirm the IPs match.

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

At scale, it's faster to match IPs against the JSON lists that Google, OpenAI, Anthropic and Perplexity each publish for their crawlers. I downloaded every list on 27 October 2026 (AEST) and counted the IP ranges in each:

Look at the dates. Google's list changed the same day I downloaded it, and OpenAI's ChatGPT-User list, with 234 ranges, changed six days before that. Old copies reject real bots. Fetch the lists fresh every run.

Anything that claims to be a bot but fails verification goes in its own bucket. Sometimes that bucket outgrows the real crawl, and then it tells you more about your bot protection than about your SEO.

How do you find crawl waste in your logs?

Group verified Googlebot requests by site section and URL pattern, then set each group's share of crawl beside its share of clicks or pipeline. A section with 40% of the crawl and 1% of the clicks is waste.

On a combined-format log, three commands give you a first pass. Run them on verified Googlebot lines only:

# Requests by top-level folder
awk '{split($7,p,"/"); print "/" p[2]}' googlebot.log | sort | uniq -c | sort -rn | head -20

# Requests to URLs with parameters
awk '$7 ~ /\?/' googlebot.log | wc -l

# Requests by status code
awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn

The usual culprits on large sites, with Google's advice on each:

One trap. Google says robots.txt blocks won't shift crawling elsewhere "unless Google is already hitting your site's crawl capacity limit". So cutting waste helps when key pages are under-crawled. It does little for a site Google already crawls in full.

How do you find orphan pages with log files?

Compare two URL lists: every URL verified Googlebot requested, and every URL your site crawler reached by following internal links alone. In the logs but missing from the crawl? Orphans. In the crawl but missing from the logs? Pages Googlebot ignores.

Add your XML sitemap as a third list and you get four groups worth acting on:

The third row hides the money. When a pricing, product or integration page goes 90 days without a verified Googlebot request, Google has little reason to think it changed. Our technical SEO checklist covers the internal-linking and sitemap checks that fix it.

Which status codes should you check in your logs?

Check what share of crawler requests on pages you want indexed get anything but a 200, because redirects, 404s and 5xx errors each cost something.

Say a deploy breaks the product template. The 5xx spike shows in your logs within hours, URL by URL, while Crawl Stats shows it later and only as a total.

What do AI crawlers do in your logs?

AI crawlers come in three groups: training crawlers, search crawlers and user-triggered fetchers. Logs show which ones reach your site, which pages they read and whether your firewall lets them in.

Sources: OpenAI's crawler page, Anthropic's crawler page and Perplexity's crawler page.

The user-triggered group plays by different rules. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, while Perplexity says its user-triggered fetcher "generally ignores robots.txt rules" too. Each ChatGPT-User hit started with a person's question.

My three checks on every large site:

1. Do OAI-SearchBot, Claude-SearchBot and PerplexityBot get 200s on your product, pricing and comparison pages? A 403 or a CDN challenge there means your bot protection is turning away the search crawlers you want to reach those pages.
2. Which sections do the training crawlers hit hardest?
3. Is ChatGPT-User fetching the pages buyers would ask about, like pricing? If it never touches them, ask why.

Whether to block any of them is a business call, and our guide on whether to block AI crawlers weighs it bot by bot.

Logs also keep "AI traffic" claims honest. Here's what we found on an enterprise B2B client's account. A client's GA4 showed AI-assistant visits jump from 354 to 6,050 in a month. Almost none of it was real. About 5,700 came in three one-hour bursts: same 1920x1080 desktop Chrome, all new users, no referrer, about 20 seconds each, landing on two pages. Real AI traffic was 385 sessions against 354 the month before, roughly flat.

Logs tell you what crawled. Analytics tells you who arrived, once someone has checked it for bots.

How often does Googlebot crawl a site?

There's no fixed schedule. Google crawls as much as your servers can take and as much as it thinks your content needs. So one URL can get hit hourly while another waits months.

Your logs give the real answer. Count verified Googlebot requests per URL over 30 days, then compare the gap between visits template by template. Hubs linked from everywhere get crawled most. Deep pages wait.

New URLs arrive as "discovery" crawls. Google's Crawl Stats help says a big batch of new content or a fresh sitemap should bring "a bump in discovery crawls". How fast can that happen on a new site? Here's one of ours. We launched a new site with 161 guides, each built for one search from the live results and People Also Ask. We submitted the sitemap and pinged IndexNow. No paid indexer. About 25 hours after Google's first crawl, 203 of 228 URLs were indexed. Indexed isn't ranked, though: 5,198 impressions in ten days and 5 clicks.

That's a site we own, not a client result. Fast crawling is step one. Ranking still comes down to the page.

Which log file analysis tools should you use?

Use Screaming Frog's Log File Analyser for one-off audits, a cloud platform for ongoing monitoring, or BigQuery if your data team runs a warehouse. Choose by log volume, and by who will act on what you find.

This is my own selection, and I run Searchmaxxed, so I put it first. Searchmaxxed is a managed service rather than software, so its row is the one where a team does the analysis and ships the fixes.

Prices come from each vendor's own pricing page, read 27 October 2026 (AEST).

Tools find problems, but only shipped and verified fixes change what Googlebot does on your site next month and the month after.

How do you turn log findings into fixes?

Write each finding as a ticket: the cause, the fix, an owner and the log metric that will prove it worked. Then rank the tickets by the revenue behind the affected pages, not by URL count.

A broken canonical on a money page outranks four hundred cosmetic warnings. Rank your list the same way.

A worked example: a B2B SaaS site with 40,000 URLs

Picture a B2B SaaS company with about 40,000 URLs across product pages, 300 integration pages, versioned docs and a help centre. These are example figures to show the method, not a client.

Thirty days of CDN logs show 120,000 verified Googlebot requests. Run the folder count and 46% of them, about 55,200, turn out to hit help-centre search results and filtered integration listings. The 300 integration pages drive most demo requests. They get 4%, and 61 of them get nothing at all.

The status code count finds the old /v1/ docs redirecting twice before reaching a page. Then the AI crawler check turns up something worse. GPTBot reads the docs thousands of times, while the CDN's bot rules hand OAI-SearchBot a 403 on /pricing.

The tickets, in order:

1. Let verified OAI-SearchBot through the CDN on product and pricing pages.
2. Link all 300 integration pages from a crawlable hub, and add them to the sitemap with lastmod.
3. Block internal search and sort parameters in robots.txt, and canonicalise filtered listings.
4. Point every /v1/ URL straight at its current page.

Thirty days later, rerun the three commands. Did the search share fall? Did all 61 integration pages finally get crawled by Googlebot? If yes, move on to rankings and demo requests in Search Console and GA4; if not, the logs show which fix didn't land.

How often should you run log file analysis?

Monthly on a large site. Run it daily through the first weeks of a migration and again after any big release or CDN change, when crawling shifts most. Conductor's guide calls it "an ongoing process".

Where does Searchmaxxed fit?

Searchmaxxed is an SEO and AI SEO agency for enterprise, B2B SaaS, venture-backed, manufacturing and public-sector companies. We do technical SEO, content, link building, digital PR, GEO, AEO and LLM SEO. The aim is to rank in Google and be the name ChatGPT, Gemini, Perplexity and Google's AI answers give for the searches that bring pipeline.

Log analysis sits inside our technical SEO. We pull your CDN or server logs and verify the bots before joining the results with a full crawl, Search Console and GA4. We rank faults by revenue. Then we fix them on the site, and next month's logs show whether each fix landed. Our enterprise SEO service shows how that runs across large, multi-team sites.

Here's how we do it. We build on a website you own, our Agentic Website. AI agents pull your Search Console and GA4 data, read the Google results and check what AI answers say about you and your competitors. That feeds the Managed Search Loop, which shows which page, fix or link should bring in the most enquiries next. We build it, check the live page and measure what changed.

Every managed package includes ongoing audits, a 90-day plan in weekly releases, the build work itself and a monthly decision memo. You own the site, repository, content and data from day one.

Searchmaxxed's work has generated more than A$10 million in revenue through search for its clients.

Is Searchmaxxed right for you?

We fit when your site is big enough for crawling to be a real constraint and you want fixes built, not reported. Need one log report, with your engineers handling the rest? A tool will do.

If you're in the right-hand column, start with Search Console's Crawl Stats report and the free version of Screaming Frog's Log File Analyser.

Frequently asked questions

What should you do next?

Send engineering the request above for 30 days of bot-filtered logs, then run the three counts: by folder, by parameter and by status code. Hold the results against your money pages. If you'd like us to run the analysis and ship the fixes on your site, let's talk pricing.