Crawl budget for large websites: when it matters and how to fix it (2026)

Crawl budget is the set of URLs Google can and wants to crawl on your website, shaped by its resources and your content. It matters when important new or updated pages wait for crawling while Google spends resources elsewhere, or your server cannot handle demand. Fix unnecessary URLs, slow responses or discovery failures first. Indexing needs its own diagnosis.

I'm Brenden House, founder of Searchmaxxed. We do technical SEO, including crawl and indexing fixes, site structure, redirects and server performance, with implementation included in our managed search work. For large sites, the useful question is whether Google reaches the pages your customers need, soon enough for those pages to matter.

In summary

• Investigate delayed crawling of important pages, rather than treating every unindexed URL as a crawl-budget problem.
• Google gives rough starting points, not a diagnosis based solely on site size.
• Use Search Console for patterns and verified Googlebot logs for individual requests, then compare affected page families rather than relying on sitewide totals. Requests and pages differ.
• Control duplicate URL generation, maintain useful internal links and sitemaps, and fix server failures before chasing a higher crawl rate.
• Measure how soon Google fetches changed pages, then check indexing and organic enquiries separately. A busy crawler proves activity, not commercial progress.

When does crawl budget matter for a large website?

Crawl budget deserves attention when Google cannot keep up with important content, especially across large or frequently changing sites. Page count alone doesn't prove that.

Google names these audiences in its crawl-budget guidance:

“Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”

“Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”

It also includes sites where “Discovered - currently not indexed” accounts for a large proportion of URLs, regardless of the headline page count. These are rough estimates.

Content freshness matters as much as the size of the inventory. A manufacturing catalogue with constantly changing specifications has different freshness needs from an equally large archive of historic public documents that rarely change.

A venture-backed SaaS business may have a modest marketing site beside an enormous documentation library. Count the URLs Google encounters, not just the pages marketing maintains.

If Google already crawls important pages promptly, focus elsewhere. Searchmaxxed's technical SEO covers both crawling and indexing, so the diagnosis should decide the work rather than the label on the brief.

What determines Googlebot's crawl rate?

Googlebot's crawling depends on crawl capacity and crawl demand. Capacity describes what Google can fetch without overloading the host; demand describes what it wants to fetch.

Google has resource limits too, which means server improvements alone cannot determine how much crawling your site receives.

Faster hosting helps when serving capacity limits crawling. It won't create demand for thousands of near-identical pages that offer no additional value to searchers.

Google separates sites by hostname. Your main website and documentation subdomain therefore have separate crawl budgets, even if a combined report hides that distinction.

Google's crawlers share host capacity, although each has its own demand, so a surge from another Google crawler can affect the capacity available. Separate crawler types before interpreting the combined request total.

How do you tell crawling problems from indexing problems?

Has Google fetched the page? A page waiting for its first crawl needs different work from a page Google fetched but declined to index.

Use the Page indexing report, then inspect representative URLs from each affected page family. Keep the purpose of each page alongside its indexing status.

An exclusion can be correct. Internal search results, duplicate views and private account pages don't all belong in Google just because a report counts them.

At Searchmaxxed, the fix follows the diagnosis rather than a target for reducing the total number of exclusions visible in a Search Console report. Removing a block helps a useful product page; it doesn't improve an empty template.

What has our own indexing work shown?

On a site we own, Search Console and URL Inspection showed the following over 29 September to 8 October 2026:

We launched a new site with 161 guides, each built for one search from the live results and People Also Ask. We submitted the sitemap and pinged IndexNow. No paid indexer. About 25 hours after Google's first crawl, 203 of 228 URLs were indexed. Indexed isn't ranked, though: 5,198 impressions in ten days and 5 clicks.

The scale matters here. This small-site observation doesn't prove a large-site crawl-budget fix, and it doesn't identify which submission action caused Google to fetch those pages.

The lesson is narrower. A report can show substantial indexing progress while search traffic remains modest, so keep the measures separate when judging the next release.

How do you diagnose crawl budget with Crawl Stats and logs?

Use Crawl Stats to locate the pattern, then use verified Googlebot requests to identify which URLs explain it. Read them together.

Open Search Console's Settings → Crawl stats. Google explains the scope and limitations in its Crawl Stats documentation, including requests that never reach your server and gaps in the examples.

Start with host status. Check robots.txt availability, DNS resolution and server connectivity before interpreting a fall in crawl requests as lost interest in your content.

Compare requests, download size and average response time across the same period, looking for changes that coincide with deployments or known server incidents. Break requests down by response code, file type, crawl purpose and Googlebot type, looking around release dates.

“Discovery” means a first crawl. “Refresh” means another crawl of a known page, so neither label tells you whether Google indexed the content afterwards.

Why isn't the total a page count?

Crawl Stats counts requests. That includes repeat requests and page resources, while redirect hops count separately and some attempted crawls never reach your server in the first place.

The examples are incomplete. A missing URL in that list doesn't prove Google never requested it, nor will its totals necessarily match your logs.

Ask engineering for edge or CDN logs as well as origin logs where caching separates them, then confirm which requests each export can show. Otherwise, an origin-only export may miss requests the CDN served.

Verify the crawler first. Google's verification instructions support published IP ranges or reverse DNS followed by a matching forward lookup.

Keep the original query string. Removing parameters before analysis hides exactly the filter and sorting patterns you're trying to diagnose, so classify URLs before combining duplicates.

Useful fields include timestamp, hostname, requested URL, status code, user agent, response time and cache status where available. Record the log source and timezone too.

Our SEO log-file analysis guide covers the detailed analysis. Here, the decision is which page family needs a change and what evidence would confirm it worked.

Which crawl-budget fixes should you make first?

Fix availability failures first, then unnecessary URL generation and discovery problems. Prioritise important pages that Google misses rather than treating every crawler request as equally valuable.

For Searchmaxxed, these are technical SEO implementation tasks: server performance, crawl controls, redirects, canonical tags, sitemaps and internal links. The order depends on the evidence.

Stop filters and parameters creating unnecessary URLs

Faceted navigation lets users narrow a collection using attributes such as material, size or application. Those choices can produce more URLs than useful search destinations.

Preserve filters that answer distinct searches. A manufacturer may need a searchable stainless-steel valve category while having no reason to expose every sort order as another search page.

Separate useful destinations from interface states before writing crawl rules. Google's faceted-navigation guidance recommends preventing crawling where filtered URLs don't need search visibility.

Parameters can serve useful purposes. Pagination, language selection and valuable filtered categories may depend on them, which makes a blanket question-mark block a poor starting point.

The implementation belongs in a URL policy. Our faceted-navigation SEO guide covers that work without repeating it here.

Choose the control that matches the job

Choose the control by purpose. Robots.txt, noindex and canonical tags solve different problems, and mixing their jobs can stop Google seeing the instruction you wanted it to follow.

Google must see the instruction. Its noindex documentation requires crawler access for the instruction to work, which a robots.txt block on the same page would prevent.

Separate existing indexed duplicates from future unwanted URLs when planning the rollout, because removing something from search requires Google to process the relevant instruction. If removal is necessary, let Google process the removal instruction before considering a long-term crawl block.

Canonical tags also need access. Google's canonical guidance treats them as signals for consolidating duplicates, rather than instructions to stop all duplicate crawling.

Keep a genuinely removed URL accessible to Google so it can receive the removal status. Blocking it instead hides that response and can prolong uncertainty.

Make sitemaps and internal links agree

XML sitemaps identify intended search destinations. Internal links should lead directly to those destinations, including important pages beyond the first listing screen.

Keep normal sitemap output focused on canonical, indexable destinations by removing redirects, missing pages and unwanted variants as part of the same release. Split large inventories into sensible page families so your team can compare their indexing behaviour.

Keep lastmod truthful. Google's sitemap guidance says it should reflect significant changes, and Google ignores priority and changefreq values.

Deployment dates aren't content updates. Updating every modification date on every build gives Google less useful information about what deserves another visit.

Use crawlable HTML links to important pages, including on mobile. A product accessible only through a search box needs a discovery path beyond that interface.

Improve the responses Google receives

Stable, fast responses can increase crawl capacity when server performance constrains it. A browser score cannot diagnose that.

Check time to first byte alongside slow database queries, cache misses and expensive templates, then trace the affected requests to their application behaviour. Review firewall rules if legitimate crawlers receive challenges or unexpected access errors.

Google's crawl guidance identifies 5xx server errors and HTTP 429 rate limiting as reasons to reduce crawling. Fix those accidental failures.

Support conditional requests for unchanged content where your application can handle them correctly. A correct 304 Not Modified response lets Google reuse unchanged content, saving transfer resources without pretending the page has changed.

Keep essential JavaScript and CSS accessible so Google can render the page. Cutting those requests can make the totals look better while preventing Google from understanding the page properly.

What would this look like for a manufacturing catalogue?

For a manufacturer, separate product pages from browsing states before deciding what Google should crawl. The business needs searchable specifications, not every possible ordering of them.

Consider an illustrative catalogue with valve product pages, material categories, technical PDFs and filter combinations. This example is illustrative.

Assign the decision before the ticket. Marketing identifies the useful destinations; engineering owns application behaviour; SEO checks discovery, canonical selection and subsequent crawling.

This is where our implementation scope matters. Searchmaxxed's managed plans include technical changes and weekly releases, so the work can move beyond an audit document into the site.

Your company keeps the website, repository, content and data, including the changes that make future releases easier to diagnose and measure against this baseline. For a catalogue with several markets or complex engineering needs, Custom & Enterprise can include dedicated engineering and custom reporting.

How do you know crawl-budget optimisation worked?

Crawl-budget optimisation works when Google reaches important new or changed pages sooner without harming availability or removing useful search destinations. Fewer requests can be progress.

Record the baseline before release, using the same definitions you will apply afterwards. Compare the same page families afterwards, allowing for changes in publishing volume, content updates and server incidents during either period.

Set the freshness window around the content. A public-sector deadline page needs different attention from an unchanged technical reference, so a single sitewide target can hide the problem.

Percentages can hide falling counts. The share of crawling going to products can rise even when Google fetches fewer products, if other requests fall faster.

Google's guidance adds an important limit: blocking unwanted URLs doesn't automatically transfer their crawl resources to other pages. Capacity and demand still apply.

Is Searchmaxxed right for your crawl-budget problem?

Searchmaxxed fits companies that need technical SEO changes implemented and measured alongside their wider search work. The scope should follow the diagnosis.

Growth System and Agentic Search Engineering include technical implementation within managed search work, so crawl fixes sit alongside the pages and internal links they support. Custom & Enterprise can cover multiple markets, dedicated engineering and custom reporting when those belong in scope.

Direct access matters here. Our managed plans connect you with the people doing the work, with no account managers, and the company retains ownership of its assets.

Our technical SEO service covers those distinctions. Bring the affected page families, Search Console access and available logs, so we can scope the work around the actual constraint.

Frequently asked questions