Skip to content

Ch 2 — Crawling infrastructure

Part II — The crawl layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md covers this in roughly six lines (Phase 1, lines 70–102: robots.txt reviewed, "Crawled/Discovered – currently not indexed" triage, ≤10-hop redirect chains, no soft-404s; glossary line 273 compresses crawl budget to one sentence). This chapter adds the operating mechanics under those checkboxes: fetch protocols and size limits, Google's code-by-code HTTP status handling, the capacity×demand crawl budget model with Google's actual thresholds, the crawler HTTP-caching contract, emergency crawl control, the Crawl Stats report read diagnostically, and the indexable file-type inventory.

Googlebot is not a mystery process; it is an HTTP client with documented protocols, documented size limits, documented reactions to every status-code class, and a documented budget model. Almost every crawl-layer incident reduces to one question: what did your server (or the infrastructure in front of it) return to Google, and how does Google's crawling infrastructure react to that class of response? The reactions are asymmetric in ways that surprise practitioners — 4xx errors deindex without throttling, 5xx and 429 throttle first and deindex later, and network timeouts are the worst outcome of all — so this chapter is organized as a reference to those reactions. One navigational fact up front: since November 2025 the authoritative crawling documentation lives on Google's dedicated crawling-infrastructure site (developers.google.com/crawling/…), not Search Central — 🟢 the crawling docs changelog logs the move on Nov 20, 2025 ("Migrated crawling documentation from the Google Search Central to a new Google crawling documentation site. The content hasn't changed, only the location," crawling docs changelog, fetched 2026-08-04), with a second batch (crawl budget, HTTP status codes, DNS/network errors, faceted navigation) logged Dec 18, 2025 in the Search updates changelog. Old /search/docs/crawling-indexing/… URLs currently 301 to the new homes (verified live 2026-08); cite only the new URLs — redirect windows are not forever.

2.1 The fetch layer: what a Googlebot request actually is

Discovery is algorithmic, not on-demand

🟢 "Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site" (How Search works, fetched 2026-08-04, last updated 2025-12-18). There is no path to demand a crawl on a schedule of your choosing; sitemaps (→ Ch 9) and internal links (→ Ch 10) feed the queue, but scheduling stays Google's. The same page states the physical scale plainly: crawling runs on "a huge set of computers," and the crawler overview adds that these clients are 🟢 "designed to be run simultaneously by thousands of machines to improve performance and scale as the web grows," distributed across worldwide datacenters — which is why your logs show many IPs (crawler overview, fetched 2026-08-04, last updated 2026-06-12).

Two crawlers, one identity

Googlebot is the umbrella name for two crawlers — Googlebot Smartphone and Googlebot Desktop. 🟢 "you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt"; both obey the same product token (Googlebot doc, fetched 2026-08-04, last updated 2026-02-03). Under mobile-first indexing the smartphone crawler makes the majority of requests; the Crawl Stats report shows you the exact split (→ §2.8). The full crawler/fetcher taxonomy — common crawlers, special-case crawlers, user-triggered fetchers, Google-Agent, Web Bot Auth, IP-range verification — is Ch 5's home; robots.txt semantics are Ch 3's.

Protocols

  • 🟢 "Google's crawlers and fetchers support HTTP/1.1 and HTTP/2. The crawlers will use the protocol version that provides the best crawling performance and may switch protocols between crawling sessions depending on previous crawling statistics" (crawler overview).
  • 🟢 "The default protocol version used by Google's crawlers is HTTP/1.1"; HTTP/2 "may save computing resources (for example, CPU, RAM) for your site and Googlebot, but otherwise there's no Google-product specific benefit to the site (for example, no ranking boost in Google Search)."
  • 🟢 Opting out of HTTP/2 is documented: "instruct the server that's hosting your site to respond with a 421 HTTP status code when Google attempts to access your site over HTTP/2. If that's not feasible, you can send a message to the Crawling team (however this solution is temporary)."
  • 🟢 FTP (RFC 959) and FTPS (RFC 4217) are supported, "however crawling through these protocols is rare."
  • ⚠️ HTTP/3 appears nowhere in the crawler documentation — verified by reading the live overview page 2026-08-04. Claims that Googlebot crawls over h3, or that h3 support helps crawling, are unsupported.

Compression

🟢 "Google's crawlers and fetchers support the following content encodings (compressions): gzip, deflate, and Brotli (br)." Supported encodings are advertised per request in the Accept-Encoding header (e.g. Accept-Encoding: gzip, deflate, br). Compression saves transfer, not fetch-limit headroom — the size limit applies to uncompressed data (→ §2.2).

Origin and politeness

  • 🟢 "Google egresses primarily from IP addresses in the United States. In case Google detects that a site is blocking requests from the United States, it may attempt to crawl from IP addresses located in other countries" (crawler overview).
  • 🟢 "When crawling from IP addresses in the US, the timezone of Googlebot is Pacific Time" (Googlebot doc).
  • ⚠️ Consequence: geo-served content, consent walls, paywalled variants, and price localization are judged from a US vantage point. And the causality runs the opposite way from the old lore: geo-blocking the US is what triggers non-US crawling — localization needs don't. Locale-adaptive page handling → Ch 11.
  • 🟢 Politeness baseline: "For most sites, Googlebot shouldn't access your site more than once every few seconds on average" (Googlebot doc). Short-term bursts can look higher; sustained rates far beyond that suggest either AdsBot (→ §2.8) or an impostor (→ Ch 5 for verification).

The crawl→render handoff

Googlebot fetches the HTML and hands it to the Web Rendering Service (WRS); WRS, "using Googlebot," downloads the referenced resources. Two crawl-layer facts follow (rendering internals → Ch 6):

  1. Render fetches are real crawls, billed to the hostname serving each resource (→ §2.5) and cached under WRS's own 30-day rule, which ignores your caching headers (→ §2.6).
  2. The crawl→render gap is usually seconds, not weeks. ⚪ MERJ and Vercel's study of over 100,000 Googlebot fetches (April 1–30, 2024; primarily nextjs.org, with supplemental data from monogram.io and basement.io; delay distribution computed on 37,000+ matched server-beacon pairs) found: median rendering delay 10 seconds, 25th percentile within 4 seconds, 75th percentile 26 seconds, 90th ~3 hours, 99th ~18 hours — long delays "were the exception and not the rule" (Vercel blog, Jul 31, 2024, fetched 2026-08-04). ⚪ Scope honestly: one vendor-run study dominated by a single high-crawl-priority site (nextjs.org). Treat it as an existence proof that the "render queue takes weeks" lore is wrong — not as universal constants for your site.

2.2 Fetch size limits: 2MB per file, 64MB for PDFs

⚠️ The "15MB limit" the industry still cites is not Googlebot-for-Search's limit. Since the February 3, 2026 documentation update (🟢 logged that day in the Search updates changelog: the move of the generic limits to the crawler docs "enabled us to be more precise about Googlebot's limits"), the Search-specific numbers are explicit: 🟢 "When crawling for Google Search, Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file" (Googlebot doc, fetched 2026-08-04, last updated 2026-02-03). The 15MB figure survives only as the generic cross-product default: 🟢 "By default, Google's crawlers and fetchers only crawl the first 15MB of a file, and any content beyond this limit is ignored. However, individual projects may set different limits for their crawlers and fetchers, and also for different file types" (crawler overview, fetched 2026-08-04). Any audit template, tool warning, or client deck quoting "15MB" for Google Search is citing the wrong crawler class.

Three mechanics matter in practice:

  1. The limit is per fetched file, not per page. Each referenced CSS/JS resource is fetched separately under its own limit. A page whose HTML is 300KB is nowhere near the limit even if its total payload with resources is tens of megabytes.
  2. It applies to uncompressed data. 🟢 "The file size limit is applied on the uncompressed data" (Googlebot doc). A 1.5MB Brotli response that decompresses to 12MB of HTML is over the limit; your compression ratio buys transfer speed, not headroom.
  3. Truncation, not rejection. 🟢 "Once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration" (Googlebot doc). The page is not dropped; the content past the cutoff simply doesn't exist to indexing. This is why critical content, structured data, and <head> elements belong early in the document.

⚪ Realistic offenders are not hand-written pages: server-side-rendered apps inlining large JSON state blobs, inlined base64 assets, generated listings, and log-like pages are the URLs that approach 2MB of markup. Audit those templates, not your articles.

2.3 HTTP status codes through Google's eyes

This section is the book's canonical home for Google's status-code handling. Source: How HTTP status codes affect Google's crawlers, fetched 2026-08-04, last updated 2026-02-04. Related homes: redirect type selection and migration mechanics → Ch 16; robots.txt-specific status handling (4xx-fail-open / 5xx-fail-closed) → Ch 3; WAF/CDN-generated responses → Ch 4.

The reference table

Code(s) Crawl-rate effect Index effect (Google Search) Doc language (all 🟢, same page)
200 none content passed to indexing pipeline — "may index the content, but that's not guaranteed" "Google passes on whatever it received to the next processing step"
201/202 none partial content may be processed "Google waits for the content for a limited time, then passes on whatever it received"; timeout "is user agent dependent"
204 none nothing to process "Google wasn't able to receive any content and therefore can't process it"
301/308 none strong canonicalization signal toward target "a strong signal that the redirect target should be processed"; 308 "Equivalent to 301"
302/303/307 none weak canonicalization signal toward target "a weak signal that the redirect target should be processed"; 307 "Equivalent to 302"
304 none previous crawl's content reused; signals may be recalculated "the content is the same as last time it was crawled… otherwise the status code has no effect on indexing"
400–428, 430+ none — "no effect on crawl rate" (frequency gradually decreases per URL) URL removed from index; new URLs not processed "All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn't exist"
429 throttles site-wide (server-error class) preserved at first, "eventually dropped" if persistent "a signal that the server is overloaded, and it's considered a server error"
5xx throttles site-wide preserved at first, "eventually dropped"; persistent errors removed "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling"
network timeout / DNS failure throttles immediately removed "within days" — terminal → §2.4

2xx — considered, not guaranteed

A 200 is an invitation to evaluate, nothing more; indexing selection is Ch 7's territory. ⚠️ A 200 whose content looks like an error is flagged: 🟢 "If the content suggests an error for Google Search, an empty page or an error message, Search Console will show a soft 404 error" (→ §2.4). The edge codes are worth knowing for API-backed and SPA setups: a 204 from an endpoint that renders fine for users is nothing to Search, and 201/202 responses race an unpublished, user-agent-dependent timeout.

3xx — hop budget and signal strength

🟢 "By default, Google's crawlers follow up to 10 redirect hops. However, specific products' crawlers may have different limits… Google Inspection Tools doesn't follow redirects." Content served on the redirecting URL itself is ignored; the target's content is processed. The strong/weak vocabulary is canonicalization language — the redirect is one input into canonical selection (full model → Ch 8; type taxonomy and migration practice → Ch 16). ⚠️ Two operational notes: chains burn the hop budget silently (the playbook's "≤1 hop" tightening exists because 10 is a ceiling, not a target), and URL Inspection showing "redirect" without following it is expected tool behavior, not an error.

4xx — deindexing without throttling

🟢 "All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn't exist." Previously indexed URLs are removed; newly encountered 404s aren't processed; "The crawling frequency gradually decreases" for the affected URLs. ⚠️ The trap Google states explicitly: 🟢 "Don't use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate." A WAF answering Googlebot with 403s throttles nothing and deindexes everything it touches — the exact inverse of the intended effect. On 404 vs 410: this doc treats them identically. 🟢 In a May 2018 office-hours hangout John Mueller said a 410 "will sometimes fall out a little bit faster than a 404. But usually, we're talking on the order of a couple days or so" — bracketed by his own caveat that "in the mid term/long term, a 404 is the same as a 410 for us" (recording, ~28:41; transcription per Search Engine Journal, May 2018, re-fetched 2026-08-04). That is a 2018 recorded statement, not current documentation — don't build process on 410 being faster.

429 — the 4xx that behaves like a 5xx

🟢 "Google's crawlers treat the 429 status code as a signal that the server is overloaded, and it's considered a server error." It throttles like a 5xx and, sustained, deindexes like a 5xx. Rate-limiting layers that emit 429s to verified Googlebot are therefore self-inflicted crawl cuts (→ Ch 4 for the WAF configuration angle, → §2.7 for when you want this behavior).

5xx — throttle first, drop later

🟢 "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped." For 500 specifically: "The decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error," and the indexing pipeline "removes from the index URLs that persistently return a server error." Recovery is symmetric and automatic: 🟢 "Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site." This controlled two-stage behavior is exactly why a deliberate 503 is the sanctioned emergency brake (→ §2.7) — and why a 5xx state left running for weeks quietly empties your index.

JavaScript on non-200 pages

🟢 Google's December 18, 2025 documentation update: "While pages with a 200 HTTP status code are sent to rendering, this might not be the case for pages with a non-200 HTTP status code" (Search updates changelog, fetched 2026-08-04). ⚠️ JS-injected noindex, canonical tags, or client-side redirects on error pages may never execute. Anything you need Google to see on an error response must be sent at the HTTP layer — status code and headers — not painted in by JavaScript (→ Ch 6).

2.4 Network errors, DNS errors, and soft 404s — the fast killers

Transport failures outrank status-code failures

🟢 "Google treats network timeouts, connection reset, and DNS errors similarly to 5xx server errors. In case of network errors, crawling immediately starts slowing down, as a network error is a sign that the server may not be able to handle the serving load" — and, critically, "already indexed URLs that are unreachable will be removed from Google's index within days" (DNS and network errors doc, fetched 2026-08-04, last updated 2025-12-18).

Google's own severity ranking comes from the Crawling December CDN post: 🟢 "Network timeouts from the CDN will cause the affected URLs to be removed from Google's search index, as these network errors are considered terminal, \"hard\" errors. Additionally they may also considerably affect your site's crawl rate because they signal our crawl infrastructure that the site is overloaded" (CDNs and crawling, Dec 24, 2024, fetched 2026-08-04). ⚠️ A firewall or WAF that silently drops Googlebot connections does more damage, faster, than any status-code mistake: a clean 503 buys you days of grace; a timeout starts deletions. If you must block or pause, answer — with a 503 (→ §2.7, Ch 4).

Debugging order per the doc: (1) firewall rules — "Make sure that Google IP addresses are not blocked by any firewall rule" (check against the published IP-range JSONs, → Ch 5); (2) packet capture — tcpdump/Wireshark; (3) DNS — A/CNAME records and every name server. ⚪ In practice, transient DNS failures at scale usually implicate the DNS provider or a propagation mistake, and Crawl Stats' host-status DNS chart (→ §2.8) is the fastest first confirmation that Google is seeing it too.

Soft 404s are a content verdict, not a status code

🟢 "A soft 404 error is when a URL that returns a page telling the user that the page does not exist and also a 200 (success) status code. In some cases, it might be a page with no main content or empty page… Such pages are excluded from Search" (troubleshoot crawling errors, fetched 2026-08-04, last updated 2025-12-18).

Detection is algorithmic and content-based: 🟢 "When Google's algorithms detect that the page is actually an error page based on its content, Search Console will show a soft 404 error in the site's Page Indexing report." The doc's listed causes deserve a close read, because two of the four are infrastructure failures, not content decisions:

  • "A missing server-side include file."
  • "A broken connection to the database."
  • "An empty internal search result page."
  • "An unloaded or otherwise missing JavaScript file." ⚠️ — the one that blindsides JS sites. A rendering/resource failure can soft-404 a perfectly good page; a soft-404 wave on a JavaScript site is usually a rendering problem, not a content problem (→ Ch 6).

Fix by the actual state of the page

  • Gone: return a real 404 or 410. A helpful custom 404 page is good UX; the status code is what Google acts on. 404s cost no crawl budget (→ §2.5) and are normal web behavior.
  • Moved: 301 to the specific replacement (→ Ch 16).
  • Still exists: URL Inspection — compare the rendered content and the returned HTTP code against what you expect; a soft-404 verdict on a live page means Google's fetch saw something you didn't.
  • ⚠️ Not a fix: blanket-redirecting dead URLs to the homepage. 🟢 "Don't redirect many old URLs to one irrelevant single URL destination, such as the home page of the new site. This can confuse users and might be treated as a soft 404 error" (site moves doc, fetched 2026-08-04, last updated 2026-06-17) — at scale you convert clean 404s into a soft-404 backlog (redirect mechanics → Ch 16).

The worst variant: error text under 200 at scale

🟢 "If Google couldn't detect the error messages as \"hard\" errors, all the pages with the same error message may be eliminated as duplicates from Google's search index. Since Google indexing has little incentive to request a recrawl of duplicate URLs, recovering from this may take more time" (CDN post). An outage page, challenge interstitial, or CMS error string served with 200 across thousands of URLs can collapse them all into one duplicate cluster — and duplicate clusters get little recrawl priority, so recovery is measured in weeks, not days. Slower, in Google's own telling, than recovering from an honest 503 outage. Duplicate clustering mechanics → Ch 8; the CDN/WAF scenarios that produce this at scale → Ch 4.

2.5 Crawl budget: capacity × demand, and who actually needs to care

Home doc: Optimize your crawl budget, fetched 2026-08-04, last updated 2026-07-22 ("polished and clarified" for terminology consistency Jul 22, 2026 per the crawling docs changelog).

Scope check first

🟢 The guide's own gate: "If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide. For Google Search specifically, keeping your sitemap up to date and checking the Page Indexing report regularly is adequate." Google's audience thresholds (rough, not exact):

  • "Large sites (1 million+ unique pages) with content that changes moderately often (once a week)"
  • "Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)"
  • "Sites with a large portion of their total URLs classified by Search Console as Discovered - currently not indexed" (→ Ch 21 for that diagnosis playbook)

⚠️ "Crawl budget optimization" as a universal deliverable for 200-page sites is a service-industry invention. The playbook glossary's "roughly 10k+ frequently-changing pages" line is the compressed version of the middle threshold; this section is what it compresses.

The unit is the hostname

🟢 "Google's crawling infrastructure defines a site as a unique hostname. For example, https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets." Not the domain, not the GSC property — the hostname. Resource fetches debit the hostname hosting the resource: 🟢 "Crawling the resources needed to render a page will chip away from the crawl budget of the hostname that's hosting the resource" (resources post, Dec 3, 2024, fetched 2026-08-04). Moving assets to cdn.example.com moves the cost, it doesn't delete it — and that post (with its Dec 6, 2024 correction) recommends against off-hosting critical JS/CSS for rendering-performance reasons, while calling the approach "worth considering" for large non-critical resources (→ Ch 4 for the hosting tradeoff in full).

Side one — crawl capacity limit (hostload)

🟢 "This limits the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration." Mechanics, all from the same doc:

  • "Every site starts with the same default, conservative crawl capacity limit." Health plus demand raises it automatically over time.
  • Slowdowns, 5xx responses, and 429s lower it (the §2.3 reactions, formalized).
  • ⚠️ 🟢 "While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others." An AdsBot spike from Dynamic Search Ads targets (→ §2.8) eats Googlebot's room on the same hostname.

Side two — crawl demand

Three documented factors:

  • Perceived inventory — "the factor that you can positively control the most." Every duplicate, faceted permutation, and infinite URL space Google knows about dilutes it (→ Ch 8, Ch 10).
  • Popularity — "URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems."
  • Staleness — recrawl scheduling tracks how often content actually changes. 🟢 "site-wide events like site moves may trigger an increase in crawl demand in order to reprocess the content under the new URLs" (→ Ch 16).

The definition that reconciles both sides: 🟢 "Taking crawl capacity and crawl demand together, Google defines a site's crawl budget as the set of URLs that Google can and wants to crawl. Even if the crawl capacity limit isn't reached, if crawl demand is low, Google will crawl your site less." ⚪ Crawl budget is a set, not a countable page quota — "you wasted 34% of your crawl budget" is tool-vendor framing, not a Google measurement.

Levers that work (all 🟢, crawl-budget doc)

  • Consolidate duplicate content (→ Ch 8).
  • robots.txt-block URL spaces you never want crawled — faceted explosions, infinite calendars (→ Ch 3; Google itself prunes some of this: 🟢 "calendars that go to the year 9999 probably don't need to be crawled in their entirety," about crawling, fetched 2026-08-04, last updated 2026-03-03).
  • Return 404/410 for permanently removed pages: "Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed."
  • "Eliminate soft 404 errors. soft 404 pages will continue to be crawled, and waste your budget" (→ §2.4).
  • Keep sitemaps up to date with <lastmod> (→ Ch 9).
  • Avoid long redirect chains (→ Ch 16).
  • Make pages efficient to load; support 304s (→ §2.6).

Anti-levers Google explicitly rejects (⚠️)

  • noindex as a crawl saver: 🟢 "Don't use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time." noindex is an index control (→ Ch 7); only a robots.txt disallow prevents the fetch. (Ch 7 also owns the long-term noindex→crawled-less-often nuance.)
  • robots.txt rotation: 🟢 "Don't use robots.txt to temporarily reallocate crawl budget for other pages… Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit." Blocking waste does not re-spend the savings unless you were capacity-limited — most sites aren't.
  • 4xx panic: 🟢 "Pages that serve 4xx HTTP status codes (except 429) don't waste crawl budget" (myths page, fetched 2026-08-04, last updated 2025-12-18). Redirecting every dead URL "to save budget" solves a non-problem and creates soft 404s (→ §2.4).

Quarantined lore (all 🟢 debunks from the myths page)

  • crawl-delay: "The non-standard \"crawl-delay\" robots.txt rule is not processed by Google's crawlers."
  • Parameters: "We can crawl parameters." The real parameter problem is inventory explosion, not capability (→ Ch 10).
  • Compressed sitemaps: "Zipped sitemaps still have to be fetched from the server, so you're not really saving much crawling time."
  • Fake freshness: "there's no additional value in making pages artificially appear to be fresh by making trivial changes and updating the page date."
  • Site size: "If a site has important content that changes often, we crawl it often, regardless of the size."
  • Crawling as ranking: "Improving your crawl rate won't necessarily lead to better positions in Google Search results… while crawling is necessary for a page to be in search results, it's not a ranking signal."

Getting more — and expectation-setting

Two documented ways only: 🟢 "Add more server resources" (when URL Inspection returns Hostload exceeded — the capacity ceiling is real), or "Optimize your content's quality for the Google product you're targeting" — for Search, "popularity, overall user value, content uniqueness, and serving capacity." For everyone else: 🟢 "for most sites, new pages will take several days minimum to be noticed; most sites shouldn't expect same-day crawling for URLs, with the exception of time-sensitive sites such as news sites" (troubleshoot crawling errors). And the client-facing reframe: 🟢 "Frequent crawling is a good sign!… To catch breaking news articles, we may recrawl news homepages every few minutes. In other cases we might have seen that nothing has changed for years, so we might wait a month to recrawl" (about-crawling page).

2.6 HTTP caching for Google's crawlers

The contract is narrow and explicit

🟢 "Google's crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header… Other HTTP caching directives aren't supported" (crawler overview, fetched 2026-08-04). 📘 These are standard HTTP conditional-request semantics (RFC 9111 territory), not a Google invention — but Google implements only this slice: no-store, private, s-maxage, stale-while-revalidate and the rest of the Cache-Control vocabulary do not steer Googlebot.

Preference order and formats

  • 🟢 "If both ETag and Last-Modified response header fields are present in the HTTP response, Google's crawlers use the ETag value as required by the HTTP standard." Google recommends ETag "as ETag doesn't have date formatting issues"; the caching blog post adds 🟢 "if you have the option, set them both: the internet will thank you. Maybe."
  • 🟢 Last-Modified, if used, should follow "Weekday, DD Mon YYYY HH:MM:SS Timezone" (e.g. Fri, 4 Sep 1998 19:15:56 GMT).
  • 🟢 "While not required, consider also setting the max-age field of the Cache-Control response header to help crawlers determine when to recrawl the specific URL" (e.g. Cache-Control: max-age=94043) — a recrawl hint, and the only Cache-Control field Google reads here.
  • 🟢 Per-crawler variance is documented: "Googlebot supports caching when re-crawling URLs for Google Search, and Storebot-Google only supports caching in certain conditions."

How the exchange works

🟢 "Google's crawlers that support caching will send the ETag value returned for a previous crawl of that URL in the If-None-Match header. If the ETag value sent by the crawler matches the current value the server generated, your server should return an HTTP 304 (Not modified) status code with no HTTP body" (HTTP caching post, Dec 9, 2024, fetched 2026-08-04). The empty body is the payoff: no content generation, no transfer; indexing reuses the previously crawled version (→ §2.3, 304 row). Two practical wrinkles:

  • 🟢 Conditional headers are not sent on every crawl: "Google's crawlers don't send the headers with all crawl attempts; it depends on the use case of the request (for example, AdsBot is more likely to set the If-Modified-Since and If-None-Match HTTP request headers)." The If-Modified-Since value, when sent, is "the date and time the content was last crawled" (troubleshoot crawling errors).
  • 🟢 The unconditional escape hatch: "Independently of the request headers, you can send a 304 (Not Modified) HTTP status code and no response body for any Googlebot request if the content hasn't changed since Googlebot last visited the URL" (same doc). Your server may decide "unchanged" on its own — no incoming validator required.

Why Google begged — and the refresh policy

🟢 "10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%" (Gary Illyes, "Crawling December: HTTP caching," Dec 9, 2024 — verified against the live post 2026-08-04). Almost nobody implements validators in a way Google can use; a large site's engineering team that does gets cheaper recrawls at exactly the scale where §2.5 says budget is real. Refresh policy: 🟢 "require a cache refresh on significant changes to your content; if you only updated the copyright date at the bottom of your page, that's probably not significant."

⚠️ The WRS exception — the marquee correction

For page resources (JS/CSS) fetched during rendering, your caching headers are ignored entirely: 🟢 "The time to live of the WRS cache is unaffected by HTTP caching directives; instead WRS caches everything for up to 30 days, which helps preserve the site's crawl budget for other crawl tasks" (resources post). "Cache-Control controls Googlebot" is therefore false twice over: for pages, only validators + max-age matter; for rendering resources, nothing in your headers matters. Corollary: 🟢 "Use cache-busting parameters cautiously: if the URLs of resources change, Google may need to crawl the resources again, even if their contents haven't changed. This, of course, will consume crawl budget." Fingerprinting every asset on every deploy re-crawls your entire resource set against the hosting hostname's budget (→ Ch 6 for WRS mechanics, → Ch 20 for release-safety checks).

2.7 Emergency crawl control

The sanctioned brake

When crawling is overwhelming the origin: 🟢 "If you need to urgently reduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests. Google's crawling infrastructure reduces your site's crawling rate when it encounters a significant number of URLs with 500, 503, or 429 HTTP response status codes" (reduce crawl rate, fetched 2026-08-04, last updated 2025-12-18). 🟢 "The reduced crawl rate affects the whole hostname of your site… both the crawling of the URLs that return errors, as well as the URLs that return content."

⚠️ The time limit is part of the mechanism, not a suggestion: 🟢 "We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products. For example, in case of Search, if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index." The Search Console help version allows slightly longer and adds the robots.txt option (effective only after up to a day — the 24-hour cache, → Ch 3): 🟢 "Be sure not to return 503 or 429 for more than two or three days, though, or it can signal Google to crawl your site less frequently in the long term" (Crawl Stats help, fetched 2026-08-04). Planned-downtime handling and the 503-vs-placeholder decision tree → Ch 4; site-wide pause procedures → Ch 20.

The rate limiter is gone

🟢 "The crawl rate limiter tool in Search Console is being deprecated on Jan 8th, 2024" (blog, Nov 24, 2023, fetched 2026-08-04). Automatic reaction replaced it: 🟢 "if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling. Similarly, Googlebot slows down automatically if the response time for requests gets significantly longer." Google simultaneously set 🟢 "the minimum crawling speed to a lower rate, comparable to the old crawl rate limits" — effectively grandfathering old limiter settings for low-demand sites. Any audit playbook that still says "set the crawl rate in GSC" is two years stale.

Last resorts — and the missing inverse

  • 🟢 If serving errors is infeasible: "file a special request to report a problem with unusually high crawl rate, mentioning the optimal rate for your site" — the Googlebot report form (search.google.com/search-console/googlebot-report) — and "it may take several days for the request to be evaluated and fulfilled" (reduce-crawl-rate doc).
  • 🟢 The inverse does not exist: "You cannot request an increase in crawl rate" (reduce-crawl-rate doc); "You can't tell Google to increase your crawl rate" (Crawl Stats help). More crawling comes only from the §2.5 levers — capacity and quality.
  • ⚠️ Before throttling anything: if the "Googlebot" hammering you doesn't verify against Google's published IP ranges, it's an impostor, and your emergency response should be a firewall rule, not a 503 policy (verification workflow → Ch 5).

2.8 Reading the Crawl Stats report diagnostically

Source: Crawl Stats report help, fetched 2026-08-04 (support articles display no revision date). Location: Settings → Crawl stats. GSC property mechanics → Ch 19.

Scope and the top-line charts

🟢 "This report is available only for root-level properties" — a Domain property or a root URL-prefix property; a /subfolder/ property has no Crawl Stats. Google's own audience gate: 🟢 "If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail." Top charts: total crawl requests ("whether successful or not"), total download size, average response time. 🟢 "Duplicate requests for the same URL are counted individually." Example URLs in each drill-down are samples "weighted by day," not exhaustive lists.

What counts — and what doesn't

  • 🟢 Same-property page resources are included; "Requests to other domains will not be shown. This includes requests for any page resources (such as images) hosted outside this property." Your CDN subdomain's fetches are invisible here — and remember they debit that hostname's budget (→ §2.5).
  • 🟢 "If a URL has a server-side redirect, each request in the redirect chain is counted as a separate request" — page1→page2→page3 shows as three requests. Client-side redirects are not counted.
  • ⚠️ The phantom-crawl diagnostic: when robots.txt is insufficiently available, 🟢 "Google counts crawls that it might have made if your robots.txt file were available, but doesn't actually make those calls." Crawl Stats totals far exceeding server-log Googlebot totals is itself a symptom of robots.txt unavailability — the report documents this exact comparison as a troubleshooting entry. (Neither logs nor this report alone is the full picture; log pipeline architecture → Ch 20.)

Host status

Three tracked categories: robots.txt fetching, DNS resolution, server connectivity. 🟢 "The chart has a dotted red line; if the metric was above the dotted line for this category (for example, if DNS resolution fails for more than 5% of requests on a given day), that is considered an issue for that category, and the status will reflect the recency of the last issue." Red = a significant issue within the last week; yellow = the last one is older than a week; green = none significant.

The robots.txt category is existential: 🟢 "Google requests this file frequently, and if the request doesn't return either a valid file (either populated or empty) or a 404 (file does not exist) response, then Google will slow or stop crawling your site until it can get an acceptable robots.txt response." The full availability algorithm — the 24-hour reuse window, 404-counts-as-success, the 12-hour/30-day 5xx ladder ending in 🟢 "If the site homepage is available, Google will act as if there is no robots.txt file, and crawl without restraints. If the site homepage is not available, Google will stop crawling the site" — is dissected with the spec in Ch 3.

The four breakdowns, read diagnostically

  • By response. Percentages of requests, not URLs — one URL fetched twice, 500 then 200, counts once in each bucket. A rising 5xx/429 share predicts capacity-limit cuts (→ §2.3, §2.5); a rising 3xx share means chains or a migration in progress (→ Ch 16).
  • By file type. HTML, image, video, JavaScript, CSS, PDF, JSON, syndication, and more. A JS/CSS-heavy mix on an HTML site means rendering fetches dominate — check for per-deploy cache-busting (→ §2.6).
  • By purpose. 🟢 "Discovery: The URL requested was never crawled by Google before. Refresh: A recrawl of a known page." Discovery spiking while your sitemap is flat = URL-space explosion (faceted navigation, parameter bugs → Ch 10). Refresh starvation on fast-changing pages = sitemap <lastmod> hygiene (→ Ch 9).
  • By Googlebot type. Smartphone, Desktop, Image, Video, 🟢 "Page resource load: A secondary fetch for resources used by your page," AdsBot, StoreBot, Other. 🟢 Media loaded as a page resource is "counted as Page resource load, not as Image" — so a low Image count doesn't mean images aren't fetched. 🟢 "AdsBot crawls URLs about every 2 weeks" — an AdsBot spike usually means someone created Dynamic Search Ads targets (URL_Equals / page feeds), and capacity is shared across all crawlers (→ §2.5).

Drops, spikes, and the URL-level gap

Documented drop causes: a new or broadened robots.txt rule, slower responses, a rising server-error rate, or lower-value/less-fresh content ("If a site has information that changes less frequently, or isn't very high quality, we might not crawl it as frequently"). Documented spike causes: new sections, unblocking large sections, AdsBot. Cross-check with URL Inspection: 🟢 Hostload exceeded "means that Googlebot can't crawl as many URLs from your site as it discovered" (troubleshoot crawling errors — this and the next quote live there, not in the Crawl Stats help) — the capacity ceiling is real, and the fix is serving capacity (→ §2.5).

URL-level history is explicitly not in Search Console: 🟢 "Search Console doesn't provide a crawl history for your site that can be filtered by URL or path, but you can inspect your site logs to see whether specific URLs have been crawled by Googlebot" (same troubleshooting doc) — Google's own pointer to log analysis, seconded by the resources post: 🟢 "The best source to analyze what resources Google is crawling is the site's raw access logs." The log pipeline (schemas, CDN-vs-origin reconciliation, verified-Googlebot filtering, alert thresholds) is Ch 20's home.

2.9 Supported and indexable file types — including PDFs

The inventory

🟢 "Google can index the content of most text-based files and certain encoded document formats. The file type is determined by the Content-Type HTTP header returned when Google crawls the file, though in some cases Google may use the file extension or re-parse the file using a different parser if the Content-Type header is missing or incorrect" (File types indexable by Google, fetched 2026-08-04, last updated 2026-02-03).

⚠️ Note the URL: the long-standing help-center address support.google.com/webmasters/answer/35287 now 301s (via a deprecated /search/docs/advanced/crawling/… hop) to this developers.google.com page — redirect chain verified live 2026-08-04. Cite the new home; serve your own Content-Type headers correctly rather than relying on Google's extension fallback.

  • Flat file types (🟢 "files where the content is stored in plain, unencoded text (though they may use markup tags)"): HTML (.htm/.html and other extensions), Text (.txt/.text and others) including source code in common languages (.bas, .c/.cc/.cpp/.cxx/.h/.hpp, .cs, .java, .pl, .py), CSV, SVG, XML, TeX/LaTeX, Google Earth KML/KMZ, GPX, WML.
  • Encoded file types (🟢 "binary files or complex containers that require a specific parser to extract the human-readable text"): PDF (.pdf), PostScript (.ps), EPUB, Hancom Hanword (.hwp), Excel (.xls/.xlsx), PowerPoint (.ppt/.pptx), Word (.doc/.docx), OpenOffice presentation/spreadsheet/text (.odp/.ods/.odt), RTF.
  • Media formats (🟢 "Google can also index the following media formats"): images — BMP, GIF, JPEG, PNG, WebP, SVG, AVIF; video — 3GP, 3G2, ASF, AVI, DivX, M2V, M3U, M3U8, M4V, MKV, MOV, MP4, MPEG, OGV, QVT, RAM, RM, VOB, WebM, WMV, XAP. Image and video search indexing proper → Ch 14.
  • Auditing tool: 🟢 the filetype: operator limits results to a format or extension — "filetype:rtf galway will search for RTF files and URLs ending in .rtf whose content contains the term 'galway'." Run filetype:pdf site:yourdomain.com to see your non-HTML index footprint before deciding policy for it.

PDF behavior worth engineering around

  • Limits: PDFs get the special 64MB fetch limit (→ §2.2) but the same truncation rule — text past the cutoff doesn't exist to indexing. ⚪ Scanned-image PDFs with no text layer index poorly — the parser extracts text, not pictures (industry observation; Google's file-types doc doesn't address it).
  • Controls: a PDF has no <head> — no title tag, no meta robots, no rel=canonical link element. Its page-level controls are HTTP headers, and both are documented: 🟢 "A response header can be used for non-HTML resources, such as PDFs, video files, and image files" for X-Robots-Tag indexing directives (block-indexing doc, fetched 2026-08-04; → Ch 7), and 🟢 the Link: <…>; rel="canonical" header applies to "non-HTML documents such as PDF files" (consolidate duplicate URLs, fetched 2026-08-04; → Ch 8).
  • ⚠️ PDF↔HTML pairs: when the same content exists as an HTML page and a PDF (spec sheets, documentation, whitepapers), Google clusters them as duplicates and picks one canonical — and the PDF can win if it earns the links. Keeping the HTML version canonical (canonical HTTP header on the PDF, internal-link preference) is Ch 8's home (→ Ch 8, PDF/HTML pairs). Don't robots.txt-block the PDF to "solve" the duplication: the block only hides the duplicate signals and the blocked URL can stay indexed anyway (→ Ch 3).

Symptoms & diagnosis

Symptom Likely cause Where
Indexed pages disappearing "within days," crawl rate collapsed Network timeouts / connection resets / DNS failures — terminal "hard" errors, worse than clean 5xx §2.4, Ch 4
Pages deindexed but Googlebot request volume unchanged 401/403/404 served to Googlebot — 4xx deindexes without throttling §2.3
Site-wide crawl slowdown, indexed URLs eroding over weeks Persistent 5xx or 429 (incl. WAF rate-limiting with 429) §2.3, §2.7
Content in the lower part of a huge HTML file not indexed Per-file fetch cutoff — first 2MB uncompressed per supported file (64MB PDF) §2.2
Soft 404 wave in Page indexing on a JS site Rendering/resource failure (missing JS file, empty rendered body), not content §2.4, Ch 6
Thousands of URLs collapsing into duplicates after an outage/WAF event Error text served with HTTP 200 at scale → duplicate elimination; slow recovery §2.4, Ch 4, Ch 8
Crawl Stats total ≫ server-log Googlebot count robots.txt insufficiently available — phantom "abandoned" crawls counted but never made §2.8, Ch 3
Crawl volume spike with no site change AdsBot (new Dynamic Search Ads targets, ~2-week cycle) or newly unblocked/added sections §2.8, §2.5
"Page resource load" dominates Googlebot-type breakdown WRS re-fetching resources — check per-deploy cache-busting fingerprints §2.6, §2.8
Discovery crawls spiking, sitemap flat URL-space explosion: faceted navigation, parameter bugs, infinite spaces §2.5, Ch 10
New pages take days to be crawled (normal-size site) Documented baseline — "several days minimum"; not a defect §2.5
Conditional requests never arrive despite ETag/Last-Modified Headers aren't sent on every crawl; verify validators, consider unconditional 304 §2.6
"Hostload exceeded" in URL Inspection Crawl capacity ceiling reached — add serving capacity §2.5, §2.8
Crawl rate must drop NOW (origin melting) Serve 503/429 site-wide (1–2 days max) or file the Googlebot report form; the GSC rate limiter died Jan 8, 2024 §2.7
Geo-personalized/consent content indexed wrong US IP egress — Google judges from a US vantage; US-blocking is what triggers non-US crawls §2.1, Ch 11
PDF outranks its HTML twin Duplicate cluster canonicalized to the PDF §2.9, Ch 8

Sources

  • Googlebot — https://developers.google.com/search/docs/crawling-indexing/googlebot (fetched 2026-08-04; page last-updated 2026-02-03)
  • Overview of Google crawlers and fetchers (user agents) — https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers (fetched 2026-08-04; page last-updated 2026-06-12)
  • Optimize your crawl budget — https://developers.google.com/crawling/docs/crawl-budget (fetched 2026-08-04; page last-updated 2026-07-22)
  • How HTTP status codes affect Google's crawlers — https://developers.google.com/crawling/docs/troubleshooting/http-status-codes (fetched 2026-08-04; page last-updated 2026-02-04)
  • Debug DNS and network errors — https://developers.google.com/crawling/docs/troubleshooting/dns-network-errors (fetched 2026-08-04; page last-updated 2025-12-18)
  • Troubleshoot Google Search crawling errors — https://developers.google.com/search/docs/crawling-indexing/troubleshoot-crawling-errors (fetched 2026-08-04; page last-updated 2025-12-18)
  • Crawl Stats report — Search Console Help — https://support.google.com/webmasters/answer/9679690 (fetched 2026-08-04; no revision date shown)
  • Reduce Google crawl rate — https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate (fetched 2026-08-04; page last-updated 2025-12-18)
  • Myths and facts about crawling — https://developers.google.com/crawling/docs/myths-about-crawling (fetched 2026-08-04; page last-updated 2025-12-18)
  • Things to know about Google's web crawling — https://developers.google.com/crawling/docs/about-crawling (fetched 2026-08-04; page last-updated 2026-03-03)
  • File types indexable by Google — https://developers.google.com/search/docs/crawling-indexing/indexable-file-types (fetched 2026-08-04 via 301 from support.google.com/webmasters/answer/35287; page last-updated 2026-02-03)
  • Crawling December: The how and why of Googlebot crawling — https://developers.google.com/search/blog/2024/12/crawling-december-resources (fetched 2026-08-04; posted 2024-12-03, updated 2024-12-06)
  • Crawling December: HTTP caching — https://developers.google.com/search/blog/2024/12/crawling-december-caching (fetched 2026-08-04; posted 2024-12-09, Gary Illyes)
  • Crawling December: CDNs and crawling — https://developers.google.com/search/blog/2024/12/crawling-december-cdns (fetched 2026-08-04; posted 2024-12-24, Martin Splitt & Gary Illyes)
  • Goodbye crawl rate limiter tool — https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebye (fetched 2026-08-04; posted 2023-11-24)
  • In-depth guide to how Google Search works — https://developers.google.com/search/docs/fundamentals/how-search-works (fetched 2026-08-04; page last-updated 2025-12-18)
  • Search Central documentation updates (Dec 18, 2025 entries: JavaScript execution on non-200 status codes, second crawling-docs migration batch; Feb 3, 2026 entry: Googlebot file size limits) — https://developers.google.com/search/updates (fetched 2026-08-04)
  • Crawling docs changelog (Nov 20, 2025 migration entry; Jul 22, 2026 crawl-budget entry) — https://developers.google.com/crawling/docs/changelog (fetched 2026-08-04)
  • Site moves with URL changes — https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes (fetched 2026-08-04; page last-updated 2026-06-17)
  • Google 404 vs 410 status codes (Mueller office-hours transcription, May 2018) — https://www.searchenginejournal.com/google-404-status/254429/ (fetched 2026-08-04; recording youtube.com/watch?v=kQIyk-2-wRg)
  • How Google handles JavaScript throughout the indexing process (MERJ/Vercel) — https://vercel.com/blog/how-google-handles-javascript-throughout-the-indexing-process (fetched 2026-08-04; posted 2024-07-31; ⚪ third-party measurement)
  • Block Search indexing with noindex — https://developers.google.com/search/docs/crawling-indexing/block-indexing (fetched 2026-08-04)
  • Consolidate duplicate URLs — https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls (fetched 2026-08-04)