Ch 4 — Hosting, CDN, WAF & DNS

Part II — The crawl layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md has no dedicated hosting/CDN/WAF/DNS lines. Its nearest coverage is Phase 1's robots.txt review (line 74), the AI-crawler policy decisions (lines 75, 198), and the glossary's one-line Googlebot definition ("What it can't fetch or render doesn't exist to Google," line 270). The playbook assumes the serving stack answers Googlebot cleanly. This chapter is the failure catalog for when it doesn't: WAF challenge pages, CDN blocklists, rate-limiting 429s, DNS outages, wildcard hostname explosions, dangling CNAMEs, and edge-modified HTML drifting away from the origin.

The crawl layer has a property the rest of SEO does not: Google reacts to what your infrastructure does, not what your team intended. A WAF rule nobody remembers writing, a challenge interstitial a CDN enabled by default, an expired CNAME, or a DNS record that stops validating will deindex pages faster than any content or ranking problem can — Google's own documentation puts the clock for unreachable URLs at "within days." And because these failures happen between Google and your origin, they are invisible in the CMS, often invisible in origin logs, and frequently invisible to every human visitor. The mental model for this chapter: Googlebot is a client your infrastructure serves, and every layer between Google and your HTML — DNS, CDN, WAF, edge worker, origin — can independently lie to it. Diagnosis means finding which layer lied.

4.1 How Google's crawl model reads your serving infrastructure

Three documented mechanics determine how infrastructure choices translate into crawl behavior; a fourth (what happens when you change infrastructure) follows from them.

4.1.1 The unit of crawl accounting is the hostname

🟢 "Google's crawling infrastructure defines a site as a unique hostname. For example, https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets" — crawl budget doc (fetched 2026-08-04, page updated 2026-07-22).

Consequences for infrastructure design:

• Every hostname you add — a cdn. subdomain for assets, a blog. on another platform, an image CDN — opens a separate crawl-budget ledger with its own capacity ceiling, its own robots.txt (→ Ch 3: rules bind only their own host/protocol/port), and its own failure surface.
• The full capacity × demand model is Ch 2's canonical home (→ Ch 2). What matters here is the serving side of the equation: 🟢 the crawl capacity limit "limits the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration" (same doc).
• 🟢 "If the site slows down (latency increases or response times become longer), or responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429), the limit goes down and Google crawls less" (same doc). Your infrastructure's latency and error rate are the crawl-rate control. There is no dial anywhere else — the GSC crawl-rate limiter was retired January 8, 2024 (→ Ch 2).

4.1.2 CDN backing is detected from the serving IP — and raises the throttling ceiling

🟢 "Our crawling infrastructure is designed to allow higher crawl rates on sites that are backed by a CDN, which is inferred from the IP address of the service that's serving the URLs our crawlers are accessing" — Crawling December: CDNs and crawling (Splitt & Illyes, Dec 24, 2024; fetched 2026-08-04).

🟢 "The threshold for this throttling is much higher when our crawling infrastructure detects that your site is backed by a CDN, and assumes that it's fine to send more simultaneous requests because your server most likely can handle it" (same post).

• This is the only documented mechanism by which a pure infrastructure choice increases crawl capacity.
• Note what it is not: it is not a ranking factor, and it is inferred from serving IPs — you cannot declare it, and switching CDN or hosting changes the inference input silently.

⚠️ The cold-cache launch caveat. The CDN's higher ceiling does not exempt your origin from the first serve of every URL: 🟢 "even if your webshop is backed by a CDN, your server will need to serve those 1,000,007 URLs at least once. Only after that initial serve can your CDN help you with its caches. That's a significant burden on your 'crawl budget' and the crawl rate will likely be high for a few days; keep that in mind if you're planning to launch many URLs at once" (same post). Two operational corollaries:

• Large launches behind a CDN still need origin capacity planning for the crawl burst — the CDN starts helping only after each URL's first serve.
• A WAF that interprets that legitimate first-crawl burst as an attack produces exactly the failure modes of §4.2, at the worst possible moment (launch week).

4.1.3 Resource hostnames shift the crawl bill; they don't erase it

🟢 "Crawling the resources needed to render a page will chip away from the crawl budget of the hostname that's hosting the resource" — Crawling December: resources post (Dec 3, 2024; fetched 2026-08-04).

• Hosting resources on a separate hostname (CDN subdomain) 🟢 "will shift crawl budget concerns to the host that's serving the resources" (same post).
• ⚠️ The post's December 6, 2024 in-line correction narrows the advice: 🟢 "we don't recommend this strategy for critical resources (such as JavaScript or CSS) that are needed for rendering a page. However, for larger non-critical resources such as video or downloads, this approach is worth considering."
• Backing the main hostname with a CDN sidesteps the tradeoff entirely: 🟢 "If you back your main host with a CDN, then you avoid this problem: one hostname to query, and the critical rendering resources are likely served from the CDN's cache," and "Google's crawling infrastructure supports either option without issues" (CDN post).
• WRS's 30-day resource cache (which ignores your HTTP caching headers) and the cache-busting-parameter trap are Ch 2/Ch 6 material (→ Ch 2, → Ch 6).

⚠️ Your monitoring has a hole exactly where your CDN hostname is. The Crawl Stats report 🟢 excludes off-property fetches: "Requests to other domains will not be shown. This includes requests for any page resources (such as images) hosted outside this property" — Crawl Stats help (fetched 2026-08-04). A misbehaving cdn.example.com can throttle rendering of the main site without appearing in the main property's Crawl Stats or the main origin's logs. Countermeasures:

• Register the resource hostname as its own GSC property (Domain property if it's a subdomain of a domain you've verified — §4.5).
• Reconcile CDN edge logs against origin logs; the edge sees Googlebot hits the origin never receives (→ Ch 20).

4.1.4 Changing hosting, CDN, or WAF is a crawl event

Swapping hosting provider, CDN, or WAF with URLs unchanged is a site move without URL changes — Ch 16 owns the procedure (→ Ch 16). The crawl-layer mechanics that make it worth care:

• Google's capacity model re-learns your infrastructure from serving behavior: latency, error rate, and the CDN-backing IP inference (§4.1.2) all change on cutover day. Expect crawl-rate adjustment; verify health in Crawl Stats host status the week after.
• ⚪ Keep the old infrastructure answering (or DNS TTLs short) through the propagation window; Google's DNS debugging doc acknowledges changes "within the last 72 hours" may still be settling (§4.4).
• ⚠️ The classic cutover injury is not the swap itself but the new layer's defaults: a new CDN's bot protection enabled out of the box, a new WAF's managed ruleset challenging crawlers, a new origin's firewall geo-blocking. Every cutover checklist should end with the §4.2 URL Inspection check.

4.2 WAF and bot-management failure modes — the canonical taxonomy

This section is the book's canonical home for WAF/challenge-page/CDN failure modes; other chapters cross-reference it rather than restating it.

The problem statement, from Google's own CDN post: 🟢 "occasionally the bots that you do want on your site may end up in your CDN's blocklist, typically in their Web Application Firewall (WAF). This prevents crawlers from accessing your site, which ultimately may prevent your site from showing up in search results" — CDNs and crawling. Google sorts the ways a WAF can answer a crawler into hard blocks (error responses) and soft blocks (challenge interstitials).

4.2.1 The four ways a WAF answers Googlebot, ranked by damage

WAF behavior toward Googlebot	Google's classification	Documented consequence
503 or 429 response	Hard block, correctly signaled	🟢 "Sending these status codes is the preferred way to signal a temporary blockage. It will give you some time to react to unintended blocks by the CDN." Crawl slows; index preserved short-term; safe ~1–2 days, destructive beyond 2–3 (→ §4.3).
Connection timeout / silent drop	Terminal "hard" error	🟢 "Network timeouts from the CDN will cause the affected URLs to be removed from Google's search index, as these network errors are considered terminal, 'hard' errors. Additionally they may also considerably affect your site's crawl rate because they signal our crawl infrastructure that the site is overloaded." Removal clock: "within days" (§4.4).
Error/block page with HTTP 200	Soft error	🟢 "this is particularly bad. If the error message is equated on Google's end to a 'hard' error (say, an HTTP 500), Google will remove the URL from Search. If Google couldn't detect the error messages as 'hard' errors, all the pages with the same error message may be eliminated as duplicates from Google's search index. Since Google indexing has little incentive to request a recrawl of duplicate URLs, recovering from this may take more time."
Challenge interstitial ("are you human?") with HTTP 200	Soft block — the challenge is your content now	🟢 "Our crawlers are in fact convinced that they're NOT human and they're not pretending to be one. They just wanna crawl. However when the interstitial shows up, that's all they see, not your awesome site."

Reading the table top to bottom: a correctly signaled block is a manageable incident; a silent one is a deindexing event; a 200-coded one is a deindexing event with a slow recovery tail.

4.2.2 Why the challenge page is the signature failure

• A JavaScript or CAPTCHA challenge served with 200 replaces your page, sitewide, with a near-identical interstitial. Mechanically this is cloaking-shaped — the crawler receives different content than human visitors — but the resemblance runs in the self-harming direction: Google's cloaking policy targets "presenting different content to users and search engines with the intent to manipulate search rankings" (spam policies, fetched 2026-08-04; → Ch 18), and no Google source treats WAF challenges as a spam violation. Do not let anyone on the team panic about penalties; the real risk is worse in a different way.
• ⚠️ The documented damage is indexing damage via the soft-error class: thousands of URLs all serving the same challenge HTML are prime "eliminated as duplicates" material, and duplicate-eliminated URLs recover slowly because Google has "little incentive to request a recrawl."
• A challenge-page incident is therefore worse than a clean outage of the same duration: a 503 outage preserves the index for days; a 200-challenge actively replaces and deduplicates it.
• ⚠️ JS-based challenges do not "pass" for Googlebot because WRS executes JavaScript. Rendering is not solving: the challenge is designed to distinguish automation, Google's crawlers openly identify as automation, and Google's own instruction (below) presumes the challenge blocks them.

4.2.3 The sanctioned configuration

🟢 "In case of these bot-verification interstitials, we strongly recommend sending a clear signal in the form of a 503 HTTP status code to automated clients like crawlers that the content is temporarily unavailable. This will ensure that the content is not removed from Google's index automatically" (CDN post).

⚪ Implementation (house standards; Google specifies only the status code):

• Verified search crawlers are either exempt from challenges or receive a plain 503 — never the challenge HTML with 200, never a silently dropped connection.
• Verification means IP/rDNS against Google's published data, not user-agent strings — fake-Googlebot traffic is routine, and a UA-string exemption is an open door (verification stack → Ch 5).
• Apply the same policy decision deliberately to non-Google crawlers you care about (Bing, AI crawlers per your Ch 5 decision matrix) — the failure taxonomy above is crawler-agnostic even though the evidence quotes are Google's.
• Exempt /robots.txt and sitemap endpoints from all challenge/rate rules (§4.3 explains why this one file can take the whole site down).

4.2.4 The WAF incident workflow

1. Confirm from Google's side, not yours. 🟢 "use the URL Inspection tool in Search Console and observe the rendered image: if it shows your page, you're good; if it shows an empty page, an error, or a page with a bot challenge, you might want to talk to your CDN about it" (CDN post). This works because URL Inspection fetches with Google's real infrastructure from Google's real IPs — it experiences your WAF exactly as Googlebot does. A browser check proves nothing: the WAF is specifically treating you and Googlebot differently.
2. Read Crawl Stats for the shape: rising 5xx/429 share, falling total requests, or "Other client error (4xx)" spikes; host status for robots.txt/connectivity failures (report mechanics → Ch 2).
3. Pull the WAF/CDN event logs and match blocked/challenged requests against Google's published IP ranges (→ Ch 5). 🟢 The post's own tip for long blocklists: look for "just the first few segments of the IP ranges" rather than full addresses (CDN post).
4. Fix at the WAF: remove the blocking rule or add the verified-crawler exemption. 🟢 Google links vendor bot-management docs for Cloudflare, Akamai, Fastly, F5, and Google Cloud Armor in the post — the setting exists in every major stack.
5. Expect gradual, not instant, recovery. 🟢 "Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site" — HTTP status codes doc. Soft-error/duplicate-eliminated URLs recover slowest (little recrawl incentive); request recrawls for the highest-value URLs via URL Inspection (→ Ch 19).
6. Make it recurring. 🟢 "Remember that the IPs may end up on a blocklist automatically, without you knowing, so checking in on the blocklists every now and then is a good idea for your site's success in search and beyond" (CDN post). ⚠️ ⚪ WAF vendors update managed rulesets and reputation feeds continuously; a site that changed nothing can start blocking Googlebot overnight. Blocklist review and crawled-with-error alerting are standing operational tasks (→ Ch 20), not setup steps.

4.3 Rate limiting Googlebot: 429s, 401/403, and the robots.txt chokepoint

4.3.1 429 is a server error in Google's taxonomy, not a polite request

• 🟢 "Google's crawlers treat the 429 status code as a signal that the server is overloaded, and it's considered a server error" — HTTP status codes doc (fetched 2026-08-04, page updated 2026-02-04).
• 🟢 "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped" (same doc).
• 🟢 The 500-family entry states the endpoint plainly: "For Google Search, Google's indexing pipeline removes from the index URLs that persistently return a server error."
• A CDN rate-limiting rule that catches Googlebot is therefore a two-stage weapon: immediate crawl throttling, then deindexing of whatever it keeps catching. (Full status-code semantics, including redirect signal strengths and the 4xx table → Ch 2.)

4.3.2 The 401/403 trap: deindexing without throttling

⚠️ 🟢 "Don't use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate" (same doc). A WAF answering Googlebot with 403:

• does not slow crawling at all — 4xx has no crawl-rate effect;
• tells the indexing pipeline "the content doesn't exist," and previously indexed URLs are removed.

Sites "rate-limiting" bots with 403s are deindexing themselves without throttling anything. If the goal is throttling, the only correct codes are 429/503/500; if the goal is exclusion, that is an indexing-controls decision (→ Ch 7), not a WAF rule.

4.3.3 Deliberate emergency throttling: sanctioned but time-boxed

• 🟢 "return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests" for urgent reduction — reduce crawl rate doc (fetched 2026-08-04, updated 2025-12-18).
• 🟢 "We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products," and "if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index" (same doc).
• 🟢 The Crawl Stats help sets the outer bound: "Be sure not to return 503 or 429 for more than two or three days, though, or it can signal Google to crawl your site less frequently in the long term" (Crawl Stats help).
• Emergency crawl control as a discipline — including the Googlebot report form and the AdsBot-overcrawl diagnosis — is Ch 2's home (→ Ch 2). Planned downtime and pausing a business (503 + Retry-After patterns) are Ch 16's (→ Ch 16).

4.3.4 The robots.txt chokepoint: your WAF can invert your robots.txt

robots.txt semantics live in Ch 3 (→ Ch 3). The infrastructure consequences belong here, because a WAF/CDN rule that intercepts /robots.txt flips the whole site's crawl state — in either direction:

• WAF serves 403 (or any 4xx except 429) on robots.txt → fail-open. 🟢 "Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist" — meaning "Google assumes that there are no crawl restrictions" at all (robots.txt spec, fetched 2026-08-04, updated 2026-07-08). The "block bots by blocking robots.txt" configuration achieves the opposite of its intent.
• WAF serves 5xx/429, a challenge, or times out on robots.txt → fail-closed. 🟢 "A robots.txt file which cannot be fetched due to DNS or networking issues, such as timeouts, invalid responses, reset or interrupted connections, and HTTP chunking errors, is treated as a server error" (same spec). And per the Crawl Stats help: 🟢 "if the request doesn't return either a valid file (either populated or empty) or a 404 (file does not exist) response, then Google will slow or stop crawling your site until it can get an acceptable robots.txt response."
• The full 12-hour → 30-day → homepage-availability escalation ladder for robots.txt 5xx is Ch 3's home (→ Ch 3 §3.7).

⚪ House rule that follows: exempt /robots.txt and sitemap endpoints from every WAF challenge, rate-limit, and geo rule; monitor robots.txt availability in Crawl Stats host status; and alert on it, because this single URL's availability gates crawling of everything else.

4.4 DNS failures, DNSSEC, and the fastest deindexing clock in the book

4.4.1 DNS errors sit in the server-error class, with a days-scale removal clock

• 🟢 "Google treats network timeouts, connection reset, and DNS errors similarly to 5xx server errors" — Debug DNS and network errors (fetched 2026-08-04, page updated 2025-12-18).
• 🟢 "In case of network errors, crawling immediately starts slowing down, as a network error is a sign that the server may not be able to handle the serving load" (same doc).
• 🟢 The clock: "already indexed URLs that are unreachable will be removed from Google's index within days" (same doc).
• Combine with the CDN post's "terminal, hard errors" framing (§4.2): transport-level failure is the fastest route out of the index that exists. A weekend DNS outage is not a weekend problem — it is an index-recovery project.

The first-party monitor is Crawl Stats host status. DNS resolution is one of its three tracked categories (robots.txt fetching, DNS resolution, server connectivity): 🟢 "The chart has a dotted red line; if the metric was above the dotted line for this category (for example, if DNS resolution fails for more than 5% of requests on a given day), that is considered an issue for that category"; the DNS chart 🟢 "shows when your DNS server didn't recognize your hostname or didn't respond during crawling" (Crawl Stats help). Reading the report as a whole → Ch 2.

4.4.2 Google's own debugging order

From the DNS/network doc (🟢 throughout), in sequence:

1. Firewall first: check firewall settings and logs — "Make sure that Google IP addresses are not blocked by any firewall rule." ⚠️ A "DNS problem" in GSC is often a firewall dropping packets — §4.2 wearing a DNS costume. Rule this out before touching DNS.
2. Capture and analyze traffic with tcpdump/Wireshark if the firewall is clean.
3. Inspect DNS records: A/CNAME records, and "Check that all your name servers are pointing to the correct IP addresses of your site."
4. Allow for propagation: changes made "within the last 72 hours" may still be settling.
5. If you run your own DNS server, verify its health; if nothing surfaces, escalate to the hosting company.

⚪ Corollaries (industry practice):

• Run authoritative DNS on a provider with an SLA unless you have a hard operational reason not to. Authoritative-DNS downtime takes every hostname, every subdomain, and your robots.txt (§4.3) down at once — it is the single widest blast radius in the stack.
• Before planned infrastructure changes (§4.1.4), lower record TTLs in advance so the cutover propagates inside the change window instead of trailing it by days.
• Monitor resolution externally from more than one resolver network; your office resolver's cache can show a healthy site hours into a global resolution failure (the DNSSEC signature of §4.4.4 makes this literal).

4.4.3 Domain expiry: the DNS failure that serves content

⚠️ An expired domain registration usually does not produce a clean NXDOMAIN. ⚪ ICANN's Expired Registration Recovery Policy (fetched 2026-08-04) requires registrars to interrupt "the existing DNS resolution path" after expiration and expressly contemplates that "the registrar directs web traffic to the domain name to a web page" during the renewal window (ERRP §2.2.2–2.2.4) — in practice a registrar-served parking/renewal page, which answers 200 with boilerplate on every URL. In Google's documented taxonomy that is not a DNS error at all; it is a sitewide content replacement: at best soft-error/duplicate elimination of every URL (§4.2's worst row), at speed, with the slow recovery tail. The renewal-lapse scenario is strictly worse than an outage of the same length. Prevention is administrative, not technical: registry-lock or auto-renew with a monitored payment method, registrant email on a monitored alias, expiry alerts in the ops calendar (→ Ch 20).

4.4.4 DNSSEC: a signing error is a total DNS outage for validating resolvers

⚠️ No Google document describes how Googlebot's resolvers handle DNSSEC — the crawling docs never mention it, and this book will not invent a mechanism. What is documented and standard:

• 📘 Under DNSSEC (RFC 4033–4035), a validating resolver that cannot validate a signed zone returns SERVFAIL rather than an answer — the domain simply fails to resolve.
• 🟢 Google documents exactly this behavior for its own public resolver: "If Google Public DNS cannot validate a response (due to misconfiguration, missing or incorrect RRSIG records, etc.), it will return an error response (SERVFAIL) instead," with validation on by default — Public DNS FAQ (fetched 2026-08-04).
• ⚪ The safe, honest composite: a DNSSEC misconfiguration (expired RRSIGs, a botched key rollover, a registrar DS record pointing at a key you no longer publish) makes your domain unresolvable to some or all of the world's resolvers. To the extent Google's crawl path is affected, what Googlebot experiences is indistinguishable from a DNS outage — which puts you on the 🟢 documented "removed within days" clock of §4.4.1.
• Diagnostic signature: users on validating resolvers go dark at the same time as crawlers, so the symptom pattern is simultaneous loss of search and direct traffic, while your own office (on a non-validating resolver, or with the record cached) sees a healthy site.

⚪ Prevention: automate RRSIG renewal; never let DS records outlive the keys they reference (rollover discipline); after any key event, test with external validators (DNSViz, Verisign Labs' DNS Analyzer — the same tools Google's own Public DNS FAQ points users to for diagnosing DNSSEC failures).

4.5 Wildcard DNS as infinite crawl space

A wildcard record (📘 defined in RFC 1034 §4.3.3) makes every otherwise-undefined label under a domain resolve: *.example.com answers for anything.example.com, including subdomains that have never existed. Combined with a webserver or CDN that accepts any Host header and returns 200, this creates the hostname edition of an infinite crawl space — every typo, every malformed link, every spam link pointing at an invented subdomain becomes a live, crawlable, indexable site.

Google's infinite-space doctrine is old and still live. 🟢 "When Googlebot crawls the web, it often finds what we call an 'infinite space'. These are very large numbers of links that usually provide little or no new content for Googlebot to index. If this happens on your site, crawling those URLs may use unnecessary bandwidth, and could result in Googlebot failing to completely index the real content on your site" — To infinity and beyond? No! (Aug 5, 2008; fetched 2026-08-04). The modern crawling primer concedes Google prunes some of this itself — 🟢 "calendars that go to the year 9999 probably don't need to be crawled in their entirety" (about crawling, fetched 2026-08-04, page updated 2026-03-03) — but detection is heuristic, and the 2008 post already warned Google "may not detect all of them."

⚠️ Wildcard subdomains are worse than wildcard paths, for a documented structural reason. Path-level infinite spaces (faceted URLs, calendars → Ch 10) at least share one hostname's crawl ledger and one robots.txt. Each wildcard-minted hostname is a separate site with a separate crawl budget and its own (probably template-served) robots.txt (§4.1.1; → Ch 3) — an unbounded hostname space multiplies sites, not URLs, and the robots.txt on www controls none of it.

🟡 John Mueller, English Webmaster Central office-hours hangout, August 2016 (per Search Engine Roundtable's transcript writeup, published Aug 4, 2016; primary video not re-fetched): "With subdomains, the main thing I'd watch out for is that you're not using wildcard subdomains because that can make crawling really, really hard. If we have to go through all of these subdomains and treat them all as separate hosts."

⚪ Wildcard-DNS infinite spaces are also an established blackhat pattern — "spider pool" networks that, per the paper's abstract, "abuse wildcard DNS to create virtually infinite sites and construct complicated loop structure to force search-engine crawlers to visit them relentlessly," documented academically in Du, Yang, Li, Duan & Zhang, "The Ever-Changing Labyrinth: A Large-Scale Analysis of Wildcard DNS Powered Blackhat SEO" (25th USENIX Security Symposium, 2016; PDF fetched 2026-08-04). Company a legitimate site does not want to keep: an accidental wildcard makes your domain look structurally like that pattern's substrate.

⚪ Prescriptions (industry practice; Google is silent on the mechanics):

• No catch-all in production DNS unless a product requires it. Undefined hostnames should fail to resolve (NXDOMAIN) — a DNS error Google walks away from — rather than resolve and 200.
• If a wildcard is required (multi-tenant SaaS, per-customer subdomains): make the server strict even though DNS is loose. Unknown tenants get a hard 404/410 — not a rendered template (recreates the infinite space one layer up), not a 200 "tenant not found" page (soft-error class, §4.2), not a redirect to the marketing homepage (soft-404 pattern → Ch 2).
• Each real tenant hostname needs its own robots.txt and carries its own crawl ledger; plan crawl and GSC coverage per tenant, not per platform.
• Staging/dev hostnames that a wildcard quietly exposes belong behind authentication — password protection, not robots.txt (→ Ch 7 §7.10).
• Detection: a GSC Domain property is the instrument, because it 🟢 "includes all subdomains (m, www, and so on) and multiple protocols" (property types, fetched 2026-08-04) — unexplained hostnames in its Page indexing and Performance data are the tell. Corroborate with edge logs, which see Host-header probes the origin may never receive (→ Ch 20).

4.6 Dangling CNAMEs and subdomain takeover — the crawl/index side

(Attack anatomy, credential/cookie exposure, and incident response → Ch 17. This section covers only what search crawling and indexing do with a taken-over subdomain.)

⚪ The mechanism (OWASP-documented industry knowledge — OWASP WSTG-CONF-10, "Test for Subdomain Takeover", fetched 2026-08-04: the victim "uses a CNAME record" to reference another domain that "expires, becoming available for registration by anyone"): a CNAME like promo.example.com → brand.saasplatform.com outlives the SaaS tenant it pointed at. The DNS record still resolves; the target name is claimable by anyone who registers the abandoned tenant/bucket/app slug. The attacker now serves arbitrary content on a hostname you own — no compromise of your servers required.

What the crawl layer does with it follows mechanically from documented behavior, and all of it favors the attacker:

• They inherit a live, crawlable site under your domain. The hostname resolves, serves 200, and is a separate "site" in Google's accounting (§4.1.1) — nothing about it looks broken to a crawler. Historical links to the subdomain fund both its discovery and its crawl demand (→ Ch 2).
• Your crawl and index controls are irrelevant. robots.txt binds per-host (→ Ch 3); meta robots lives in markup the attacker writes. There is no crawl-layer control you retain over a hostname whose serving you lost.
• It surfaces in your Search Console — if you verified the right property type. A Domain property 🟢 covers "all subdomains (m, www, and so on)" (property types); takeover content shows up in its indexing and performance data, and site:example.com -site:www.example.com queries corroborate. ⚪ A URL-prefix-only verification setup is blind to all of it — one more reason the playbook prefers Domain properties (→ playbook Phase 1, line 73; → Ch 19).
• Policy exposure is real even though you're the victim. Google's spam policies flag abuse riding on established hosts — site reputation abuse is a tactic "where third-party content is published on a host site mainly because of that host's already-established ranking signals," and policy-circumvention includes "Using existing or creating new subdomains, subdirectories, or sites with the intention of continuing to violate our policies" (spam policies, fetched 2026-08-04) — and hacked-content/Safe Browsing enforcement can flag the domain (→ Ch 17, → Ch 18). Treat takeover content as a deindexing emergency, not just a security ticket.

Crawl/index remediation (⚪ sequencing; each step's mechanism 🟢-anchored):

1. Delete or repoint the dangling DNS record. Once the hostname stops resolving, Google's documented DNS-error handling (§4.4.1) removes the attacker's URLs "within days."
2. If you reclaim the hostname instead, serve 404/410 across the attacker's URL space — 4xx tells indexing "the content doesn't exist" and removes previously indexed URLs (→ Ch 2).
3. For content that must leave results faster than recrawl allows, use the Removals tool (→ Ch 7 §7.8).
4. Then run the security track: Security Issues review, cleanup, review request if flagged (→ Ch 17).

⚪ Prevention is DNS lifecycle hygiene: deprovision in the right order (delete the DNS record before canceling the service it points to); keep an owned inventory of every CNAME/A record and its downstream dependency; audit on a schedule for records whose targets return provider "unclaimed" pages or NXDOMAIN. Operationally this is Ch 20 material (→ Ch 20); the point here is that every dangling record is pre-provisioned crawl space for someone else.

4.7 Edge SEO, reverse proxies, and origin-vs-edge drift

⚠️ Provenance warning up front: there is no Google documentation for "edge SEO." No Google doc endorses, forbids, or even names the practice of modifying HTML or headers in CDN workers; searches of official sources during drafting confirmed the silence. Everything in this section is ⚪ industry practice except where a documented Google mechanic is cited. The closest official statement is architectural indifference: Google's crawling "supports either option without issues" regarding CDN-backed hostnames (🟢, CDN post, §4.1.3) — Google processes the response the edge serves, full stop.

4.7.1 What edge SEO is

⚪ The term dates to 2018 (Dan Taylor, Search Engine Journal, Dec 19, 2018 — fetched 2026-08-04): using CDN-layer compute (Cloudflare Workers and equivalents) to modify requests and responses in flight — injecting or rewriting titles, meta robots, canonicals, hreflang, redirects, security headers, even robots.txt — without touching origin code. Legitimate uses cluster where the origin is unchangeable: legacy platforms, restrictive SaaS ecommerce, emergency fixes shipped in minutes instead of release cycles.

⚪ A sibling architecture is the reverse-proxy subfolder: serving example.com/blog/ by proxying a separate backend, consolidating content onto one hostname — with the crawl-accounting consequences of §4.1.1 (one hostname, one crawl ledger, one robots.txt) and the subdomain-vs-subdirectory debate left where it belongs (→ Ch 10).

⚪ Mechanically Google is indifferent — it crawls URLs and processes responses; it neither knows nor cares that a worker assembled them — provided the responses are coherent. Incoherence is the entire risk, and it has a name:

4.7.2 Origin-vs-edge drift: the failure taxonomy

⚪ Drift = the origin's truth and the edge's output diverge, and every diagnostic that looks at the origin lies to you. The recurring variants:

1. Double signals. The edge injects a canonical/robots/hreflang tag while the origin already emits one → two conflicting tags in the served HTML. Conflicting canonicals degrade the signal (→ Ch 8); conflicting robots directives resolve restrictively (→ Ch 7). The CMS shows one tag; Google sees two.
2. Stale edge cache. The origin is fixed (a noindex removed, a title corrected) but edge caches keep serving the old body for their TTL — or indefinitely, when purge-on-deploy was never wired. Origin checks pass while Google keeps ingesting the stale version.
3. Partial coverage. Worker routes or POP rollouts that apply the transform to only part of the URL space or network → the same URL class serves different markup on different requests. Nondeterministic serving lands in canonicalization's ambiguity bucket (→ Ch 8) and makes incidents irreproducible.
4. Edge-served robots.txt or redirects drifting from origin intent — §4.3.4's chokepoint, now self-inflicted: the team edits the origin file; the edge keeps serving its own.
5. Crawler-conditional transforms. Workers can key on user-agent or verified-bot status. Serving Googlebot different content than users is the cloaking boundary — 🟢 "presenting different content to users and search engines with the intent to manipulate search rankings" (spam policies); intent is the test, but a UA-conditional content transform is intent made config (→ Ch 18). Keep edge transforms client-agnostic.

4.7.3 Controls

⚪ House standards — Google offers none:

• Edge code and configuration live in version control and ride the same review/release gates as origin code (→ Ch 20 §20.2). An edge worker is production code with sitewide blast radius; treat "quick edge fix" with the same ceremony as a deploy, because it is one.
• Every deploy purges or versions the affected edge caches; cache invalidation is part of the release, not an afterthought.
• Parity testing: scheduled external fetches of representative URLs, diffing edge output against origin output for the SEO-critical elements — status code, canonical, robots directives, title, hreflang, structured data. Add a crawler-UA vs. browser-UA diff to prove no conditional serving crept in.
• URL Inspection is the arbiter of what Google actually received (→ Ch 19) — the only tool in the stack that observes the full DNS → CDN → WAF → edge → origin chain from Google's side.

4.8 Hosting lore, quarantined

Infrastructure attracts durable myths because causation is invisible from the outside. The ones this chapter's domain generates, against what is actually documented:

• "Host on a US server (or in Google's country) for better crawling/rankings." No such guidance exists anywhere in Google's documentation. What is documented: 🟢 "Google egresses primarily from IP addresses in the United States. In case Google detects that a site is blocking requests from the United States, it may attempt to crawl from IP addresses located in other countries" — overview of Google crawlers (fetched 2026-08-04, page updated 2026-06-12). US egress describes where Google crawls from — it implies nothing about where you should host. Server location's real effects route through latency (CWV → Ch 15) and geo-blocking mistakes (blocking US IPs blocks the crawler — → Ch 2, → Ch 11 for locale-adaptive serving).
• "A CDN is a ranking factor" / "CDNs hurt SEO." Neither appears in any Google source. The documented CDN effects are exactly two: the crawl-capacity inference of §4.1.2 (🟢) and whatever the CDN does to latency and availability, which feed the capacity model (§4.1.1) and page experience (→ Ch 15). A CDN that serves fast, correct responses is invisible to ranking; one that challenges crawlers is §4.2.
• "Upgrade hosting to HTTP/2 (or HTTP/3) for a ranking boost." 🟢 "crawling over HTTP/2 may save computing resources (for example, CPU, RAM) for your site and Googlebot, but otherwise there's no Google-product specific benefit to the site (for example, no ranking boost in Google Search)" (same doc). HTTP/3 does not appear in the crawler documentation at all. Protocol negotiation mechanics, including the 421 opt-out, are Ch 2's home (→ Ch 2).
• "Shared hosting puts you in a 'bad IP neighborhood' that Google penalizes." ⚪ No official source supports IP-neighborhood penalties for ordinary shared hosting, and the only documented IP-derived crawl behavior is the CDN-backing inference (§4.1.2) — which, note, raises trust in shared serving IPs rather than lowering it. The legitimate shared-infrastructure risks are operational: a noisy neighbor's load affecting your latency (capacity model, §4.1.1), and a shared WAF/CDN policy blocking crawlers for everyone at once (§4.2).
• "Moving hosts resets your SEO" / "hosting migrations are penalized." No penalty mechanism exists; a clean swap is a documented, supported operation (site move without URL change → Ch 16). What actually goes wrong is §4.1.4: the new stack's defaults, not the move.

Symptoms & diagnosis

Symptom	Likely cause	Where
Pages dropping from index; URL Inspection rendered image shows a bot challenge or empty page	WAF/CDN challenge interstitial served to Googlebot with 200	§4.2
Sitewide duplicate-elimination or soft-404 wave; recovery slow after the fix ships	Block/error pages served with HTTP 200 (soft errors) deduplicated across URLs	§4.2
Indexed URLs removed within days; Crawl Stats shows connectivity failures; origin logs show nothing	CDN/WAF dropping Googlebot connections — timeouts are terminal "hard" errors; origin never sees the requests	§4.2, §4.4
Crawl rate collapses but 4xx rates look normal	429/5xx from a rate-limiting rule (server-error class) shrinking the capacity limit	§4.3
URLs deindexed but crawl rate never slowed	WAF answering Googlebot with 403/401 — no rate effect, straight removal	§4.3
Whole site's crawling slows or stops; robots.txt availability red in Crawl Stats host status	robots.txt 5xx/challenge/timeout behind the WAF (fail-closed)	§4.3
Disallowed sections being crawled anyway	robots.txt served 403/404 by WAF/CDN (fail-open: "as if no robots.txt")	§4.3, → Ch 3
Search and direct traffic drop together; domain intermittently unresolvable	DNS outage — or DNSSEC validation failure (expired RRSIG/stale DS) returning SERVFAIL to validating resolvers	§4.4
DNS resolution above the dotted red line in Crawl Stats	Nameserver failure — or a firewall blocking Google; check firewall first per Google's own debug order	§4.4
Every URL suddenly serving the same thin 200 page	Expired domain repointed to a registrar parking page — sitewide content replacement	§4.4.3
Unknown hostnames accumulating in Domain-property indexing data	Wildcard DNS + catch-all vhost minting crawlable subdomains	§4.5
Spam content indexed under a subdomain nobody remembers creating	Dangling CNAME subdomain takeover	§4.6, → Ch 17
Fixed at origin, but Google keeps indexing the old markup	Stale edge cache or edge-injected tag overriding origin (origin-vs-edge drift)	§4.7
Two canonicals or conflicting robots tags in served HTML that the CMS can't explain	Edge worker injecting tags on top of origin output	§4.7, → Ch 8
Googlebot served different content than browsers by design of an edge rule	Crawler-conditional edge transform — the cloaking boundary	§4.7, → Ch 18

Sources

• Crawling December: CDNs and crawling — https://developers.google.com/search/blog/2024/12/crawling-december-cdns (fetched 2026-08-04; posted 2024-12-24; Splitt & Illyes)
• Crawling December: The how and why of Googlebot crawling — https://developers.google.com/search/blog/2024/12/crawling-december-resources (fetched 2026-08-04; posted 2024-12-03, in-post update 2024-12-06)
• Optimize your crawl budget — https://developers.google.com/crawling/docs/crawl-budget (fetched 2026-08-04; page last-updated 2026-07-22)
• How HTTP status codes affect Google's crawlers — https://developers.google.com/crawling/docs/troubleshooting/http-status-codes (fetched 2026-08-04; page last-updated 2026-02-04)
• Debug DNS and network errors — https://developers.google.com/crawling/docs/troubleshooting/dns-network-errors (fetched 2026-08-04; page last-updated 2025-12-18)
• Reduce Google crawl rate — https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate (fetched 2026-08-04; page last-updated 2025-12-18)
• How Google interprets the robots.txt specification — https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec (fetched 2026-08-04; page last-updated 2026-07-08)
• Crawl Stats report — Search Console Help — https://support.google.com/webmasters/answer/9679690 (fetched 2026-08-04; no revision date shown)
• Overview of Google crawlers and fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers (fetched 2026-08-04; page last-updated 2026-06-12)
• Things to know about Google's web crawling — https://developers.google.com/crawling/docs/about-crawling (fetched 2026-08-04; page last-updated 2026-03-03)
• To infinity and beyond? No! — https://developers.google.com/search/blog/2008/08/to-infinity-and-beyond-no (fetched 2026-08-04; posted 2008-08-05)
• Spam policies for Google web search — https://developers.google.com/search/docs/essentials/spam-policies (fetched 2026-08-04; page last-updated 2026-05-15)
• Search Console property types — https://support.google.com/webmasters/answer/34592 (fetched 2026-08-04; no revision date shown)
• Google Public DNS FAQ (DNSSEC/SERVFAIL behavior) — https://developers.google.com/speed/public-dns/faq (fetched 2026-08-04)
• ICANN Expired Registration Recovery Policy (ERRP) — https://www.icann.org/resources/pages/errp-2013-02-28-en (fetched 2026-08-04; §2.2.2–2.2.4 DNS-interruption and renewal-page provisions)
• Google On How They Treat Subdomains vs Subdirectories — Search Engine Roundtable (Barry Schwartz) — https://www.seroundtable.com/google-treat-subdomains-vs-subdirectories-22485.html (fetched 2026-08-04; published 2016-08-04; Mueller office-hours transcript — 🟡 secondary source)
• What Is Edge SEO & Why You Should Adopt It — Search Engine Journal (Dan Taylor) — https://www.searchenginejournal.com/edge-seo/273983/ (fetched 2026-08-04; published 2018-12-19; ⚪ industry source)
• RFC 1034 §4.3.3 (wildcard DNS records); RFC 4033–4035 (DNSSEC) — 📘 standards, referenced not quoted
• "The Ever-Changing Labyrinth: A Large-Scale Analysis of Wildcard DNS Powered Blackhat SEO" (Du, Yang, Li, Duan & Zhang) — 25th USENIX Security Symposium, 2016 — https://www.usenix.org/system/files/conference/usenixsecurity16/sec16_paper_du.pdf (PDF fetched 2026-08-04; title, venue, authors, and wildcard-DNS "spider pool" findings verified against the paper)
• OWASP Web Security Testing Guide, WSTG-CONF-10: Test for Subdomain Takeover — https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/10-Test_for_Subdomain_Takeover (fetched 2026-08-04)