Ch 20 — Operations: logs, monitoring & release safety¶
Part VII — Measurement & diagnosis · The Technical SEO Reference
Playbook coupling: the playbook has no operations layer — it audits states, not change processes. This chapter exists because the book's own traps catalog is, read honestly, a list of things that shipped by accident: noindex to production, robots.txt overwritten, canonical template bugs, hreflang matrices breaking on deploy. Almost everything here is ⚪ industry engineering practice — labeled as such throughout — anchored to 🟢 Google mechanics where they exist (Crawl Stats, crawler verification, error-driven throttling).
The cheapest technical-SEO incident is the one that can't ship. The second-cheapest is the one detected in hours. Everything in Chapters 2–17 describes failure states; this chapter is the operational program that makes those states either impossible (release gates) or short-lived (monitoring), plus the log pipeline that turns crawl behavior into evidence.
20.1 Server-log analysis for SEO (⚪ practice on 🟢 anchors)¶
Why logs, when Crawl Stats exists: 🟢 the GSC Crawl Stats report (→ Ch 2 §2.7) is sampled, host-aggregated, and ~free; ⚪ logs are complete, URL-level, real-time, and joinable. Use Crawl Stats for trend triage, logs for forensics.
Pipeline shape (⚪): 1. Collect — origin access logs and CDN logs (⚠️ on CDN-fronted sites, origin logs miss every cache-hit crawl — the CDN log is the true crawl record; → Ch 4). Ship to durable storage (BigQuery/ELK/object store). Retention: 13+ months to cover YoY diagnosis (playbook Phase 7's 16-month window argues for 16). 2. Verify — 🟢 filter "Googlebot" lines through reverse-DNS or the published IP-range JSONs (→ Ch 5 §5.4); ⚠️ unverified UA strings are the classic log-analysis error — scrapers fake Googlebot constantly. 3. Normalize — parse UA into crawler tokens (Googlebot Smartphone/Desktop/Image/Video, GoogleOther, AI crawlers per Ch 5's inventory), status, bytes, URL pattern (templated: product/category/facet/parameter classes).
The working queries (⚪ — each anchored to a chapter's mechanic): - Crawl share by URL class over time — facet/parameter overcrawl detection (→ Ch 10 §10.8: is the robots.txt pattern actually reducing hits?). - Status mix per crawler per day — 5xx/429 spikes that trigger 🟢 host-wide crawl throttling (→ Ch 16 §16.9); 404 storms from a bad release. - Crawled-but-not-in-sitemap and sitemap-but-never-crawled deltas (→ Ch 9 segmentation). - Discovery latency: first-crawl timestamp for new URLs vs publish time — the number the "indexing is slow" conversation actually needs. - Render-fetch pattern: WRS resource fetches (→ Ch 6 §6.1) hitting blocked/erroring assets. - AI-crawler audit: who's fetching what volume (→ Ch 5 §5.9's policy decisions need this baseline).
20.2 Release safety — making regressions unshippable (⚪)¶
CI/CD gates, in order of catastrophe prevented:
- robots.txt diff gate — any change to robots.txt fails the build until explicitly approved; block deploys where staging's
Disallow: /pattern would ship (→ Ch 3's availability matrix makes the failure mode precise: a 5xx robots.txt can halt crawling entirely). - noindex-leak assertion — post-build, assert production templates contain no
noindexmeta/header outside the sanctioned list. The staging→production noindex leak is the single most expensive recurring SEO incident (⚠️ and its inverse: staging shipping without noindex → Ch 7 §7.10, Ch 16 §16.5). - Canonical unit tests — per template: self-canonical present, absolute URL, exactly one tag, matches the served host (→ Ch 8's failure taxonomy; the Dec 2025 JS-canonical rules in Ch 6 §6.7).
- hreflang integrity test — generated matrix is reciprocal, self-inclusive, valid codes (→ Ch 11 §11.3–11.4 — remember
en-UKhalf-works, which is why only a test catches it). - Structured-data validation — parse all JSON-LD; fail on syntax errors; warn on types outside Ch 12's live gallery (dead-feature markup shipping fresh).
- Status-code contract tests — error routes actually return 404/410/503 (not 200 soft-404 shells → Ch 6 §6.6/Ch 2 §2.4); redirect map spot-checks (single hop, correct type → Ch 16).
- Post-deploy diff — fetch a sentinel URL set (per template × per locale) after every release; diff title/canonical/robots/hreflang/JSON-LD/status against the pre-release baseline; alert on unexplained change. This catches what unit tests can't: edge/CDN-layer drift (→ Ch 4 §4.7's origin-vs-edge divergence).
20.3 Monitoring & alerting (⚪ thresholds on 🟢 signals)¶
| Signal | Source | Alert on |
|---|---|---|
| Page indexing totals by status | GSC API (→ Ch 19 §19.4) | Step-change in noindexed/excluded counts; "Crawled – currently not indexed" trend break |
| Crawl Stats: total requests, avg response, 5xx share | GSC | 🟢 sustained 5xx/429 (throttling risk), response-time doubling, request-volume cliff |
| robots.txt fetch status | Log pipeline / synthetic fetch | Any non-200 (🟢 a 5xx robots.txt can suspend crawling → Ch 3) |
| Sentinel-URL element diff | §20.2's post-deploy differ, run on schedule too | Any unexplained tag change (catches CMS/plugin updates, not just deploys) |
| Security Issues / Manual Actions | GSC email + API | Any entry — page the on-call, not the marketing channel (→ Ch 17/18) |
| Certificate expiry | Synthetic | 30/7/1-day warnings (→ Ch 17 §17.2's canonical-flip consequence) |
| CoA/migration health | Sitemap-indexed crossover (→ Ch 16 §16.10) | Old-sitemap counts not falling on schedule |
| GSC verification changes | GSC email | ⚠️ Unknown-owner alerts are incident pages (→ Ch 17 §17.7) |
⚪ Alert discipline: every alert maps to a chapter's diagnosis section (the symptom tables) — an alert without a runbook row is noise and gets deleted.
20.4 The operations calendar (⚪)¶
- Every deploy: §20.2 gates + post-deploy diff.
- Daily: automated — status mix, robots.txt fetch, sentinel diff, security alerts.
- Weekly: crawl-share by URL class; discovery latency; new-404 review.
- Monthly: playbook Phase 7's GSC review (line 206); token/permission audit sweep (→ Ch 19 §19.2); certificate inventory.
- Quarterly: citation-rot sweep of your own docs/runbooks against the deprecation ledger (Appendix B/C) — Google's tooling churn (crawl-rate limiter, Mobile-Friendly Test, CrUX Dashboard, International Targeting…) has killed at least one audit-template dependency per year since 2022.
Symptoms & diagnosis¶
| Symptom | Likely cause | Where |
|---|---|---|
| Traffic collapse hours after a release | noindex/robots leak — check sentinel diff first | §20.2 |
| Crawl Stats fine, origin logs show no Googlebot | CDN absorbing crawls — you're reading the wrong log | §20.1 |
| "Googlebot" hammering search/filter URLs | Unverified impostor, or real facet overcrawl — verify first | §20.1 |
| Indexing statuses drifting with no deploys | CMS/plugin auto-update, edge-layer change — scheduled sentinel diff catches it | §20.3 |
| Migration monitoring shows stalled crossover | Redirect gaps — pull the log pipeline's old-URL status mix | §20.3, Ch 16 |
| Alert fatigue, real incidents missed | Alerts without runbook rows — prune to the symptom tables | §20.3 |
Sources¶
⚪ This chapter is industry engineering practice; its Google anchors are cited in their home chapters: Crawl Stats (Ch 2), crawler verification JSONs (Ch 5), robots.txt availability semantics (Ch 3), error-driven throttling (Ch 16 §16.9), staging noindex (Ch 16 §16.5). No claims here carry 🟢 beyond those anchors.