Ch 7 — Indexing controls¶

Part IV — The index layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md covers this in ~5 lines (Phase 1 "Access & indexation", lines 77–81: verify indexation via the Page indexing report + URL Inspection — never site: counts; no stray noindex on money pages; no noindex + robots.txt-block combos) plus the Phase 6 baseline (lines 190–193) that name-drops the snippet controls and the Search Console "Search generative AI" control. This chapter adds: the complete robots-rule inventory with per-rule semantics and surface scope, the mechanics of noindex (why it requires crawlability, discovery timing, the JavaScript asymmetry, the long-term nofollow effect), snippet controls as the AI-input control layer, conflict resolution and per-crawler targeting, indexifembedded, the four-tier removal ladder, status codes as indexing signals, staging protection, URL Inspection as an instrument, and the complete Page indexing status taxonomy.

Indexing controls are the second rulebook of the crawl→index boundary: robots.txt (→ Ch 3) decides whether Googlebot may fetch a URL; the rules in this chapter decide whether a fetched URL may be indexed and served, and how much of it may be shown or fed into AI surfaces. The single mental model that prevents most of this domain's failures: every rule in this chapter is delivered inside the HTTP response — so Google can only obey it if Google is allowed to fetch the page. Crawl-blocking and index-blocking are not layered defenses; combined on the same URL they cancel each other. Everything else — the removal ladder, the status codes, the Search Console reports — is machinery for observing and accelerating what these rules do.

7.1 Two delivery mechanisms: robots `meta` tag and `X-Robots-Tag`¶

🟢 There are exactly two equivalent carriers for page-level indexing rules: "There are two ways to implement noindex: as a <meta> tag and as an HTTP response header. They have the same effect; choose the method that is more convenient for your site and appropriate for the content type" — Block Search indexing with noindex (fetched 2026-08-04; page last updated 2025-12-10). The same page closes a historical door: "Specifying the noindex rule in the robots.txt file is not supported by Google" (robots.txt Noindex: lines died in Sept 2019 → Ch 3).

🟢 The header is the only option for non-HTML content: "To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead" — Robots meta tag specifications (fetched 2026-08-04; last updated 2026-03-24).

Mechanics of the header form (all 🟢, same spec page): - "Multiple X-Robots-Tag headers can be combined within the HTTP response, or you can specify a comma-separated list of rules." - "The HTTP header, the user agent name, and the specified values are not case sensitive." (For the meta tag, both the name and content attributes are case-insensitive too.) - Server-config deployment is documented with regex patterns — Apache: <Files ~ "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </Files>; NGINX: location ~* \.pdf$ { add_header X-Robots-Tag "noindex, nofollow"; } — "The benefit of using an X-Robots-Tag with HTTP responses is that you can specify crawling rules that are applied globally across a site." - ⚠️ Per-file Apache targeting has a placement gotcha, in Google's own example comment: "the htaccess file must be placed in the directory of the matched file."

⚠️ Body placement is honored — since March 2026, officially. 🟢 "Google Search doesn't enforce placement of meta robots in the HTML head and will respect robots meta tags in the body section of an HTML document as well" (robots-meta-tag spec). The changelog entry (March 24, 2026) is explicit that this is documentation catching up, not a behavior change: "The behavior didn't change but was previously undocumented" — docs changelog. ⚪ Keep robots meta in <head> anyway: other engines and most SEO crawlers still assume head placement — but stop reporting body-placed robots meta as a Google-indexing bug.

7.2 The complete rule inventory¶

🟢 All definitions below are verbatim from the robots meta tag specifications (fetched 2026-08-04). Framing from the spec: the rules are "also available in machine-readable format" (useful for CI parsers → Ch 20); "Multiple rules may be combined in a comma-separated list or in separate meta tags. These rules are case-insensitive." Portability caveat: "It is possible that these rules may not be treated the same by all other search engines."

Rule	Google's semantics (verbatim core)	Notes
`all`	"There are no restrictions for indexing or serving. This rule is the default value and has no effect if explicitly listed."	⚠️ Google documents no `index` or `follow` values — indexing and following are defaults, not rules you can assert.
`noindex`	"Do not show this page, media, or resource in search results."	Full mechanics § 7.4.
`nofollow`	"Do not follow the links on this page. If you don't specify this rule, Google may use the links on the page to discover those linked pages."	The block-indexing doc calls it "a `nofollow` hint" when combined with noindex.
`none`	"Equivalent to `noindex, nofollow`."
`nosnippet`	"Do not show a text snippet or video preview in the search results for this page."	Full AI-surface scope § 7.3.
`indexifembedded`	"Google is allowed to index the content of a page if it's embedded in another page through iframes or similar HTML tags, in spite of a `noindex` rule."	§ 7.6. Inert without `noindex`.
`max-snippet:[number]`	"Use a maximum of [number] characters as a textual snippet for this search result."	`0` = "No snippet is to be shown. Equivalent to `nosnippet`."; `-1` = "Google will choose the snippet length that it believes is most effective to help users discover your content and direct users to your site." "This rule is ignored if no parseable [number] is specified."
`max-image-preview:[setting]`	"Set the maximum size of an image preview for this page in search results."	Values: `none` ("No image preview is to be shown"), `standard` ("A default image preview may be shown"), `large` ("A larger image preview, up to the width of the viewport, may be shown"). `large` is the Discover-preview lever (→ Ch 14).
`max-video-preview:[number]`	"Use a maximum of [number] seconds as a video snippet for videos on this page in search results."	`0` = "At most, a static image may be used, in accordance to the `max-image-preview` setting."; `-1` = "There is no limit." Ignored without a parseable number.
`notranslate`	"Don't offer translation of this page in search results."	"If the user clicks the translated title link, all further user interaction with the page is through Google Translate, which will automatically translate any links followed." (→ Ch 11 for translated results.)
`noimageindex`	"Do not index images on this page. If you don't specify this value, images on the page may be indexed and shown in search results."	Page-scoped, not file-scoped; blocking the image files themselves is a robots.txt job (§ 7.7, → Ch 14).
`unavailable_after:[date/time]`	"Do not show this page in search results after the specified date/time."	📘 "The date/time must be specified in a widely adopted format including, but not limited to RFC 822, RFC 850, and ISO 8601." ⚠️ "The rule is ignored if no valid date/time is specified" — an invalid date silently disables it. Side-effect: "Googlebot will decrease the crawl rate of the URL considerably after the specified date and time." It promises non-display, not index removal.

Historical and unused rules (🟢, kept on the spec page "because people have often asked about them or we used them in the past" — "The following rules aren't used by Google Search and are ignored"): - noarchive — "no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists." Retired to the historical section Oct 2, 2024; changelog adds "You don't need to remove the meta tag, as other search engines and services may be using it." - nocache — "The nocache rule isn't used by Google Search." - nositelinkssearchbox — "no longer used… as the feature no longer exists." Archived Nov 29, 2024 alongside the sitelinks-search-box docs removal.

⚠️ Audit-tool correction: flagging noarchive/nocache as "content protection" or recommending them against AI reuse is dead advice at Google — the actual controls Google names are nosnippet, data-nosnippet, max-snippet, noindex (§ 7.3), plus the GSC generative-AI control.

7.3 Snippet controls as the AI-surface control layer¶

(Canonical home. Ch 5 covers only the AI-policy decision layer; Ch 14 only image-specific preview interplay; Ch 13 owns how snippets are generated.)

Snippet rules govern display extraction, not ranking input: 🟢 "Robots meta tags govern the amount of content that Google extracts automatically from web pages for display as search results" (robots-meta-tag spec). Since AI Overviews and AI Mode build their answers from the same extraction layer, these display rules became the de-facto AI-input controls. Google added AI Mode to the spec on March 5, 2025 ("AI Mode is now available in Search Labs" — changelog).

Per-rule surface scope, exactly as documented (🟢, robots-meta-tag spec, fetched 2026-08-04): - nosnippet — "This applies to all forms of search results (at Google: web search, Google Images, Discover, AI Overviews, AI Mode) and will also prevent the content from being used as a direct input for AI Overviews and AI Mode." A caveat rides along: "A static image thumbnail (if available) may still be visible, when it results in a better user experience." - max-snippet — "This applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant, AI Overviews, AI Mode) and will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode." - max-image-preview — "applies to all forms of search results (such as Google web search, Google Images, Discover, Assistant)". ⚠️ Note what's absent: the image-preview rule carries no AI-direct-input clause — the AI-input language is attached only to the text-snippet rules. - max-video-preview — "(at Google: web search, Google Images, Google Videos, Discover, Assistant)". Also no AI clause.

🟢 The AI-features doc names the full control set: "To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls" — AI features and your website (fetched 2026-08-04; last updated 2025-12-10). Eligibility runs the other way through the same gate: "To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet."

⚠️ The permission bypass. The snippet limits are not absolute: 🟢 "this limit does not apply in cases where a publisher has separately granted permission for use of content. For instance, if the publisher supplies content in the form of in-page structured data or has a license agreement with Google, this setting does not interrupt those more specific permitted uses" (max-snippet definition; max-image-preview carries the same clause). And structured data is its own lane: "Robots meta tag limitations don't affect the use of that structured data, with the exception of article.description and the description values for structured data specified for other creative works." Even inside a suppressed block: "structured data remains usable for search results when declared within a data-nosnippet element." If you mark up content as structured data, snippet controls do not fence it (→ Ch 12).

data-nosnippet: element-level suppression¶

🟢 All from the spec page: "You can designate textual parts of an HTML page not to be used as a snippet… with the data-nosnippet HTML attribute on span, div, and section elements." Traps, each documented: - ⚠️ "The data-nosnippet is considered a boolean attribute. As with all boolean attributes, any value specified is ignored" — Google's own example shows <div data-nosnippet="false"> still suppressed ("all values are ignored"). - ⚠️ "To ensure machine-readability, the HTML section must be valid HTML and all appropriate tags must be closed accordingly" — the example comments that an unclosed div "will include all content afterwards." - ⚠️ "extraction of data-nosnippet may happen both before and after rendering. To avoid uncertainty from rendering, do not add or remove the data-nosnippet attribute of existing nodes through JavaScript. When adding DOM elements through JavaScript, include the data-nosnippet attribute as necessary when initially adding the element to the page's DOM." Custom elements must be wrapped in div/span/section to use it.

The Search Console "Search generative AI" control — status as of 2026-08-04¶

🟢 The dedicated opt-out sits at Settings → Search generative AI and governs "AI Overviews, AI Mode, Generative AI features in Google Discover", with Google noting "We expect to update this list over time as we develop Google Search" — Search generative AI control (fetched 2026-08-04). Three states: Include ("This is the default control for all properties"), Exclude, Inherit control from parent (default when a parent property exists; property mechanics → Ch 19). Boundaries, verbatim: - "this control isn't used as a ranking or inclusion signal affecting other parts of Search." - "This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended. To block your content from appearing in Google Search completely, use noindex." (Google-Extended scope → Ch 5.) - Propagation: "Content will be excluded within 1-2 days after the control goes live, but some content may take longer to be excluded due to caching and propagation across Google systems."

Availability: 🟢 the help doc still opens with "We're rolling out this control to a subset of website owners, allowing for thorough testing before rolling it out further" — unchanged on 2026-08-04, no regions named. 🟢 Google's only official scope statement is the June 3, 2026 launch post (Mrinalini Loew, GM Google Search Ecosystem, blog.google): "We are beginning to roll these features out to a subset of website owners in the UK, allowing for thorough testing before rolling them out to website owners globally," adding "This control will not be used as a ranking signal for search results outside of these generative AI Search features." 🟡 In early July 2026 the control began appearing for properties outside the UK, including US sites — Barry Schwartz, "Google Search Generative AI Controls Rolling Out Beyond UK Sites (Not Everyone Yet…)", Search Engine Roundtable, July 9, 2026; partial rollout, no Google announcement. As of Aug 4, 2026 there is no official confirmation of the beyond-UK expansion. The layered model to teach: snippet controls = per-page/per-element content limits; the GSC control = property-level AI-surface exclusion; noindex = out of Search entirely; Google-Extended = Gemini training, not Search (→ Ch 5).

7.4 noindex mechanics¶

🟢 The guarantee: "When Googlebot crawls that page and extracts the tag or header, Google will drop that page entirely from Google Search results, regardless of whether other sites link to it" — block-indexing. This is the exact property robots.txt lacks (§ 7.7): a seen noindex beats external links; a crawl block does not.

🟢 The crawlability precondition (this book's most-cited interlock): "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it." Stated generically for every rule in this chapter: "If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored" and "If indexing or serving rules must be followed, the URLs containing those rules cannot be disallowed from crawling" (robots-meta-tag spec).

The sequencing consequence, as a procedure (⚪ sequence is house practice; each step's mechanics are Google-documented): 1. Serve noindex (meta or header) on the section; leave robots.txt open. 2. Wait until the URLs accumulate under "URL marked 'noindex'" in the Page indexing report (§ 7.12) — Google is extracting the rule. 3. Only then, if crawl savings matter, add the robots.txt disallow. The pages are already out; the rule being invisible from now on no longer matters — but any new URL born under the disallow can still leak in as a link-only result (§ 7.7). 4. Skip step 3 entirely for sections that change membership often — URLs cycling in and out need the noindex to stay visible.

🟢 Timing is crawl-bound (block-indexing): "We have to crawl your page in order to see <meta> tags and HTTP headers. If a page is still appearing in results, it's probably because we haven't crawled the page since you added the noindex rule. Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page." Monitoring at scale: "You can also use the Page Indexing report in Search Console to monitor the pages on your site from which Googlebot extracted a noindex rule."

The JavaScript asymmetry (render mechanics → Ch 6): 🟢 "When Google encounters the noindex tag, it may skip rendering and JavaScript execution, which means using JavaScript to change or remove the robots meta tag from noindex may not work as expected" — JavaScript SEO basics (fetched 2026-08-04; last updated 2026-03-04). The Dec 15, 2025 changelog entry sharpened it: "While Google may be able to render a page that uses JavaScript, the behavior of this is not well defined and might change. If there's a possibility that you do want the page indexed, don't use a noindex tag in the original page code." The reverse direction is a documented, supported pattern: 🟢 "You can use JavaScript to add a robots meta tag to a page or change its content" — e.g., injecting noindex when an API call fails or a product is gone. ⚠️ So: JS may add noindex; JS must never be relied on to remove one. If the server HTML ships noindex, assume the page stays out.

Long-term noindex ends link-following. 🟢 (recorded statement, 2017 — not in current docs): in a 2017 English Google Webmaster Central office-hours hangout on the official Google Search Central YouTube channel (9GNg8R-X8LQ, ~55:00; channel and title verified via YouTube oEmbed 2026-08-04; transcription via Sitebulb), John Mueller said: "If we see the noindex there for longer then we think this page REALLY doesn't want to be used in search so we will remove it completely. And then we won't follow the links anyway. So noindex and follow is essentially the same as a noindex, nofollow. There's no really big difference there in the long run." ⚠️ Google's current noindex documentation contains no statement of this behavior — it survives solely as this dated recording. Consequence for architecture: "noindex,follow" as a permanent strategy (classic on paginated archives or filtered lists, expecting link equity and discovery to keep flowing) has no documented support and a recorded statement against it. If a page's links matter long-term, the page must be indexable — or the links must exist elsewhere (→ Ch 10).

7.5 Combining rules, conflicts, and per-crawler targeting¶

🟢 All from the robots-meta-tag spec: - Combination: "You can create a multi-rule instruction by combining robots meta tag rules with commas or by using multiple meta tags." Comma list and separate tags are equivalent; the same is true across placements (header + HTML). - Conflict resolution: "In the case of conflicting robots rules, the more restrictive rule applies. For example, if a page has both max-snippet:50 and nosnippet rules, the nosnippet rule will apply." ⚠️ This is why a stray template-level nosnippet silently overrides every per-page max-snippet, and why one leftover noindex in any of the three placements (meta head, meta body, header) wins over everything permissive. - Meta-tag targeting: "Google supports two user agent tokens in the robots meta tag; other values are ignored: googlebot: for all text results. googlebot-news: for news results." Example: <meta name="googlebot-news" content="nosnippet"> hides snippets only in News. ⚠️ <meta name="bingbot"> does nothing at Google — and generic name="robots" binds all search crawlers. - Header targeting: "The X-Robots-Tag may optionally specify a user agent before the rules" (X-Robots-Tag: googlebot: nofollow), and "Rules specified without a user agent are valid for all crawlers." - Non-search crawlers need naming: "The <meta name="robots" content="noindex"> rule applies to search engine crawlers. To block non-search crawlers, such as AdsBot-Google, you might need to add rules targeted to the specific crawler (for example, <meta name="AdsBot-Google" content="noindex">)." (Crawler taxonomy → Ch 5.)

7.6 indexifembedded: the embed-only indexing switch¶

The one rule that relaxes noindex instead of restricting further. 🟢 Spec: "Google is allowed to index the content of a page if it's embedded in another page through iframes or similar HTML tags, in spite of a noindex rule. indexifembedded only has an effect if it's accompanied by noindex."

🟢 The launch post (Jan 21, 2022, posted by Weizi Wang and Gary Illyes — New robots tag: indexifembedded, fetched 2026-08-04) states the use case and trigger: - "The indexifembedded tag addresses a common issue that especially affects media publishers: while they may want their content indexed when it's embedded on third-party pages, they don't necessarily want their media pages indexed on their own. Because they don't want the media pages indexed, they currently use a noindex tag in such pages. However, the noindex tag also prevents embedding the content in other pages during indexing." - "The new robots tag, indexifembedded, works in combination with the noindex tag only when the page with noindex is embedded into another page through an iframe or similar HTML tag, like object." - Syntax (both forms): <meta name="googlebot" content="noindex,indexifembedded"> or X-Robots-Tag: googlebot:noindex,indexifembedded (separate tags/headers equally valid). - ⚠️ "Presently, only Google supports the indexifembedded tag."

🟢 Production validation exists: the docs changelog (Jan 25, 2023) added "a new case study about how Vimeo improved Video SEO at scale for their customers by using the indexifembedded rule combined with noindex and adding structured data." Embedder-side iframe attribution → Ch 6; video watch-page architecture → Ch 14.

7.7 Crawl-blocking vs deindexing: the decision boundary¶

(robots.txt syntax and semantics → Ch 3. This section owns the interaction with indexing.)

🟢 robots.txt is a crawl valve, not a removal tool: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page" — Introduction to robots.txt (fetched 2026-08-04; last updated 2025-12-10). The mechanism of the leak: "A page that's disallowed in robots.txt can still be indexed if linked to from other sites… we might still find and index a disallowed URL if it is linked from other places on the web," and such a result "won't have a description." The GSC-visible symptom is the "Indexed, though blocked by robots.txt" warning (§ 7.12), whose fix is documented and counterintuitive: unblock, then serve noindex — never tighten robots.txt.

⚠️ The anti-pattern is named in the docs: "Combining multiple crawling and indexing rules might cause some rules to counteract other rules" (robots/intro). Disallow + noindex on one URL = the noindex is never seen (§ 7.4).

The media exception — where robots.txt IS the indexing control: 🟢 "Google only indexes images and videos that Googlebot is allowed to crawl. To prevent Googlebot from accessing your media files, use robots.txt rules to block the files" — Control what you share with Google (fetched 2026-08-04; last updated 2025-12-10). robots/intro concurs for media files: robots.txt can "prevent image, video, and audio files from appearing in Google Search results," though "This won't prevent other pages or users from linking to your image, video, or audio file." Media files can't carry a meta tag and Google doesn't index what it can't fetch for media — the leak path that plagues HTML pages doesn't produce indexed content here. (Image removal specifics → Ch 14.)

The decision boundary, compressed (decision-tree figure → Appendix D):

Goal	Correct control	Wrong control (and why)
Keep an HTML page out of results	`noindex` on a crawlable URL, or auth wall	robots.txt (leaks via external links, § 7.7)
Reduce server load / crawl waste	robots.txt disallow (→ Ch 3)	401/403 ("no effect on crawl rate", § 7.9); noindex (still crawled until seen, then less)
Both: out of results AND crawl savings	`noindex` first; disallow only after the noindex is processed	Both at once (the noindex is never seen)
Keep an image/video file out of results	robots.txt block on the file	Meta tag (media files can't carry one; § 7.7 media exception)
Truly confidential content	Access control, full stop	Everything else — 🟢 "Removing content from your site is the best way to ensure that it won't appear in Google Search and anywhere else on the Internet" (control-what-you-share)
Out of AI surfaces, in Search otherwise	Snippet controls and/or GSC generative-AI control (§ 7.3)	`noarchive`/`nocache` (ignored, § 7.2); Google-Extended (Gemini, not Search → Ch 5)

7.8 The removal ladder¶

Four tiers, from fastest-and-weakest to strongest. The tool names confuse practitioners because the fast tier is named "Removals" but doesn't remove anything.

Tier 0 — Removals tool (fast hide, ~6 months). 🟢 "For quick removals, use the Removals tool to remove a page hosted on your site from Google's search results within a day" — Remove a page hosted on your site from Google (fetched 2026-08-04). Its true nature, from the Removals tool help (fetched 2026-08-04): "A successful request lasts only about six months. After that, your information can appear on Google search results," and "Blocking a URL does not prevent Google from crawling your page, only from showing it in Search results." The help page is blunt that it's "only one step in this process to remove a URL permanently. Using the tool alone won't work." Operational details worth knowing: - Matching modes: exact URL vs "Remove all URLs with this prefix… This blocks all URLs beginning with the specified prefix, both www and non-www." ⚠️ Prefix mode can hide a directory — or a site — for six months from one request. On any site takeover, audit removal-request history (visible for the past 6 months). - ⚠️ Self-canceling on dead URLs: "If your URL is unreachable by Google (404, 502/3) when you use this tool, it will assume that the page is gone, and your block request will expire. Any page found at that URL at a later time will be considered a new page." - Second function — "Clear snippet in search": "Wipes out the page description snippet in Search results until the page is indexed again… the page description will say something like 'No page description available.'" URL matching is exact ("including the page extension"; anchors never matched). - Documented misuses, verbatim from the help page: not for canonicalization ("It won't keep your favorite version of a page; instead, could remove all versions (http/https and www/non-www) of a URL"), not for cleaning up crawl errors or 404 "cruft" ("those pages will naturally drop out"), not for taking a hacked site fully offline ("block any new URLs that the hacker created… But we don't recommend blocking your entire site"). - 🟢 Request lifecycle statuses (history covers the past 6 months): "Processing request" → then "Temporarily removed" ("You should make removal permanent or the page could appear again after about six months"), "Removal expired" ("the page is eligible to appear in Search results again unless you file another removal request"), "Cleared" ("The snippet clear request has completed"), "Request denied" ("typically because there is another identical request already in force"), or "Request canceled." The same tool also houses the SafeSearch Filtering tab — user-reported adult-content labeling of your URLs, with its own Processing/Canceled/Denied/"Filtered" statuses — worth checking on any takeover audit alongside removal history.

Tier 1 — permanent methods. 🟢 remove-information ranks them: (1) "Remove or update the content on your page. This is the most secure way to prevent your information from appearing in other search engines that might not respect the noindex tag" — with the server returning "either a 404 (Not Found) or 410 (Gone) HTTP status code" (Removals help; "Non-HTML files (like PDFs) should be completely removed from your server"); (2) "Password-protect your page. Limiting access to your page enables the right users to view your page, while preventing Googlebot and other web crawlers from accessing it" — the Removals help phrases it "Block access to the content, for example by requiring a password"; (3) noindex — "A noindex tag only blocks your page from showing up in Google search results. Users and other search engines that don't support noindex can still access your page"; the Removals help adds "This is less secure than the other methods." And for both docs: "Don't use robots.txt as a way to block your page." 🟢 Combining tiers correctly: file the temporary block first, make the change, and if the page "was recrawled after blocking… unblock and then reblock the page. This clears the page from the index."

Tier 2 — Refresh Outdated Content (third-party path). 🟢 For searchers, not owners: "If content was deleted from a site but still comes up in Google Search results, the page description or cache might be outdated" — Refresh outdated content (fetched 2026-08-04); "If you've already updated a live page, you don't have to fill out the form." Owner side: 🟢 "The Remove Outdated Content tool is used by non-site-owners to update search results when Google Search shows information that is no longer present on the site" — requests surface in the Removals tool's Outdated content tab (Removals help) with two request types: "Outdated snippet removal: The page still exists, but some content has been removed. Clears the page snippet in the result until the next crawl" and "Outdated page removal: The page no longer exists, and has been cleared from the Google index and search results." Denial reasons are diagnostic in themselves — "Denied: Content still on page," "Denied: Outdated content not in index," "Denied: Page not indexed," "Denied: Duplicate request," "Denied: Page not removed." ⚠️ It cannot remove live content; owners should use their own Removals tool.

Tier 3 — legal removals. 🟢 Per-product and Google-scoped: "If the content you are reporting appears in multiple Google products, please submit a separate notice for each relevant product," and "We can restrict access to content that appears in Google products and services, but that content may still exist elsewhere on the web" — Report Content for Legal Reasons (fetched 2026-08-04). (Personal-information removals follow their own policy flow; hacked-content cleanup → Ch 17.)

7.9 Status codes as indexing signals: 404 vs 410 (and 401/403/5xx)¶

(Full HTTP status handling matrix → Ch 2. This section owns the removal question.)

🟢 Official doctrine — all 4xx are one bucket: "All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn't exist. In the case of Google Search, the indexing pipeline removes the URL from the index if it was previously indexed. Newly encountered 404 pages aren't processed. The crawling frequency gradually decreases" — HTTP status codes doc (fetched 2026-08-04; last updated 2026-02-04; ⚠️ note the URL — the old /search/docs/crawling-indexing/http-network-errors address now 301s to this /crawling/docs/troubleshooting/ home). 404 and 410 sit in the same list, and the Removals help blesses "either a 404 (Not Found) or 410 (Gone)" for permanent removal.

🟢 (recorded statement, May 2018 — the only Google-sourced delta): in a May 2018 English Google Webmaster Central office-hours hangout on the official Google Search Central channel (kQIyk-2-wRg, t≈28:41; verified via YouTube oEmbed 2026-08-04; transcription via Search Engine Journal, May 23, 2018), John Mueller: "The subtle difference here is that a 410 will sometimes fall out a little bit faster than a 404. But usually, we're talking on the order of a couple days or so" — always paired with his own caveat from the same answer: "From our point of view, in the mid term/long term, a 404 is the same as a 410 for us." ⚠️ Lore correction: "you must return 410 to deindex" overstates a days-level, sometimes-only difference that current docs don't even acknowledge. Choose 410 when it's semantically true (deliberately gone); never re-engineer a platform for it.

The neighboring codes, each load-bearing for removals and staging: - 🟢 URLs are never forgotten: "Googlebot will probably continue to try this URL for some period of time; there is no way to tell Googlebot to permanently forget a URL, although it will crawl it less and less often" — Page indexing report help. Deleted URLs reappearing in crawl logs years later is normal, not a bug. - 🟢 403 is technically mislabeled but effective: "HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed" (same page). - 🟢 Never throttle with auth codes: "Don't use 401 and 403 status codes for limiting the crawl rate. The 4xx status codes, except 429, have no effect on crawl rate" (HTTP status codes doc; 429/emergency crawl control → Ch 2). - ⚠️ 5xx keeps dead pages longer than 4xx: "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped." A removed page that 500s lingers; one that 404s drops. - 🟢 429 is the one 4xx that behaves like a 5xx: "Google's crawlers treat the 429 status code as a signal that the server is overloaded, and it's considered a server error" (→ Ch 2 for emergency crawl control). - 🟢 Soft 404s: an error page served with 200 ("If the content suggests an error for Google Search, an empty page or an error message, Search Console will show a soft 404 error" — → Ch 2 for the full class; SPA soft-404 patterns → Ch 6).

7.10 Staging and dev environment protection¶

Ranked by evidence class:

Access control — the Google-documented method. 🟢 "If you have confidential or private content on your site, you need to password protect it to ensure only authorized users can access it" (control-what-you-share); the Removals help lists "Block access to the content, for example by requiring a password" as a permanent method (remove-information's wording: "Password-protect your page"). It works because "Googlebot never provides credentials" (§ 7.9) — an auth wall is invisible-by-construction, needs no crawl to be seen (unlike noindex), and binds every crawler, not just Google's. ⚪ Serve 401 (or 407) rather than 403 for correctness; Google states 403-walled pages won't be indexed either, just that the semantics are "incorrect."
noindex — documented, but leaky by design. It keeps pages out of results, yet requires every URL to stay crawlable and be fetched (existence leaks into logs, GSC, and any crawler that ignores the rule), and it creates the classic launch-day disaster: the staging template — with its noindex — ships to production. The playbook's "no stray noindex on money pages" (Phase 1, line 80) is the production-side tripwire; CI-level noindex-leak gates → Ch 20.
robots.txt-only — documented as insufficient. 🟢 Disallowed pages "can still be indexed if linked to from other sites" (§ 7.7), and ⚪ the robots.txt file itself publicly advertises every staging path it "protects." The mirror-image launch disaster: the staging Disallow: / file ships to production (robots.txt deploy gates → Ch 20; the 4xx/5xx robots.txt behavior matrix → Ch 3).

⚪ The layered house pattern (industry synthesis — Google documents no staging playbook): HTTP auth as the primary wall, IP allowlist/VPN where practical, noindex as a backstop for anything that escapes, a separate GSC property on the staging host to catch leaks, and log monitoring for verified-Googlebot hits (→ Ch 20).

⚠️ The two launch-day disasters are mirror images, and both are config-promotion failures, not SEO failures: - Staging config ships to production — the deploy carries staging's noindex template or Disallow: / robots.txt onto the live site. Symptom: indexed pages start flipping to "URL marked 'noindex'" / "URL blocked by robots.txt" (§ 7.12) after a release. - Production config ships to staging — the staging host loses its wall and gets indexed; users start landing on staging URLs from Search. Post-leak cleanup combines the tiers: Removals-tool block for the fast hide (prefix mode is legitimate here), then auth wall or 404/410 for permanence (§ 7.8), then verify via URL Inspection.

7.11 URL Inspection as the index-state oracle¶

🟢 Scope: "The URL Inspection tool provides information about Google's indexed version of a specific page, and also allows you to test whether a URL might be indexable" — URL Inspection help (fetched 2026-08-04). Two instruments in one, with different truths:

The indexed view is historical. 🟢 "This is not a live test. The results shown are from most recently indexed version of a page, not the live version on the web… Your page may have changed or become unavailable since Google last saw it," and under Last crawl: "All information shown in this tool is derived from this last crawled version." ⚠️ And a green verdict is not a SERP guarantee: "'URL is on Google' doesn't actually guarantee that your page will appear in Search results. The report doesn't check all conditions for appearing on Google" — the documented definitive test is searching for the URL.

Field semantics (🟢, indexed view — each row a distinct diagnosis): - Indexing status — indexed / not indexed plus one of the § 7.12 reasons; a third value exists only here: "Page is not indexed: URL is unknown to Google: This means that Google hasn't seen this URL before." - Sitemaps — only sitemaps submitted via the Sitemaps report or listed in robots.txt are recognized here ("Sitemaps discovered through other means won't be listed" → Ch 9). - Referring page — "A page that Google possibly used to discover this URL… If this value is absent it doesn't mean that no referring page exists, just that this information might not be available to the URL Inspection tool at this time." - Last crawl / Crawled as — timestamp plus the user agent type (desktop or mobile). - Crawl allowed? — the robots.txt gate. - Page fetch — "Whether or not Google could actually get the page from your server. Fetching can be successful even if the page is not indexed for another reason." Values: "Successful", "Failed" plus a status reason, or "N/A: A failure that doesn't fall into one of the known status reasons." - Indexing allowed? — "Whether or not your page explicitly disallowed indexing. If indexing is disallowed, the reason is shown here" (it also reports indexifembedded: with noindex, "the page will be indexed only when it is embedded"). ⚠️ The interlock made visible: "If your page is blocked by robots.txt (see Crawl allowed?), then Indexing allowed? will always be 'Yes' because Google can't see and respect any noindex directives." - User-declared canonical / Google-selected canonical — with "You can determine the canonical version only in the indexed data; the live test cannot predict whether or not the tested version will be considered canonical" (→ Ch 8).

Quota note (🟢): "There is a daily limit of inspection requests for each property that you own," and separately "There is a per-property daily limit of live inspections" (API quota figures → Ch 19).

View crawled page shows the RAW response, not a render. 🟢 "To see additional response data such as the raw HTML returned, the HTTP headers, JavaScript console output, and any page resources loaded, click View crawled page," available "only for URLs with a status of URL is on Google or URL is on Google, but has issues." ⚠️ The help page never calls this panel "rendered HTML" — the word "rendered" is reserved for the screenshot, and "A screenshot of the rendered page is available only in a live test." (Google's JavaScript docs are what direct you to URL Inspection to check "the rendered HTML" — that workflow and its evidence live in Ch 6.)

Live-test blind spots (🟢): "The live test does not test for all possible indexing issues, including whether this is a duplicate or alternate page. Duplicate pages aren't indexed." It also skips sitemaps/referring pages, "quality and security guidelines, manual actions, content removals, or temporarily blocked URLs," and it "first follows any redirects implemented by the page, then tests the page… the test does not indicate that it has followed a redirect, nor will it display the final URL that was tested." A valid live result therefore doesn't promise indexing — the help page's own list of remaining conditions ends with: "The page quality must be high enough to warrant indexing."

Request indexing (🟢): "Indexing can take up to a week or two"; "Indexing typically takes only a day or so, but can take much longer in some cases"; "Submitting a request does not guarantee that the page will appear in the Google Index"; "There is a daily limit to how many index requests you can submit" — at volume, "your best choice is to submit a sitemap" with accurate lastmod (→ Ch 9). URL Inspection API quota → Ch 19.

The documented noindex-debug workflow (🟢, Page indexing report help): inspect the URL → "Under Coverage > Indexing > Indexing allowed? the report should show that noindex is preventing indexing" (or grep the source/headers "for the word 'noindex'") → click Test live URL → "see if the noindex directive is still detected. If noindex is no longer present, you can click Request Indexing… If noindex is still present, you must remove it in order for the page to be indexed."

7.12 The Page indexing status taxonomy — complete¶

(Canonical home for the status inventory. The "currently not indexed" diagnosis playbook → Ch 21; canonical-cluster remediation → Ch 8.)

🟢 All statuses below were enumerated from the live Page indexing report help (fetched 2026-08-04). Framing first: the report "shows the Google indexing status of all URLs that Google knows about in your property"; every URL is either Indexed or Not indexed, and Google's own reading guidance is "Remember that Not indexed is not necessarily bad" and "You should not expect all URLs on your site to be indexed, only the canonical pages." Asked "Is it OK if a page isn't indexed?", the page answers: "Absolutely. Google doesn't index pages that are blocked by a robots.txt rule or noindex tag, or pages that are duplicates of other pages on your site, or pages that are inappropriate to index."

Reading calibration, all documented on the same page (🟢): - The totals are authoritative but won't match your CMS: "The indexed + not indexed totals above the chart are complete and accurate from Google's perspective, but small discrepancies can occur for various reasons." - Indexed ≠ visible: "Just because a page is indexed doesn't guarantee that it will show up in your search results… if Search Console says a URL is indexed, but it doesn't turn up in your search results, you can assume that it is indexed and eligible to appear in search results." - Each non-indexing reason carries a Source value: it "shows whether the source of the issue is Google or the website. In general, you can fix only issues where the source is listed as 'Website'." Triage rule from the doc: "prioritize fixing issues that are in validation state 'failed' or 'not started' and source 'Website'."

"Why pages aren't indexed" reasons — the complete live list (15):

Status	Meaning (Google's description, condensed)	First response
Server error (5xx)	"Your server returned a 500-level error when the page was requested."	Ch 2 §5xx; if persistent, expect eventual index drop (§ 7.9).
Redirect error	One of: "A redirect chain that was too long; A redirect loop; A redirect URL that eventually exceeded the max URL length; A bad or empty URL in the redirect chain."	Fix the chain (→ Ch 16).
URL blocked by robots.txt	Crawl-blocked. "Note that this does not guarantee that the page won't be indexed through some other means… there is a very small chance that the page might still be indexed."	Intentional? Fine. Must deindex? "remove the robots.txt block and use a 'noindex' directive."
URL marked 'noindex'	"When Google tried to index the page it encountered a 'noindex' directive… If you do not want this page indexed, congratulations!"	If wrong, run the § 7.11 debug workflow.
Soft 404	"returns a user-friendly 'not found' message but not a 404 HTTP response code."	Return a real 404/410, or fix thin/empty rendering (→ Ch 2, Ch 6).
Blocked due to unauthorized request (401)	"blocked to Googlebot by a request for authorization (401 response)."	Correct for staging (§ 7.10); on production, drop the wall or verify-and-allow Googlebot.
Not found (404)	Discovered without request; "404 responses are not necessarily a problem, if the page has been removed without any replacement." Includes the never-forget doctrine (§ 7.9).	Moved content → 301. Gone content → leave it.
Blocked due to access forbidden (403)	"Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed."	Often WAF/bot-management, not your app (→ Ch 4).
URL blocked due to other 4xx issue	"a 4xx error not covered by any other issue type."	Reproduce via URL Inspection live test.
Crawled – currently not indexed	"The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling."	At scale → the Ch 21 playbook.
Discovered – currently not indexed	"found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty."	Capacity/demand question (→ Ch 2 crawl budget; Ch 21).
Alternate page with proper canonical tag	"marked as an alternate of another page… correctly points to the canonical page, which is indexed, so there is nothing you need to do." ⚠️ "Alternate language pages are not detected by Search Console" (hreflang ≠ this status → Ch 11).	None. Working as intended.
Duplicate without user-selected canonical	Duplicate that "doesn't indicate a preferred canonical page. Google has chosen the other page as the canonical."	Declare a canonical or differentiate content (→ Ch 8).
Duplicate, Google chose different canonical than user	"marked as canonical for a set of pages, but Google thinks another URL makes a better canonical."	The Ch 8 signal-stack audit; the help page's own three-way comparison procedure.
Page with redirect	"a non-canonical URL that redirects to another page. As such, this URL will not be indexed." Note: "A canonical URL with a redirect can be indexed."	Expected post-migration (→ Ch 16).

Warnings — indexed but impaired (listed under "Improve page experience" on the summary page; "These issues don't prevent a page from being indexed, but they do reduce Google's ability to understand and index your pages"):

Status	Meaning	First response
Indexed, though blocked by robots.txt	"indexed despite being blocked… Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page… any snippet shown… will probably be very limited."	Documented fix: to block from Search, "robots.txt is not the correct mechanism… remove the robots.txt block and use 'noindex'"; to unblock, fix robots.txt (§ 7.7).
Page indexed without content	"appears in the Google index, but for some reason Google could not read the content. Possible reasons are that the page might be cloaked to Google or the page might be in a format that Google can't index. This is not a case of robots.txt blocking."	Inspect; suspect WAF/challenge pages (→ Ch 4) or rendering failure (→ Ch 6).

And Indexed — the good state, with a browsable sample via "View data about indexed pages" (up to 1,000 example URLs; data limits → Ch 19).

Operational notes (🟢, same page): validation semantics are generous — during a fix-validation pass, "If the page is not available to Google for any reason (page removed, marked noindex, requires authentication, and so on), the issue will be considered as fixed for that URL." You can't look up a URL in this report ("You can't search or filter by URL"); that's URL Inspection's job. ⚠️ Two long-lived legacy headings are absent from the live taxonomy: "Blocked by page removal tool" no longer appears as a status (removal requests are tracked only in the Removals tool, § 7.8), and the old "Excluded by 'noindex' tag" label is now "URL marked 'noindex'" — audit templates citing the old names are stale.

Symptoms & diagnosis¶

Symptom	Likely cause	Where
Page still in results weeks after adding noindex	Not recrawled yet ("may take months"), or the URL is robots.txt-blocked so the rule is never seen	§7.4, §7.7
PDFs/images indexed despite sitewide noindex meta template	Non-HTML can't carry a meta tag — needs `X-Robots-Tag` (or robots.txt for media files)	§7.1, §7.7
"Indexed, though blocked by robots.txt" in GSC	External links to a crawl-blocked URL; robots.txt used as a removal tool	§7.7, §7.12
Deindexed pages reappear ~6 months after "removal"	Removals-tool block expired with no permanent method behind it	§7.8
Whole directory vanished from Search, site otherwise healthy	Prefix-mode removal request (possibly by a previous owner/agency)	§7.8
Money page carries noindex after a release	Staging template or config shipped to production	§7.10, →Ch 20
Staging site indexed	robots.txt-only or noindex-only protection; no auth wall	§7.10
JS site: noindex "removed" in the browser but page stays out	Render-skip on noindex in server HTML; JS removal is unreliable by design	§7.4, →Ch 6
Snippet/AI answer shows text you tried to suppress	Content also exposed via structured data or a license permission lane; or `data-nosnippet` on an invalid/unclosed element	§7.3
`nosnippet` page still shows a thumbnail	Documented: "A static image thumbnail (if available) may still be visible"	§7.3
Dead pages linger in index despite being "removed"	Server answers 5xx (index preserved initially) or a soft 404, not 404/410	§7.9
"Page indexed without content"	Cloaking/WAF challenge served to Googlebot, or unindexable format	§7.12, →Ch 4
"URL is on Google" but the page never appears	Verdict ≠ serving guarantee; check quality/duplication layers	§7.11, →Ch 21
Crawled/Discovered – currently not indexed at scale	Not an error state by definition; capacity/quality investigation	§7.12, →Ch 21
`unavailable_after` deadline passed, page still shown	Invalid date format (rule silently ignored) — or expecting index removal from a display-only rule	§7.2
Content still in AI Overviews days after GSC exclusion	Documented propagation lag ("within 1-2 days… some content may take longer"); or content re-exposed via structured data/license lanes	§7.3

Sources¶

All fetched live 2026-08-04.

Robots meta tag, data-nosnippet, and X-Robots-Tag specifications — https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag (last updated 2026-03-24)
Block Search indexing with noindex — https://developers.google.com/search/docs/crawling-indexing/block-indexing (last updated 2025-12-10)
Remove a page hosted on your site from Google — https://developers.google.com/search/docs/crawling-indexing/remove-information (last updated 2025-12-10)
Removals and SafeSearch reports tool — https://support.google.com/webmasters/answer/9689846
Refresh outdated content — https://support.google.com/websearch/answer/6349986
Report Content for Legal Reasons — https://support.google.com/legal/answer/3110420
Page indexing report — https://support.google.com/webmasters/answer/7440203
URL Inspection tool — https://support.google.com/webmasters/answer/9012289
Search generative AI control — https://support.google.com/webmasters/answer/16908024
AI features and your website — https://developers.google.com/search/docs/appearance/ai-features (last updated 2025-12-10)
Introduction to robots.txt — https://developers.google.com/search/docs/crawling-indexing/robots/intro (last updated 2025-12-10)
Control what you share with Google — https://developers.google.com/search/docs/crawling-indexing/control-what-you-share (last updated 2025-12-10)
JavaScript SEO basics — https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics (last updated 2026-03-04)
HTTP status codes, and network and DNS errors — https://developers.google.com/crawling/docs/troubleshooting/http-status-codes (last updated 2026-02-04; old /search/docs/crawling-indexing/http-network-errors URL 301s here)
New robots tag: indexifembedded — https://developers.google.com/search/blog/2022/01/robots-meta-tag-indexifembedded (posted 2022-01-21)
Google Search Central documentation changelog — https://developers.google.com/search/updates (entries: 2024-10-02 noarchive; 2024-11-29 nositelinkssearchbox; 2025-03-05 AI Mode; 2025-12-15 noindex+JS; 2026-03-24 body-placement note; 2023-01-25 Vimeo case study)
New opportunities, control and insights for website owners — https://blog.google/products-and-platforms/products/search/new-controls-website-owners/ (posted 2026-06-03, Mrinalini Loew)
English Google Webmaster Central office-hours hangout (Mueller, 2017; long-term noindex) — https://www.youtube.com/watch?v=9GNg8R-X8LQ (channel/title verified via oEmbed; transcription via Sitebulb)
English Google Webmaster Central office-hours hangout (Mueller, May 2018; 410 vs 404) — https://www.youtube.com/watch?v=kQIyk-2-wRg (channel/title verified via oEmbed; transcription via Search Engine Journal, 2018-05-23)
🟡 Google Search Generative AI Controls Rolling Out Beyond UK Sites — Barry Schwartz, Search Engine Roundtable, 2026-07-09 (secondary; site 403s to fetchers — per V0 verification record)