Ch 9 — Sitemaps & discovery¶

Part IV — The index layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md covers this in one line (Phase 1, line 82): XML sitemap lists the canonical URLs you want in results, submitted in GSC — plus glancing mentions of hreflang-via-sitemap (line 93), video sitemaps (line 137), and updating sitemaps during migrations (line 223). This chapter adds: the full limit stack and all four format families, the lastmod trust model, the death of every general push channel (ping retired, IndexNow unsupported, Indexing API narrow and abuse-enforced), news-surface eligibility mechanics, cross-domain submission, segmentation-as-dashboard, and the Sitemaps report read forensically.

A sitemap is a discovery input to the crawl pipeline — nothing else. It does not command crawling, it does not cause indexing, and no Google document ties it to ranking. Links remain Google's primary discovery mechanism; sitemaps backfill what internal linking misses and carry one metadata signal Google actually uses (lastmod, and only when it stays honest). The rest of the metadata the protocol defines is ignored, and every "push your URLs to Google" mechanism the industry sells is either dead (the ping endpoint), never supported (IndexNow), or so narrow that using it generally is an enforced abuse vector (the Indexing API). The working mental model: a sitemap is a machine-readable statement of which canonical URLs you want considered, and — via Search Console — the cheapest per-section indexation dashboard you can build.

9.1 The contract: a discovery hint, nothing more¶

🟢 Google's definition: "A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them" — sitemaps overview. The same page states the limit of the contract: "A sitemap helps search engines discover URLs on your site, but it doesn't guarantee that all the items in your sitemap will be crawled and indexed."

🟢 The build doc is blunter still: "Keep in mind that submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site" — build and submit a sitemap.

🟢 Not every site needs one. Google's own criteria (overview):

You might not need a sitemap if your site is "small" — "about 500 pages or fewer," counting only "pages that you think need to be in search results" — and "comprehensively linked internally," with few media files or news pages you want in results.
You likely do need one if the site is large, "new and has few external links to it," or heavy on rich media / news content.
Baseline: "If your site's pages are properly linked, Google can usually discover most of your site."

⚠️ "Missing XML sitemap" as a universal audit defect is lore. On a 200-page, well-linked brochure site it is a non-finding by Google's own criteria. It is rarely wrong to have one — a sitemap costs little and buys the Search Console diagnostics in §9.11 — but it is not a compliance item.

🟢 What belongs in it: the canonical URLs you want in results. "When creating a sitemap, you're telling search engines about which URLs you prefer to show in search results. These are the canonical URLs" — and for duplicate content, "choose the URL you prefer and include that in the sitemap instead of all URLs that lead to the same content" (build-sitemap). Sitemap inclusion is itself one of the canonicalization hints Google weighs (→ Ch 8). Feeding non-canonical variants, redirecting URLs, or 404s into a sitemap injects noise into exactly the system you are trying to steer.

🟢 URLs are taken literally: "Use fully-qualified, absolute URLs in your sitemaps. Google will attempt to crawl your URLs exactly as listed" — no relative URLs, no protocol drift, no host variants. Order is irrelevant: "You don't have to worry about the order of the URLs in your sitemap, it doesn't matter to Google" (build-sitemap).

⚠️ The inverse operation does not exist: removing a URL from a sitemap — or deleting the whole sitemap from Search Console — is not a removal mechanism. 🟢 "Deleting a sitemap removes the sitemap from this report, but Google won't forget the sitemap or any URLs listed in it" (Sitemaps report help). Deindexing runs through noindex, the Removals tool, or access control (→ Ch 7).

9.2 Protocol 📘 vs implementation: lastmod is the only metadata Google uses — conditionally¶

📘 The sitemaps.org protocol (v0.9, page last updated November 21, 2016 — frozen for a decade) defines three optional per-URL metadata tags: <lastmod>, <changefreq>, <priority>. Google implements exactly one of them, and only conditionally.

priority and changefreq: dead weight¶

🟢 "Google ignores <priority> and <changefreq> values" — build-sitemap (page updated 2026-07-08). The June 26, 2023 blog post is categorical: "Google still doesn't use the changefreq or priority elements at all" — changefreq is "conceptually overlapping with lastmod," and priority "is a heavily subjective field" that "generally doesn't accurately reflect the actual priority of a page relative to other pages on a site" (Sitemaps ping endpoint is going away, Gary Illyes).

📘 Even the protocol undercuts its own tags: changefreq "is considered a hint and not a command"; for priority, "the priority you assign to a page is not likely to influence the position of your URLs in a search engine's result pages" and "assigning a high priority to all of the URLs on your site is not likely to help you"; "The default priority of a page is 0.5" (sitemaps.org).

⚪ Generators still emit both tags and audit tools still grade them. Leaving them in is harmless — they are valid XML per the protocol, and other engines may read them — but tuning them for Google is zero-yield work.

lastmod: a crawl-scheduling signal with a trust condition¶

🟢 "Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate" (build-sitemap). Its function is crawl scheduling, not indexing: "nowadays lastmod is indeed useful in many cases and we're using it as a signal for scheduling crawls to URLs that we previously discovered" (2023 blog). It reprioritizes recrawl of known URLs; it is not a lever on whether anything gets indexed.

🟢 The lying penalty is documented, verbatim: "if your page changed 7 years ago, but you're telling us in the lastmod element that it changed yesterday, eventually we're not going to believe you anymore when it comes to the last modified date of your pages" (2023 blog). ⚠️ A CMS or sitemap plugin that stamps generation time into every lastmod on every rebuild is systematically burning the one sitemap signal Google uses — sitewide.

🟢 "Last modification" means significant modification: "The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not" (build-sitemap). The blog adds the operational escape hatch: partial coverage is fine — "You can use a lastmod element for all the pages in your sitemap, or just the ones you're confident about" — and for aggregate pages (homepage, categories) whose modification date is hard to compute, "it's fine to leave out lastmod for those pages."

📘 Format: W3C Datetime; per sitemaps.org, "This format allows you to omit the time portion, if desired, and use YYYY-MM-DD" — and per the GSC help page, "if you do specify a time, you must also specify a time zone." 🟢 Search Console flags unsupported formats on submission (2023 blog).

⚠️ The book's rule for this whole domain: the protocol defines syntax; Google's docs define semantics. sitemaps.org still documents priority, changefreq, and even the ping method (§9.5) exactly as it did in 2016. Reading the protocol as a description of Google's current behavior is itself the trap.

9.3 Formats: XML, feeds, plain text — and WebSub¶

🟢 Google accepts three format families, with no ranking of preference among them (build-sitemap):

XML sitemap — the full-featured default; the only format that carries extensions (image/video/news, §9.6) and hreflang alternates (§9.7). Con per Google's own comparison: "Can be cumbersome to work with" and complex to maintain "on larger sites, or sites where the URLs change often."
RSS 2.0 / Atom 1.0 / mRSS — "Google accepts RSS 2.0 and Atom 1.0 feeds." Most CMSes emit them automatically, "however keep in mind that this feed only provides information on recent URLs." mRSS specifically "can … provide Google details about video content on your site."
Plain text — "a common text file that contains one URL per line," web page URLs only, any filename "provided it has a .txt extension (for instance, sitemap.txt)."

The minimal valid XML sitemap, for reference — one <urlset>, one <url>, only <loc> required, <lastmod> the only optional tag worth emitting for Google:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/widgets/blue-widget</loc>
    <lastmod>2026-07-30</lastmod>
  </url>
</urlset>

🟢 Rules common to all formats: UTF-8 encoding ("The sitemap file must be UTF-8 encoded"), absolute canonical URLs, entity-escaping of XML data values ("As with all XML files, all tag values must be entity escaped"). 📘 sitemaps.org: "any data values (including URLs) must use entity escape codes" for the five XML-reserved characters (& ' " > <), and <loc> values must be under 2,048 characters and include the protocol. ⚠️ The single most common parse error in practice is an unescaped & in a parameterized URL — Search Console reports it as Invalid URL / Parsing error (§9.11).

⚠️ Feeds-as-sitemaps have a silent failure mode: because a feed "only provides information on recent URLs," a site that submits only its RSS feed has no sitemap coverage of its back catalog. The sound pattern is a pair: full XML sitemap for coverage + feed for freshness. ⚪ This pairing is standard practice; Google documents the components, not the pattern.

🟢 WebSub is the one surviving push channel for general content: "If you use Atom or RSS, you can use WebSub to broadcast your changes to search engines, including Google" (build-sitemap). Note the scope: WebSub rides on feeds — there is no WebSub for a plain XML sitemap. After the ping retirement (§9.5) this sentence is the entire documented surface of push-style notification to Google Search outside the Indexing API's niche (§9.12).

9.4 Limits, index files, and the directory-scope rule¶

The complete, live-verified limit stack:

Limit	Value	Source
URLs per sitemap file	50,000	🟢 build-sitemap
Bytes per sitemap file	50MB uncompressed (📘 52,428,800 bytes)	🟢 build-sitemap + 📘 sitemaps.org
`<loc>` tags per sitemap index file	50,000	🟢 large-sitemaps
Sitemap index files per site in Search Console	500	🟢 large-sitemaps
Index-file nesting	prohibited	🟢 Sitemaps report help
`<image:image>` tags per `<url>`	1,000	🟢 image-sitemaps
`<news:news>` tags per sitemap	1,000	🟢 news-sitemap
`<loc>` URL length	< 2,048 characters	📘 sitemaps.org

🟢 "All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs" (build-sitemap). ⚠️ Gzip does not raise the cap — compression saves transfer bandwidth only; the 50MB figure is uncompressed size, and Search Console has a dedicated Sitemap file size error for files "larger than 50MB when uncompressed" (§9.11).

⚠️ Child elements don't count. 🟢 "Child elements don't count towards the URL limit for sitemaps" (localized versions) — only <loc> entries consume the 50,000 budget; xhtml:link hreflang alternates, image:image, video:video, and news:news children do not. 🟢 Google added this clarification to the docs on November 8, 2022 (docs changelog: "Clarified in the sitemap documentation for localized versions that child elements don't count towards the total number of URLs in a sitemap file"). Tools that count every tag make teams shard international sitemaps far earlier than the limits require.

🟢 Index files. Past 50,000 URLs/50MB you split into multiple sitemaps and optionally submit one index file. "A sitemap index file may have up to 50,000 loc tags" and "You can submit up to 500 sitemap index files for each site in your Search Console account" (large-sitemaps). The structure:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap-products-1.xml</loc>
    <lastmod>2026-08-01</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap-articles-1.xml</loc>
  </sitemap>
</sitemapindex>

⚠️ No nesting: "A sitemap index file can't list other sitemap index files, only sitemap files" — Search Console reports this as the Nested sitemap indexes error (Sitemaps report help). Maximum addressable space per index is therefore 50,000 × 50,000 URLs — 2.5 billion; nobody exhausts the format, they exhaust crawl demand first (→ Ch 2).

🟢 Placement rules for index members: "Sitemaps that are referenced in the sitemap index file must be in the same directory as the sitemap index file, or lower in the site hierarchy," and "The referenced sitemaps must be hosted on the same site as your sitemap index file. This requirement is waived if you set up cross-site submission" (§9.9) — large-sitemaps. ⚠️ Index entries must also be complete URLs: a relative entry like sitemap.xml makes Google look for it in the index file's own directory, and a miss surfaces as Invalid URL in sitemap index file: incomplete URL (§9.11).

⚠️ The directory-scope rule — the quiet one that bites robots.txt-only submitters: 🟢 "You can host your sitemaps anywhere on your site, but unless you submit your sitemap through Search Console, a sitemap affects only descendants of the parent directory. Therefore, a sitemap posted at the site root can affect all files on the site, which is where we recommend posting your sitemaps" (build-sitemap). A sitemap at /blog/sitemap.xml discovered only via robots.txt cannot carry /shop/ URLs — Search Console reports out-of-scope entries as URL not allowed (§9.11). Host at the root, or submit through Search Console to lift the restriction. 📘 The underlying protocol rule is stricter still: "all URLs listed in the Sitemap must use the same protocol … and reside on the same host as the Sitemap" (sitemaps.org) — with host matching literal enough that http://example.com/, www.example.com/, and https://www.example.com/ are three different hosts from a http://www.example.com/sitemap.xml file's perspective (GSC help). Google's Search Console verification and robots.txt cross-submission mechanisms (§9.9) are the documented relaxations.

9.5 Submission today: three channels — and the corpse of the ping endpoint¶

🟢 The current, exhaustive list of ways to make a sitemap known to Google (build-sitemap, page updated 2026-07-08):

Search Console Sitemaps report — "This will allow you to see when Googlebot accessed the sitemap and also potential processing errors." The only channel with diagnostics (§9.11). Requires owner permission on the property.
Search Console API — programmatic submission of the same thing.
robots.txt — "Insert the following line anywhere in your robots.txt file … We will find it the next time we crawl your robots.txt file: Sitemap: https://example.com/my_sitemap.xml." Also: "You can specify multiple sitemap lines, and there's no limit to the number of sitemaps you can include in your robots.txt file." (robots.txt syntax and fetch semantics → Ch 3.)
WebSub for RSS/Atom feeds only (§9.3).

That list contains no ping endpoint and no IndexNow — a verified negative finding, not an omission (the string "IndexNow" appears nowhere in the page; checked against raw HTML 2026-08-04).

⚠️ The ping endpoint is dead. The history, all from Google's own record:

📘 The protocol still documents it: "To submit your Sitemap using an HTTP request … issue your request to the following URL: <searchengine_URL>/ping?sitemap=sitemap_url" (sitemaps.org) — a live example of protocol text diverging from every major implementation.
🟢 June 26, 2023 — Gary Illyes announces the deprecation: "these unauthenticated sitemap submissions are not very useful. In fact, in the case of Google Search, the vast majority of the submissions lead to spam. To wit, we're deprecating our support for sitemaps ping and the endpoint will stop functioning in 6 months," after which pings "will result in a 404 error" (Sitemaps ping endpoint is going away). The post cites corroborating findings from "other search engines such as Bing."
🟢 Leftover plugin code is inert, not harmful: "Any existing code or plugins which use this endpoint will not cause problems for Google Search; you don't need to make any changes (but using the endpoint will also not do anything useful)."
🟢 The post now carries the banner "The sitemaps ping endpoint deprecation is complete." ⚪ The six-month sunset put the cutover in January 2024; industry reporting (Barry Schwartz, Search Engine Roundtable, Jan 10, 2024: "Google has officially deprecated the Sitemaps ping endpoints") dates the observed 404s to early January, but Google never published a precise cutover date.
🟢 Live test, 2026-08-04: GET https://www.google.com/ping?sitemap=… returns HTTP 404 with the body "Sitemaps ping is deprecated. See https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping."

That 404 is the one-line rebuttal to any tool or service still selling ping-based "instant indexing" for Google. The deeper point is doctrinal: Google's stated reason for killing ping — unauthenticated push submission attracts overwhelming spam — is the same structural logic that explains why Google never adopted IndexNow (§9.13) and why the Indexing API is quota-gated and spam-screened (§9.12). Google accepts pull discovery from files you host under ownership you can prove, and almost nothing else.

9.6 Extensions: image, video, news — and combining them¶

Extensions attach media metadata to <url> entries in an XML sitemap. 🟢 "Sitemap extensions are a great way to tell Google about the different kinds of content and their metadata that you're using on your site" (combining sitemap extensions).

Image¶

Namespace http://www.google.com/schemas/sitemap-image/1.1. 🟢 "Image sitemaps are a way of telling Google about other images on your site, especially those that we might not otherwise find (such as images your site reaches with JavaScript code)"; "Each <url> tag can contain up to 1,000 <image:image> tags" (image-sitemaps).

⚠️ The extension is now two tags — <image:image> wrapping <image:loc> — because Google deleted the metadata tags: 🟢 "We removed the following tags and attributes from our documentation: <image:caption>, <image:geo_location>, <image:title>, <image:license>" (image-sitemaps; ⚪ the removal is commonly dated to May 2022 — corroborated by Wayback snapshots: the May 1, 2022 capture still documents the tags with no removal note, the June 5, 2022 capture carries the "We removed…" note (archive-diffed 2026-08-04) — but the live docs changelog carries no image-sitemap entry and the doc states the removal without a date). SEO tools still recommending sitemap captions "for image SEO" are recommending tags Google removed; image licensing now lives in structured data/IPTC (→ Ch 14).

🟢 CDN-hosted images are fine cross-domain: "In some cases, the image URL may not be on the same domain as your main site. This is fine, as long as you verify both domains in Search Console" (image-sitemaps).

Video¶

Namespace http://www.google.com/schemas/sitemap-video/1.1 (doc updated 2026-05-20). 🟢 Purpose is discovery of video "content that was recently added or that we might not otherwise discover with our usual crawling mechanisms"; "Google recommends using video sitemaps, however we also support mRSS feeds" (video-sitemaps).

🟢 Required per video: <video:video>, <video:thumbnail_loc>, <video:title>, <video:description> (max 2048 characters; must match the on-page description "but it doesn't need to be a word-for-word match"), and at least one of <video:content_loc> or <video:player_loc>. ⚠️ Deprecated: <video:category>, <video:gallery_loc>, <video:tvshow>, <video:price>, and the autoplay/allow_embed attributes. In mRSS, "You must specify at least one of <media:player> or the url attribute in <media:content>."

A video sitemap aids discovery only — the indexing gates (dedicated watch page, stable thumbnail) live in the video documentation and are Ch 14's territory (→ Ch 14).

News: a rolling 48-hour window, updated in place¶

Namespace http://www.google.com/schemas/sitemap-news/0.9. Operationally unlike every other sitemap type — it is a rolling window, not an archive. 🟢 From news-sitemap:

"Only include recent URLs for articles that were created in the last two days. Once the articles are older than two days, either remove those URLs from the news sitemap or remove the <news:news> metadata in your sitemap from the older URLs."
"Update your news sitemap with fresh articles as they're published. Don't create a new sitemap with each update. Google News crawls news sitemaps as often as it crawls the rest of your site."
Cap: "a sitemap may have up to 1,000 news:news tags"; beyond that, "split your sitemap into several smaller sitemaps."
⚠️ Empty is fine — by design: "You may see an Empty Sitemap warning in Search Console, but this is just to make sure it was intentional on your behalf. It won't cause any problems with Google Search if the file is empty." Publishers who "fix" this warning by stuffing old articles back in are committing the actual error.
Required tags: one <news:news> per <url>, <news:publication>, <news:name> — "It must exactly match the name as it appears on your articles on news.google.com, omitting anything in parentheses" — <news:language> (ISO 639 two- or three-letter code; exception: zh-cn/zh-tw), <news:publication_date>, <news:title>.
⚠️ <news:publication_date> is the original publication time: "Specify the original date and time when the article was first published on your site. Don't specify the time when you added the article to your sitemap." Generators that timestamp at generation silently violate this — the news-sitemap twin of the lastmod-inflation trap (§9.2).
Separate vs combined: "Either option is fine with Google, however creating a separate sitemap just for your news articles may enable better tracking of your content in … Search Console."

Combining extensions¶

🟢 One <url> entry can carry news + video + image + hreflang children at once; declare each namespace you use on <urlset>, and "The order of the extension in the sitemap is irrelevant after the <loc> tag" (combine-sitemap-extensions). Skeleton:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:news="http://www.google.com/schemas/sitemap-news/0.9"
        xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"
        xmlns:image="http://www.google.com/schemas/sitemap-image/1.1"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">
  <url>
    <loc>https://www.example.com/story/launch</loc>
    <news:news>…</news:news>
    <image:image>…</image:image>
    <video:video>…</video:video>
    <xhtml:link rel="alternate" hreflang="de" href="https://www.example.com/de/story/launch"/>
  </url>
</urlset>

The documented cost: 🟢 "Combining sitemap extensions increases the file size of your sitemap significantly" — size pressure the child-elements-don't-count rule (§9.4) partially offsets, since only <loc> entries consume the URL budget; the bytes still count toward 50MB.

9.7 hreflang in sitemaps¶

The sitemap is one of the three official hreflang delivery methods (HTML link tags, HTTP headers, sitemap — full spec, code rules, and return-link semantics → Ch 11). What belongs here is the sitemap-specific mechanics, from localized versions (page updated 2025-12-22):

🟢 Each <url> element "must have a child element <xhtml:link rel="alternate" hreflang="…"> that lists every alternate version of the page, including itself." Every alternate gets its own <url> entry carrying the identical full set of alternates — this is how the bidirectional-confirmation requirement (→ Ch 11) manifests in sitemap form.
🟢 "The order of these child <xhtml:link> elements doesn't matter, though you might want to keep them in the same order to make them easier for you to check for mistakes."
🟢 "Child elements don't count towards the URL limit for sitemaps" (§9.4) — the November 2022 clarification that made sitemap-delivered hreflang viable at scale. A 10-locale site of 40,000 pages per locale is 400,000 <loc> entries (8+ shards) but the ~4 million xhtml:link children cost nothing against the URL caps — only file bytes.

⚪ At-scale tradeoff (house practice, consistent with the docs): sitemap-delivered hreflang keeps annotations out of page templates and lets internationalization changes ship without code deploys, at the price of very large files and slower error surfacing (annotations are only reprocessed when the sitemap is re-read). For sites past a handful of locales it is usually the only maintainable method — the per-page <head> grows by one link element per locale per page in the HTML method, but by locale-count × page-count rows in the sitemap method, where bytes are cheap.

9.8 News surfaces: eligibility is automatic — the news sitemap is a tool, not a gate¶

⚠️ The most persistent belief in publisher SEO — that a news sitemap (or Publisher Center registration) is required for Google News, Top Stories, or the News tab — is wrong on every surface. Verified against Google's Publisher Center Help, fetched 2026-08-04:

🟢 No submission requirement, any surface: "Publishers don't need to submit their site to be eligible to show content on Google News surfaces. Google algorithmically discovers news content through Search technologies. However, publishers need to be compliant with our content policies to appear on Google News surfaces" — Help Google News discover your web crawled content.
🟢 Top Stories / News tab: "Publishers are automatically considered for 'Top stories' or the News tab of Search. They just need to produce high-quality content and comply with Google News content policies." Top Stories "is a section that appears within Google Search when we detect a search query is news-oriented," and "Content is automatically selected for this feature" — Appear on other News surfaces.
🟢 Publisher Center is no longer an intake channel at all. "Starting today, publishers can no longer set up Google News publications in Publisher Center" — announcement of April 25, 2024 (Publisher Center announcements). The follow-through, "Google News transitions to automatically-generated publication pages" (published February 10, 2025, updated March 20, 2025): "Google News is fully transitioning to automatically generated publication pages in late March 2025" and "Google News will no longer use RSS feeds or web locations that were submitted in Publisher Center." Eligibility is unchanged and automatic: "Content from publishers that adheres to our content policies is automatically eligible for consideration in Google News and across news surfaces." Publisher Center's residual role is commercial products only: "For Google News Showcase and Reader Revenue Manager, publishers will continue to submit logos through Publisher Center" — answer/15898024.

So what is the news sitemap for? Freshness. 🟢 "You can use a Google News sitemap to help improve how quickly our crawler finds your content and can resolve issues" (answer/9606634). For a publisher whose economics run on being crawled within minutes of publishing, that is reason enough to run one well (§9.6) — but present it to stakeholders as a latency optimization, never as an eligibility requirement.

The correct framing of the whole stack: eligibility = content policies + quality (automatic, no instrument to submit); discovery latency = news sitemap (optional, recommended); diagnostics = the Google News performance report in Search Console. Opting out is the only action with a mechanism: 🟢 "If you don't want to surface in Google News, you can block access to content on your site without affecting your continued indexing and ranking in Google Search," and "blocking Googlebot-News excludes your content from Google News" (News content across Google) (crawler taxonomy → Ch 5).

9.9 Cross-site and cross-domain submission¶

Multi-domain portfolios can centralize sitemap hosting. 🟢 Both patterns are documented in build-sitemap:

Option A — one sitemap, many domains, via Search Console. "A single sitemap that includes URLs for multiple websites, including sites from different domains" is allowed: "Make sure that you have verified ownership of all the sites that you will add in the sitemap," then submit via Search Console (optionally as an index file). Ownership verification is what substitutes for the protocol's same-host rule.

Option B — per-site sitemaps hosted centrally, via robots.txt. "Create one or more sitemaps for each individual site. For each individual sitemap file, make sure you include only URLs from that particular site." Upload them all to one host you control (e.g. https://sitemaps.example.com), then "For each individual site, make sure that the robots.txt file references the sitemap for that individual site":

# robots.txt file of https://example.com
Sitemap: https://sitemaps.example.com/sitemap-example-com.xml

Here the robots.txt reference is the ownership proof — 📘 the same mechanism the protocol itself defines for cross-submission.

🟢 The index-file same-site hosting requirement "is waived if you set up cross-site submission" (large-sitemaps). ⚪ Choosing between A and B: A concentrates everything in one property's Sitemaps report (convenient, but per-domain diagnostics get coarser); B keeps each site's report clean and survives Search Console access changes — house preference, not Google guidance. The image-CDN case is simpler still: cross-domain image URLs need both domains verified, nothing more (§9.6).

9.10 Segmentation as an indexation dashboard¶

The 50,000-URL cap is almost never the real reason to split sitemaps. The real reason is measurement. 🟢 "You can submit multiple sitemaps and sitemap index files to Google. This may be useful if you want to track the search performance of each individual sitemap in Search Console" (build-sitemap); 🟢 "You can filter the Page indexing report by sitemap to see how many page URLs in this sitemap have been indexed" (Sitemaps report help).

⚪ The practice that follows: shard by template or section (products / categories / articles), by year, or by priority tier, so each sitemap's indexed-to-submitted ratio isolates one hypothesis. When /products/ shows 92% indexed and /products-archive/ shows 11%, you have localized an indexation problem to a template before opening a crawl. Useful segmentation axes:

By template — separates "this page type doesn't get indexed" from sitewide problems.
By publication year or recency band — separates freshness decay from structural exclusion.
By migration wave — during a URL migration, old-URL and new-URL sitemaps tracked side by side show replacement progress (→ Ch 16).
By indexation priority — a "money pages" shard whose ratio you alert on (→ Ch 20).

The specific schemes are practitioner convention built on the documented per-sitemap filtering; Google endorses the mechanism, not any particular taxonomy. ⚠️ The related lore — "keep sitemaps under 10,000 URLs for better crawling" — has no Google basis: the only documented limits are 50,000/50MB, and shard size buys diagnostics, not crawl rate.

The crawl-budget interplay is modest and documented: 🟢 "Keep your sitemaps up to date. Google reads your sitemap regularly, so be sure to include all the content that you want Google to crawl. If your site includes updated content, we recommend including the <lastmod> tag" (crawl budget guide). And crawl budget concerns only large sites — 🟢 "Large sites (1 million+ unique pages)" with weekly-changing content, "Medium or larger sites (10,000+ unique pages)" with daily-changing content, or sites with a large share of Discovered – currently not indexed; "The numbers given here are a rough estimate" (full crawl-budget mechanics → Ch 2; the Discovered – currently not indexed diagnosis playbook → Ch 21).

9.11 The Sitemaps report, forensically¶

🟢 "Use the Sitemaps report to tell Google about any new sitemaps for your property, to see your sitemap submission history, and to see any errors that Google encountered when parsing your submitted sitemaps" (Sitemaps report help). Submission requires owner permission; without it, the robots.txt route still works. Read the report as two distinct layers:

Layer 1 — the report list (last request per sitemap): "Success: The sitemap was fetched and read without any errors." / "Couldn't fetch: Google couldn't fetch the sitemap." / "Sitemap had X errors: The sitemap could be fetched and partially read, but encountered some errors."

Layer 2 — the per-sitemap details page (status of the latest crawl): "Success: The sitemap was loaded and processed successfully with no errors. All URLs will be queued for crawling." / "Has errors: The sitemap could be fetched, but has one or more errors. Any URLs that could be parsed without errors from the sitemap will be queued for crawling." / "Couldn't fetch: The sitemap could not be fetched for some reason."

The two layers separate the failure planes — fetch (can Google retrieve the file?) from parse (can Google read what it retrieved?). A partially-broken sitemap still works: URLs that parse cleanly are queued despite errors elsewhere in the file, and 🟢 "Issues affecting individual URLs within a sitemap won't prevent Google from continuing to read the sitemap, as long as the sitemap can be fetched and read in general."

Fetch failures ("Couldn't fetch")¶

🟢 The help page's documented causes, two of which almost nobody checks:

Cause	Detail (verbatim where quoted)	Fix path
robots.txt block	"The sitemap is blocked by your robots.txt file. Google respects robots.txt when fetching sitemaps."	→ Ch 3
⚠️ Manual action	"Sitemaps are not read when a site has an unresolved manual action."	→ Ch 18
Wrong URL	"HTTP 404 error: sitemap not found at the address provided"	verify in browser
Transient server errors	"server unavailability … Some of these errors can be transient"	→ Ch 2, Ch 4
⚠️ Low crawl demand	"There is low crawl demand for the sitemap. The higher the quality of the site's content, the higher the crawl demand."	quality, not plumbing

The last two rows reframe a "sitemap problem" as something else entirely: an unresolved manual action silently disables sitemap processing sitewide, and a perpetually-unfetched sitemap on a technically healthy site is a crawl-demand (quality) verdict, not a configuration bug. 🟢 Google's prescribed debugging path: run the sitemap URL itself through the URL Inspection tool's live test and check "Crawl allowed? = 'Yes', Page fetch = 'Successful.'"

Parse failures¶

🟢 The documented error taxonomy (verbatim names; help page): URLs not accessible, URLs not followed (too many redirects, or relative URLs — "use absolute or complete links"), URL not allowed (out of directory/host scope, §9.4), Compression error, Empty sitemap, Sitemap file size error (>50MB uncompressed), Invalid attribute value, Invalid date (W3C Datetime; time requires a timezone), Invalid tag value, Invalid URL ("unsupported characters, spaces, or other characters such as quotes, or it might be incorrectly formatted (for example, htp:// instead of http://)"), Invalid URL in sitemap index file: incomplete URL, Invalid XML: too many tags (duplicate tags in one <url>), Nested sitemap indexes (§9.4), Parsing error ("Often, this problem is caused by an unescaped character in the URL"), missing required news tags (§9.6), and Sitemap contains urls which are blocked by robots.txt — this last one meaning the listed content is robots-blocked, distinct from the sitemap file itself being blocked.

Report semantics that mislead¶

All 🟢 from the help page:

Discovered pages = "The number of page URLs that were parsed from the sitemap. If this is a sitemap index, the number is the count of all URLs in all child sitemaps. Duplicate URLs are counted only once. There is no guarantee that a page URL discovered in a sitemap has been or will be crawled or indexed by Google." Discovered ≠ crawled ≠ indexed — the report counts parse output, nothing downstream (per-URL indexing status taxonomy → Ch 7).
Refresh cadence: on success, "Google will continue to recrawl the sitemap at a pace independent of the site crawl schedule."
⚠️ Fetch-failure decay: "If a sitemap fetch or parse fails, Google will continue trying to fetch and process the sitemap for a few days. If the attempts continue to fail, Google will stop trying to crawl that URL." A sitemap that 500s or sits behind a robots.txt block for weeks quietly falls out of the refresh cycle — fix, then resubmit; don't assume Google is still polling.
⚠️ Display cap: "The report can show a maximum of 1,000 submitted requests. You may submit more, but only 1,000 can be shown here (which are not shown in order of submission). There currently isn't any way to see the additional requests in Search Console." Very large sites that submit thousands of shards individually lose report visibility; submit the index file and let the report expand its children.
Media blind spot: "You can submit image, video, or news URLs in your sitemap. However, the report doesn't currently show any data for those types of URLs" — video URLs surface via Discovered videos and the Video indexing report (→ Ch 14).
Redirect literalism: the listed Sitemap URL is "The exact URL specified when the sitemap was submitted. Redirects are not followed." A sitemap URL that 301s reads as broken — submit the final URL.
⚠️ Deletion does nothing to URLs (§9.1) — and Google may keep visiting the sitemap file itself at its old URL. The documented ways to stop that: "Use a robots.txt rule to block Google from reading it," delete "the sitemap file … from your website" (so the URL no longer resolves), or add a noindex rule — "For a sitemap, you'll need to use a response header."

9.12 The Indexing API: narrow eligibility, real enforcement¶

⚠️ The Indexing API is not a general fast-indexing tool, and as of September 2024 misusing it is an enforced abuse vector, not a growth hack. All 🟢 from the Indexing API quickstart (page updated 2026-07-16):

Scope: "The Indexing API allows site owners to directly notify Google when their job posting or livestreaming video pages are added or removed." Hard eligibility: "The Indexing API can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject."
Enforcement: "All submissions through the Indexing API undergo rigorous spam detection. Any attempts to abuse the Indexing API, including the use of multiple accounts or other means to exceed usage quotas, may result in access being revoked."
Quota: "The Indexing API provides a default 200 quota for API onboarding and submission testing, and it requires additional approval for usage and resource provisioning." The default quota is a sandbox, not production capacity. Batching exists but doesn't change quotas: "combining up to 100 calls into a single HTTP request."
Relationship to sitemaps: for sites "with many short-lived pages like job postings or livestream videos … we recommend using the Indexing API instead of sitemaps because the Indexing API prompts Googlebot to crawl your pages sooner than updating the sitemap. However, we still recommend submitting a sitemap for coverage of your entire site." Even in its niche, the API is a latency layer on top of a sitemap, not a replacement.

🟢 The 2024 tightening is on the record in the docs changelog: September 4, 2024 — "Clarified that the default quota is for setting up the Indexing API, and how to request approval and quota" (the same entry corrects that DefaultRequestsPerMinutePerProject has "always been a 380 quota"); September 11, 2024 — "Clarified that submissions to the Indexing API are subject to spam detection … attempts to exceed quotas may result in revoked access."

⚪ Context: the "Indexing API for any page" hack was mass-marketed by indexing SaaS and rank-tracking tools through 2023–2024 — service accounts added as delegated owners to client properties, ordinary URLs pushed through URL_UPDATED notifications. The September 2024 language is Google's direct answer to it. Submitting ordinary pages through the API now risks the access of the property doing it — advise clients accordingly, and treat any vendor whose product depends on Indexing API misuse as a liability.

9.13 IndexNow: not a Google channel¶

⚠️ Google does not support IndexNow. The evidence is a clean negative plus a doctrine:

🟢 Google's current submission inventory — Search Console, Search Console API, robots.txt Sitemap: lines, WebSub for feeds (§9.5) — is exhaustive, and "IndexNow" appears nowhere in build-sitemap (verified against raw HTML, 2026-08-04) or any other Google Search documentation.
⚪ IndexNow's own site lists its participating endpoints: Amazon (indexnow.amazonbot.amazon), Bing, Naver, Seznam.cz, Yandex, and Yep, plus a shared global endpoint that relays "across all IndexNow-enabled search engines" — Google is not among them and is not mentioned anywhere on the page (indexnow.org/faq, fetched 2026-08-04).
🟡 The one Google statement on record is old and led nowhere: on November 9, 2021 a Google spokesperson told Search Engine Land "We're encouraged by work to make web crawling more efficient, and we will be testing the potential benefits of this protocol" (🟡 Google spokesperson via Search Engine Land, Nov 9, 2021 — primary article fetched 2026-08-04; a statement reported by SEL, not a Google publication). No adoption was ever announced, and none has appeared in nearly five years of Search Central changelogs.
The structural reading: 🟢 Google retired its own unauthenticated push endpoint because "the vast majority of the submissions lead to spam" (§9.5). IndexNow is a key-in-URL push protocol whose ownership proof (a hosted key file) is weaker than the verification Google demands for Search Console or robots.txt cross-submission. Google adopting the pattern it just killed would be incoherent — ⚪ this inference is ours, flagged as such, but it has predicted Google's behavior correctly since 2021.

Practical posture: implementing IndexNow is harmless and may genuinely help on Bing, Naver, Seznam, Yandex, and Yep. Any product, plugin, or agency deck that claims IndexNow (or any "instant indexing" service) accelerates Google indexing is selling either a 404 (ping-based, §9.5), a policy violation (Indexing API misuse, §9.12), or nothing.

Symptoms & diagnosis¶

Symptom	Likely cause	Where
Sitemap submitted; URLs still not indexed	Sitemap is a hint, not a directive — indexation gated on quality/demand/canonicalization	§9.1; → Ch 7, Ch 21
Google crawls pages long after they changed, despite lastmod	lastmod stamped on every regeneration → signal distrusted sitewide	§9.2
Audit tool demands priority/changefreq tuning	Tool grades tags Google ignores	§9.2
Back-catalog URLs missing from sitemap coverage	RSS/Atom feed submitted alone — feeds carry recent URLs only	§9.3
Parsing error / Invalid URL on an otherwise-valid sitemap	Unescaped `&` (or other reserved character) in parameterized URLs	§9.3, §9.11
Sitemap URLs outside its directory ignored (URL not allowed)	robots.txt-discovered sitemap scoped to parent directory; not submitted via GSC	§9.4
Nested sitemap indexes error in GSC	Index file lists another index file — prohibited	§9.4
International sitemaps sharded absurdly early	Tool counts child elements against the 50k URL cap; only `<loc>` counts	§9.4, §9.7; → Ch 11
Plugin still "pinging Google" after publish	Ping endpoint retired — returns 404 "Sitemaps ping is deprecated" (live-tested 2026-08-04)	§9.5
hreflang via sitemap ignored	Missing self-reference or incomplete alternate sets per `<url>`	§9.7; → Ch 11
"Empty Sitemap" warning on news sitemap	Intentional-by-design for a 48h rolling window with no fresh articles — not a defect	§9.6
Articles slow to appear on news surfaces	Discovery latency (crawlability, news sitemap freshness, publication_date stamping) — never a "submission" problem	§9.6, §9.8
Stakeholder claims site "isn't registered for Google News"	No registration exists: eligibility automatic; Publisher Center intake closed Apr 2024 / Mar 2025	§9.8
Couldn't fetch despite the sitemap loading fine in a browser	robots.txt block on the sitemap file, unresolved manual action, or low crawl demand	§9.11; → Ch 3, Ch 18
Sitemap "last read" date weeks old after an outage	Fetch-failure decay — Google stopped polling after repeated failures; fix and resubmit	§9.11
Submitted sitemaps missing from GSC report	1,000-row display cap, unordered — submit the index file instead of shards	§9.11
Sitemap submitted at a URL that redirects	Report follows no redirects; submit the final URL	§9.11
URL deleted from sitemap but still indexed	Sitemaps don't remove; use noindex/Removals/blocking	§9.1; → Ch 7
"Indexing API" service quota revoked / no effect on normal pages	JobPosting/BroadcastEvent-only eligibility + Sept 2024 spam enforcement	§9.12
Vendor promises IndexNow speeds up Google indexing	Google does not support IndexNow — Bing/Naver/Seznam/Yandex/Yep only	§9.13

Sources¶

What is a sitemap? — https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview (fetched 2026-08-04; page last-updated 2025-12-10)
Build and submit a sitemap — https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap (fetched 2026-08-04, incl. raw HTML; page last-updated 2026-07-08)
Manage your sitemaps with a sitemap index file — https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps (fetched 2026-08-04; page last-updated 2025-12-10)
Sitemaps ping endpoint is going away — Google Search Central Blog, Gary Illyes — https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping (fetched 2026-08-04, raw HTML; posted 2023-06-26; banner "The sitemaps ping endpoint deprecation is complete.")
Retired ping endpoint live test — https://www.google.com/ping?sitemap=https://www.example.com/sitemap.xml (fetched 2026-08-04; HTTP 404, "Sitemaps ping is deprecated.")
Image sitemaps — https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps (fetched 2026-08-04; page last-updated 2025-12-10)
Video sitemaps and alternatives — https://developers.google.com/search/docs/crawling-indexing/sitemaps/video-sitemaps (fetched 2026-08-04; page last-updated 2026-05-20)
News sitemaps — https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap (fetched 2026-08-04, incl. raw HTML; page last-updated 2025-12-10)
Combining sitemap extensions — https://developers.google.com/search/docs/crawling-indexing/sitemaps/combine-sitemap-extensions (fetched 2026-08-04; page last-updated 2025-12-10)
Localized versions of your pages — https://developers.google.com/search/docs/specialty/international/localized-versions (fetched 2026-08-04; page last-updated 2025-12-22)
Sitemaps report — Search Console Help — https://support.google.com/webmasters/answer/7451001 (fetched 2026-08-04, incl. raw HTML; no last-updated date shown)
Indexing API Quickstart — https://developers.google.com/search/apis/indexing-api/v3/quickstart (fetched 2026-08-04; page last-updated 2026-07-16)
Large site owner's guide to managing your crawl budget — https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget (fetched 2026-08-04; page last-updated 2026-07-22)
sitemaps.org — Protocol v0.9 — https://www.sitemaps.org/protocol.html (fetched 2026-08-04; page last-updated 2016-11-21)
Google Search documentation updates (changelog) — https://developers.google.com/search/updates (fetched 2026-08-04, raw HTML; entries cited: 2024-09-04, 2024-09-11, 2022-11-08)
Help Google News discover your web crawled content — Publisher Center Help — https://support.google.com/news/publisher-center/answer/9606634 (fetched 2026-08-04; no last-updated date shown)
Appear on other News surfaces — Publisher Center Help — https://support.google.com/news/publisher-center/answer/9607026 (fetched 2026-08-04; no last-updated date shown)
News content across Google — Publisher Center Help — https://support.google.com/news/publisher-center/answer/9607025 (fetched 2026-08-04; no last-updated date shown)
Google News transitions to automatically-generated publication pages — Publisher Center Help — https://support.google.com/news/publisher-center/answer/15898024 (fetched 2026-08-04; published 2025-02-10, updated 2025-03-20)
Publisher Center announcements index — https://support.google.com/news/publisher-center/announcements/10146168 (fetched 2026-08-04; "An update on publication pages" published 2024-04-25)
IndexNow FAQ — https://www.indexnow.org/faq (fetched 2026-08-04, incl. raw HTML; no last-updated date shown)
Google is testing the IndexNow protocol for sustainability — Search Engine Land, Nov 9 2021 (secondary source for the Google spokesperson statement) — https://searchengineland.com/google-is-testing-the-indexnow-protocol-for-sustainability-375932 (fetched 2026-08-04)
Google Sitemaps Ping Endpoints Officially No Longer Work — Search Engine Roundtable, Barry Schwartz, Jan 10 2024 (secondary source for the January 2024 cutover observation) — https://www.seroundtable.com/google-sitemaps-ping-endpoints-no-longer-work-36692.html (fetched 2026-08-04 via Wayback snapshot 2025-02-09)
Image sitemaps doc Wayback snapshots (dating the metadata-tag removal between 2022-05-01 and 2022-06-05) — https://web.archive.org/web/20220501005409/https://developers.google.com/search/docs/advanced/sitemaps/image-sitemaps and …/web/20220605000616/… (archive-diffed 2026-08-04)