Ch 5 — AI Crawlers & Bot Management

Part II — The crawl layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md covers this in ~12 lines (Phase 1, lines 75–76: AI crawler policy is a deliberate decision; Google-Extended governs Gemini training/grounding only. Phase 6, lines 189–198: AI-feature eligibility = indexed + snippet-eligible; the GSC "Search generative AI" control; per-vendor crawler-doc checks). This chapter adds: the full three-category Google crawler taxonomy with per-category robots.txt semantics, the 2026 verification mechanics (relocated IP-range files, Web Bot Auth), a vendor-quoted inventory of every major third-party AI bot split by role, the Cloudflare–Perplexity dispute as a provenance case study, llms.txt doctrine, and a blocking-tradeoff decision framework priced in each vendor's own words.

The mental model for this domain: bot identity is now a three-axis problem — who sent it, what role it plays, and whether it obeys robots.txt — and each axis is decided per token, not per company. Google operates roughly two dozen tokens in three categories with three different robots.txt postures; every major AI vendor has copied that structure, splitting training crawlers (obey robots.txt, prospective-only) from search-index crawlers (obey, but blocking costs visibility) from user-triggered fetchers (documented as ignoring robots.txt by design). A bot policy written at the company level ("block OpenAI") or enforced at the wrong layer (robots.txt against a non-compliant crawler) fails silently. This chapter inventories the tokens, the verification machinery that separates real bots from impostors, and the decision framework; robots.txt parsing law lives in → Ch 3, and hard enforcement mechanics (WAF/CDN) live in → Ch 4.

5.1 Google's crawler taxonomy: three categories, three robots.txt postures

🟢 Google's crawler documentation — which moved in Nov–Dec 2025 from /search/docs/crawling-indexing/ to the standalone developers.google.com/crawling/ site because the crawl infrastructure serves products beyond Search (→ Ch 2) — divides all Google fetching into three categories with distinct robots.txt behavior (overview, last updated 2026-06-12, fetched 2026-08-04):

Category	Google's definition	robots.txt posture
Common crawlers	"The common crawlers used for Google's products (such as Googlebot)."	"They always respect robots.txt rules for automatic crawls."
Special-case crawlers	"used by specific products where there's an agreement between the crawled site and the Google product about the crawl process"	Partial: "For example, AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission."
User-triggered fetchers	"part of tools and product functions where the end user triggers a fetch"	"Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules."

🟢 Two infrastructure facts that shape every bot-management decision (same overview page): "Google egresses primarily from IP addresses in the United States. In case Google detects that a site is blocking requests from the United States, it may attempt to crawl from IP addresses located in other countries," and "The default protocol version used by Google's crawlers is HTTP/1.1." Geo-blocking and consent-wall behavior are therefore judged from a US egress point (→ Ch 2, → Ch 6).

Common crawlers (roster as of 2026-07-14)

🟢 Per the common crawlers page (last updated 2026-07-14, fetched 2026-08-04): Googlebot (governs Search including Discover and Search features), Googlebot-Image, Googlebot-Video, Googlebot-News (no separate UA string — uses the Googlebot strings), Storebot-Google (Shopping), Google-InspectionTool (Rich Results Test + URL Inspection), GoogleOther (generic crawler for various product teams), GoogleOther-Image, GoogleOther-Video, Google-CloudVertexBot (site-owner-requested Vertex AI crawls), and Google-Extended (a control token only — §5.2).

🟢 The Chrome version in Googlebot UA strings is a moving placeholder: "If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number" (same page). ⚠️ Log filters pinned to an exact Chrome build silently stop matching.

Special-case crawlers (roster as of 2026-02-11)

🟢 Per the special-case crawlers page (last updated 2026-02-11, fetched 2026-08-04): "The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process." Current roster: APIs-Google, AdsBot-Google-Mobile, AdsBot-Google, Mediapartners-Google (AdSense), Google-Safety.

• ⚠️ 🟢 For APIs-Google, AdsBot-Google-Mobile, AdsBot-Google, and Mediapartners-Google, "the global user agent (*) is ignored" — User-agent: * / Disallow: / does not stop ad-quality or AdSense crawling; these tokens must be named explicitly.
• 🟢 Google-Safety "ignores robots.txt rules" entirely; it handles "abuse-specific crawling, such as malware discovery for publicly posted links on Google properties."
• 🟢 Retired special-case crawlers, per the same page: AdsBot Mobile Web (iPhone), Duplex on the web, Google Favicon, Mobile Apps Android, Web Light. ⚠️ Tools and blog posts still citing these are dead weight.

User-triggered fetchers (roster as of 2026-07-16)

🟢 Per the user-triggered fetchers page (last updated 2026-07-16, fetched 2026-08-04), these "generally ignore robots.txt rules." Current roster: Google-CWS (Chrome Web Store), FeedFetcher-Google, Google-GeminiNotebook (Gemini Notebook), Google-Agent, GoogleMessages (link previews), Google-Pinpoint, GoogleProducer (Publisher Center), Google-Read-Aloud (deprecated legacy agent: google-speakr), Google-Site-Verification.

Three 2025–2026 additions matter for bot management:

1. 🟢 Google-Agent (added March 20, 2026 per the crawling docs changelog): "Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request. It uses IP ranges from user-triggered-agents.json." Agentic traffic is now a distinct, separately verifiable category (§5.4), and a subset of its requests is cryptographically signed (§5.5).
2. ⚠️ 🟢 The NotebookLM rename. Google-NotebookLM (added Oct 9, 2025) became Google-GeminiNotebook on July 16, 2026; the fetchers page lists the "Former agent (supported until August 2026): Google-NotebookLM". Robots rules and log filters keyed to the old token stop matching when support ends — the changelog explicitly warns to update hardcoded strings.
3. 🟢 The changelog is the authoritative feed for new tokens: Google-CWS (Nov 3, 2025), Google-Pinpoint (Nov 12, 2025), GoogleMessages (Jan 21, 2026), Google-Agent (Mar 20, 2026). Subscribe to developers.google.com/crawling/docs/changelog, not trade-press summaries.

⚠️ There is no AI Overviews crawler. No separate Google token exists for AI Overviews or AI Mode; those surfaces run on the normal Googlebot crawl and the Search index (§5.3). Any tool reporting "AI Overviews bot" traffic is mislabeling something.

5.2 Google-Extended: exact scope, and what it is not

🟢 Google-Extended is a robots.txt control token, not a crawler: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity" (common crawlers, fetched 2026-08-04). ⚠️ You will never see "Google-Extended" in server logs; log-based "Google-Extended traffic" reports are fabrications.

🟢 Its full scope, verbatim from the same page: "Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI." The grounding parenthetical dates from an April 25, 2025 documentation clarification made "based on publisher feedback" (🟢 Search Central docs changelog, entry dated April 25, 2025 — the entry lives in the Search updates changelog, not the crawling docs changelog, which starts later).

🟢 And its limit: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

⚠️ The headline correction: blocking Google-Extended does NOT remove a site from AI Overviews or AI Mode. Those are Search features whose eligibility is "indexed and eligible to be shown in Google Search with a snippet" (§5.3). Google-Extended governs Gemini-model training and Gemini/Vertex grounding — a different product family. The GSC control doc draws the same line from the other side: "This control doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended" (🟢 Search generative AI control, fetched 2026-08-04).

5.3 The AI-policy decision layer for Google's own surfaces

For Google's AI surfaces the policy question is never "which crawler do I block" — it is "which of three stacked controls do I set." (The snippet-control rule inventory — nosnippet, data-nosnippet, max-snippet, and their exact AI-surface semantics — is → Ch 7; this section covers only the decision layer.)

🟢 Eligibility baseline. "To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. There are no additional technical requirements," and "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary" (AI features and your website, last updated 2025-12-10, fetched 2026-08-04). 🟢 "Our generative AI features on Google Search are rooted in our core Search ranking and quality systems" (AI optimization guide, last updated 2026-07-10, fetched 2026-08-04). Normal indexability is the whole eligibility story — you exercise the tradeoff through controls, never by blocking Googlebot.

The three-layer control stack:

Layer	Instrument	Governs	Home
Content-level	nosnippet / data-nosnippet / max-snippet / noindex	What Search (including AI Overviews/AI Mode) may show and use as direct input	→ Ch 7
Property-level	GSC "Search generative AI" control	Whether the site's content appears in / grounds AI Overviews, AI Mode, generative AI in Discover	this section
Model-training	Google-Extended robots.txt token	Gemini training + Gemini/Vertex grounding — not Search	§5.2

🟢 The GSC control (Settings → Search generative AI; help doc, fetched 2026-08-04): Exclude means the site's content is prevented from being visible in Search generative AI features, including being linked and grounding responses — "You won't receive any traffic or impressions from these features." It is not a broader signal: "This control only affects whether your content can appear in certain Search generative AI features; this control isn't used as a ranking or inclusion signal affecting other parts of Search." Changes apply within "1-2 days after the control goes live, but some content may take longer to be excluded due to caching and propagation across Google systems." Child properties inherit from the closest configured parent. For total removal the doc points elsewhere: "To block your content from appearing in Google Search completely, use noindex."

🟢 Rollout status (checked 2026-08-04): the help doc still says "We're rolling out this control to a subset of website owners, allowing for thorough testing before rolling it out further," naming no countries. The launch announcement (blog.google, June 3, 2026, Mrinalini Loew) described "beginning to roll these features out to a subset of website owners in the UK, allowing for thorough testing before rolling them out to website owners globally," in the context of engaging "with regulators like the UK's Competition and Markets Authority," and stated "This control will not be used as a ranking signal for search results outside of these generative AI Search features" (announcement, fetched 2026-08-04). 🟡 Expansion beyond the UK — to "select sites in the US and other countries" — is reported only by trade press (first by Barry Schwartz, Search Engine Roundtable, July 9, 2026; corroborated by other secondary outlets since); Google has published no confirmation as of 2026-08-04. ⚠️ Do not promise a client the toggle exists on their property until you have opened Settings on that property.

🟢 The exclusion is visibility-priced and unilateral: "Content from other sites will still be available in those features, and it may appear similar to yours" (help doc). Opting out removes your traffic from those surfaces without removing the surfaces.

5.4 Verifying genuine Google traffic

⚠️ User-agent strings are trivially spoofed; fake-Googlebot traffic is endemic (even Common Crawl warns of CCBot impostors — §5.6). Never allowlist or analyze by UA string alone. Google documents two verification methods (Verify requests from Google, last updated 2026-03-20, fetched 2026-08-04) — note the URL: the old /search/docs/crawling-indexing/verifying-googlebot path 301s to this new verify-google-requests slug.

🟢 Method 1 — reverse DNS + forward confirmation: run a reverse DNS lookup on the accessing IP with host; "Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com"; run a forward lookup on that hostname and confirm it resolves back to the original IP. The rDNS masks identify the category:

Category	Reverse-DNS mask
Common crawlers	crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com
Special-case crawlers	rate-limited-proxy-***-***-***-***.google.com
User-triggered fetchers	***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com

🟢 Method 2 — published IP ranges. Five JSON files (CIDR format), all under https://developers.google.com/static/crawling/ipranges/:

• common-crawlers.json — common crawlers
• special-crawlers.json — special-case crawlers
• user-triggered-fetchers.json — user-triggered fetchers running on Google Cloud infrastructure
• user-triggered-fetchers-google.json — user-triggered fetchers controlled by Google
• user-triggered-agents.json — user-triggered agents (Google-Agent)

🟢 Trust nuance from the verify doc: "IPs in the user-triggered-fetchers-google.json object … resolve to a google.com hostname. IPs in the user-triggered-fetchers.json object resolve to gae.googleusercontent.com hostnames" — the latter range includes fetches originated by third-party software hosted on Google Cloud, so treat it with less trust than Google-controlled ranges.

⚪ Other Google-origin traffic that is not a crawler or fetcher (e.g., Apps Script, Google Cloud egress) matches the broader Google IP list at https://www.gstatic.com/ipranges/goog.json (returned 200 with creationTime 2026-08-03 when checked 2026-08-04). A request from a Google IP is therefore not automatically "Googlebot" — category membership comes from the five crawler files above, not from Google ownership of the address.

⚠️ The relocation trap. The files moved from /search/apis/ipranges/ to /crawling/ipranges/ (changelog dates the move Feb 11, 2026), and the Googlebot file was renamed: googlebot.json no longer exists. Live-verified 2026-08-04: the old https://developers.google.com/static/search/apis/ipranges/googlebot.json returns a 301 to common-crawlers.json. Google's announcement ("New Location for the Google Crawlers' IP Range Files," Search Central Blog, Mar 31, 2026, posted by Gary) is explicit that the grace period ends: "We will eventually phase out the old locations and redirect them to the new ones within 6 months" (🟢 blog post, fetched 2026-08-04). Any WAF automation, allowlist script, or monitoring job still fetching googlebot.json or the /search/apis/ paths is running on borrowed time.

5.5 Web Bot Auth: cryptographic bot identity (experimental)

🟢 Google published its Web Bot Auth documentation on May 4, 2026 (crawling docs changelog; doc page, last updated 2026-05-04, fetched 2026-08-04): "Web Bot Auth is an experimental cryptographic protocol used to authenticate requests sent by bots." Its promise, per the doc: "Move beyond easily spoofed headers to a verified identity and decouple agent identity from IP addresses."

🟢 Mechanics (per the same doc): participating requests carry a Signature-Agent HTTP header set to g="https://agent.bot.goog"; you fetch Google's public key set from https://agent.bot.goog/.well-known/http-message-signatures-directory (caching per its Cache-Control header, deleting keys that disappear from the file, "as they might be expired or revoked"), then verify the Signature header against Signature-Input following the HTTP Message Signatures standard (📘 RFC 9421; the directory format follows the IETF drafts draft-meunier-http-message-signatures-directory and draft-meunier-web-bot-auth-architecture). The directory endpoint returned HTTP 200 on 2026-08-04.

🟢 Scope today is narrow and explicitly experimental: "Not all Google user agents are using Web Bot Auth. Google is not yet signing every request of agents using the protocol." Concretely: "A subset of requests made by the Google-Agent are signed with Web Bot Auth; in these cases, they are authenticated as https://agent.bot.goog." Google's own recommendation: "in addition to Web Bot Auth you continue relying on IP addresses, reverse DNS, and user-agent strings as we gradually roll out signed traffic," and when verifying yourself, "Don't forget to fall back to IP-based verification, as not all requests are signed."

⚪ Web Bot Auth is an ecosystem play, not a Google-only one: the draft-meunier-* IETF drafts are lead-authored by Cloudflare's Thibault Meunier, who introduced the proposals on the Cloudflare blog ("Forget IPs: using cryptography to verify bot and agent traffic," May 15, 2025, fetched 2026-08-04). ⚠️ Draft-status drift: on the IETF datatracker (checked 2026-08-04) draft-meunier-web-bot-auth-architecture is expired/archived, superseded by draft-meunier-webbotauth-httpsig-protocol (rev. March 2, 2026) — Google's doc still cites the older draft names; major bot-detection services/CDNs/WAFs support it (per Google's doc); and Cloudflare's Aug 2025 Perplexity post (§5.7) cites OpenAI signing ChatGPT Agent requests with it. Treat it as the direction of travel for agent identity — supplementing, not replacing, IP verification for years yet.

5.6 The third-party AI-crawler inventory

Every vendor roster below was fetched live from official vendor documentation on 2026-08-04. Vendor docs are not Google docs: tag-wise these are ⚪-grounded-in-vendor-primary — cited here with exact quotes because the vendor's own wording is the only authoritative statement of each bot's role. The recurring pattern is a three-role split: a training crawler, a search-index crawler, and a user-triggered fetcher — with the user-triggered class documented by the vendors themselves as ignoring robots.txt.

OpenAI — four bots (developers.openai.com/api/docs/bots; platform.openai.com/docs/bots 301s here)

• GPTBot (training): "used to crawl content that may be used in training our generative AI foundation models." Opt-out semantics: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." IPs: openai.com/gptbot.json (200 on 2026-08-04).
• OAI-SearchBot (search): "used to surface websites in search results in ChatGPT's search features." The cost of blocking, in OpenAI's words: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." IPs: openai.com/searchbot.json.
• ChatGPT-User (user-triggered): used for certain user actions in ChatGPT and Custom GPTs; "Because these actions are initiated by a user, robots.txt rules may not apply." IPs: openai.com/chatgpt-user.json.
• OAI-AdsBot (ad safety): validates the safety of web pages submitted as ads on ChatGPT; "data collected by OAI-AdsBot is not used to train generative AI foundation models." IPs: openai.com/adsbot.json. ⚪ OpenAI publishes no introduction date for it (earliest archive trace ~July 2026).
• Version strings observed Aug 2026: GPTBot/1.4, OAI-SearchBot/1.4 — they change; match tokens, not versions.

Anthropic — three tokens (support.claude.com article 8896518, article dated April 7, 2026; support.anthropic.com 301s to support.claude.com)

• ClaudeBot (training): "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." Blocking signals "that the site's future materials should be excluded from our AI model training datasets."
• Claude-SearchBot (search): "navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses."
• Claude-User (user-triggered): "Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent." Blocking "may reduce your site's visibility for user-directed web search."
• Compliance statement: "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." IPs: claude.com/crawling/bots.json (fetched live 2026-08-04; creationTime 2026-05-01; ipv4Prefix entries only — no UA strings in the file).
• ⚪ Volume context for the compliance story: iFixit CEO Kyle Wiens reported ClaudeBot hitting iFixit roughly one million times in 24 hours in July 2024, and that "the crawling stopped after we added them to our robots.txt" (The Register, July 30, 2024, fetched 2026-08-04) — an attributed anecdote, but the standing illustration that a compliant training crawler can still be an infrastructure problem until you state a policy.
• ⚪ Full Mozilla/5.0 … ClaudeBot/1.0-style UA strings appear nowhere in Anthropic's current documentation — they are log-observed only. 🟡 The legacy tokens ANTHROPIC-AI and CLAUDE-WEB likewise appear in no current Anthropic doc; their retirement rests on an unnamed Anthropic spokesperson's statement to The Register (July 30, 2024, fetched 2026-08-04): "The 'ANTHROPIC-AI' and 'CLAUDE-WEB' user agents are no longer in use. We have configured ClaudeBot, our centralized user agent, to respect any existing robots.txt directives that were previously set for these deprecated user agents."

Perplexity — two bots (docs.perplexity.ai/guides/bots)

• PerplexityBot (search): "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Perplexity recommends "allowing PerplexityBot in your site's robots.txt file." UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). IPs: www.perplexity.com/perplexitybot.json (200 on 2026-08-04). ⚠️ Preserve the nuance: the page recommends allowing the bot and offers robots.txt controls, but nowhere explicitly promises that PerplexityBot obeys disallow rules.
• Perplexity-User (user-triggered): visits pages when users ask questions; not used for crawling or training. Perplexity states on the record: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." IPs: www.perplexity.com/perplexity-user.json.

Meta — five crawlers (developers.facebook.com/docs/sharing/webmasters/web-crawlers)

• Meta-ExternalAgent (training + indexing, bundled): "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." ⚠️ Meta is the exception to the three-role split: training and index-building share one token — you cannot allow one and block the other.
• Meta-WebIndexer (search): "navigates the web to improve Meta AI search result quality for users" (UA meta-webindexer/1.1). ⚪ No introduction date published.
• Meta-ExternalAds: "crawls the web for use cases such as improving advertising and other business-related products and services" (UA meta-externalads/1.1). ⚪ No introduction date published.
• Meta-ExternalFetcher (user-triggered): "fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities"; "may bypass robots.txt rules."
• FacebookExternalHit (link previews): crawls content shared on Meta apps; "might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content."
• robots.txt caching: "allow up to 24 hours for changes to robots.txt to take effect because crawlers may cache the contents of robots.txt for up to 24 hours." Contact: webmasters@meta.com.

Apple — one crawler + one control token (support.apple.com/en-us/119829)

• Applebot powers "the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari"; its data "may also be used to help train Apple foundation models powering generative AI features across Apple products."
• Applebot-Extended is a Google-Extended-style control token: "Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file."
• Distinctive extras, all verbatim from the page: "Web publishers can opt out of their content being used in these broad world knowledge answers by applying the nosnippet meta tag to specific content"; ⚠️ "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions" — a Googlebot-only disallow silently blocks Apple's crawler too; and "Applebot does not follow crawl-delay."
• Verification: reverse DNS in the *.applebot.apple.com domain, or match the published Applebot IP CIDR JSON.

Amazon — three tokens (developer.amazon.com/amazonbot)

• Amazonbot (training): "used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models."
• Amzn-SearchBot (search): "used to improve search experiences in Amazon products and services. By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa." It "does not crawl content for generative AI model training."
• Amzn-User (user-triggered): "supports user actions, such as responding to Alexa queries that require up-to-date information"; "it may not follow all robots.txt directives."
• Compliance: the bots respect "the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives," but "They do not support the crawl-delay directive," and Amazon may "use a cached copy from the last 30 days" of robots.txt — ⚠️ the slowest robots.txt refresh of any major vendor; expect up to a month of lag after a policy change.

Common Crawl — CCBot (commoncrawl.org/ccbot)

• UA: CCBot/2.0 (https://commoncrawl.org/faq/); respects robots.txt (User-agent: CCBot / Disallow: / is the documented block); "CCBot is now run on dedicated IP address ranges with reverse DNS," published at index.commoncrawl.org/ccbot.json.
• Impostor warning, verbatim: "We are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity."
• ⚪ Policy weight: the Common Crawl corpus is a major upstream training source for many LLMs, so a CCBot disallow is a broad, slow-acting training opt-out affecting many downstream models — not one vendor.

ByteDance — Bytespider (⚪ no official documentation exists)

⚪ As of August 2026, ByteDance publishes no official public documentation for Bytespider — nothing on bytedance.com, toutiao.com, or the Toutiao webmaster platform describes the crawler. That absence is itself the fact to record. Its UA string — Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) — is known only from server logs and third-party bot databases; its robots.txt compliance is disputed in webmaster and bot-management-vendor reports rather than addressed by ByteDance. ⚠️ With no documented compliance and no published IP ranges, a robots.txt disallow for Bytespider is a statement of intent only; enforcement belongs at the WAF/CDN layer (→ Ch 4).

Role summary (tokens as of 2026-08-04)

Vendor	Training	Search/retrieval index	User-triggered (ignores/may ignore robots.txt)	Other
Google	Google-Extended (control token)	Googlebot	Google-Agent, Google-GeminiNotebook, etc. (§5.1)	Google-Safety (ignores robots.txt)
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User	OAI-AdsBot
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User	—
Perplexity	— (stated: none)	PerplexityBot	Perplexity-User	—
Meta	Meta-ExternalAgent (bundled w/ indexing)	Meta-WebIndexer	Meta-ExternalFetcher	Meta-ExternalAds, FacebookExternalHit
Apple	Applebot-Extended (control token)	Applebot	—	nosnippet honored for "world knowledge answers"
Amazon	Amazonbot	Amzn-SearchBot	Amzn-User	—
Common Crawl	CCBot (corpus feeds many models)	—	—	—
ByteDance	Bytespider (⚪ undocumented)	—	—	—

5.7 The Cloudflare–Perplexity dispute: where "user-triggered" ends

The August 2025 collision between Cloudflare and Perplexity is the canonical case study in why the user-triggered-fetcher carve-out is contested — and in keeping label provenance straight.

⚪ Cloudflare's allegations (attributed; "Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives," Cloudflare blog, Aug 4, 2025, by Gabriel Corral, Vaibhav Singhal, Brian Mitchell, and Reid Tatoris; fetched live 2026-08-04 — the post shows a "modified" date of July 15, 2026): "Perplexity is repeatedly modifying their user agent and changing their source ASNs to hide their crawling activity, as well as ignoring — or sometimes failing to even fetch — robots.txt files," and "Perplexity uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked." Cloudflare's scale attribution: 20–25M daily requests from the declared crawler plus 3–6M daily from the alleged stealth crawler, "across tens of thousands of domains." Actions: Perplexity de-listed from Cloudflare's Verified Bots program; stealth-crawler signatures added to managed rules for all customers including the free tier. The post holds up OpenAI as the well-behaved contrast — respecting robots.txt, stopping when blocked, and signing ChatGPT Agent requests with Web Bot Auth (§5.5).

⚪ Perplexity's rebuttal (attributed; the post "Agents or Bots? Making sense of AI on the open web" still returned HTTP 403 to fetchers on 2026-08-04 — characterization rests on dated secondary reports, e.g. The Register, "Perplexity vexed by Cloudflare's claims its bots are bad," Aug 5, 2025): Perplexity denied wrongdoing, suggested Cloudflare "needed a clever publicity moment," attributed the disputed 3–6M daily requests to unrelated traffic from a third-party cloud-browser provider (BrowserBase, with Perplexity's own usage put under 45,000 daily requests), and argued that user-driven agent fetches are not crawling and therefore not subject to robots.txt.

⚠️ Neither side's claims are independently adjudicated. "Stealth crawling" is Cloudflare's label from its own traffic fingerprinting; "not crawling at all" is Perplexity's framing. Present both as attributed allegations, never as settled fact.

📘 The doctrinal gap underneath: RFC 9309 (the Robots Exclusion Protocol, → Ch 3) governs automatic clients — crawlers. Every major vendor in §5.6, Google included, has carved out a user-triggered class that per their own documentation partially or wholly ignores robots.txt. The dispute is over where that carve-out ends — one user question fanning out to thousands of fetches is operationally indistinguishable from crawling — and it is why cryptographic identity (§5.5) rather than UA-string etiquette is where bot management is heading.

5.8 llms.txt: proposal vs. reality

⚪ /llms.txt is "a proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time" — Jeremy Howard, September 3, 2024 (llmstxt.org, fetched 2026-08-04). Format: Markdown with a required H1, blockquote summary, and optional H2 link sections. The proposal's own scoping: "Our expectation is that llms.txt will mainly be useful for inference … as opposed to for training." It is explicitly "open for community input" — a proposal, not a standard.

🟢 Google's position is a flat no for Search: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them" (AI optimization guide, last updated 2026-07-10, fetched 2026-08-04). The docs changelog added this clarification on June 15, 2026, adding "it's fine if you want to maintain these files for other services or systems that use them" (🟢 Search updates changelog, fetched 2026-08-04).

🟡 John Mueller (Reddit, 2025, per Search Engine Journal / Search Engine Land writeups — primary post not fetched): "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)," comparing it to the keywords meta tag and noting a divergent llms.txt could function like a cloaking vector.

⚪ Corroborating absence: none of the official vendor crawler docs fetched for §5.6 (OpenAI, Anthropic, Perplexity, Meta, Apple, Amazon, Common Crawl) documents reading llms.txt, as of 2026-08-04.

5.9 Writing and enforcing the AI-bot policy

The decision matrix

The blocking tradeoff differs per role, and each vendor prices its own blocks:

1. Training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot) — blocking is a prospective-only corpus decision. ⚠️ Every vendor's wording is forward-looking ("future generations of Gemini models," Anthropic's "future materials"): nothing already crawled and trained on is removed. Low visibility cost today (except Meta, where the same token also does product indexing).
2. Search/retrieval tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot, Meta-WebIndexer) — blocking is an immediate visibility decision, in the vendors' own words: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers" (OpenAI); blocking Claude bots "may reduce your site's visibility" (Anthropic). For Google's AI surfaces the analogue is the control stack of §5.3, never a Googlebot block.
3. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent) — a robots.txt rule here is a preference signal, because the vendors document these as ignoring or possibly ignoring robots.txt. Hard enforcement requires IP-verified rules at the WAF/CDN layer (→ Ch 4).

Condensed as a policy matrix (⚪ framework; each cell's consequence is vendor-stated in §5.6):

Goal	robots.txt action	Effect timing	Residual exposure
Stay visible everywhere, feed no training	Disallow training tokens only (Policy A)	Prospective only — nothing already trained on is removed	Meta bundles training with indexing; Bytespider undocumented; user fetchers unaffected
Invisible in AI search engines too	Add the search tokens (Policy B)	Immediate for retrieval; vendors state the visibility cost	User fetchers still fetch on request; Google AI surfaces need §5.3 controls instead
Stop agentic/user fetches	robots.txt is only a preference	—	Requires WAF rules keyed to published IP files (→ Ch 4)
Out of Google's AI surfaces, in Search	GSC toggle / snippet controls — not robots.txt	"1-2 days" per Google (§5.3)	Competitors' content still fills the feature

⚪ Worked robots.txt patterns (grounded in the vendor docs above):

# Policy A: block AI training, keep AI-search visibility
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

# (search tokens — OAI-SearchBot, Claude-SearchBot, PerplexityBot,
#  Amzn-SearchBot, meta-webindexer — remain allowed by omission)

# Policy B: block everything AI — add the search tokens too,
# accepting invisibility in those engines.

The parsing traps (law lives in → Ch 3)

• ⚠️ 🟢 One group per crawler. "Google's crawlers determine the correct group of rules by finding in the robots.txt file the group with the most specific user agent that matches," and "Only one group is valid for a particular crawler" (robots.txt spec, last updated 2026-07-08, fetched 2026-08-04). Adding a User-agent: Googlebot group means the * group's rules stop applying to Googlebot entirely — shared disallows must be repeated in every group. Most AI vendors implement REP-style matching, so write the file as if this rule applies to all of them.
• ⚠️ 🟢 robots.txt errors invert intent: 4xx on /robots.txt (except 429) is treated "as if a valid robots.txt file didn't exist" (same robots.txt spec page) — i.e., crawl everything — while 5xx pauses crawling (→ Ch 3 for the full matrix). A WAF that challenges bot fetches of robots.txt un-blocks the site.
• ⚠️ crawl-delay is dead across major crawlers: Google ignores it as an unsupported rule (→ Ch 3), "Applebot does not follow crawl-delay" (Apple), and Amazon's bots "do not support the crawl-delay directive." Rate control belongs at the server/CDN (→ Ch 4), and for Googlebot via 429/503 semantics (→ Ch 2).
• ⚠️ Remember the Applebot fallback (§5.6): a Googlebot-only rule set governs Apple's crawler too unless Applebot is named.
• ⚠️ Propagation is not instant: Google caches robots.txt up to 24 hours, Meta up to 24 hours, Amazon up to 30 days. Sequence policy changes accordingly.

The enforcement and licensing layer (⚪ — not Google doctrine)

⚪ robots.txt compliance is voluntary (📘 RFC 9309 has no enforcement mechanism), so "protection" via robots.txt alone is soft — the Bytespider record and the Cloudflare–Perplexity dispute are the standing exhibits. Publishers wanting hard guarantees use IP-verified WAF rules (→ Ch 4) and, increasingly, commercial gatekeeping: Cloudflare launched pay per crawl in private beta on July 1, 2025 ("leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access"), and announced the same day it was "changing the default to block AI crawlers unless they pay creators for their content" for participating domains (Cloudflare blog, July 1, 2025, fetched 2026-08-04). Its Verified Bots program is the allowlist the Perplexity de-listing was executed against (§5.7). None of this is Google doctrine; it is the market building the enforcement layer robots.txt never had.

Symptoms & diagnosis

Symptom	Likely cause	Where
"Googlebot" traffic passes UA filters but behaves abusively	Spoofed UA; never allowlist by string — rDNS + forward-confirm or CIDR match	§5.4
Googlebot allowlist script broke / returns 404 or empty ranges	IP files moved to /crawling/ipranges/ (Feb 2026); googlebot.json renamed common-crawlers.json; old paths on a 6-month redirect clock	§5.4
Robots rules for NotebookLM silently stopped matching	Token renamed Google-GeminiNotebook (Jul 16, 2026); old token supported only until Aug 2026	§5.1
Site still appears in AI Overviews / AI Mode after blocking Google-Extended	Wrong control: Google-Extended governs Gemini training/grounding, not Search AI surfaces — use snippet controls (→ Ch 7) or the GSC toggle	§5.2–5.3
Tool reports "Google-Extended traffic" in server logs	Fabrication — the token has no UA string; crawling happens under normal Google UAs	§5.2
GSC property has no "Search generative AI" setting	Partial rollout ("subset of website owners" as of Aug 2026) — not an error	§5.3
User-agent: * Disallow: / but ad/AdSense crawlers still fetching	AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google, APIs-Google ignore * and must be named; Google-Safety ignores robots.txt entirely	§5.1
Blocked GPTBot but pages still appear/cited in ChatGPT search	Different token: OAI-SearchBot does ChatGPT search; ChatGPT-User does live fetches	§5.6
Apple/Siri visibility dropped after a Googlebot-only disallow	Applebot follows Googlebot rules when not named itself	§5.6
AI bot fetches pages despite disallow	User-triggered fetcher class (documented as ignoring robots.txt) or non-compliant crawler (e.g., Bytespider) — enforce at WAF with verified IPs	§5.6, §5.9
Bot ignores crawl-delay	Unsupported by Google, Applebot, Amazonbot — rate-limit at server/CDN	§5.9
robots.txt change seems to have no effect for weeks	Vendor caching: Google/Meta up to 24h, Amazon up to 30 days	§5.9
Signed-looking requests with Signature-Agent: g="https://agent.bot.goog"	Google-Agent traffic under Web Bot Auth — verify against the key directory; unsigned Google-Agent requests still verify via user-triggered-agents.json	§5.4–5.5

Sources

• Overview of Google's crawlers and fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers (fetched 2026-08-04; last updated 2026-06-12)
• Google's common crawlers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers (fetched 2026-08-04; last updated 2026-07-14)
• Google's special-case crawlers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers (fetched 2026-08-04; last updated 2026-02-11)
• Google's user-triggered fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers (fetched 2026-08-04; last updated 2026-07-16)
• Verify requests from Google crawlers and fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests (fetched 2026-08-04; last updated 2026-03-20)
• Authenticate requests with Web Bot Auth (experimental) — https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth (fetched 2026-08-04; last updated 2026-05-04)
• Google Crawling docs changelog — https://developers.google.com/crawling/docs/changelog (fetched 2026-08-04)
• How Google interprets the robots.txt specification — https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec (fetched 2026-08-04; last updated 2026-07-08)
• New Location for the Google Crawlers' IP Range Files — https://developers.google.com/search/blog/2026/03/crawler-ip-ranges (fetched 2026-08-04; posted 2026-03-31)
• AI features and your website — https://developers.google.com/search/docs/appearance/ai-features (fetched 2026-08-04; last updated 2025-12-10)
• AI optimization guide — https://developers.google.com/search/docs/fundamentals/ai-optimization-guide (fetched 2026-08-04; last updated 2026-07-10)
• Search generative AI control (Search Console Help) — https://support.google.com/webmasters/answer/16908024 (fetched 2026-08-04)
• New opportunities, control and insights for website owners — https://blog.google/products-and-platforms/products/search/new-controls-website-owners/ (fetched 2026-08-04; posted 2026-06-03)
• Google Search documentation updates (changelog) — https://developers.google.com/search/updates (fetched 2026-08-04)
• OpenAI bots — https://developers.openai.com/api/docs/bots (fetched 2026-08-04)
• Does Anthropic crawl data from the web…? — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (fetched 2026-08-04; dated 2026-04-07)
• Perplexity crawlers — https://docs.perplexity.ai/guides/bots (fetched 2026-08-04)
• Meta web crawlers — https://developers.facebook.com/docs/sharing/webmasters/web-crawlers (fetched 2026-08-04)
• About Applebot — https://support.apple.com/en-us/119829 (fetched 2026-08-04)
• Amazonbot — https://developer.amazon.com/amazonbot (fetched 2026-08-04)
• CCBot — https://commoncrawl.org/ccbot (fetched 2026-08-04)
• The /llms.txt file proposal — https://llmstxt.org/ (fetched 2026-08-04; dated 2024-09-03)
• Perplexity is using stealth, undeclared crawlers… — https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ (fetched 2026-08-04; posted 2025-08-04, modified 2026-07-15)
• Forget IPs: using cryptography to verify bot and agent traffic — https://blog.cloudflare.com/web-bot-auth/ (fetched 2026-08-04; posted 2025-05-15, modified 2026-07-15)
• Introducing pay per crawl — https://blog.cloudflare.com/introducing-pay-per-crawl/ (fetched 2026-08-04; posted 2025-07-01)
• Content Independence Day — https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ (fetched 2026-08-04; posted 2025-07-01)
• Anthropic on ANTHROPIC-AI/CLAUDE-WEB retirement (spokesperson via The Register) — https://www.theregister.com/2024/07/30/taming_ai_content_crawlers/ (fetched 2026-08-04; posted 2024-07-30)
• Perplexity vexed by Cloudflare's claims its bots are bad (secondary for Perplexity's rebuttal) — https://www.theregister.com/software/2025/08/05/perplexity-vexed-by-cloudflares-claims-its-bots-are-bad/448660 (posted 2025-08-05; cited ⚪ — Perplexity's own post 403s to fetchers)
• Google Search Generative AI Controls Rolling Out Beyond UK Sites — https://www.seroundtable.com/google-search-generative-ai-controls-41653.html (NOT fetchable — 403s to fetchers; cited 🟡 per V0 verification, dated 2026-07-09)
• Live endpoint checks 2026-08-04: googlebot.json 301 → common-crawlers.json; common-crawlers.json (creationTime 2026-08-03); user-triggered-agents.json 200; claude.com/crawling/bots.json (creationTime 2026-05-01); www.perplexity.com/perplexitybot.json 200; openai.com/gptbot.json 200; agent.bot.goog/.well-known/http-message-signatures-directory 200; www.gstatic.com/ipranges/goog.json 200 (creationTime 2026-08-03)