Ch 3 — robots.txt: the full spec

Part II — The crawl layer · The Technical SEO Reference

Playbook coupling: seo-checklist.md covers robots.txt in 3 lines (Phase 1, lines 74, 81, 88): review the file so nothing important is blocked, never combine noindex with a robots.txt block, and disallow faceted-filter URL spaces. This chapter adds the entire rule engine underneath those checkboxes: Google's implementation and RFC 9309 read side by side — group selection, longest-match precedence, wildcard semantics, the 4xx/5xx/unreachable behavior matrix, caching, the 500 KiB limit — plus testing and the failure taxonomy.

robots.txt is a crawl-admission protocol, not an index-control mechanism — it decides which URLs a compliant crawler may fetch, and nothing else. Almost every robots.txt disaster is one of two category errors: using the file to keep pages out of Google (it can't — blocked URLs are indexable from links alone), or assuming the file fails safe (it doesn't — a 404 on robots.txt opens the whole site to crawling, while a 500 shuts crawling down entirely). Since September 2022 the protocol is a real standard, 📘 RFC 9309 (IETF Proposed Standard, published September 2022; authors M. Koster, G. Illyes, H. Zeller, L. Sassman — three of the four at Google), and Google publishes its exact implementation, which is RFC-conformant but more specific in several places. This chapter cites both and marks which is which. Everything here is the canonical home for robots.txt syntax and semantics; Ch 2 (crawl mechanics, HTTP status handling), Ch 5 (which crawlers obey which rules), and Ch 7 (index controls) reference this chapter rather than restating it.

3.1 The doctrine: crawl control is not index control

🟢 Google's own definition sets the boundary: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests"; and in the same breath, "it is not a mechanism for keeping a web page out of Google" — robots.txt intro.

Why the file cannot remove pages: Google can index a URL it has never fetched. 🟢 "A page that's disallowed in robots.txt can still be indexed if linked to from other sites" (intro) — Google builds an entry from the linking pages' anchor text and context. The visible symptom: 🟢 "If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description" (intro). The spec page states the same mechanic from the disallow rule's side: 🟢 "Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet" — robots.txt spec.

This produces the classic Search Console warning, "Indexed, though blocked by robots.txt": 🟢 "The page was indexed despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page… we can still index it, using the information from the page that links to your blocked page" — Page indexing report. Google's prescribed fix is explicit: 🟢 "If you do want to block this page from Google Search, robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'." ⚠️ Note the order of operations: the block must be removed for the noindex to work, because a noindex on a disallowed URL is never fetched and therefore never seen. noindex mechanics, the removal ladder, and the full Page indexing status taxonomy (including the sibling not-indexed reason "URL blocked by robots.txt") live in → Ch 7.

The decision rule, stated once:

Goal	Correct tool	Wrong tool and why
Reduce crawl traffic / keep crawlers out of URL spaces	robots.txt disallow	noindex — 🟢 Google "will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time" (crawl budget; → Ch 2)
Keep an HTML/PDF page out of results	noindex (crawlable!) or password protection	robots.txt — URL indexable from links (above)
Keep content private	authentication	robots.txt — 🟢 "The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them" (intro); ⚪ the file is also public and enumerates your "hidden" paths for anyone who looks

Two legitimate exceptions to "robots.txt never removes things":

1. Media files. 🟢 "Use a robots.txt file to manage crawl traffic, and also to prevent image, video, and audio files from appearing in Google Search results" (intro) — media is indexed from its fetched bytes, not from third-party anchor text, so a disallow does keep it out of Google Images/Video (though 🟢 "this won't prevent other pages or users from linking to your image, video, or audio file"). Image/video specifics → Ch 14.
2. Embedded assets of blocked pages. 🟢 "Image files, video files, PDFs, and other non-HTML files embedded in the blocked page will be excluded from crawling, too, unless they're referenced by other pages that are allowed for crawling" (intro).

⚠️ The rendering trap. Disallowing CSS/JS breaks Google's ability to render pages that depend on them. 🟢 "You can use a robots.txt file to block resource files such as unimportant image, script, or style files, if you think that pages loaded without these resources won't be significantly affected by the loss. However, if the absence of these resources make the page harder for Google's crawler to understand the page, don't block them, or else Google won't do a good job of analyzing pages that depend on those resources" (intro). Rendering pipeline and WRS resource fetching → Ch 6; the crawl-budget consequences of resource crawling → Ch 2.

Finally, robots.txt is per-crawler law only for crawlers that obey it. 🟢 Google's common crawlers "always respect robots.txt rules for automatic crawls" and "AdsBot ignores the global robots.txt user agent (*)" (crawler overview); its user-triggered fetchers "generally ignore robots.txt rules" (user-triggered fetchers); and "The Google-Safety user agent ignores robots.txt rules" (special-case crawlers) — the full obedience matrix is → Ch 5.

3.2 Where the file lives and what it governs

🟢 "You must place the robots.txt file in the top-level directory of a site, on a supported protocol" — and scope is exact: "The rules listed in the robots.txt file apply only to the host, protocol, and port number where the robots.txt file is hosted" (spec). 📘 RFC 9309 §2.3 concurs: "The rules MUST be accessible in a file named '/robots.txt' (all lowercase) in the top-level path of the service."

Consequences, all from the spec page's validity table (🟢, spec):

File at	Governs	Does not govern	Rule
https://www.example.com/robots.txt	https://www.example.com/…	https://example.com/…, other subdomains	Subdomains are separate. Every host — including CDN subdomains like cdn.example.com — needs its own file
https://example.com/robots.txt	https://example.com/…	http://example.com/…	Protocols are separate (relevant mid-HTTPS-migration → Ch 16)
ftp://example.com/robots.txt	ftp://example.com/…	https://example.com/…	FTP governs only FTP
https://example.com:443/robots.txt	https://example.com/…	https://example.com:444/…	🟢 "Standard port numbers (80 for HTTP, 443 for HTTPS, 21 for FTP) are equivalent to their default hostnames"
https://example.com:8181/robots.txt	https://example.com:8181/…	https://example.com/…	Non-standard ports are separate origins
https://212.96.82.21/robots.txt	that IP as hostname	domains hosted on that IP	🟢 "It isn't automatically valid for all websites hosted on that IP address"
https://example.com/pages/robots.txt	nothing	everything	🟢 "Crawlers don't check for robots.txt files in subdirectories"

🟢 "Your site can have only one robots.txt file" per host, and 🟢 "The file must be named robots.txt" (create-robots-txt). ⚠️ The subdomain-scoping row is the operationally dangerous one: sites that block their staging or CDN hostnames "in robots.txt" by editing the main domain's file have blocked nothing — and conversely, a staging robots.txt with Disallow: / that gets deployed to production is a site-wide crawl stop (CI diff gates → Ch 20; staging protection done right → Ch 7).

🟢 Default state: no file, no restrictions — "Unless you specify otherwise in your robots.txt file, all files are implicitly allowed for crawling" (create-robots-txt). 📘 RFC 9309 §2.2.2 encodes the same default at rule level: "If no match is found amongst the rules in a group for a matching user-agent or there are no rules in the group, the URI is allowed."

3.3 File format and the 500 KiB limit

🟢 "The robots.txt file must be a UTF-8 encoded plain text file and the lines must be separated by CR, CR/LF, or LF" (spec). The parser is deliberately forgiving:

• 🟢 "Google ignores invalid lines in robots.txt files, including the Unicode Byte Order Mark (BOM) at the beginning of the robots.txt file, and use only valid lines. For example, if the content downloaded is HTML instead of robots.txt rules, Google will try to parse the content and extract rules, and ignore everything else." ⚠️ Salvage parsing cuts both ways: a CMS error page or an HTML file accidentally served at /robots.txt with HTTP 200 is not "no rules" — Google will mine it for anything that parses as a rule.
• 🟢 "Similarly, if the character encoding of the robots.txt file isn't UTF-8, Google may ignore characters that are not part of the UTF-8 range, potentially rendering robots.txt rules invalid."
• Syntax errors don't invalidate the fetch: 🟢 "If the file has syntax errors in it, the request is still considered successful, though Google might ignore any rules with a syntax error" — Crawl Stats report.

Size limit. 🟢 "Google enforces a robots.txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored" (spec). 📘 RFC 9309 §2.5 sets the floor Google implements as a ceiling: "The parsing limit MUST be at least 500 kibibytes." ⚠️ Truncation is silent and positional: rules after the cutoff simply don't exist, so a generated file that appends its most important disallows last (common with plugin-assembled robots.txt) loses exactly those. Google's mitigation advice: 🟢 "You can reduce the size of the robots.txt file by consolidating rules that would result in an oversized robots.txt file. For example, place excluded material in a separate directory."

Line syntax. 🟢 "A valid robots.txt line consists of a field, a colon, and a value. Field names are case-insensitive… Spaces are optional, but recommended to improve readability. Space at the beginning and at the end of the line is ignored. To include comments, precede your comment with the # character" — general form <field>:<value><#optional-comment> (spec).

3.4 Fields: the four that exist and the graveyard of those that don't

🟢 "Google supports the following fields (other fields such as crawl-delay aren't supported): user-agent, allow, disallow, sitemap" (spec). Case sensitivity is asymmetric by field: 🟢 the user-agent field name and its value are case-insensitive; allow/disallow/sitemap field names are case-insensitive but their values (paths, URLs) are case-sensitive. 🟢 Rules without a path are ignored ("Crawlers ignore the rule without a path"), and 🟢 "The path value must start with / to designate the root."

Unsupported fields:

• ⚠️ crawl-delay — 🟢 "The non-standard 'crawl-delay' robots.txt rule is not processed by Google's crawlers" — Myths and facts about crawling. 🟢 Bing, per its own Webmaster Blog, has honored crawl-delay since the MSNBot era ("Bing supports the directives of the Robots Exclusion Protocol (REP)… the crawl-delay directive accepts only positive, whole numbers as values" — Crawl delay and the Bing crawler, MSNBot, Aug 10, 2009, fetched 2026-08-04), which is why tools still emit it; on Google it controls nothing. Crawl-rate management that actually works (server responses, the Googlebot report form) → Ch 2.
• ⚠️ noindex / nofollow in robots.txt — dead since September 1, 2019. 🟢 "we're retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019" — Gary Illyes, A note on unsupported rules in robots.txt (official blog, July 2019). The post's rationale: these rules "were never documented by Google" and "their usage was contradicted by other rules in all but 0.001% of all robots.txt files on the internet." The doc's current wording that unlisted fields "aren't supported" says the same thing generically; 🟢 the docs changelog dates that clarification to October 7, 2024: "Clarifying support for robots.txt fields… Clarified that fields that aren't listed in our robots.txt documentation aren't supported" (Search Central updates).
• 📘 The RFC anticipates extensions without endorsing any: "Crawlers MAY interpret other records that are not part of the robots.txt protocol — for example, 'Sitemaps'," provided parsing them doesn't interfere with the defined records (RFC 9309 §2.2.4).

The sitemap field is the one supported cross-cutting extension: 🟢 "Google, Bing, and other major search engines support the sitemap field in robots.txt, as defined by sitemaps.org." Its semantics differ from rules in three ways (🟢, spec): the value must be "a fully qualified URL, including the protocol and host"; 🟢 "The URL doesn't have to be on the same host as the robots.txt file" (cross-host submission — one of only two sitemap cross-host authorization mechanisms, → Ch 9); 🟢 "You can specify multiple sitemap fields, with no limit"; and 🟢 "The sitemap field isn't tied to any specific user agent and may be followed by all crawlers, provided it isn't disallowed for crawling" — it belongs to the file, not to any group, no matter where it sits.

3.5 Group selection: which rules a crawler actually obeys

A file is a list of groups: one or more user-agent lines followed by rules. 🟢 A group can name several crawlers by stacking user-agent lines (spec shows user-agent: e / user-agent: f / disallow: /g as one group covering both).

How groups are delimited. 🟢 The spec's own example (spec):

user-agent: a
disallow: /c

user-agent: b
disallow: /d

user-agent: e
user-agent: f
disallow: /g

user-agent: h

🟢 "In this example there are four distinct rule groups": one for a, one for b, one covering both e and f, and one (empty — everything allowed) for h. Blank lines are cosmetic; a group runs from its user-agent line(s) to the next user-agent line that starts a new block of rules.

Selection algorithm (Google, 🟢 all from the spec):

1. "Only one group is valid for a particular crawler." Google finds "the group with the most specific user agent that matches the crawler's user agent. Other groups are ignored." (Googlebot-Image obeys a googlebot-image group if present, else googlebot, else * — product-token hierarchy per crawler → Ch 5.)
2. Token matching is exact-token, not substring-fancy: "All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot)." ⚠️ Corollary: wildcards carry no meaning inside a user-agent value — you cannot write user-agent: Google* to catch a family of crawlers; each product token must be named, and * alone means "every crawler not otherwise named." Matching is case-insensitive (📘 RFC 9309 §2.2.1: "Crawlers MUST use case-insensitive matching to find the group that matches the product token").
3. "The order of the groups within the robots.txt file is irrelevant."
4. Duplicate groups merge: "If there's more than one specific group declared for a user agent, all the rules from the groups applicable to the specific user agent are combined internally into a single group." 🟢 The spec's example: user-agent: googlebot-news / disallow: /fish plus a later user-agent: googlebot-news / disallow: /shrimp is read as one group disallowing both /fish and /shrimp. (📘 RFC: "the matching groups' rules MUST be combined" — same behavior, RFC 9309 §2.2.1.)
5. ⚠️ But specific and global never merge: "User agent specific groups and global groups () are not combined." 📘 The RFC frames the same rule as a fallback: "If no matching group exists, crawlers MUST obey the group with a user-agent line with the '' value" (§2.2.1) — the * group applies only when no named group matched.

Worked selection example — a file with three groups, in the spec's own scenario (🟢 outcomes as stated on the spec page):

user-agent: googlebot-news
(rules 1)

user-agent: *
(rules 2)

user-agent: googlebot
(rules 3)

Crawler	Group obeyed	Google's stated reason
Googlebot News	rules 1	🟢 "group 1 is the most specific group"
Googlebot (web)	rules 3	most specific matching token
Storebot-Google	rules 2	🟢 "because there is no specific Storebot-Google group"
Any unlisted crawler	rules 2	no specific match → * fallback

Note what rule 5 implies here: Googlebot News obeys only rules 1 — nothing in the * group applies to it, no matter how universal those rules look.

⚠️ The invisible-*-group trap (rule 5 is the most damaging line in the protocol): adding

User-agent: Googlebot
Allow: /

to a file whose * group carries your disallows silently exempts Googlebot from every one of them — the * group ceases to exist for Googlebot. The converse is equally sharp: adding a Googlebot group with one targeted disallow removes all * protections you assumed still applied to Googlebot. Any named group must restate everything the crawler should still obey.

⚠️ Non-rule lines don't split groups. 🟢 "Rules other than allow, disallow, and user-agent are ignored by the robots.txt parser" — the spec's example shows that a stray unsupported line (or a sitemap: line) between two user-agent blocks does not terminate a group; the parser sees one group and applies its rules to both tokens. Don't rely on blank lines or decorative fields as separators; a group ends where the next user-agent line begins.

One Googlebot-specific note from → Ch 2: Googlebot Smartphone and Googlebot Desktop share one product token, so robots.txt cannot target them separately.

3.6 Rule matching: paths, wildcards, precedence

Path matching. Rules match by prefix against the URL path (plus query string), case-sensitively. 🟢 The spec's matching table (spec):

Pattern	Matches (🟢 examples from the spec)	Notes
/fish	/fish, /fish.html, /fish/salmon.html, /fishheads, /fish.php?id=anything	"Matches any path that starts with /fish. Note that the matching is case-sensitive." ⚠️ Prefix, not word: /fishheads matches
/fish*	same as /fish	🟢 "Equivalent to /fish. The trailing wildcard is ignored."
/fish/	/fish/, /fish/?id=anything, /fish/salmon.htm	"Matches anything in the /fish/ folder." ⚠️ Does not match /fish itself
/*.php	/index.php, /folder/filename.php?parameters, /folder/any.php.file.html	"Matches any path that contains .php."
/*.php$	/filename.php, /folder/filename.php	"Matches any path that ends with .php." Not /filename.php?parameters
/fish*.php	/fish.php, /fishheads/catfish.php?parameters	"Matches any path that contains /fish and .php, in that order."

Wildcards. 🟢 Google: * "designates 0 or more instances of any valid character"; $ "designates the end of the URL" (spec). 📘 RFC 9309 §2.2.3 defines the identical two special characters (* "Designates 0 or more instances of any character", $ "Designates the end of the match pattern") — what began as a Google/Bing extension is now standard.

Precedence. 🟢 "When matching robots.txt rules to URLs, crawlers use the most specific rule based on the length of the rule path. In case of conflicting rules, including those with wildcards, Google uses the least restrictive rule" (spec). 📘 RFC 9309 §2.2.2 in standards language: "The most specific match found MUST be used. The most specific match is the match that has the most octets"; and for exact ties, "If an 'allow' rule and a 'disallow' rule are equivalent, then the 'allow' rule SHOULD be used."

So: longest rule path wins; on a tie (or wildcard-induced conflict), allow wins. ⚠️ Order in the file is irrelevant — intuitions imported from firewall rules or .htaccess ("first match wins" / "last match wins") are both wrong. The spec's worked examples (🟢, all six rows):

URL	Rules	Winner	Google's stated reason
/page	allow: /p · disallow: /	allow	"it's more specific" (2 > 1 chars)
/folder/page	allow: /folder · disallow: /folder	allow	tie → "Google uses the least restrictive rule"
/page.htm	allow: /page · disallow: /*.htm	disallow	"the rule path is longer and it matches more characters in the URL, so it's more specific"
/page.php5	allow: /page · disallow: /*.ph	allow	conflict → least restrictive
/	allow: /$ · disallow: /	allow	"it's more specific"
/page.htm	allow: /$ · disallow: /	disallow	"the allow rule only applies on the root URL"

⚠️ Row 3 is the counter-intuitive one: a site-wide wildcard disallow (/*.htm, 6 characters) beats a deliberate allow: /page (5 characters) for /page.htm purely on length. Auditing a robots.txt means running the algorithm, not eyeballing intent — use the parser (§3.9).

The allow field itself is Google/RFC-era canon but was not in the original 1994 de-facto protocol — 📘 the 1994 document defines only User-agent and Disallow ("The record starts with one or more User-agent lines, followed by one or more Disallow lines," A Standard for Robot Exclusion, 1994, fetched 2026-08-04) — so ⚪ very old third-party crawlers may not implement it, one reason 🟢 Google warns "Different crawlers interpret syntax differently" (intro).

3.7 Fetch-time behavior: redirects, errors, and the availability matrix

How Google fetches the file matters as much as what's in it. The asymmetry to internalize: 4xx fails open (crawl everything), 5xx fails closed (crawl nothing) — the opposite of what most engineers assume a "blocking" file does.

The operational algorithm (🟢, Crawl Stats report): "Before Google crawls your site, it first checks if there's a recent successful robots.txt request (less than 24 hours old)." Successful = "HTTP 200 and a robots.txt file (the file can be valid, invalid, or empty)" or "HTTP 403/404/410 (the file does not exist). Your site is not required to have a robots.txt file." Unsuccessful = "HTTP 429/5XX (connection issue)." On an unsuccessful or stale response Google re-requests the file; 🟢 "If successful, the crawl can start. If not successful: For the first 12 hours, Google will stop crawling your site, but will continue to request your robots.txt file. From 12 hours to 30 days, Google will use the last successfully fetched robots.txt file… After 30 days: If the site homepage is available, Google will act as if there is no robots.txt file, and crawl without restraints. If the site homepage is not available, Google will stop crawling the site."

The full behavior matrix, Google 🟢 vs standard 📘:

Fetch outcome	Google's behavior (spec; the homepage-availability fork wording is from the Crawl Stats help, quoted in the algorithm above)	RFC 9309
2xx	🟢 "process the robots.txt file as provided by the server"	parse per spec
3xx	🟢 "Google follows at least five redirect hops as defined by RFC 1945 and then stops and treats it as a 404" → fail-open. 🟢 "Google doesn't follow logical redirects in robots.txt files (frames, JavaScript, or meta refresh-type redirects)"	📘 "The crawlers SHOULD follow at least five consecutive redirects, even across authorities" (§2.3.1.2)
4xx except 429	🟢 "Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions" → fail-open	📘 "unavailable": "the crawler MAY access any resources on the server" (§2.3.1.3)
429	🟢 excluded from the 4xx rule — grouped with server errors (429-as-server-error doctrine → Ch 2)	📘 no 429 carve-out (all 400–499 = unavailable) — Google is stricter than the RFC here
5xx / 429	🟢 12 hours: stop crawling, keep retrying → 30 days: "use the last good version" → after 30 days: homepage-availability decides (above). 🟢 "A 503 (service unavailable) error results in fairly frequent retrying. If there's no cached version available, Google assumes there's no crawl restrictions"	📘 "unreachable": "the crawler MUST assume complete disallow" (§2.3.1.4); after "a reasonably long period of time (for example, 30 days)" crawlers "MAY assume that the robots.txt file is unavailable" or keep using a cached copy
DNS/network failure	🟢 "A robots.txt file which cannot be fetched due to DNS or networking issues, such as timeouts, invalid responses, reset or interrupted connections, and HTTP chunking errors, is treated as a server error"	📘 same class: unreachable

⚠️ Fail-open traps. A WAF or auth layer returning 403 on /robots.txt does not "lock the site down" — it deletes every crawl restriction you have (403 = file does not exist = no rules). A redirect loop or 6-hop chain on robots.txt does the same via the treated-as-404 rule, and 🟢 the fail-open "also applies to any disallowed URLs in the redirect chain, since the crawler couldn't fetch rules due to the redirects" (spec). WAF/CDN interaction patterns → Ch 4.

⚠️ Fail-closed traps. A robots.txt that 500s takes the whole site's crawling down with it — 🟢 "Google requests this file frequently, and if the request doesn't return either a valid file (either populated or empty) or a 404 (file does not exist) response, then Google will slow or stop crawling your site until it can get an acceptable robots.txt response" (Crawl Stats). This is why robots.txt availability is one of the three host-status categories in the Crawl Stats report (→ Ch 2). Diagnostic fingerprint: during robots.txt unavailability 🟢 "Google counts crawls that it might have made if your robots.txt file were available, but doesn't actually make those calls" — so a Crawl Stats total far above your server logs' verified-Googlebot count is itself evidence of a robots.txt availability problem (→ Ch 2, → Ch 20).

⚠️ The nuance inside fail-closed: if Google has no cached good version when the 5xx streak starts (new site, first fetch fails), it "assumes there's no crawl restrictions" — fail-closed protects only sites Google has successfully read before.

The unavailability timeline, in one table (Google behavior; sources as above):

Time since last successful fetch	Google has a last-good copy	Google has no cached copy
< 24 h	uses the cached copy (normal operation)	first fetch decides: 200/403/404/410 → proceed; 5xx/timeout → next rows
0–12 h of failures	🟢 stops crawling the site, keeps retrying robots.txt	🟢 "assumes there's no crawl restrictions"
12 h – 30 days	🟢 crawls using "the last good version", keeps retrying	same as above
> 30 days	🟢 homepage reachable → "act as if there is no robots.txt file, and crawl without restraints"; homepage unreachable → "stop crawling the site" — robots.txt still re-requested periodically	same fork

📘 RFC comparison: the RFC's "unreachable → MUST assume complete disallow" corresponds to Google's 0–12 h behavior; the RFC's 30-day escape hatch ("MAY assume that the robots.txt file is unavailable" or "continue to use a cached copy", §2.3.1.4) is exactly the fork Google implements with the homepage-availability test as its tiebreaker. Google's 12 h→30 d last-good-version window is an implementation choice the RFC permits but does not require — other RFC-conformant crawlers may go fully fail-closed for the entire outage. ⚪ Do not assume Bing/others mirror Google's exact windows.

Deliberately serving 503 on robots.txt as an emergency crawl brake, and why it must not run longer than a couple of days, → Ch 2 (emergency crawl control).

3.8 Caching and propagation

🟢 "Google generally caches the contents of robots.txt file for up to 24 hours, but may cache it longer in situations where refreshing the cached version isn't possible (for example, due to timeouts or 5xx errors). The cached response may be shared by different crawlers. Google may increase or decrease the cache lifetime based on max-age Cache-Control HTTP headers" — spec. 📘 RFC 9309 §2.4: "Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable."

Operational consequences:

• Edits are not live. 🟢 "Google's crawlers notice changes you made to your robots.txt file and update the cached version every 24 hours. If you need to update the cache faster, use the Request a recrawl function of the robots.txt report" — submit-updated-robots-txt. Plan unblocking releases a day ahead; don't judge a fix's effect from logs within the first 24 hours.
• A bad file lingers exactly as long. An accidental Disallow: / deployed to production keeps suppressing crawling for up to a day after you revert it — request a recrawl (§3.9) instead of waiting. ⚪ Shipping robots.txt changes through the same review gates as code (diff alerts in CI) is the standard prevention; → Ch 20.
• One cache, many crawlers. The shared cache means Googlebot, Googlebot-Image, and the rest generally act on the same snapshot — a mid-day edit does not create per-crawler divergence for long.
• ⚪ Cache-Control: max-age on robots.txt is honored directionally (🟢 "increase or decrease"), but no minimum/maximum bounds are documented; don't design a "fast-propagating robots.txt" around it.

3.9 Testing and monitoring

Search Console robots.txt report (Settings → robots.txt): 🟢 "The robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, the last time they were crawled, and any warnings or errors encountered" — robots.txt report help. What it gives you:

• Fetch status per file: 🟢 "Fetched", "Not Fetched - Not found (404)" (🟢 "Not having a robots.txt file is fine, and means that Google can crawl all URLs on your site"), "Not Fetched - Any other reason".
• Parse issues: 🟢 "Errors prevent a rule from being used. Warnings do not prevent a rule from being used"; 🟢 "Google ignores the lines with issues and uses those that it can parse."
• The exact cached content Google is using, plus per-file version history: 🟢 "To see fetch requests for a given robots.txt file in the last 30 days, click the file in the files list in the report, then click Versions" — and it records changes, not every fetch: 🟢 "A request is included in the history only if the retrieved file or fetch result is different from the previous file fetch request" (report help) — invaluable for "what did Google see during the incident" forensics.
• Request a recrawl — the cache-refresh lever from §3.8; 🟢 the help page scopes it to "You changed your robots.txt rules to unblock some important URLs" or a fixed fetch error.
• 🟢 Availability: "This report is available only for properties at the domain level" (Domain properties, or URL-prefix properties at root) — → Ch 19 on property types.

⚠️ The old standalone robots.txt Tester tool is gone (🟡 the robots.txt report was announced Nov 15, 2023 and the legacy Tester was removed Dec 12, 2023 — dates per industry coverage of the launch, e.g. Search Engine Land, Nov 15, 2023; no Google-primary changelog entry located as of 2026-08-04. Some Google help pages still reference "the robots.txt tester" by name — verified live on the Page indexing help, 2026-08-04). Rule-level testing now happens in the report or offline:

• Google's open-source parser — 🟢 the create-doc points to "Google's open source robots.txt library… which is also used in Google Search" (create-robots-txt; repo: github.com/google/robotstxt). This is the ground truth for matching questions: feed it your file and candidate URLs and you're running the same C++ matcher production Google runs. ⚪ Wire it (or a port) into CI to assert "URL X must remain crawlable / blocked" on every deploy → Ch 20.
• ⚪ Third-party testers (Screaming Frog's, various web tools) are convenient but reimplement the matcher — verify surprising verdicts against the Google library.
• URL Inspection shows crawl-allowed status for a live URL, including which state ("Blocked by robots.txt") applied at last crawl → Ch 7, Ch 19.
• ⚪ Raw check: curl -A "Googlebot" https://example.com/robots.txt from outside your network — confirms what an external client actually receives (WAF rules can serve crawlers a different response than your browser sees → Ch 4).

Writing the file (create-robots-txt, 🟢): name must be exactly robots.txt, UTF-8, uploaded to the host root; "Rules are case-sensitive"; hosted platforms may not expose the file directly (Wix/Blogger-class CMSes offer settings instead — Shopify's robots.txt.liquid → Appendix A).

Canonical patterns (🟢 all from useful robots.txt rules, which carries the warning "Remember, don't use robots.txt to block access to private content; use proper authentication instead"):

# Block the whole site            # Block a directory tree
User-agent: *                     User-agent: *
Disallow: /                      Disallow: /calendar/

# Allow only one crawler          # Block site except one subtree
User-agent: Googlebot-News        User-agent: *
Allow: /                          Disallow: /
User-agent: *                     Allow: /public/
Disallow: /

# Block one filetype (note $)     # Block Google Images only
User-agent: Googlebot             User-agent: Googlebot-Image
Disallow: /*.gif$                 Disallow: /

⚠️ When copying the "allow only one crawler" pattern, remember §3.5: the named group replaces * entirely for that crawler — every rule the named crawler should still obey must be restated inside its group. AI-crawler policy files (GPTBot, ClaudeBot, PerplexityBot groups and what each vendor honors) → Ch 5.

3.10 Triage: the robots.txt incident runbook

⚪ House workflow, built entirely on the mechanics above — run it top to bottom whenever crawling behaves as if the robots.txt were different from the one you think you shipped:

1. Fetch the file as an outsider. curl -sI https://example.com/robots.txt from outside your network. Anything but 200 or 403/404/410 is an availability incident (§3.7): a 5xx/timeout is throttling your whole site right now; a redirect chain is heading toward treated-as-404 fail-open. Repeat with a Googlebot user-agent string — a WAF that answers crawlers differently than browsers is a Ch 4 problem wearing a robots.txt costume.
2. Ask what Google actually holds. GSC robots.txt report → fetch status, last fetch time, and the cached content (§3.9). Diagnose against the copy Google is using, not the file on your server — during the 24 h cache window and 5xx last-good-version windows the two legitimately differ.
3. Check the version history. The report's ~30-day version list answers "what did Google see during the incident window" — the question logs alone can't answer, because abandoned crawls during unavailability never reach your origin (§3.7).
4. Run the matcher, don't eyeball it. Feed the cached file plus the affected URLs to Google's open-source parser (§3.9). Most "robots.txt is being ignored" reports dissolve here into group-selection (§3.5) or longest-match (§3.6) surprises.
5. Cross-check the blast radius. Crawl Stats: robots.txt host status, response-code mix, and total-crawls vs your verified-Googlebot log count (a large excess = phantom abandoned crawls, §3.7) → Ch 2. Page indexing report: growth in "Indexed, though blocked by robots.txt" (§3.1) or in the blocked-by-robots.txt reason → Ch 7.
6. Fix, then force propagation. Correct the file (or the serving layer), request a recrawl in the robots.txt report, and re-verify the report's fetched version — not just your origin — before declaring the incident closed (§3.8).

Symptoms & diagnosis

Symptom	Likely cause	Where
Crawling site-wide slowed or stopped; Crawl Stats host status flags robots.txt	robots.txt returning 5xx/429 or timing out — fail-closed	§3.7, → Ch 2
Disallowed sections being crawled anyway	robots.txt returns 403/404, or a redirect chain/loop → treated as nonexistent — fail-open	§3.7
URL in results with no snippet / "no information is available"	Disallowed but indexed from external links	§3.1, → Ch 7
GSC warning "Indexed, though blocked by robots.txt"	robots.txt used for index control; noindex unreachable behind the block	§3.1, → Ch 7
Googlebot ignoring rules that "clearly" block it	Named Googlebot group exists → * group invisible to Googlebot; or rule shadowed by a longer allow	§3.5, §3.6
One URL unexpectedly blocked/allowed	Longest-match precedence with wildcards (e.g. /*.htm beats allow: /page)	§3.6
robots.txt edit shows no effect	Up-to-24h cache; request a recrawl in the robots.txt report	§3.8, §3.9
Rules silently not applied though file fetches fine	Syntax errors (report shows them), non-UTF-8 encoding, or rules past the 500 KiB cutoff	§3.3, §3.9
crawl-delay (or robots.txt noindex) having no effect on Google	Unsupported fields — noindex retired Sept 1, 2019	§3.4
Crawl Stats total far exceeds verified Googlebot hits in server logs	Phantom "counted but not made" crawls during robots.txt unavailability	§3.7, → Ch 2
Pages render blank/broken in URL Inspection; rankings sag	Rendering-critical CSS/JS disallowed	§3.1, → Ch 6
Rules behave differently on Google vs other engines	Divergent implementations (allow support, wildcard handling, crawl-delay)	§3.6, §3.4

Sources

• How Google interprets the robots.txt specification — https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec (fetched 2026-08-04; page last-updated 2026-07-08). Old URL developers.google.com/search/docs/crawling-indexing/robots/robots_txt verified 301-redirecting here (checked 2026-08-04).
• Introduction to robots.txt — https://developers.google.com/search/docs/crawling-indexing/robots/intro (fetched 2026-08-04; last-updated 2025-12-10)
• RFC 9309: Robots Exclusion Protocol — https://datatracker.ietf.org/doc/html/rfc9309 (fetched 2026-08-04; published September 2022)
• How to write and submit a robots.txt file — https://developers.google.com/crawling/docs/robots-txt/create-robots-txt (fetched 2026-08-04; last-updated 2025-11-21)
• List of useful robots.txt rules — https://developers.google.com/crawling/docs/robots-txt/useful-robots-txt-rules (fetched 2026-08-04; last-updated 2026-06-12)
• Submit an updated robots.txt file — https://developers.google.com/crawling/docs/robots-txt/submit-updated-robots-txt (fetched 2026-08-04; last-updated 2025-11-21)
• robots.txt report — Search Console Help — https://support.google.com/webmasters/answer/6062598 (fetched 2026-08-04; no revision date shown)
• Crawl Stats report — Search Console Help — https://support.google.com/webmasters/answer/9679690 (fetched 2026-08-04; no revision date shown)
• Page indexing report — Search Console Help — https://support.google.com/webmasters/answer/7440203 (fetched 2026-08-04; no revision date shown)
• Myths and facts about crawling — https://developers.google.com/crawling/docs/myths-about-crawling (fetched 2026-08-04; last-updated 2025-12-18)
• A note on unsupported rules in robots.txt (Gary Illyes, official blog, July 2019) — https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt (fetched 2026-08-04)
• Google's robots.txt open-source parser — https://github.com/google/robotstxt (link verified from create-robots-txt page, 2026-08-04)
• Large site owner's guide to managing your crawl budget — https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget (fetched 2026-08-04; last-updated 2026-07-22)
• Overview of Google crawlers and fetchers (user agents) — https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers (fetched 2026-08-04; last-updated 2026-06-12)
• Google's user-triggered fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers (fetched 2026-08-04; last-updated 2026-07-16)
• Google's special-case crawlers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers (fetched 2026-08-04; last-updated 2026-02-11)
• Google Search Central documentation changelog — https://developers.google.com/search/updates (fetched 2026-08-04; Oct 7, 2024 entry "Clarifying support for robots.txt fields" re-verified live 2026-08-04)
• Crawl delay and the Bing crawler, MSNBot — Bing Webmaster Blog — https://blogs.bing.com/webmaster/August-2009/Crawl-delay-and-the-Bing-crawler,-MSNBot (fetched 2026-08-04; posted 2009-08-10)
• A Standard for Robot Exclusion (1994 de-facto protocol) — https://www.robotstxt.org/orig.html (fetched 2026-08-04)
• Google Search Console adds robots.txt report — Search Engine Land — https://searchengineland.com/google-search-console-adds-robots-txt-report-434708 (fetched 2026-08-04; published 2023-11-15; 🟡 secondary source for report launch / Tester retirement dates)