# robots.txt — Walk Another Mile # Tells search engine crawlers what they may index. # Spec: https://www.robotstxt.org/robotstxt.html # Last reviewed: 2026-09-02 — added explicit Allow: /regions/ (21 region pages # were previously only covered by the top-level Allow: /, not individually # spelled out like /tours/, /guides/, /temples/ and /blog/ are; harmless as-is, # but spelling it out removes any doubt for crawlers that resolve conflicts by # most-specific-match rather than most-specific-Allow-wins). User-agent: * Allow: / # Content-Signal, 2026-08-20. # # Cloudflare injects its own signal above this file which reads # search=yes,ai-train=no,use=reference # and notably OMITS ai-input. Under the Content-Signals spec, a use that is not # named is "neither granted nor restricted" — which leaves AI grounding and # retrieval-augmented answers in an ambiguous state. # # This site's position (confirmed 2026-09-29): # search=yes index us and return links and excerpts # ai-input=yes use our pages to ground live AI answers, which cite and # link back — a referral channel # ai-train=yes our content may also be used for model training # Content-Signal: search=yes,ai-input=yes,ai-train=yes # Explicit allow for PWA + render-critical assets. Some crawlers default-deny # manifest or dotfile-looking paths; spelling them out makes the contract # clear and prevents accidental blocking by overzealous catch-all rules. Allow: /site.webmanifest Allow: /web-app-manifest-192x192.png Allow: /web-app-manifest-512x512.png Allow: /apple-touch-icon.png Allow: /favicon.svg Allow: /favicon.ico Allow: /favicon-96x96.png Allow: /shared.css Allow: /shared.js Allow: /image-manifest.js # Page-type stylesheets — added 2026-08-20. # These were previously unlisted, so the old `Disallow: /*?v=` rule caught them # (Google resolves conflicts by longest matching pattern, so a 5-character # Disallow beat the 1-character `Allow: /`). That silently blocked Googlebot # from rendering 399 page-loads: all 82 tour itineraries, 52 city guides, # 21 region pages, 45 temple pages and 31 journal posts. Allow: /destination.css Allow: /itinerary.css Allow: /cityguide.css Allow: /temples.css Allow: /blog.css Allow: /images/ # Content sub-folders (HTML) — individual tour itineraries and city/destination guides Allow: /tours/ Allow: /guides/ Allow: /temples/ Allow: /blog/ Allow: /regions/ # Private / non-public endpoints — keep out of search indexes. # # NOTE (2026-08-20): /admin.html is deliberately NOT listed here any more. # robots.txt is public, so naming a private path in it advertises that path to # anyone reading the file. /admin.html already carries # # which is strictly stronger: a Disallow stops Google CRAWLING the page, which # means it can never SEE the noindex tag. Access control belongs in front of # the page (Cloudflare Access / Supabase RLS), not in this file. # # /WAM-Platform-Copy-Kit.html and /WAM-Social-Strategy.html were also removed — # neither file exists (both return 404), so the lines advertised nothing but # their own names. Disallow: /*.bak$ Disallow: /*.sql$ # Block URL-parameter variants from cluttering the index. # # NOTE (2026-08-20): `Disallow: /*?v=` was REMOVED. It was intended to stop # cache-busted URLs being indexed, but no page on this site links to a # `.html?v=` URL, so it protected nothing — while silently blocking every # versioned stylesheet not named in the Allow list above. Canonical tags # already de-duplicate parameter variants, which is the correct mechanism. Disallow: /*?source= Disallow: /*?utm_ # IMPORTANT: do NOT block CSS or JS — Google needs to fetch them to render # and understand the page. Previously these were disallowed which hurt SEO. # Image manifest is allowed for the same reason (used at render time). # Sitemap location — index references the page, video, and image sitemaps # AI/LLM index Allow: /llms.txt # Only the index is declared. It already chains sitemap.xml, sitemap-video.xml # and sitemap-images.xml, so declaring all four made Google discover each child # sitemap twice and cluttered the Search Console Sitemaps report. (2026-08-20) Sitemap: https://www.walkanothermile.com/sitemap-index.xml # ══════════════════════════════════════════════════════════════════════ # NOTE (2026-08-10) — POLICY CHANGE, and a WARNING. # # This file now ALLOWS AI *retrieval* agents (they fetch a page live to # answer a user question and cite us back) while still BLOCKING AI # *training* crawlers. Retrieval agents are a referral channel. # # ⚠️ THIS FILE IS NOT THE WHOLE STORY. # Cloudflare injects an additional managed block ABOVE this content at the # edge. As of 2026-08-10 the live robots.txt served to crawlers ALSO # contained Cloudflare-generated rules blocking: # ClaudeBot, GPTBot, Google-Extended, Applebot-Extended, CCBot, # Bytespider, Amazonbot, meta-externalagent, # CloudflareBrowserRenderingCrawler # plus Content-Signal: search=yes,ai-train=no,use=reference # # Those Cloudflare rules OVERRIDE the intent of this file. Editing this # file alone will NOT unblock AI retrieval agents. # # TO COMPLETE THIS FIX you must also, in the Cloudflare dashboard: # Security → Settings → "AI Scrapers and Crawlers" → review/disable # Security → Settings → "Manage Content Signals policy" → set ai-input=yes # Then re-fetch https://www.walkanothermile.com/robots.txt and confirm the # Cloudflare "BEGIN Cloudflare Managed content" block no longer disallows # the retrieval agents listed in the ALLOW section below. # ══════════════════════════════════════════════════════════════════════ # ────────────────────────────────────────────────────────────────────── # SEO TOOLING — allowed so we can audit our own site. # (Blocking these does not stop competitors analysing us; it only blinds us.) # Reviewed 2026-08-10. # ────────────────────────────────────────────────────────────────────── User-agent: SemrushBot Allow: / User-agent: AhrefsBot Allow: / # Low-value link scrapers — still blocked to save bandwidth User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / # ────────────────────────────────────────────────────────────────────── # AI RETRIEVAL / CITATION AGENTS — ALLOWED. # These fetch a page live to answer a user's question and cite the source # back to us. They are a discovery and referral channel, not training. # Reviewed 2026-08-10. # ────────────────────────────────────────────────────────────────────── User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bingbot Allow: / # ────────────────────────────────────────────────────────────────────── # AI CRAWLERS — OPENED 2026-09-09. # Previously blocked on the reasoning that training crawls take content # and return no traffic. That reasoning held while retrieval and training # were cleanly separable. In practice the major assistants increasingly # answer from a blend of both, and being absent from the training corpus # means being absent from the answer even when the retrieval bot is # allowed in. For a business whose buyers research via AI assistants, # invisibility is the larger cost. Content-Signal below is updated to # ai-train=yes to stay consistent with what is actually permitted here. # Purely extractive scrapers that return nothing remain blocked further down. # ────────────────────────────────────────────────────────────────────── User-agent: GPTBot Allow: / User-agent: CCBot Allow: / User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: meta-externalagent Disallow: / User-agent: MistralAI-User Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Timpibot Disallow: /