# Everview robots.txt — deliberate answer-engine (GEO) policy, not inherited.
# See docs/website-system.md (GEO pass) and CLAUDE.md. Consistency between this
# file and each page's <meta name="robots"> state is enforced at build time by
# scripts/verify-prerender.mjs — do not hand-drift the two apart.
#
# Owner policy: ALLOW answer-engine / AI crawlers across the whole site so
# Everview is citable in AI answers, EXCEPT the 404 page. Pages excluded here
# carry <meta name="robots" content="noindex"> and are kept out of sitemap.xml;
# the Disallow rules below mirror that state.
#
# /capabilities/* was Disallowed here while its copy was an interim AI draft.
# Owner released the four pages for indexing 2026-08-27, so they are now
# index,follow and listed in sitemap.xml — the Disallow is gone with them.
# The Disallow set below must equal seo-policy.mjs NOINDEX_PREFIXES exactly;
# verify-prerender.mjs asserts it, so re-adding a page means editing both.

# ── Answer-engine & AI crawlers (explicit allow, minus interim/noindex pages) ──
# Grouped user-agents share one rule block (RFC 9309). Tokens verified against
# each engine's live crawler docs (OpenAI, Anthropic, Perplexity, Google, Apple,
# Microsoft, Common Crawl, Amazon, Meta, ByteDance, DuckDuckGo, Cohere, Mistral).
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Applebot-Extended
User-agent: Bingbot
User-agent: CCBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: DuckAssistBot
User-agent: cohere-ai
User-agent: MistralAI-User
Allow: /
Disallow: /404

# ── Everything else (incl. Googlebot/Bingbot search indexing) ──
# Deliberately NOT disallowing the noindex pages here: general search engines
# must be able to CRAWL them to SEE and honor their noindex meta tag. Blocking a
# noindex URL in robots.txt would prevent the crawler from ever reading the
# noindex directive.
User-agent: *
Allow: /

Sitemap: https://everview.ai/sitemap.xml