# ahmeego.com — Crawler & AI Agent Directives # Updated: 2026-10-11 # Wikidata: https://www.wikidata.org/wiki/Q139620020 # Content Signals (per Cloudflare standard) # search=yes: allow search indexing # ai-input=yes: allow AI retrieval/grounding (RAG, Cloudflare AI agents, etc.) # ai-train=no: do NOT use for model training/fine-tuning User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no Allow: / Allow: /solutions/ Allow: /learn/ Allow: /blog/ Allow: /tools/ Allow: /tools/argus/ Allow: /industries/ Allow: /videos/ Allow: /course/ Allow: /training/ Allow: /setup/ Allow: /about/ Allow: /contact/ Allow: /sitemaps/ Allow: /llms.txt Allow: /llms-full.txt Allow: /.well-known/ai-plugin.json Allow: /agents.json Allow: /agents.txt Disallow: /api/ Disallow: /admin/ Disallow: /training/admin/ Disallow: /tools/auditor/admin Disallow: /tools/auditor/admin.html Disallow: /training/exam.html Disallow: /training/results.html Disallow: /training/cert.html Disallow: /*?sort= Disallow: /*?filter= # ============================================================ # AI training crawlers — BLOCKED (do not use content for model training) # ============================================================ User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: Diffbot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: Timpibot Disallow: / User-agent: PetalBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: img2dataset Disallow: / User-agent: AI2Bot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: PiplBot Disallow: / User-agent: PanguBot Disallow: / # ============================================================ # AI retrieval / grounding bots — ALLOWED (citation, not training) # These are the bots that bring live traffic via AI answers. # # NOTE: GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended are # intentionally NOT listed here — those are TRAINING crawlers and are # blocked by Cloudflare's managed robots.txt (consistent with the # ai-train=no policy above). The retrieval bots below are what power # AI-search citations and live answers, and they stay fully allowed. # ============================================================ User-agent: Claude-User Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: Claude-SearchBot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: OAI-SearchBot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: ChatGPT-User Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: GoogleOther Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: PerplexityBot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: Perplexity-User Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: YouBot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: DuckAssistBot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no User-agent: Bingbot Allow: / Content-Signal: search=yes,ai-input=yes,ai-train=no # Sitemap index: core pages, solutions, industries, videos, blog Sitemap: https://ahmeego.com/sitemap.xml