# sflix.today — robots.txt — v3 P2-8 AI bot split policy + honeypot trap # Last regen: dynamic per request (FastAPI) # =========================================================== # Default group (catches uncommon crawlers + smaller engines) # =========================================================== User-agent: * Disallow: /v1/csp/ Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /search? Disallow: /search/ Disallow: /filter? Disallow: /filter Disallow: /sort= Disallow: /movies? Disallow: /tv? Disallow: /anime? Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?session= Disallow: /*?yclid= Disallow: /*?gclid= Disallow: /*?fbclid= Disallow: /*?msclkid= Disallow: /drops Disallow: /account Disallow: /wallet Disallow: /signup Disallow: /login # 2026-07-12 SEO: locale (/xx/) versions of the faceted + account junk (root covered above) Disallow: /*/search Disallow: /*/filter Disallow: /*/movies? Disallow: /*/tv? Disallow: /*/anime? Disallow: /*/drops Disallow: /*/account Disallow: /*/wallet Disallow: /*/signup Disallow: /*/login Disallow: /_next/data/ Disallow: /scraper-bait- Disallow: /actor/ Disallow: /*/actor/ Disallow: /director/ Disallow: /*/director/ # 2026-06-07 SEO: /_next/image is now Allow (was Disallow) — unblocks Google Images + crawler-side LCP signal # =========================================================== # Google + AdsBot family (all ALLOW) # =========================================================== User-agent: Googlebot Disallow: /v1/csp/ Allow: /_next/image Allow: /_next/static/ Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /search? Disallow: /search/ Disallow: /*/search Disallow: /filter? Disallow: /filter Disallow: /*/filter Disallow: /sort= Disallow: /movies? Disallow: /tv? Disallow: /anime? Disallow: /*/movies? Disallow: /*/tv? Disallow: /*/anime? Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?session= Disallow: /*?yclid= Disallow: /*?gclid= Disallow: /*?fbclid= Disallow: /*?msclkid= Disallow: /drops Disallow: /account Disallow: /wallet Disallow: /signup Disallow: /login Disallow: /*/drops Disallow: /*/account Disallow: /*/wallet Disallow: /*/signup Disallow: /*/login Disallow: /_next/data/ Disallow: /scraper-bait- Disallow: /actor/ Disallow: /*/actor/ Disallow: /director/ Disallow: /*/director/ User-agent: Googlebot-Image Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Googlebot-Video Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Googlebot-News Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: AdsBot-Google Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: AdsBot-Google-Mobile Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Mediapartners-Google Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: APIs-Google Disallow: /api/ Disallow: /v1/csp/ Allow: / # Google Gemini / AI Overview training opt-in — BLOCK jul28-2026 (training-only; per Google docs does NOT affect Search/AI-Overviews which use Googlebot; piracy brand gains nothing from Gemini training) User-agent: Google-Extended Disallow: / # =========================================================== # Bing (ALLOW) # =========================================================== User-agent: Bingbot Disallow: /v1/csp/ Allow: /_next/image Allow: /_next/static/ Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /search? Disallow: /search/ Disallow: /*/search Disallow: /filter? Disallow: /filter Disallow: /*/filter Disallow: /sort= Disallow: /movies? Disallow: /tv? Disallow: /anime? Disallow: /*/movies? Disallow: /*/tv? Disallow: /*/anime? Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?session= Disallow: /*?yclid= Disallow: /*?gclid= Disallow: /*?fbclid= Disallow: /*?msclkid= Disallow: /drops Disallow: /account Disallow: /wallet Disallow: /signup Disallow: /login Disallow: /*/drops Disallow: /*/account Disallow: /*/wallet Disallow: /*/signup Disallow: /*/login Disallow: /_next/data/ Disallow: /scraper-bait- Disallow: /actor/ Disallow: /*/actor/ Disallow: /director/ Disallow: /*/director/ User-agent: AdIdxBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: MSNBot-Media Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: BingPreview Disallow: /api/ Disallow: /v1/csp/ Allow: / # =========================================================== # Yandex (ALLOW) + Clean-param + site-wide rules # =========================================================== User-agent: Yandex Disallow: /v1/csp/ Allow: /_next/image Allow: /_next/static/ Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /search? Disallow: /search/ Disallow: /*/search Disallow: /filter? Disallow: /filter Disallow: /*/filter Disallow: /movies? Disallow: /tv? Disallow: /anime? Disallow: /*/movies? Disallow: /*/tv? Disallow: /*/anime? Disallow: /drops Disallow: /account Disallow: /wallet Disallow: /signup Disallow: /login Disallow: /*/drops Disallow: /*/account Disallow: /*/wallet Disallow: /*/signup Disallow: /*/login Disallow: /scraper-bait- Disallow: /actor/ Disallow: /*/actor/ Disallow: /director/ Disallow: /*/director/ User-agent: YandexImages Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: YandexVideo Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: YandexMobileBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: YandexMetrika Disallow: /api/ Disallow: /v1/csp/ Allow: / # Yandex-exclusive — consolidates URL variants (huge win at our scale) Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content Clean-param: gclid&yclid&fbclid&msclkid&ref&affiliate_id Clean-param: session&sid&PHPSESSID Clean-param: dub /anime/ Clean-param: page /filter Clean-param: sort&order / # =========================================================== # Other real search engines — ALLOW # =========================================================== User-agent: DuckDuckBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Petalbot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: PetalSearch Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Applebot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Bravebot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: MojeekBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Seznambot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Yeti Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Yep Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Naverbot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Baiduspider Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Sogou web spider Disallow: /api/ Disallow: /v1/csp/ Allow: / # =========================================================== # AI crawlers — 2026 split policy # BLOCK training crawlers; ALLOW search/agent crawlers (citation surface) # =========================================================== # OpenAI: BLOCK training, ALLOW search + per-user User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: ChatGPT-User Disallow: /api/ Disallow: /v1/csp/ Allow: / # Anthropic three-bot framework User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-SearchBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Claude-User Disallow: /api/ Disallow: /v1/csp/ Allow: / # Perplexity — 110:1 crawl-to-referral, best AI bot ROI User-agent: PerplexityBot Disallow: /api/ Disallow: /v1/csp/ Allow: / User-agent: Perplexity-User Disallow: /api/ Disallow: /v1/csp/ Allow: / # Common Crawl (feeds every other AI training dataset) User-agent: CCBot Disallow: / # ByteDance/TikTok — notorious robots.txt non-compliance; also nginx UA-filter User-agent: Bytespider Disallow: / # Apple Intelligence training (no citation surface) User-agent: Applebot-Extended Disallow: / # Meta LLaMA training User-agent: Meta-ExternalAgent Disallow: / User-agent: FacebookBot Disallow: / # Amazon Alexa + training User-agent: Amazonbot Disallow: / # Diffbot — catalog scraper sold to LLM trainers User-agent: Diffbot Disallow: / # Other AI scrapers User-agent: AI2Bot-Dolma Disallow: / User-agent: Cotoyogi Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / # =========================================================== # SEO scrapers — BLOCK (bandwidth waste, zero ranking impact) # =========================================================== User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: SerpstatBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: rogerbot Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: AspiegelBot Disallow: / User-agent: SiteAuditBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: SeekportBot Disallow: / User-agent: BacklinksExtendedBot Disallow: / User-agent: TurnitinBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: NextGenSearchBot Disallow: / # =========================================================== # Sitemaps (all engines that read this line will fetch) # =========================================================== Sitemap: https://sflix.today/sitemap.xml # =========================================================== # 2026-08-13 - /actor/ (and 2026-08-25 /director/) is off-limits to the AI/search-agent # crawlers too. They each have their OWN group above, and a # crawler with its own group ignores `User-agent: *` entirely, # so they were never covered by the /actor/ Disallow there. # Measured ~700k /actor/ fetches a day, led by Applebot, # meta-webindexer and Claude-SearchBot. Actor pages are # `noindex, follow` by design, so blocking the FETCH costs no # ranking and no citation surface - it only saves origin CPU. # RFC 9309: groups repeating a user-agent MUST be merged, so # these combine with the groups above rather than replacing them. # =========================================================== User-agent: Applebot User-agent: Applebot-Extended User-agent: Claude-SearchBot User-agent: ClaudeBot User-agent: Claude-User User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: GPTBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Meta-ExternalAgent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Google-Extended Disallow: /actor/ Disallow: /*/actor/ Disallow: /director/ Disallow: /*/director/