S1 Bot Shield — every bot
Blocks or redirects any request classified as a bot, in every category. Use it on domains where no bot is welcome — a tracking domain that should never be indexed.
Bot Filtering
Smart Shield sorts bots into categories at the Cloudflare edge, so one rule can show a whitepage to ad reviewers, another can stop AI crawlers, and search engines can still index your site. This page explains what counts as a bot, how rules match, and which preset fits which job.
How a Request Is Classified
Every request that reaches your primary domain passes through the same checks, in this order. The first check that fires decides the category.
- Empty or very short User-Agent (under 10 characters) — a bot, category “other unverified”. Real browsers always send one.
- Known bot signature — the User-Agent matches a curated list: Googlebot, Bingbot, YandexBot; GPTBot, ClaudeBot, CCBot; UptimeRobot, Pingdom; facebookexternalhit, AdsBot-Google, TikTokBot; TelegramBot, WhatsApp, Slackbot and more. The list decides the category.
- Verified by Cloudflare — Cloudflare marks the request as coming from a bot in its verified registry, and 301.st maps Cloudflare’s category onto its own.
- Generic bot words — the User-Agent contains
bot,crawl,spiderorslurp. Category “other unverified”.
Separately from all of the above, every request gets a network type: datacenter when the network owner is a hosting or cloud provider (AWS, Google Cloud, Azure, DigitalOcean, Hetzner, OVH and others), residential otherwise. It is set for bots and people alike.
The Seven Bot Categories
In a rule, these are the checkboxes under Bot category. Pick one or several; a rule matches a bot in any of the ticked categories.
| Category | Who ends up there | Typical decision |
|---|---|---|
| Search engines | Googlebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, Applebot | Let through — blocking them removes the site from search results |
| AI training crawlers | GPTBot, ClaudeBot, CCBot, PerplexityBot, Bytespider, Amazonbot, Google-Extended | Block — they collect content, they do not bring visitors |
| Monitoring | UptimeRobot, Pingdom, StatusCake, Datadog, New Relic | Let through, or they report your site as down |
| Ad reviewers | facebookexternalhit, AdsBot-Google, TikTokBot, LinkedInBot, PinterestBot | Whitepage — the moderator sees a compliant page |
| Link previews (social) | TelegramBot, WhatsApp, Slackbot, Discordbot, Twitterbot | Whitepage or let through — decides what a shared link preview shows |
| Other verified bots | Bots Cloudflare verifies that fall outside the categories above: archivers, feed fetchers, security scanners | Usually let through |
| Other unverified bots | Empty User-Agents and anything that names itself a bot, crawler or spider | Block or redirect |
Bot Rules Run First
A rule with a bot condition is checked before every other rule on the domain, whatever its priority — Smart Shield or SmartLink. A geo or UTM rule placed above it cannot catch bot traffic first. In the Streams table a lock icon next to the block counter marks this; its tooltip names the rules. Bot rules can still be reordered among themselves.
One Flag, Every Bot
Ticking Bot traffic only without choosing a category matches all seven categories — search engines, link previews and uptime monitors included. A redirect rule like that keeps the domain out of search results and makes a shared link preview show the redirect target. The Streams table labels such a rule “all bots”. If that is not what you want, narrow the rule with categories.
Presets for Common Jobs
Each preset is one rule with its conditions filled in. Among bot rules, the narrower presets run before the broad Bot Shield.
S6 Cloaking Standard — whitepage
Sends ad reviewers, search engines and link previews to a whitepage you choose. AI training crawlers are not included — they get their own preset.
S7 AI Guard — AI crawlers
Blocks or diverts AI training crawlers only. Search engines are untouched, so the site stays indexed while its content stays out of training sets.
S8 Datacenter Cloak — hosting networks
Diverts every request from a datacenter network, whether or not it looks like a bot. Checkers often run from cloud servers with a perfectly normal browser User-Agent; this is the preset that catches them. It has no bot condition, so it runs after your bot rules, not before them.
Narrow Any Rule by Category
Presets are a starting point, not a limit. Any Smart Shield rule can be tuned in the editor:
- Open the rule in Streams and click edit.
- Expand More signals — OS, browser, bot details.
- Tick the Bot category boxes the rule should act on. Optionally set Network type to datacenter or residential.
- Save and apply. The Conditions column now lists the categories instead of “all bots”.
Example: a Bot Shield rule that should spare search engines and link previews — tick AI training crawlers, Monitoring, Ad reviewers and both “other” categories, and leave Search engines and Link previews unticked.
What Detection Does Not Catch
Classification works on what a request says about itself and on the network it comes from. It is not a fingerprinting or behavioural system, so some traffic passes as human:
- Scripts with a normal-looking User-Agent — curl, python-requests and similar tools whose User-Agent does not mention a bot.
- Headless browsers (Puppeteer, Playwright) and anything that copies a real browser’s User-Agent.
- Residential proxies — the network type reads residential, and the User-Agent looks like a person.
- People on hosting networks — a VPN or corporate proxy running in a datacenter is marked datacenter, so S8 diverts those visitors too.
For traffic from cloud servers, add S8 Datacenter Cloak next to your bot rules. The label “verified” means Cloudflare vouched for the bot; a bot recognised by its User-Agent signature is matched by category but is not marked verified.