AI crawlers are reshaping the internet.
From content indexing to model training and answer generation, AI bots are scanning websites at an accelerating pace. But not every organization wants blanket access and not every AI bot should be treated the same.
That’s why we’re excited to introduce Granular AI Bot Controls inside the Convesio Cloudflare Metadata UI.
You can now selectively allow or block specific AI bots per hostname with full visibility, auditability, and security control.
Precision Control, Not a Blanket Block
Previously, blocking AI bots often meant enabling a broad “block all AI crawlers” rule. That approach lacks nuance and can create unintended consequences.
With this new feature, Convesio customers can:
- Choose whether to block specific high-impact AI bots
- Apply policies per hostname
- Maintain SEO-safe configurations
- Inspect and audit enforcement behavior
Instead of an all-or-nothing toggle, you get strategic control.
How It Works
Inside the Convesio control plane, you can now set an ai_bot_block_profile metadata value for any hostname.
Available profiles:
- none – No AI bots blocked
- top_ai_bots – Blocks a defined set of major AI crawlers
When top_ai_bots is enabled, our WAF custom rules reference this metadata and block only the specified bots.
If set to none, those bots are not blocked.
If metadata is missing or invalid, the system safely defaults and emits a log/metric for observability.
Security-first. Fail-safe by design.
Bots Currently Supported
The initial AI bot block profile includes:
- GPTBot
- CCBot
- ClaudeBot / Claude-Web
- Google-Extended
- PerplexityBot
We intentionally do not block BingBot, as doing so would negatively impact search visibility and Copilot indexing.
This ensures your SEO posture remains intact while still giving you control over AI training crawlers.
Full Auditability and Debug Visibility
Enterprise-grade control requires transparency.
With this release:
- All changes to ai_bot_block_profile are auditable
- The effective AI bot profile applied to any request can be inspected
- Debug headers and logs surface enforcement behavior
- Security teams can validate rule application in real time
This makes it easy to answer:
- Was this bot blocked?
- Which profile was applied?
- Is the configuration correct for this hostname?
Operational clarity is built in.
Why This Matters
AI bots introduce new strategic considerations:
- Content licensing concerns
- Training data control
- Traffic load management
- Competitive intelligence
- Regulatory and compliance risk
Different hostnames may require different policies. A public marketing site may allow broader access. A gated application, research portal, or regulated healthcare environment may require tighter control.
Now, you can manage that distinction directly inside Convesio.
Built on Cloudflare, Managed by Convesio
This capability is implemented through Convesio’s Cloudflare Metadata integration and enforced via WAF custom rules.
You get:
- Cloudflare-grade edge enforcement
- Convesio-managed control plane simplicity
- Security engineering flexibility
- Centralized observability
No custom rule writing required.
A More Intentional Internet Strategy
The web is changing. AI access policies should not be accidental.
With Convesio’s Granular AI Bot Controls, you can:
- Protect strategic content
- Maintain SEO integrity
- Preserve performance
- Make policy decisions intentionally
This is another step in our ongoing investment in modern security controls and intelligent edge infrastructure.
