✦ Key takeaway
Since September 15, 2026, Cloudflare classifies Googlebot, Bingbot and Applebot as “multipurpose” crawlers: blocking AI training, including via the “Block AI bots” option, can therefore block them too. The possible result: 403 errors and disappearing from Google, Bing and AI answers. It’s best to check your CDN settings, Search Console and server logs, and to block training crawlers one by one rather than by category.
Yes. Since September 15, 2026, a site that blocks training AI crawlers via Cloudflare may also block Googlebot, Bingbot, and Applebot. The reason: these bots are classified as multipurpose because they serve both traditional indexing and artificial intelligence. This article details what has changed, the risks to your visibility, and the checks you should perform to avoid disappearing from search results.
What has changed with Cloudflare’s new AI crawler rule?
Cloudflare now classifies AI crawlers into three categories. The first includes search crawlers, which power search engines and generated answers. The second concerns AI agents, which visit sites on behalf of conversational assistants. The third gathers training crawlers, which collect data to improve language models.
Since September 15, 2026, Cloudflare blocks training and agent crawlers by default on pages displaying ads. This setting applies to new domains, new sites on existing accounts, and free accounts that had not modified their settings. Importantly, any site blocking training is affected by the multipurpose crawler rule, including via the old “Block AI bots” option. If you enabled it with one click, you risk blocking bots essential for your SEO.
The timeline spans a year. In July 2025, Cloudflare launched one-click AI bot blocking and the Pay-Per-Crawl model. On July 1, 2026, the company announced the new classification and default settings. On September 15, 2026, these settings took effect. Cloudflare’s official blog details these rules, which may still evolve: always check the latest documentation before making changes.
Why can blocking AI crawlers also block Google?
Googlebot, Bingbot, Applebot: multipurpose crawlers
Googlebot is no longer just for indexing pages for classic search results. It also powers Google’s AI Overviews and AI Mode. Bingbot and Applebot follow the same logic, serving both search and AI features in their ecosystems.
Cloudflare classifies these three bots in the “search + training” category and applies the strictest rule. In practice, if you block training, they are blocked too, even if search crawlers are still allowed. Note that Google-Extended, which lets you exclude your content from Google’s model training, has no effect on Google Search. Blocking it does not affect your indexing, unlike blocking Googlebot.
Blockages already reported by publishers
This is not just theoretical. Several publishers reported as early as August 2026 that Cloudflare’s AI blocking was returning 403 errors to Googlebot, as noted by Playwire and RankX AI. Some administrators only notice when they see a sudden spike in crawl errors in Search Console.
Crawler | Cloudflare category | Effect of blocking on search | Effect on AI answers |
Googlebot | Search + training | Pages not crawled, so missing from Google Search | Missing from AI Overviews and AI Mode |
Google-Extended | Training | None | Content excluded from Google’s model training |
Bingbot | Search + training | Pages not crawled by Bing | Reduced visibility in AI services powered by Bing |
Applebot | Search + training | Pages not crawled by Apple | Reduced visibility in Apple’s search and AI features |
AI firewall: what are the risks for your site’s visibility?
Blocking the wrong crawlers leads to several direct consequences:
- Pages not crawled by Google or Bing, and thus excluded from classic search results
- Absence from AI Overviews and AI Mode, which are capturing a growing share of attention
- Loss of citations in ChatGPT or Perplexity if their search crawlers are blocked
- Inconsistency between robots.txt and the firewall, with bots allowed by one but blocked by the other
You will also see these losses in your stats: a sudden drop in traffic from LLMs after a configuration change should be a red flag.
Cloudflare is not the only one affected. Other CDNs, application firewalls, or WordPress plugins offer anti-AI settings, sometimes without distinguishing between training and search. The more protection layers you stack, the higher the risk of accidental blocking.
Robots.txt and the firewall are two separate mechanisms. The former tells bots which pages not to crawl, but it is just a request that some ignore. The latter technically blocks access by returning a 403 error. A bot allowed in robots.txt can still be blocked by the firewall: both layers must remain consistent.
How to check that your firewall is not blocking the right AI crawlers?
- Check your CDN’s AI settings. Identify which categories are blocked (search, agent, training) and see whether the old “Block AI bots” option is still active.
- Compare these settings with your robots.txt. Make sure no essential bot is blocked by either layer without a valid reason.
- Monitor crawl errors in Search Console. A sudden spike in 403 errors for Googlebot often signals a firewall issue.
- Analyze your server logs. Look for Googlebot, Bingbot, or Applebot user-agents and check whether they are receiving 403 errors.
- Test prompts in ChatGPT, Perplexity, or Google AI Mode. A sharp drop in citations after a configuration change indicates a block.
Prompt tracking from Search AI lets you quickly spot a drop in citations after a settings change.
For a more comprehensive diagnosis of your visibility in generative engines, a GEO audit reviews all the technical and non-technical obstacles.
To dive deeper into the technical settings that matter for being cited, check out our article on technical GEO.
Frequently Asked Questions
Does blocking training crawlers affect my Google ranking?
Not if you only block Google-Extended. However, on Cloudflare, blocking the training category also blocks Googlebot, which is classified as a multipurpose crawler. Always check what the enabled setting covers.
Can I allow Googlebot while blocking other AI crawlers?
Yes, but not with Cloudflare’s category-based blocking alone, which drags Googlebot along with training. Cloudflare planned an option to disable this rule in the security settings: check whether it is currently available. Otherwise, block training crawlers one by one (like GPTBot or ClaudeBot) rather than by category.
Do Cloudflare’s default settings block Googlebot?
They can on ad pages for new domains, new sites, and free accounts that had not changed their settings, since training is blocked by default there. And regardless of account type, any site that enabled training blocking, including via the old “Block AI bots” option, is affected.
How do I know if my site is blocked by an AI firewall?
Check Search Console for 403 errors on Googlebot, analyze your server logs for blocked requests, and test prompts in conversational tools to verify that your content is still being cited.
Should I block all AI crawlers to protect my content?
No. Blanket blocking of all AI crawlers reduces your visibility in generated answers and can affect your classic indexing. Prefer a selective approach, blocking only training crawlers if you want to protect your data, and check that this block does not include Googlebot.
About the author
Antonia
R&D & IA, CPO Search Ai
Antonia is Chief Product Officer (CPO) of Search Ai, a Generative Engine Optimization (GEO) platform for measuring, managing, and optimizing brand visibility on generative AI engines such as ChatGPT, Gemini, Perplexity, ... With over 10 years of experience as an R&D engineer, she continues to work at the intersection of R&D and business, transforming AI advancements into concrete functionalities for marketing teams.