--- title: "Is AI Blocked From Your Website? How to Check" description: "The Bulgarian hosts we checked don't say if they block GPTBot. How to check your own site, what to ask your host, and what Cloudflare changed on 15 September." url: https://wonderworksdesign.net/worth-knowing/is-ai-blocked-from-your-website language: en bulgarian: https://wonderworksdesign.net/bg/worth-knowing/is-ai-blocked-from-your-website updated: 2026-10-03 publisher: Wonder Works (Wonder Works Design Ltd), Blagoevgrad, Bulgaria --- [Worth Knowing](https://wonderworksdesign.net/worth-knowing) # Something Might Be Blocking AI From Your Website And You Cannot See It By [Colin Rooney](https://wonderworksdesign.net/authors/colin-rooney) · Published 6 August 2026 · Updated 3 October 2026 · 8 min read **Check your own robots.txt first, then your raw access logs. On 6 August 2026 our log showed Meta's crawler making 1,600 requests in 24 hours, Googlebot 4, GPTBot 1, and ClaudeBot, PerplexityBot and OAI-SearchBot none.** In this piece 1. How do I know if AI can read my website? 2. Where would a block be hiding? 3. Does this affect me if I am not on Cloudflare? 4. What should I ask my hosting company? 5. What did Cloudflare change on 15 September 2026? 6. The checklist 7. Sources 8. Related reading ## How do I know if AI can read my website? Open your own robots.txt in a browser, type your domain followed by /robots.txt, and read it. If you see a Disallow line under GPTBot, ClaudeBot, CCBot, PerplexityBot or Google-Extended, you are blocking AI. That check takes ten seconds, and you may never have done it. The second check is your raw access logs, which is the only place you see what actually arrived rather than what you permitted. We ran ours on 6 August 2026 and the numbers are worth sitting with. In the previous 24 hours, Meta's crawler made 1,600 requests and transferred 1.17 MB. Googlebot made 4 requests. GPTBot made 1. ClaudeBot, PerplexityBot and OAI-SearchBot made none at all. On that day our settings allowed every AI crawler and our robots.txt named the main ones one by one with an explicit Allow, so nothing was being turned away. Permission is not traffic. Meta crawls hard, Google crawls efficiently because it already has us, and the assistants people quote at you in conversation barely visit. Anyone who tells you a robots.txt edit will get you crawled by ChatGPT this week is guessing. ## Where would a block be hiding? On a Bulgarian small business site, almost always in WordPress. W3Techs data dated 30 September 2026 puts WordPress on 70.4 percent of .bg sites whose CMS it can detect, which makes Yoast, Rank Math and Wordfence the likeliest hiding places for a rule you never wrote. Yoast and Rank Math both generate a virtual robots.txt, so the file you are reading in the browser may not exist on disk anywhere. Editing it means going into the plugin, not the file manager. Wordfence is the other common one, because its blocking and rate limiting rules can throttle or refuse a crawler on user agent without ever appearing in robots.txt. A crawler that is rate limited into failure looks, from the outside, exactly like a site that has nothing to say. The third place is .htaccess, where a developer or a previous agency may have pasted a bot blocking snippet years ago. Those snippets get copied around forums, they name whatever bots were annoying at the time, and nobody ever removes them. ## Does this affect me if I am not on Cloudflare? Yes, and probably more than you think, because if you are not behind Cloudflare then nobody is filtering bots on your behalf and nobody is publishing what happens either. On Bulgarian shared hosting the block, if there is one, is either in your own site or invisible in your host's network configuration. W3Techs data for the .bg top level domain dated 30 September 2026 shows team.blue hosting 41.6 percent of .bg sites, Delta.bg 6.3 percent and Hetzner 3.7 percent. Read that as a share of the sites W3Techs can see, not as audited market share, because W3Techs measures what it can fingerprint and nothing else. That team.blue number matters because it is more consolidated than it looks. SuperHosting.bg, Host.bg and ICN.bg now all sit inside team.blue: SuperHosting bought Host.bg in 2017 and joined team.blue in 2020, and team.blue bought ICN.bg's hosting and domain business in 2021. So three names a Bulgarian owner thinks of as separate suppliers sit under one group. SuperHosting places no CDN or proxy in front of customer sites by default, and its own help page says that switching Cloudflare on through cPanel only worked on older versions of the control panel and that its servers run the newer ones. That is genuinely useful for AI crawling: nothing is intercepting the request. But their help documentation says nothing at all about AI crawlers. Their robots.txt and bad bot articles were last updated in 2022, and the bad bots they name are Surfbot, ChinaClaw and Zeus. No GPTBot, no ClaudeBot, no Google-Extended, because none of those were a topic in 2022. Jump.bg is the more interesting case. It filters what it calls grey bots for its customers unless they switch the protection off, names SemrushBot, Ahrefs, DotMoz and Majestic SEO, and confirms that Googlebot and Bingbot are not blocked. That page was updated on 24 April 2026, so it is current. What it does not say, either way, is anything about AI crawlers. A Jump.bg customer cannot tell from published documentation whether GPTBot reaches their site. This is our honest limit. We cannot tell you what your host blocks, because the Bulgarian hosts we checked do not publish it. We can tell you exactly which question to ask and how to read the answer. ## What should I ask my hosting company? Ask two things in writing, by email or ticket, so you have the reply on record. First: do you block any bots or crawlers at server or network level, and can I see the list. Second: are GPTBot, ClaudeBot and PerplexityBot on that list. Ask in writing rather than on the phone, because a support agent saying "no, everything is open" in a chat window is worth nothing in three months when you are trying to work out why nothing changed. A ticket reply is a record you can hand to whoever fixes it next. If the answer is that they block grey bots by default and the agent cannot say whether AI crawlers are included, that is not a dishonest answer, it is an undocumented one, and it is your cue to ask them to check with the network team rather than to accept it. ## What did Cloudflare change on 15 September 2026? On 15 September 2026 Cloudflare replaced its single "Block AI bots" switch with separate controls for Search, Training and Agent crawlers, and added a setting called Disallow AI Training. Disallow AI Training writes a no training line into your robots.txt and blocks training crawlers, but keeps the search crawlers Cloudflare labels Accountable, which include Googlebot, Bingbot and Applebot, so you stay in search. Existing sites had their old settings moved across so they work the way they did before, with an old training block turned into Disallow AI Training rather than Block, and Cloudflare's own advice to existing customers was that in almost every case they need to do nothing. New domains are now offered a preset when they join: search, training and agents all allowed for a site without ads, and for a site that carries ads, training disallowed and agents blocked on the pages that show ads. Cloudflare appears on 93.7 percent of .bg sites that use a reverse proxy at all, according to W3Techs on 30 September 2026, and that caveat is not decoration. It does not mean 93.7 percent of Bulgarian sites use Cloudflare. It means that among the minority of .bg sites sitting behind a reverse proxy, almost all of them chose Cloudflare. Anyone quoting that figure as national adoption is wrong, and you will see it quoted that way. Then there is the trap, which is the part worth writing down. Googlebot is a multi purpose crawler, collecting pages for search and for AI training at once, and since 15 September Cloudflare's Block setting for Training applies to multi purpose crawlers too. In Cloudflare's own words about Googlebot, Bingbot and Applebot, "Selecting Block stops them entirely." So if you are behind Cloudflare and you have been told to block AI training to protect your content, choose Disallow AI Training rather than Block, or you may be paying for it with your Google visibility. ## The checklist Work through this in order. It costs you an hour and it is the difference between assuming you are visible and knowing. - Open your own robots.txt in a browser and look for Disallow lines naming GPTBot, ClaudeBot, CCBot, PerplexityBot or Google-Extended. - If robots.txt is plugin generated, edit it in Yoast under Tools and File editor, or in Rank Math under General Settings. - Check .htaccess in cPanel File Manager with hidden files shown, and look for user agent rules. - Check Wordfence under Blocking and under Rate limiting. - If you are on Cloudflare, check the Search, Training and Agent settings in your security settings, and make sure Training says Disallow AI Training or Allow rather than Block. - Email your host and ask in writing whether they block any bots at server level, and whether GPTBot, ClaudeBot and PerplexityBot are on that list. - Check your raw access logs for AI user agents, so you know what is actually arriving. If the pages themselves are the problem rather than the permissions, the fix is a build question, and we set out what AI needs from a page in [the ultimate guide to GEO](https://wonderworksdesign.net/worth-knowing/ultimate-guide-to-geo). If you would rather someone just went and looked, that is part of [the web design work](https://wonderworksdesign.net/services/web-design). ## Sources - [W3Techs, web hosting providers among .bg sites, 30 September 2026](https://w3techs.com/technologies/segmentation/tld-bg-/web_hosting): team.blue 41.6 percent, Delta.bg 6.3 percent, Hetzner 3.7 percent. - [W3Techs, content management systems among .bg sites, 30 September 2026](https://w3techs.com/technologies/segmentation/tld-bg-/content_management): WordPress on 70.4 percent of .bg sites whose CMS is detected. - [W3Techs, reverse proxy services among .bg sites, 30 September 2026](https://w3techs.com/technologies/segmentation/tld-bg-/proxy): Cloudflare on 93.7 percent of .bg sites whose reverse proxy is detected. - [SuperHosting.BG, "The Acquisition of Host.BG, the Whole Story", 31 October 2017](https://blog.superhosting.bg/en/host-bg.html), and [Invest Sofia, "SuperHosting.bg Joins the Leading European Hosting Group team.blue", 15 July 2020](https://investsofia.com/en/superhosting-bg-joins-belgian-team-blue-leading-european-hosting-group/). - [CEE Legal Matters, interview with ICN.bg's CEO on the sale of its hosting and domain business to team.blue, 23 June 2021](https://www.ceelegalmatters.com/bulgaria/17808-deal-5-internet-corporated-networks-ceo-julian-borislavov-on-sale-of-business). - SuperHosting.bg help pages (in Bulgarian): [robots.txt, updated 18 March 2022](https://help.superhosting.bg/robots-txt-seo.html); [crawlers and blocking bad bots, updated 12 October 2022](https://help.superhosting.bg/bots.html), naming Surfbot, ChinaClaw and Zeus; [What is Cloudflare, updated 12 August 2022](https://help.superhosting.bg/what-is-cloudflare.html), on switching Cloudflare on through cPanel. - [Jump.bg, "How We Protect You From Malicious Bots", updated 24 April 2026](https://www.jump.bg/en/blog/how-we-protect-you-from-malicious-bots): SemrushBot, Ahrefs, DotMoz and Majestic SEO filtered; Googlebot and Bingbot not blocked. - [Cloudflare changelog, "New options to manage AI traffic", 1 July 2026](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/), announcing the Search, Agent and Training controls and the 15 September date. - [Cloudflare blog, "Have it both ways: stay discoverable in search while disallowing AI training", Bryan Becker, 15 September 2026](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/): Disallow AI Training, Block applying to Googlebot, Bingbot and Applebot, migration of existing settings, presets for new domains. - [Cloudflare press release, "Cloudflare helps end the search or AI training tradeoff", 15 September 2026](https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/). - Wonder Works first party crawler log, 6 August 2026, previous 24 hours. ## Related reading - [You Can Finally See If AI Search Is Mentioning Your Business](https://wonderworksdesign.net/worth-knowing/see-if-ai-search-mentions-your-business) - [The Ultimate Guide to Generative Engine Optimization (GEO) in 2026](https://wonderworksdesign.net/worth-knowing/ultimate-guide-to-geo) - [Your Google Business Profile Is Now Your Homepage For AI](https://wonderworksdesign.net/worth-knowing/google-business-profile-ai-search) [Back to: Worth Knowing](https://wonderworksdesign.net/worth-knowing)