Goes deeper: the 15 September crawler change, in full
Check your own robots.txt first, then your raw access logs. On 6 August 2026 our log showed Meta's crawler making 1,600 requests in 24 hours, Googlebot 4, GPTBot 1, and ClaudeBot, PerplexityBot and OAI-SearchBot none.
How do I know if AI can read my website?
Open your own robots.txt in a browser, type your domain followed by /robots.txt, and read it. If you see a Disallow line under GPTBot, ClaudeBot, CCBot, PerplexityBot or Google-Extended, you are blocking AI. That check takes ten seconds and most owners have never done it.
The second check is your raw access logs, which is the only place you see what actually arrived rather than what you permitted. We ran ours on 6 August 2026 and the numbers are worth sitting with. In the previous 24 hours, Meta's crawler made 1,600 requests and transferred 1.17 MB. Googlebot made 4 requests. GPTBot made 1. ClaudeBot, PerplexityBot and OAI-SearchBot made none at all.
Our settings allow every AI crawler and our robots.txt names twelve of them with an explicit Allow, so nothing was being turned away. Permission is not traffic. Meta crawls hard, Google crawls efficiently because it already has us, and the assistants people quote at you in conversation barely visit. Anyone who tells you a robots.txt edit will get you crawled by ChatGPT this week is guessing.
Where would a block be hiding?
On a Bulgarian small business site, almost always in WordPress. W3Techs data dated 16 June 2026 puts WordPress on 70.7 percent of .bg sites whose CMS it can detect, which makes Yoast, Rank Math and Wordfence the likeliest hiding places for a rule you never wrote.
Yoast and Rank Math both generate a virtual robots.txt, so the file you are reading in the browser may not exist on disk anywhere. Editing it means going into the plugin, not the file manager. Wordfence is the other common one, because its blocking and rate limiting rules can throttle or refuse a crawler on user agent without ever appearing in robots.txt. A crawler that is rate limited into failure looks, from the outside, exactly like a site that has nothing to say.
The third place is .htaccess, where a developer or a previous agency may have pasted a bot blocking snippet years ago. Those snippets get copied around forums, they name whatever bots were annoying at the time, and nobody ever removes them.
Does this affect me if I am not on Cloudflare?
Yes, and probably more than you think, because if you are not behind Cloudflare then nobody is filtering bots on your behalf and nobody is publishing what happens either. On Bulgarian shared hosting the block, if there is one, is either in your own site or invisible in your host's network configuration.
W3Techs data for the .bg top level domain dated 8 August 2026 shows team.blue holding 42.6 percent of detected .bg hosting, Delta.bg at 6.4 percent and Hetzner at 3.6 percent. Read that as a share of the sites W3Techs can detect, not as audited market share, because W3Techs measures what it can fingerprint and nothing else.
That team.blue number matters because it is more consolidated than it looks. SuperHosting.bg, Host.bg and ICN.bg are now all part of team.blue: SuperHosting acquired Host.bg in 2017, joined team.blue in 2020, and acquired ICN.bg's hosting business in 2021. So three names a Bulgarian owner thinks of as separate suppliers sit under one group.
SuperHosting places no CDN or proxy in front of customer sites by default, and its own help page states that cPanel based Cloudflare activation is no longer available on their servers. That is genuinely useful for AI crawling: nothing is intercepting the request. But their help documentation says nothing at all about AI crawlers. Their robots.txt and bot articles were last updated in 2022, and the bad bots they name are Surfbot, ChinaClaw and Zeus. No GPTBot, no ClaudeBot, no Google-Extended, because none of those were a topic in 2022.
Jump.bg is the more interesting case. It blocks grey bots at network level by default, names SemrushBot, Ahrefs, DotMoz and Majestic SEO, and confirms that Googlebot and Bingbot are not blocked. That page was updated on 24 April 2026, so it is current. What it does not say, either way, is anything about AI crawlers. A Jump.bg customer cannot tell from published documentation whether GPTBot reaches their site.
This is our honest limit, and it is the reason the article exists. We cannot tell you what your host blocks, because most Bulgarian hosts do not publish it. We can tell you exactly which question to ask and how to read the answer.
What should I ask my hosting company?
Ask two things in writing, by email or ticket, so you have the reply on record. First: do you block any bots or crawlers at server or network level, and can I see the list. Second: are GPTBot, ClaudeBot and PerplexityBot on that list.
Ask in writing rather than on the phone, because a support agent saying "no, everything is open" in a chat window is worth nothing in three months when you are trying to work out why nothing changed. A ticket reply is a record you can hand to whoever fixes it next.
If the answer is that they block grey bots by default and the agent cannot say whether AI crawlers are included, that is not a dishonest answer, it is an undocumented one, and it is your cue to ask them to check with the network team rather than to accept it.
What is the Cloudflare change on 15 September?
From 15 September 2026 Cloudflare sets new defaults: Training and Agent crawlers are blocked by default on pages that display ads, while Search crawlers are still allowed. Per Cloudflare's press release of 1 July 2026, this applies to existing free customers who have not changed their settings, not only to new sign ups.
Cloudflare appears on 93.6 percent of .bg sites that use a reverse proxy at all, and that caveat is not decoration. It does not mean 93.6 percent of Bulgarian sites use Cloudflare. It means that among the minority of .bg sites sitting behind a reverse proxy, almost all of them chose Cloudflare. Anyone quoting the 93.6 percent as national adoption is wrong, and you will see it quoted that way.
Then there is the trap, which is the part worth writing down. Cloudflare treats Googlebot as a multi purpose crawler, and it resolves defaults to the most restrictive matching rule. Cloudflare's own dashboard says of the Training setting, and we quote it verbatim: "This option will block all Training crawlers, even those who use the same bot for Search." So ticking block training also blocks Google. If you are behind Cloudflare and you have been told to block AI training to protect your content, understand that you may be paying for it with your Google visibility.
The checklist
Work through this in order. It costs you an hour and it is the difference between assuming you are visible and knowing.
- Open your own robots.txt in a browser and look for Disallow lines naming GPTBot, ClaudeBot, CCBot, PerplexityBot or Google-Extended.
- If robots.txt is plugin generated, edit it in Yoast under Tools and File editor, or in Rank Math under General Settings.
- Check .htaccess in cPanel File Manager with hidden files shown, and look for user agent rules.
- Check Wordfence under Blocking and under Rate limiting.
- Email your host and ask in writing whether they block any bots at server level, and whether GPTBot, ClaudeBot and PerplexityBot are on that list.
- Check your raw access logs for AI user agents, so you know what is actually arriving.
If the pages themselves are the problem rather than the permissions, the fix is a build question, and we wrote up how we did our own in how we built our website for AI search. If you would rather someone just went and looked, that is part of the web design work.
Sources
- W3Techs, .bg top level domain hosting shares, dated 8 August 2026: team.blue 42.6 percent, Delta.bg 6.4 percent, Hetzner 3.6 percent, measured as a share of sites W3Techs can detect.
- W3Techs, .bg content management systems, dated 16 June 2026: WordPress on 70.7 percent of .bg sites whose CMS is detected.
- W3Techs, .bg reverse proxy services: Cloudflare on 93.6 percent of .bg sites that use a reverse proxy at all.
- SuperHosting.bg help documentation on robots.txt and bad bots, last updated 2022, naming Surfbot, ChinaClaw and Zeus, and its help page stating that cPanel based Cloudflare activation is no longer available on their servers.
- Jump.bg documentation, updated 24 April 2026, on network level blocking of SemrushBot, Ahrefs, DotMoz and Majestic SEO, and confirming Googlebot and Bingbot are not blocked.
- Cloudflare press release, 1 July 2026, on new default settings from 15 September 2026, and the Cloudflare dashboard text for the Training setting.
- Wonder Works Design first party crawler log, 6 August 2026, previous 24 hours.