Cloudflare, AI Crawlers and Googlebot: Why Small Business Websites Need Regular Checks

Most small business owners do not spend their week looking at crawler rules, bot settings, Cloudflare dashboards or technical SEO warnings.

That is completely understandable. You have customers to look after, work to deliver, staff to manage and a business to keep moving.

The problem is that some of the biggest changes affecting your website do not look like changes at all. Your website may still load. Your contact form may still work. Your homepage may still look exactly the same. But behind the scenes, a setting designed to control AI bots could affect whether search engines are able to crawl your site properly.

That is why Cloudflare’s latest AI crawler announcement is worth paying attention to.

What has Cloudflare announced?

Cloudflare has announced new tools and classifications to give website owners more control over how AI companies, bots and automated agents access their content. Its message is simple: your content, your rules.

According to Cloudflare, automated agents and bots now drive more than half of all web requests, which shows just how much of the modern internet is no longer made up of normal human visitors clicking around websites.

Cloudflare is also separating crawler behaviour into different categories, including search, agent use and training. In plain English, that means website owners can make more informed choices about whether a bot is visiting to help people find content through search, acting on behalf of a user, or gathering content to train an AI model.

That sounds useful, and for many website owners it will be. Businesses should have more say over how their content is used.

But there is a catch.

The Googlebot problem

Search Engine Journal has highlighted an important issue with the way some mixed-purpose crawlers may be handled. Some bots are not neatly limited to one purpose. A crawler might be involved in search, but also connected with AI training or other AI-related use.

Under Cloudflare’s stricter approach, if a website blocks training, mixed-purpose crawlers may also be blocked. Search Engine Journal names Googlebot, Applebot and Bingbot as examples of crawlers that could be affected if they are treated as carrying out more than one role.

For a small business website, that is the bit that matters.

You might reasonably think, “I do not want AI companies training on my website content.” But if the setting used to do that also blocks Googlebot, your site may become harder for Google to crawl.

That does not mean your website will disappear overnight. It does not mean every Cloudflare user is suddenly in trouble. It does mean that crawler settings now need a bit more care, especially for businesses that rely on Google search for enquiries.

Why this matters for small businesses

A large publisher may have a legal team, an SEO department, an in-house developer and someone whose job is to keep an eye on these changes.

Most small businesses do not.

For a plumber, café, consultant, care provider, salon, builder, charity, local shop or professional service business, the website is often simply there to bring in enquiries. It needs to be found, trusted and easy to use.

If Google cannot crawl your site properly, you may not notice immediately. The first sign could be fewer calls, fewer form submissions, fewer quote requests or fewer new customers saying, “I found you on Google.”

That is what makes technical SEO issues so frustrating. They often do not break a website in an obvious way. They just quietly make it less effective.

This is not just about Cloudflare

Cloudflare is in the news because it is making a specific change around AI crawler controls, with default changes due from 15 September 2026 for some customers and sites.

But the wider point is bigger than Cloudflare.

Small business websites are affected by all sorts of quiet technical changes, including:

• Search engine crawler changes
• AI crawler rules
• Security plugin settings
• Firewall rules
• Robots.txt changes
• Sitemap errors
• WordPress plugin updates
• Hosting changes
• Caching and performance settings
• Google Search Console warnings

Any one of those can affect whether your website is easy for search engines and AI tools to understand.

The website may look fine to you, while Google is seeing something very different.

Should small businesses block AI crawlers?

There is no one-size-fits-all answer.

Some businesses may want to restrict AI tools from using their content, especially if they publish original articles, guides, reports, pricing data, training material or paid resources.

Other businesses may decide that being visible in AI search and answer engines is more valuable than blocking everything. For many small businesses, discoverability matters. You may want your services, location, opening hours, expertise and helpful content to appear when someone asks an AI tool for recommendations.

The important thing is that this should be a decision, not an accident.

Cloudflare’s own announcement talks about the growth of Answer Engine Optimisation, or AEO, as businesses start looking not just at whether search engines can crawl their content, but whether AI tools cite and surface that content in answers.

That is likely to become more important over the next few years.

What should you check?

If your website uses Cloudflare, now would be a sensible time to review the settings.

The most important checks are:

1. Whether AI crawler blocking is switched on
2. Whether older “block AI bots” style rules are still active
3. Whether search crawlers such as Googlebot are still allowed
4. Whether your robots.txt file is giving the right instructions
5. Whether your sitemap is working correctly
6. Whether Google Search Console is reporting crawl or indexing problems
7. Whether traffic or enquiries have changed after technical updates

For WordPress websites, it is also worth checking security plugins, SEO plugins, caching plugins and server-level firewall settings, because those can all affect how bots and search engines access your site.

How O’Brien Media can help

At O’Brien Media, this is exactly the kind of change we keep an eye on through our website support and SEO services.

Small business owners should not have to become experts in Cloudflare, Googlebot, robots.txt files or AI crawler classifications just to make sure their website can still be found.

A technical SEO check can help identify whether your site is being crawled correctly, whether important pages are indexed, whether security settings are too aggressive, and whether recent changes may have affected your visibility.

Our website support can also help keep WordPress websites updated, monitored and checked so that issues like this are less likely to go unnoticed for months.

This is not about chasing every internet trend. It is about making sure the basics keep working as the internet changes around your business.

A quiet setting can have a big impact

Cloudflare’s “your content, your rules” approach is a good principle. Website owners should have more control over how their content is accessed and used.

But rules need checking.

For small businesses, the priority is simple: protect your website where needed, but do not accidentally block the search engines and tools that help customers find you.

If your website is important to your business, it is worth reviewing these settings before they become a problem, get in touch to find out how we can help.