Cloudflare, a leading content delivery network, is pushing for changes in Google’s artificial intelligence (AI) search crawling methods. According to Cloudflare CEO Matthew Prince, the company wants Google to provide ways to block Answer Box and AI Overview without blocking classic search indexing. This move aims to improve integration and transparency in AI-driven search results.
Currently, Google’s AI-powered search crawlers are not easily distinguishable from traditional search indexing methods. As a result, 70% of searches may be affected by the lack of transparency in AI-driven search results. Cloudflare is seeking ways to separate these crawlers for better integration and user experience.
If negotiations with Google fail, Prince has threatened to push for legislation requiring the tech giant to break out its crawlers and announce routes. This move would significantly impact the way users interact with search engines and could lead to a more transparent online environment.
Brad Stephenson
@Shuttlecock
·
Jul 9 Every other site I’ve visited this past week seems to have activated the Cloudflare gateway to make sure I’m a human visitor.
AI data crawlers putting a strain on web servers?
Dane Knecht
@dok2001
·
Jun 12 We let down our customers at
@Cloudflare
today. Our Workers KV service failed and the downstream products that rely on that service had outages of their own. We will publish a full postmortem soon.
I know that these kinds of incidents have real and serious impact for teams
Wes Bos
@wesbos
·
Jun 12 2.5 hour cloudflare outage today was due to KV going down, which many of their products are built on.
KV went down because of an upstream vendor. Is this google as they were down at the same time?
Kalicube
@TeamKalicube
·
Jul 8 Cloudflare now blocks #AI bots by default & if you haven’t opted in, you may be invisible to the very systems shaping the future of brand discovery.
Take back control—before AI locks you out >>
https://kalicube.com/learning-spaces/faq-list/generative-ai/cloudflare-blocks-ai-bots-from-scraping-web-content-why-that-is-a-huge-loss-for-digital-marketing/…
#Kalicube #AIAssistiveEngines #Cloudflare #AIOptimization
frank
@solvingfrank
·
Jul 8 yeah they had some big issues 4 hours ago, rest api server went down temporarily
Jon Stokes
@jonst0kes
·
Jul 9 Browser agents are actually NOT why they’re doing this — or why Perplexity has a browser now. Every one of these AI company browsers is about one thing & one thing only: web scraping.
Cloudflare has shut down their ability to get tokens off the open web. Hence these browsers.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
