Cloudflare's Control Over AI Bots and Content Monetization

Cloudflare empowers content creators to control bot access and monetize their content used by AI models.

### Control Over Content Scraping: Cloudflare's New Measures for Website Owners
* 🌐 Approximately **50% of website traffic** consists of automated bots, with AI bots representing around **25% of that total**, highlighting the **prevalence of content scraping**.
* 🔍 Cloudflare has developed advanced technology to **distinguish between human visitors and bot traffic**, allowing website owners to **monitor how their content is accessed**.
* 🛠 The new features enable users to **control which bots can access their sites**, giving them the option to **block all or specific bots** from scraping their content.
* 💰 A significant initiative announced involves creating a **content marketplace** where content creators can receive **compensation for their work** that aids in **training large language models**.
* 📊 Cloudflare currently serves over **20% of the web**, making these tools accessible for free to all its customers, particularly benefiting **large media companies overwhelmed by bot activity**.
* 🇧🇷 In response to recent events in Brazil, Cloudflare clarified that it did not work directly with Brazilian authorities to make the platform formerly known as Twitter accessible, attributing the situation to a **simple coincidence related to IP address changes**.
* 📅 The timing of the events surrounding the transition was unexpected for both Cloudflare and the Brazilian authorities, emphasizing that sometimes **outcomes are merely coincidental**.
* 🔄 Ultimately, the ability for Brazilian authorities to **block content remains unchanged**, reinforcing the complexities of **internet governance and access**.

Just set the scene for us for a moment. The Bloomberg website, any website. How often are we finding its content is being touched on, used mind by large language models on a daily basis? Yeah. So for the average website, about half of the traffic that it receives is from some sort of automated bot, and the large language models, the AI bots, are growing like crazy. They're about 25% of the total bot traffic. So if you're putting content online, it's getting scraped, it's being included in these LLMs. And up until Cloudflare launched what we did today, you didn't have any control over that. And now how technically do you have control over it? What are you doing behind the scenes to make sure that it's either stopped or at least monitored? Yeah. So Cloudflare has been from its earliest days the best in the world at identifying what's a human visitor to a site and what's an automated bot or

a bot visitor to a site. So what we've done is taken that same technology and then classified all of the bots that are being used by these AI companies. And then the first step is giving anyone with a website, anyone with content visibility into how their content is being accessed by these bots. The second thing we do is we let you pick and choose either I don't want any of these bots to access my site or maybe I don't want certain of these bots access the site. Once those two things are in place, the last step, which we've announced today but are going to be rolling out over the months ahead, essentially a content marketplace where we can return the quid pro quo, where content creators can get actually paid for the content they're there creating that's helping train these large language models. Matthew, in the first instance who's signed up for this list to the early customers are. Well. So this is available for free for every

single Cloudflare customer. And Cloudflare sits in front of more than 20% of the Web. So, a huge percentage of the Web is using this. What we've heard, though, is from our largest customers, large media companies that are creating large amounts of content that they can't keep up with these bots. And so, Cloudflare is in a unique position to be able to say, Listen, if you're scraping this content, if you're getting value from this content as an AI company, you should be compensating the content creators. And yes, the large companies like Reddit are able to get these deals done, but we want to bring that to anyone who's creating content anywhere on the Web. Matthew, I want to go to the recent news of last week. We covered it from the ground in Brazil that Xe, the platform formerly known as Twitter, used your technology to briefly make the service accessible again in

that country. My specific question is Brazilian authorities have said that Cloudflare, your company, have worked with the government to understand how they did it. Why did you work with the Brazilian authorities and what did that look like? To be honest, I don't know what the Brazilian authorities are talking about because we didn't specifically work with them to block X or to make X available in Brazil. But I would analogize this to everything that's on the Web. At the end of the day, has a number associated with it, which is the IP address. It's sort of like your telephone number. And you can imagine in the days before you could port your telephone number if you switched from one cell phone provider to another cell phone provider, your number would switch. Same thing happened with X. We want to deal with them. Where they started using our service, stopped using a competitor of ours, and in the process, the IP address switched. Brazil was simply blocking the old IP.

address that made them temporarily available. Brazil was then able to block the new IP address, which was accessible. There was nothing that Cloudflare did to facilitate that. There was nothing that X did to make that harder on Brazil. It was simply a coincidence of us winning an enterprise customer. The timing is a coincidence though. Were they one worldwide or just in Brazil? How much was this a one-off transition? No, it was across the country. And again, you know, sometimes things are just coincidences. But in this case, obviously, I think they were surprised. We were surprised by the attention. And again, there was nothing that X asked us to do in terms of eliminating the ability for Brazil to block the content inside of Brazil. And there was nothing that we did to further facilitate the ability for Brazil to block what they were doing. And today, my understanding is that

Cloudflare CEO: We Did Not Facilitate X Skirt Brazil Ban

Welcome to the Bloomberg Technology YouTube channel. Subscribe for original series and mini-docs on game-changing technologies, profiles on innovators, live conversations with experts, and breaking technology news & analysis from Bloomberg TV.

Bloomberg Technology

Control Over AI Bot Traffic: Cloudflare's New Initiative

Automated bots account for about half of website traffic, with AI bots making up 25% of that.

Cloudflare is using its technology to classify and monitor these bots, giving website owners visibility and control over their access.

The company is also launching a content marketplace to offer payment to content creators for their work in training large language models.

Cloudflare's Role in Content Creation and Recent Events in Brazil

Cloudflare serves over 20% of the Web and is addressing the issue of content scraping by AI companies, aiming to help content creators get compensated.

The platform's technology was used by Xe (formerly Twitter) to briefly restore service in Brazil, leading to confusion about Cloudflare's involvement with the Brazilian authorities.

The IP address switch during Xe's transition to Cloudflare's service is what led to the blocking of the old IP in Brazil.

Cloudflare's new IP address, which Brazil was able to block, was a result of winning an enterprise customer, making the timing a coincidence.

The transition was not a one-off, as it occurred across the country.

Both Cloudflare and X were surprised by the attention, and there was no collaboration between them to influence Brazil's ability to block content.

Access has been blocked in Brazil once again.

This recurring issue highlights ongoing challenges with internet accessibility in the country.

The situation raises concerns about the impact on users and businesses relying on online services.