Cloudflare introduces AI audit tool to block data scraping AI bots from accessing websites

Cloudflare has unveiled AI Audit, a new suite of tools designed to give website owners and bloggers more control over their content and how it is used by automated systems. The security tool will enable users to block or allow artificial intelligence (AI) bots that frequently crawl websites to scrape data. The company is providing this tool to all its existing customers without any extra charge. Cloudflare is also adding a feature that will allow website owners to see detailed analytics about which bots are visiting the platform and what their behavior patterns are.

With the increasing popularity of generative AI technology, there is a race to train large language models (LLMs) with human-generated data. It is important not only to create a foundation model but also to further develop and improve it. However, the main issue here is access to publicly available data sources. Most AI companies have already studied these datasets, and now need more data to train AI models.

AI bots are designed to help developers gather more information to train AI models. They can be understood as systematic programs imitating a real user that can enter websites and copy text, image and video data. These AI bots can mine large amounts of data in a short period of time and feed it to AI models. In recent times, several media firms and big websites have filed lawsuits against AI firms accusing them of plagiarism and illegally using data to feed LLMs.

Cloudflare’s AI audit tool comes as a protective layer that can prevent such bots from accessing websites. The company revealed in a press release that the tool has also been improved to give users more control over which bots will be restricted and which bots will be allowed access. This is useful in cases where the platform has made a deal with an AI firm and has no objection to its bots taking the data. Alternatively, the website owner may want to give access to certain AI models that attribute the source of the data to gain better access.

Cloudflare highlighted that it is also creating a workflow where website owners can set fair prices on their content. On the other hand, AI bot owners will be able to transact with this firewall and once the amount is paid, they will be given the right to scan the content. The company highlights that its marketplace-like tool could be beneficial for users who don’t have the bandwidth or resources to interact and make deals with every AI firm that visits their website.