AI site Perplexity uses “stealth tactics” to flout no-crawl edicts, Cloudflare says.

According to a statement by Cloudflare, an American company that provides internet security and distributed DNS services, the popular search engine AI Perplexity is currently violating a series of internet regulations that have been in place for over thirty years. More specifically, Cloudflare claims that the company in question uses “stealth” bots (designed to avoid detection) and other tactics to circumvent website deindexing guidelines.

The above was communicated in a post on the company's blog; a team of researchers from the latter explained that they had received several complaints from dissatisfied customers. The latter had blocked Perplexity's scraping bots by implementing settings in the robots.txt files of their sites, but the AI company continued to have access to the content on the pages in question.

The team's research subsequently revealed the truth: Perplexity's crawlers, when faced with blocks from robots.txt files or firewall rules, were replaced by “invisible” bots specialized in masking their movements.