Perplexity, an AI (artificial intelligence) chat product available globally, has been accused of taking data from websites even after being blocked to do so. This was said by Cloudflare, a global web security services company. Cloudflare, in a blog post confirmed that Perplexity bots are crawling and taking information from websites even when the websites have blocked them from doing so. What Perplexity is doing is that it is using stealth tactics to scrape website data even from the websites have explicitly blocked the following bots - Perplexity-User and PerplexityBot from doing so. The researchers at Cloudflare were able to confirm this by creating new test domains.
Read More - Apple is Developing ChatGPT Like Product
These domains were not indexed by any of the search engines or even made publicly accessible or discoverable. The researchers even put a robots.txt file on these domains to discourage the bots from accessing any part of the website. When the researchers went to Perplexity and asked about these domains, Perplexity was able to scrape the data from the domains and produce it as if no bot was ever blocked.
Read More - Bharti Airtel Offers Free Perplexity Pro AI Subscription to All Customers
As per Cloudflare, Perplexity's bots or user agents took several steps to bypass the directions from the website. Even after the denial from the robots.txt file to scourage the website, Perplexity went ahead and took the information. In case the website has implemented a web application firewall (WAF) for blocking the bot, Perplexity starts using a generic browser agent for impersonating Google Chrome or macOS. This undeclared bot from Perplexity is also claimed to use multiple IPs not listed in the official IP range of Perplexity for tricking the website.
Cloudflare was successful in intercepting the undeclared bots from Perplexity. After that, the results from the Perplexity about these domains weren't as good.
