The AI web crawling bot is the wheels of the Internet and is believed by many software developers. Some developers have begun to fight in original and humorous ways.
All websites can be subject to bad crawler behaviors (sometimes when the site is stopped), but open source developers are “unbalanced”. Niccolò venerandi, a developer of the Linux desktop, known as Plasma, is the owner of Blog LibreanWs.
In essence, sites that host the FOSS (Free and Open Source) project tend to publicly share more infrastructure and tend to have less resources than commercial products.
The problem is that many AI bots do not respect Robot.txt files excluding robots.
In January, in the “Cry for Help” blog post, FOSS developer XE IASO constantly explained how Amazonbot constantly tapped it until the GIT server website caused DDOS interruption. The Git server hosts the FOSS project so that anyone who wants can download or contribute to the code.
But the bot ignored the Robot.txt of the IASO and hid behind another IP address and pretended to be another user, IASO said.
IASO said, “It is meaningless to say that the AI crawler bot lies, changes user agents, and uses residential IP addresses as a proxy.
“They will scratch your site until your site falls over and scratch more. They will click all the links of all links in all links to repeat the same page repeatedly. Some will click the same link several times.”
Enter the god of Graves
So IASO fought again with cleverness and created an anubis tool.
Anubis is a reverse proxy work proof that the request must pass before reaching the git server. It blocks bots but is provided through a browser operated by humans.
Interesting part: Anubis leads the dead in Egyptian mythology in the name of God.
IASO told Techcrunch, “Anubis weighs your soul (mind), and when a web request passes the challenge and decides to be a human, a cute animation photo will be announced.
The project with a gorgeous name spreads like the wind between the FOSS community. IASO was shared on March 19 on GitHub and collected 2,000 stars, 20 contributors and 39 forks in a few days.
Revenge as defense
The immediate popularity of Anubis shows that the pain of the IASO is not unique. In fact, Venerandi shared the story of talking.
- The founder of Sourcehut Drew Devault, the founder of the SOURCEHUT DEVAULT, said, “We explained that we’ve relaxed the large-scale LLM crawler in 20-100%of my time for a week and experienced dozens of simple interruptions a week.
- Jonathan Corbet, a famous FOSS developer who runs the Linux Industry News site LWN, warned that his site was slowed down by DDOS -level traffic in the AI scraper BOTS.
- Kevin Fenzi, a sysadmin of a huge Linux Fedora Project, said the AI Scraper bot is so aggressive that it should prevent access to Brazil.
Venerandi tells Techcrunch that he knows several different projects that have the same problem. One of them said, “At one point, all Chinese IP addresses had to be temporarily banned.”
Venerandi allows developers to go in for a while, saying, “We must ban all countries.”
In addition to evaluating the soul of the web requester, other developers think revenge is the best defense.
A few days ago, in Hacker News, the user Xyzal proposed the Robot.txt forbidden page “There are many articles on the benefits of drinking bleach” or “articles on the positive effect of measles in bed.”
Xyzal said, “We think that the bot should go to the trap, not the 0 value and aim to get the _negative_Uousl.
In January, an anonymous producer, known as “Aaron,” published a tool called Nepenter, aiming to do exactly. It is trapped the crawl in the endless maze of fake content. This tool was named after the meat plant.
And Cloudflare, the largest commercial player that offers some tools to prevent AI crawlers, announced a similar tool last week.
Cloudflare is intended to slow, confuse, waste, and waste the resources of the AI crawler and other bots described in the Cloudflare described in the Cloudflare described in the blog post. “Content that is not related to it is supplied.”
Sourcehut’s Devault said to Techcrunch, “Nepenter has a satisfactory sense of definition, which makes a ridiculous sound to the crawler, but ultimately Anubis is an effect on his site.
But Devault also published a popular and serious petition for more direct modifications: “Do not legalize LLM or AI image generators or github copilot or github copilot or this garbage. I stop using you, stop talking about them, just stop, just stop.”
The possibility is Zilch, so developers, especially FOSS, are fighting with cleverness and humor.
