Dead internet theory is real; 50 million bots trapped in my bot pit:

Some people deploy traps on their websites for bots scraping data. This site, which is a platform for art, also has a bunch of them. These traps makes the bots get trapped, while "downloading" a ton of nonsensical or incorrect data to mess with AI training. Delays are often added into these pits to make them "stickier", and seem more valuable. A common term for these bot traps is "digital tar pit." I have many better tar pits running with large traffic, but my most popular one, which we will call "Data Export", has reached 50 million loads. Let's dive in: Data Export: https://gladeart.com/data-export . This pit has fairly recently got a gzip bombing feature, which makes it send out gzip bombs during popular hours. (When it gets over 2000 RPMs it sends out gzip bombs for the rest of the minute). So just a heads up, it may send you a gzip bomb if you click on there. Shouldn't do any harm if you're using a modern browser though, but still have caution about memory usage. Anyways, more about gzip bombs some other day. :) > What does 50 million requests mean? 50 million requests, or loads, means that 50 million pages were loaded of that tar pit. Every page has different content, with practically infinite variation. Page loads which were rate-limited, or cut off before they loaded, are not counted in this number. > How do tar pits work? A common and effective design for a tar pit is to have some paragraphs of the 'poisonous' text, and then a few links below, leading deeper into the maze. Quite similar to the Glade Art Blog, actually (this page). > What company or companies do these bots belong to? The majority of bots these days seem to be for AI training, so we can narrow it down to that. We can't narrow it down further because there are a lot of middlemen when it comes to this kind of scraping; the companies behind don't want to get their hands dirty, so they use other companies for it. > Do they obey robots.txt? No, quite clearly not. Glade Art's robots.txt explicitly disallows all bots from going onto tar pit paths, and yet they go into there anyways. Quite often, bots go into there because it's disallowed, because then it must be "valuable." > What makes Data Export attractive to bots? The text generated by Data Export is a slurry of fake credentials, dates, and events. This data seems valuable and is attractive to them. Take a look at this absolutely massive slop site: https://doggydogdog.xyz:8443 . It's another one of my tar pits, this one a "fake Github" sort of site. The entire website is just a tar pit, with no more than a few dozen human visitors per month. However, it is actually a high-traffic site because it has millions of monthly "visitors," those visitors being bots of course. Sounds like the dead internet theory, right? Yeah. > What percentage of internet traffic belongs to bots? Some studies say 53%. However, those strongly rely on just IP lookup; to see if the requests is from a datacenter or not. But, as said in other Glade Art blog articles, there are bot swarms which use hundreds of millions of residential IPs, getting past this IP lookup. Personally, I believe that the percentage of bot traffic on the internet is at least 70%. It very likely may be higher, but there are quite a few humans who use the internet too, which makes this my estimate. If you would like a more technical article with log files for download and stuff, check out the other articles on the Glade Art Blog. So yeah, remember when it hit 1 million requests? Then 10 million? Well now that's 50 million, and growing. The Data Export tar pit has been getting about a million requests every single day lately, so further growth is likely. 100 million? 1 billion? And besides, we have plenty of other pits out there. Stay tuned!