Proof of Work Challenges Are Actually Very Effective Against Bots. Here Is Some Data Showing It:

As stated in other blog article(s), Glade Art has many tar pits. We will assume that you know what a digital tar pit is in this article. If you don't, then this post can help explain: https://gladeart.com/blog/the-bot-situation-on-the-internet-is-actually-worse-than-you-could-imagine-heres-why These "tar pits" attract bot swarms by the millions, so their requests provide some useful information. In this post we will be talking about bots solving PoW (proof of work) challenges in particular. And so, let's begin. Some quick notes: * A PoW challenge is often a SHA 256 JS challenge similar to HashCash. Here we are experimenting with Anubis in particular: https://anubis.techaro.lol/ * We are going to be analyzing the logs from the /data-export tar pit, all parts of its link tree, and their subdomains. | What do you mean by link tree? gladeart.com/data-export has a bunch of other links which use the same generator, but have different links. For example, instead of /data-export/7r8787, it could be /archive/49847, /files/4757r8, or other. * On these logs, Anubis was at difficult 4. * The last time logs were shared, there was some confusion regarding the IPs. In the US, public IPs are considered public information. Sharing them for the purpose of illegal activity is of course illegal, but we are just analyzing bot swarm patterns in a tar pit here. * The tar pit has some fairly heavy server-side delays which cause many bots to time out. The logs are not from the Anubis or Nginx layer; they are from the server application itself. This makes 499s and stuff to not show in them; only clean, successfully loaded serves are logged. * The link branch /download has Anubis enabled, while the rest have nothing, and no JS requirements. /download is included as a link at the bottom of the tar pit just as any other link branch. * The /data-export tar pit has over 9 million requests, so for ease of access we will be sharing only the latest 1 million lines. (Each line is a request). The other 8 million lines bring very similar data. Download the log here: https://mega.nz/file/C0AD2I5L#flcKbbBviMAwC10s0cvWX5drJd6x8Jto9rswh9LsWhQ And so, let's take a look at the logs. Doing a search in the 1m log file, we can see 5 requests for /download (all from the same IP). Doing a search in the full, 9 million request logs we get 19 requests to /download. Only 19 out of 9 million requests just because Anubis was enabled there. And out of those 19 it is fully possible that they were mostly just curious humans who got redirected to the /download page from /data-export. So we can clearly see that Anubis did a great job at keeping them out. So the question is, are actual PoW challenges required, or is just a simple JavaScript requirement sufficient? Well for most use cases, a simple JS requirement through Nginx or however you want to implement it should be sufficient. However, we can see with the good search engine crawler "googlebot" that it executes JS but doesn't solve challenges. So it all depends on what protection level you want. Why was Anubis so effective? Bot swarms operate at such massive scales, that almost none of them execute JS, let alone solve PoW challenges. So yeah, we see that PoW challenges are quite effective as anti-bot measures. While I wouldn't recommend using something such as Altcha over a traditional captcha puzzle like hCaptcha for something sensitive such a registration endpoint, a PoW challenge here and there does a good job, while being very clean and more UX-friendly. Have a swell day and thanks for reading!