2026-10-14
(Inspired by this really old list of problems with email spam solutions. Whenever you see someone propose or describe a solution for LLM scrapers, simply copy/paste this list, replace the bracketted underscores with Xes where they apply, and send the marked version to your interlocutor. They will become enlightened.)
You've come up with a
(_) technical (_) licence-based (_) legislative (_) market-based (_) cryptocurrency-based
idea for fighting LLM web scrapers. Maybe you've implemented it on your site, or just described it. Your idea does not work, or works only for the majority of people whilst excluding one or more minorities. For starters, it has these issues:
(_) Attackers can easily get legitimate users blacklisted.
(_) It's too trigger-happy.
(_) It only works if barely anyone does it.
(_) It only works if everyone does it.
(_) It requires too much cooperation from scrapers.
(_) That's not how copyright works.
(_) Slop companies could simply bribe lawmakers not to write the law you want.
(_) Nobody will learn how to use your libertarian crypto techbro nonsense.
(_) It makes legitimate use annoying.
(_) It makes legitimate use impossible.
(_) It requires too much manual oversight from sysadmins.
(_) It doesn't even appreciably stop scrapers.
(_) It requires people to identify themselves.
(_) It invests too much power in a central authority.
(_) It creates a new type of fraud.
(_) Adding another MITM erodes security.
(_) It blocks archive.org.
(_) It breaks link previews.
(_) It blocks HEAD requests.
(_) It blocks search engine crawlers.
(_) The webmaster would have to pay a massive bill after an attack.
(_) Tarpits still consume significant resources.
(_) Your site is too obscure and small for scrapers to be a problem anyway.
(If applicable:) It excludes people who:
(_) Don't have IPv6.
(_) Reject proprietary software.
(_) Browse the web with JavaScript disabled.
(_) Are blind.
(_) Use proxies, like TOR or VPNs.
(_) Use some other privacy-preserving tool or setting.
(_) Have slow computers.
(_) Are unlucky enough to fall in a blocked IP range.
(_) Use a browser that isn't based on one of the 2 major engines.
(_) Can't afford to pay all the fees.
Specifically, your plan fails to account for:
(_) How poorly written some scrapers are.
(_) How well written some scrapers are.
(_) There's only one P in "scraper".
(_) Pale Moon.
(_) Browsers that aren't Firefox, Chrome, or a fork thereof.
(_) Legitimate user agents that aren't browsers.
(_) Screen readers.
(_) Legitimate POST requests with large entity bodies.
(_) Even a small amount of friction scares away a surprising amount of humans.
(_) Eternal arms race involved in all filtering approaches.
(_) Scrapers using optimised, hand-coded assembly or GPU implementations of the PoW instead of your JavaScript one.
(_) Someone benefits from the PoW computation itself, creating perverse incentives.
(_) Scrapers don't usually request the same page more than once.
(_) CGNAT.
(_) IP address spoofing.
(_) User agent spoofing.
(_) Referrer spoofing.
(_) Scrapers adding fake tracking tags to URLs.
(_) Any message intended for scrapers will be seen by legitimate users.
(_) That includes shock images.
(_) Machine learning companies don't follow laws.
(_) Machine learning companies don't always use identifiable IP address ranges.
(_) Botnets where each IP only sends a small number of requests.
(_) Pages that have a large amount of images in them.
(_) Attackers posting images where the src attribute is a link that blacklists the user.
(_) Sysadmins rushing to solve things as quickly as possible.
(_) Webmasters want captcha pages to have the same theme as the rest of their site.
(_) Proxies having shared exit nodes.
(_) Scraper traffic dying down when the slop bubble bursts.
And the following philosophical objections may also apply:
(_) You should not have to run proprietary software to access a web page.
(_) You should not have to run JavaScript to access a web page.
(_) You should not have to identify yourself to access a web page.
(_) You should not have to turn off your VPN to access a web page.
(_) You should not have to pay money to access a web page.
(_) You should not have to run a graphical environment to access a web page.
(_) You should not have to use a mainstream browser to access a web page.
(_) You should not have to be sighted to access a web page.
(_) You should not have to have a fast computer to access a web page.
(_) You should not have to provide training data to access a web page.
(_) The previous 10 also apply to registering an account.
(_) You should not have to register an account to access a web page.
(_) I don't want CloudFlare reading my traffic.
(_) People should be able to freely host their own site, on their own machine, in their own house, without using a VPS or any middleman.
(_) The corporation you suggest using is probably running scrapers of its own.
(_) Slop companies shouldn't be able to get my data in the first place, regardless of whether or not they DDoS me.
(_) People who are logged in shouldn't be rate limited.
(_) Blacklists suck.
(_) Whitelists suck.
(_) IP bans should be temporary.
(_) Polluting LLM datasets with nonsense is a much better idea than trying to keep them from getting authentic input.
(_) Not all bots should be blocked.
(_) Putting tracking tags in URLs is a threat to privacy.
(_) Killing them that way is not slow and painful enough.
Furthermore, this is what I think about you:
(_) You're well-intentioned but naive.
(_) Your stubborn insistence that this is the only solution infuriates me.
(_) Nice try, assh0le! I'm going to find out where you live and burn your house down! With the lemons!
(_) Go back to Hacker News.