Brief · Both
The CFAA double standard: researchers get raids, AI labs get funding rounds
Poke at a website the owner didn't invite you to, and you might meet the Computer Fraud and Abuse Act. Point a fleet of bots at the whole internet, ignore the "please don't" signs, and you get a Series F. We are truly in the dumbest timeline.
The researcher side
The CFAA (18 U.S.C. § 1030) makes it a crime to access a computer "without authorization" or to "exceed authorized access." Security people have been nervous about those words for decades, and with reason.
In 2010 Andrew "weev" Auernheimer collected iPad users' email addresses from a public AT&T web page. He was convicted under the CFAA. The Third Circuit vacated it in 2014, but on venue, not because anyone decided pulling data off a public URL was fine.
In 2021 a St. Louis Post-Dispatch reporter found teacher Social Security numbers sitting in the HTML source of a Missouri state website. He told the state and waited for a fix before he published. The governor called him a hacker and asked for a criminal investigation. That was a state law, not the CFAA, but it's the same reflex. The prosecutor declined to charge four months later. He pressed View Source. Four months.
The honest part
The "pentesters get thrown in jail" line is weaker than it was. Credit where due:
- Van Buren v. United States (2021). The Supreme Court held 6-3 that "exceeds authorized access" means getting into files or areas that are off limits to you, not using allowed access for a bad reason. Gates up or gates down.
- DOJ's May 2022 policy. Federal prosecutors should decline to charge good-faith security research. It's a policy, not a statute. It doesn't cover civil suits or state laws, and good faith is whatever a prosecutor decides it is.
- hiQ v. LinkedIn (2022). The Ninth Circuit said scraping public pages probably isn't access "without authorization," even after a cease-and-desist. That was at the preliminary injunction stage, and hiQ still lost later on contract claims.
So scraping a public page is mostly not a federal crime, and a good-faith researcher mostly won't get charged. Fine. The gap is the "mostly." The researcher still lives with the threat. The lab gets the benefit of the doubt every time.
The lab side
- Reddit v. Anthropic. Reddit sued in June 2025. It says Anthropic's bots hit the site more than 100,000 times after Anthropic said it had stopped, and that this broke the User Agreement and robots.txt. Anthropic disputes it. Note what Reddit actually sued over: contract and unfair competition, not hacking.
- Perplexity and Cloudflare. Cloudflare reported that when Perplexity's declared crawlers got blocked, traffic showed up from an undeclared agent pretending to be Chrome on a Mac, from IPs outside Perplexity's published ranges. Perplexity says that was user-driven fetching. Ok.
Rotating IPs and faking your user agent to get past a block is what we in the industry call evasion. A pentester who did that outside the scope would get a very awkward phone call. A lab calls it product.
Here's the thing
I don't want new AI laws. I don't want a doom commission. Do these people know you can just apply the laws we already have?
Same rules, same standard. If getting around a technical block is a gate under Van Buren when a researcher does it, it's a gate when a crawler does it. If ToS and robots.txt don't count, they don't count for anybody, and we should stop scaring researchers with them. Pick one. The only position that doesn't hold up is the one we have now, where the law has teeth for the individual and goes soft at scale.
And no, I'm not waiting for the courts to save you. Security is also your job. robots.txt is a polite request, not a control.
What to do: Treat AI crawlers like any other untrusted traffic. Rate-limit, block by behavior and not just user agent, put anything that matters behind auth, and log enough to prove what hit you if you ever need to sue.
Sources
- 18 U.S.C. § 1030 (Cornell LII)
- Van Buren v. United States, opinion (June 3, 2021)
- DOJ Justice Manual 9-48.000, CFAA charging policy
- hiQ Labs v. LinkedIn, Ninth Circuit (Apr. 18, 2022)
- Farella Braun + Martel: what later hiQ rulings say about scraping
- United States v. Auernheimer, Third Circuit (Apr. 11, 2014)
- NPR: Missouri vows to prosecute journalist who found security flaw (Oct. 14, 2021)
- STLPR: Prosecutor found no criminal intent by reporter (Feb. 2022)
- The Verge: Reddit sues Anthropic (June 2025)
- Cloudflare: Perplexity is using stealth, undeclared crawlers (Aug. 2025)