Crawler tarpit

A defensive endpoint that traps and slows abusive automated crawlers by serving deliberately slow, deep, or endless linked content instead of hard-blocking them.

grounded in: commit 9f3034e 'self-improve: Trap link-following harvesters with a nofollow tarpit anchor'

Connected concepts

Honeypot / decoy trap, AI crawler directives, Bot mitigation & scraping defense, Data poisoning defense, AI training crawler