AI crawler directives

Machine-readable consent files (robots.txt, and emerging AI-specific equivalents like llms.txt) that declare which crawlers and AI scrapers may access a site's content.

grounded in: Commit 43abfb1 'Gloss robots.txt as the site's honest scraping-defense line', combined with the trends theme 'Scraping & bot-traffic arms race: residential proxies and scrapers reshaping who gets to a