umami_wasabi@lemmy.mltoTechnology@lemmy.world•AI haters build tarpits to trap and trick AI scrapers that ignore robots.txtEnglish
7·
3 days agoI think to use it defensively, you should put the path into robots.txt, and only those doesn’t follows the rule will be greeted with the maze. For proper search engine crawler, that’s should be the standard behavior.
Is the worth as news? Ain’t that an open secret already?