We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
We have paused all crawling as of Feb 6th, 2025 until we implement robots.txt support. Stats will not update during this period.
You’re definitely right that I went a bit extreme with what I used as a reason against it, but I feel like the point still stands about “just ask before you slam people’s servers with yet another bot on the pile of millions of bots hitting their F2B system”