------- Original Message ------- On Friday, August 18th, 2023 at 12:35, Paul Koning via cctalk <cctalk@classiccmp.org> wrote:
> Really? It would be interesting to have evidence supporting that, because if > so, they > could be subjected to pain for violating an explicit order not to do so. There are some of us elsewhere on the Net (in Fedi, if you're around) who, for various reasons are pushing back against the big G and Bing due to the generally lousy state of search these days, and so dropped their crawlers into robots.txt (per those search engines' documented entries for said file) to tell their crawlers to go away. It was subsequently discovered that their crawlers (Google's for sure, Bing's less so) spidered and indexed new stuff on those sites anyway. Said new stuff still comes up in search results, just without text summaries. So, we are now looking into iptables and pf rules for blocking their crawlers. That's the extent of what I know right now, because my day job hasn't afforded me as much continuous time to devote to the discourse. The Doctor [412/724/301/703/415/510] WWW: https://drwho.virtadpt.net/ Don't be mean. You don't have to be mean.
