-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 19/09/12 06:36, grarpamp wrote:
>> People use robots.txt to indicate that they don't want their site >> to be added to indexes. > > They use it to indicate that they don't want their site to be > crawled. In almost all cases (99% or higher), robots.txt is used to indicate that a site shouldn't be crawled, *because* they don't want it to be indexed. The intention is painfully clear... - -- Mike Cardwell https://grepular.com/ http://cardwellit.com/ OpenPGP Key 35BC AF1D 3AA2 1F84 3DC3 B0CF 70A5 F512 0018 461F XMPP OTR Key 8924 B06A 7917 AAF3 DBB1 BF1B 295C 3C78 3EF1 46B4 -----BEGIN PGP SIGNATURE----- iQGGBAEBCgBwBQJQWX7AMBSAAAAAACAAB3ByZWZlcnJlZC1lbWFpbC1lbmNvZGlu Z0BwZ3AuY29tcGdwbWltZTgUgAAAAAAVABpwa2EtYWRkcmVzc0BnbnVwZy5vcmdt aWtlLmNhcmR3ZWxsQGdyZXB1bGFyLmNvbQAKCRCdJiMBwdHnBOh6B/0bF8fGP3xt UmflWZ3wCK0dBj0AQyQpdvrbqI6yHEVlzvs1M3qKtAaSInuz/Bb/tFOsd8JVhoEG /bu+YsnOwk5qEifywe34EFjFcUIeLu+badF+O57aCJoFdu4YcK4k7/7ZdmxAvpj8 9U73BywJO2HM3fe2pdMeEf0eNZUkOkjFJ2DvpaiRU4T/rv5x/zP1HrsEyRaHsLVp pbZTSb8QEgNJDUJ+r5+1BQVgKX1lw84Qp+MOD1VfyzqZImdu4ah4y3zrEhSweyLa qlNDqs1BZot7rIJhJKrJIuGSWnyWj1DWSkv9cyXmhwYEgDcK8K1+sDrrVGoQdSBq 7ms4gtZhioaH =btT8 -----END PGP SIGNATURE----- _______________________________________________ tor-talk mailing list tor-talk@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-talk