Barry Warsaw wrote: > >On Aug 31, 2009, at 1:15 PM, C Nulk wrote: > >> As for using robots.txt, hmm, it is not the legitimate search >> engines I >> care about, it is the search engines/crawlers that do not respect my >> robots.txt file that I care about. If I had an effective way to >> consistently identify those non-legitimate crawlers, I would add >> what I >> needed to drop them into my firewall as I recognized them. > >Agreed.
The point in the original post about robots.txt was that if you think obfuscation is undesirable and don't do it, but you get complaints from people who find their unobfuscated addresses on your pages via legitimate search engines, you can use robots.txt to keep the search engines out. However, robots.txt is not completely effective in this. You can use it to prevent Google from crawling your site or portions thereof, but it won't prevent Google from indexing your pages that it finds via external links. To prevent this, you need a <meta name="robots" content="noindex"> tag on the pages themselves. -- Mark Sapiro <m...@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan _______________________________________________ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
