>>>>> "Mary" == Mary Gardiner <[EMAIL PROTECTED]> writes:
Mary> On Thu, Oct 23, 2008, Tony Sceats wrote: >> Can't you use robots.txt (or the modern equiv, is there anything >> newer actually?) to stop mass indexing, perhaps point it to pages >> you want indexed and also tell it to exclude images etc etc? Mary> As I understand it, robots.txt is still the way to do Mary> this. Slurp also has a specific extension by which you should be Mary> able to suggest it crawls less often: Mary> http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html That's what I figured. As the content is divided mostly between a WIKI and some mailing list archives (both of which change fairly often), and our audience is mostly Linux hackers, I'm going to use robots.txt to stop yahoo and msn from indexing. Yahoo gave us ~130 hits last month, compared to ~32000 from Google; MSN gave us 4 hits. -- Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au http://www.ertos.nicta.com.au ERTOS within National ICT Australia -- SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/ Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
