I know I should read about Robots from the Robot FAQ web site. However, I am a little pressed for time right now. What do I need to web sites to stop Robots reading my web sites I maintain? Thank you.
Create a text file and name it 'robots.txt'
Paste the following code, save and upload to the root directory of each site you want to ban search engines from indexing:
User-agent: * Disallow: /
The first line identifies the search engine spiders (user-agents) the directive applies to (all)
The second line specifies the directories and files to be excluded, (all-from the root directory)
-- Andy Kirkwood | Creative Director
MOTIVE | web.design.integrity http://www.motive.co.nz/ ph: +64 4 3 800 800 fx: +64 4 970 9693 mob: 021 369 693 93 Rintoul St, Newtown PO Box 7150, Wellington South, New Zealand ****************************************************** The discussion list for http://webstandardsgroup.org/
See http://webstandardsgroup.org/mail/guidelines.cfm for some hints on posting to the list & getting help ******************************************************
