The file must be plural as in "robots.txt" not "robot.txt" Sample configuration: ========================== User-agent: * Disallow: /secure/ Disallow: /images/ Disallow: /db/ Disallow: /data/ Disallow: /database/ Disallow: /related/ Disallow: /reviews/ Disallow: /cfide/ Disallow: /flashcom
# don't let search engines see the RSS feed, it's just confusing. # User-agent: Googlebot User-agent: InfoNaviRobot User-agent: TV33_Mercator User-agent: AVSearch User-agent: Mercator User-agent: Scooter User-agent: Slurp User-agent: SearchengineLicenceSheep User-agent: shadow User-agent: MultiText User-agent: FAST-WebCrawler User-agent: Lycos_Spider User-agent: Atomz User-agent: htdig User-agent: spider00.logika.net User-agent: NetMechanic User-agent: libwww-perl User-agent: Teleport Pro Disallow: / # updated 2002-03-22 (disallow rtestprob links) # updated 2002-06-25 (disallow info/slides links, info/robots/) # updated 2002-07-25 (disallow /searchtools/ which is an alias) ============================================ ====================================== Stop spam on your domain, use our gateway! For hosting solutions http://www.clickdoug.com ISP rated: http://www.forta.com/cf/isp/isp.cfm?isp_id=772 ====================================== If you are not satisfied with my service, my job isn't done! ----- Original Message ----- From: "Randell B Adkins" <[EMAIL PROTECTED]> To: "CF-Community" <[EMAIL PROTECTED]> Sent: Thursday, July 17, 2003 7:43 AM Subject: Search Engine / robot.txt | Does anyone have a good resource for building a robot.txt | file for search engine bots? | | | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~| Archives: http://www.houseoffusion.com/cf_lists/index.cfm?forumid=5 Subscription: http://www.houseoffusion.com/cf_lists/index.cfm?method=subscribe&forumid=5 Signup for the Fusion Authority news alert and keep up with the latest news in ColdFusion and related topics. http://www.fusionauthority.com/signup.cfm Unsubscribe: http://www.houseoffusion.com/cf_lists/unsubscribe.cfm?user=89.70.5
