At 2:43 AM -0500 12/1/01, [EMAIL PROTECTED] wrote: >where do i put robots.txt file(s)? in every url directory i want to search? >(i cannot put it in apache, et al).
Put one file in the root directory for each host (i.e. www.example.com and specialhost.example.com). That's where the the robots will look for the robots.txt file, they will ignore it anywhere else. For more info: <http://www.robotstxt.org> Avi -- Complete Guide to Search Engines for Web Sites and Intranets <http://www.searchtools.com> _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
