According to Geoff Hutchison: > On Thu, 6 Dec 2001 [EMAIL PROTECTED] wrote: > > I think you've answered me already, but this question was suppose to > > be directed at excluding all engines except htdig (?). > > > > There is no other way (the last response threw me a bit) to do that > > except the robots.txt -is that correct? > > That's correct. Moreover, some engines seem to ignore the robots.txt file > completely. So unless you have other means of protecting the server > (e.g. requiring authorization or using local_urls indexing), there's > little you can do to exclude other engines.
If you know the addresses or domains of specific search engines you want to keep out, you can list them in "deny" records in a .htaccess file. You'd need one of these in each user's "public_html" directory. E.g. you could put this in .htaccess: <Limit GET> deny from .altavista.com deny from .google.com </Limit> You could also set it up to "deny from all" and allow specific domains only, and with Apache you can also have it fall back to password protection for denied domains. E.g., I have something like the following in one of my .htaccess files: AuthUserFile /path/to/htpasswd.private AuthGroupFile /dev/null AuthName "Private Stuff" AuthType Basic <Limit GET> order deny,allow deny from all allow from .scrc.umanitoba.ca require valid-user Satisfy any </Limit> which lets users from our domain access files without a password, but users outside our domain have to give a valid user name and password. Read up on Apache's basic authorization capabiilties, in apache.org's documentation, if you want to implement something like this yourself. Needless to say, this is WAY beyond the scope of the htdig-general mailing list. -- Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil Dept. Physiology, U. of Manitoba Phone: (204)789-3766 Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930 _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
