This is what I needed to know. Thanks.
Gilles Detillieux wrote:
>
> According to J. op den Brouw:
> > I got the "htmerge: Unable to open word list file" error
> > and to my opinion it showed when I hit a server that uses
> > a robots.txt file and I could not get a file, then somewhere
> > htmerge crashes because nothing is indexed.
> >
> > I only wanted to index that site, no other.
>
> Yup, if htdig doesn't index anything at all, htmerge will give that error.
> If you're only indexing that one site, and their robots.txt file blocks
> out htdig (or any robots), then htdig will get 0 documents.
>
> You should contact the administrators of the site you want to index (a
> good courtesy to follow in any case), and ask them if they'll allow you
> to index their site. If they allow access to the User-agent "htdig"
> in their robots.txt file, you'll be able to index the directories to
> which they give htdig access.
>
> If they want to allow you, but not just any old htdig client, you can
> agree on some other user agent ID that they'll put in their robots.txt
> file, for you to use. You can configure this ID in your htdig.conf file,
> with the "user_agent" parameter.
>
> --
> Gilles R. Detillieux E-mail: <[EMAIL PROTECTED]>
> Spinal Cord Research Centre WWW: http://www.scrc.umanitoba.ca/~grdetil
> Dept. Physiology, U. of Manitoba Phone: (204)789-3766
> Winnipeg, MB R3E 3J7 (Canada) Fax: (204)789-3930
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
