Hello Rick,
On Wed, Mar 31, 2004 at 06:50:00AM +0100, Rick wrote:
> Hello htdiggers ...
>
> Just installed htdig-3.2.0b5, but it has some
> difficulty reading this robots.txt file ...
>
> User-agent: grub-client
> Disallow: /
> User-agent: *
> Disallow:
>
> I believe that the robots.txt syntax is correct.
> Release 3.1.6 has no problem with it.
>
> Any ideas/solutions/workarounds ?
http://www.robotstxt.org
or
http://www.robotstxt.org/wc/norobots.html
tells me you are right (I wondered about the missing "/"..)
What does ht://Dig say, when you run it with multiple "v" (as for "verbose")?
Maybe a permission problem?
Yours,
Martin
--
--------------------------------------------------------
arago AG, Institut fuer komplexes Datenmanagement
Am Niddatal 3, 60488 Frankfurt/Main, [EMAIL PROTECTED]
Tel. 069/405680, Fax 069/40568111, http://www.arago.de
--------------------------------------------------------
pgp00000.pgp
Description: PGP signature
