Hi,
this was not a feature request :)! I just wanted to know if it was implanted in
ht://Dig 3.1.6 for whatever reason. Thanks for the excerpt from www.robotstxt.org. I
had overlook it. So I guess the response is no.
Bye.
-----Message d'origine-----
De : Gabriele Bartolini [mailto:[EMAIL PROTECTED]]
Envoy� : 6 f�vrier, 2003 12:57
� : Marchand Robert; [EMAIL PROTECTED]
Objet : Re: [htdig] wildcards in robot.txt?
Ciao!
At 12.15 06/02/2003 -0500, [EMAIL PROTECTED] wrote:
>User-agent: htdig-udem
>Disallow: *.pdf
>Disallow: *.PDF
>
>Is this possible with ht://Dig 3.1.6?
Well ... robots.txt is an 'international' and de-facto standard and it is
not an ht://Dig's feature; I snipped these word from the
'www.robotstxt.org' site:
Note also that regular expression are not supported in either the
User-agent or Disallow lines. The '*' in the User-agent field is a special
value meaning "any robot". Specifically, you cannot have lines like
"Disallow: /tmp/*" or "Disallow: *.gif".
I don't know whether it is a good idea to override this. What about you guys?
Ciao
-Gabriele
--
Gabriele Bartolini - Web Programmer - ht://Dig & IWA Member - ht://Check
maintainer
Current Location: Prato, Tuscany, Italia
[EMAIL PROTECTED] | http://www.prato.linux.it/~gbartolini | ICQ#129221447
> find bin/laden -name osama -exec rm {} \;
-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a
subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
