Hi,
as Daniël states, it would be nice if websec behaved like a proper web
robot. Baruch, are you aware that the change is basically a two-liner?
LWP, the web library you're using, provides a subclass of UserAgent that
respects the robots.txt protocol, see here:
* Thomas Themel [EMAIL PROTECTED] [070310 16:56]:
Hi,
as Dani??l states, it would be nice if websec behaved like a proper web
robot. Baruch, are you aware that the change is basically a two-liner?
I'm not even sure that it should go for robots.txt, the idea of websec
is that it behaves more
Hi,
Baruch Even ([EMAIL PROTECTED]) wrote on 2007-03-10:
I'm not even sure that it should go for robots.txt, the idea of websec
is that it behaves more like a browser, it doesn't crawl the website, it
doesn't try to go anywhere it's not directly instructed to and if a user
insists on
3 matches
Mail list logo
