Re: Robots.txt parser

2004-11-11 Thread Craig McClanahan
As to interest, you might consider incorporating this in the httpclient project but packaging it as a separate JAR for people who want to use some other HTTP client implementation. That way the HttpClient developers could ensure that it meets your needs, without forcing users of this parser to imp

Robots.txt parser

2004-11-11 Thread Henri Yandell
Following on from a mail on the httpclient-dev list; I'm interested in submitting a codebase of mine into Commons that parses robots.txt files. It would definitely be of use with HttpClient's future plans but I'd like to keep it stand-alone and not hidden away inside HttpClient. Is there any inter