I found this: http://www.osjava.org/norbert/changes-report.html Any comments on using this? Or any other suggestions?
Thanks, Jeetu On Fri, 2005-12-23 at 10:55 +0100, Ingo Meyer wrote: > > -----Ursprüngliche Nachricht----- > > Von: Jeetendra Mirchandani [mailto:[EMAIL PROTECTED] > > Gesendet: Freitag, 23. Dezember 2005 10:38 > > An: [email protected] > > Cc: Jeetendra Mirchandani > > Betreff: Robots.txt support? > > > > Hi, > > > > Can some one point me to some extension to HTTP Client to > > honor robots.txt? > > How should HttpClient do that??? > This is not in scope of HttpClient. > > You should load the robots.txt and parse it. > > > > > Please CC me on the reply as I am not on the list. > > > > Thanks, > > Jeetu > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: > > [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
