> -----Ursprüngliche Nachricht----- > Von: Jeetendra Mirchandani [mailto:[EMAIL PROTECTED] > Gesendet: Freitag, 23. Dezember 2005 10:38 > An: [email protected] > Cc: Jeetendra Mirchandani > Betreff: Robots.txt support? > > Hi, > > Can some one point me to some extension to HTTP Client to > honor robots.txt?
How should HttpClient do that??? This is not in scope of HttpClient. You should load the robots.txt and parse it. > > Please CC me on the reply as I am not on the list. > > Thanks, > Jeetu > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
