On Jan 27, 2011, at 7:19pm, Albretch Mueller wrote:
~ I did a search on the mailing list archives and I could only find a few pieces of code segments of people that were coding multithreaded crawlers using hc. However I could not find fullfledge crawlers based on hc. I know hc is only about baseline functionality relating to the http protocol, but fullfledge crawlers and proxies could be derived from it. ~ Does anyone know of such projects?
Bixo uses HttpClient 4.0 - see http://openbixo.org and https://github.com/bixo/bixo/blob/master/src/main/java/bixo/fetcher/SimpleHttpFetcher.java Apache Droids also uses HttpClient 4.x And Nutch uses HttpClient 3.1. -- Ken -------------------------- Ken Krugler +1 530-210-6378 http://bixolabs.com e l a s t i c w e b m i n i n g --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
