Yes, thanks, you seem to be right. If I use more threads on the same host although the process seems to go faster I get alot more http errors so it ends up being slower (and probably more disruptive to the site.)
--- EM <[EMAIL PROTECTED]> wrote: > Go with 1 thread per host. > > For my small area of internet where I fetch my > pages, almost all hosts > starts refusing requests on 3+ threads, some of > them, even at 1+. > > Bandwidth-wise, if you go with higher value on > fetcher.threads.per.host, > your fetcher will have hard time connecting and the > target server will save > on bandwidth in fact ;) > > > -----Original Message----- > From: Raymond Creel [mailto:[EMAIL PROTECTED] > Sent: Monday, July 25, 2005 4:00 PM > To: nutch-user@lucene.apache.org > Subject: fetch bandwidth settings > > I have read that you don't want to make more than 1 > or > 2 requests per second to the same host, or else you > will start adversely affecting their bandwidth. Is > this a good rule of thumb? > > Along those lines, what would be the best values to > put in the nutch config file to maximize speed of > fetching without hammering the site? I'm thinking > something like this: > > fetcher.server.delay: 1.0 > fetcher.threads.per.host: 2 > > thanks, > raymond > > > > ____________________________________________________ > Start your day with Yahoo! - make it your home page > http://www.yahoo.com/r/hs > > > > __________________________________ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html
