beware you could create a kind of Denial of Service attack if you search the site too quickly.
2009/8/26 Paul Tomblin <[email protected]> > On Wed, Aug 26, 2009 at 10:55 AM, Kirby Bohling<[email protected]> > wrote: > > Paul, > > > > I'd read the nutch-default.xml file, I believe the properties you'd > > like to examine start in the section labelled <!-- fetcher properties > > --> > > > > fetcher.threads.per.host > > fetcher.server.delay > > fetcher.server.min.delay > > fetcher.max.crawl.delay > > > > Ok, that is interesting. Since I'm only crawling one site at a time, > I guess there is no point having more threads that > "fetcher.threads.per.host" - or more to the point, I should increase > fetcher.threads.per.host up to the number of threads. > > > -- > http://www.linkedin.com/in/paultomblin > -- -MilleBii-
