All, Does anyone have a magic number for the number of fetcher threads per host so that the site doesn't think they are under attack? I'm trying to maximize the crawls without overwhelming some sites.
Thanks in advance, Alex
All, Does anyone have a magic number for the number of fetcher threads per host so that the site doesn't think they are under attack? I'm trying to maximize the crawls without overwhelming some sites.
Thanks in advance, Alex