On 5/24/07, Vishal Shah <[EMAIL PROTECTED]> wrote:
Hi Dogacan,I haven't yet checked it with the 474 patch. Thanks for that update! What I was trying to say was that even if fetcher2 works perfectly, it can only give you a significant performance boost when the number of threads per task is much less than the number of hosts that will be fetched by that task.
I don't really want Fetcher2 to be faster. I just want it to be as fast as Fetcher with a smaller number of threads which is a reasonable expectation :)
Do you have any numbers about the number of hosts from which you are fetching?
I have ~30000 urls with ~1000 hosts. Hosts have at most 500 urls and there are 23 hosts that have 500 urls. I generally run Fetcher with 100-200 threads and Fetcher2 with 50 threads.
-vishal.
[snip] -- Doğacan Güney
