Hi Vishal, On 5/23/07, Vishal Shah <[EMAIL PROTECTED]> wrote: > Hi Dogacan, > > Fetcher2 gives a better performance when the number of hosts per task is > more than the number of threads that the task can use. In this case, fetcher > might block on some hosts, whereas fetcher2 will use that idle time in > crawling some other host. > > It could be that the number of hosts per task is not significantly higher > than the number of threads per task. In that case, ideally you should see a > similar performance from fetcher2 and fetcher (assuming same url list and > network bandwidth). > > Also, as Andrzej suggested - it would be good to have some more debugging > info.
Have you tested Fetcher2 after NUTCH-474? There were a couple of bugs in Fetcher2 that made it work just like Fetcher (because lib-http still blocked threads, making Fetcher2's queue logic useless). Looking at the code, I can't see any other bugs, but I am still testing, perhaps I will find a couple more(or perhaps, I will find out that something in my conf is broken). > > Regards, > > -vishal. > > -----Original Message----- > From: Dogacan Güney [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 23, 2007 8:21 PM > To: [EMAIL PROTECTED] > Subject: Re: [Nutch-general] Fetcher2 slowness? > > On 5/23/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > So what was Fetcher2's performance like when its number of threads was the > same as that of Fetcher? > > It is still slower. I tried giving Fetcher2 more threads,it is still > worse than Fetcher but a bit better than fewer-threaded > Fetcher2(Fetcher finished in 1 hour, Fetcher2 in about 2.5). Though I > have performed other tests where their performance is similar(and I > have no idea why). I am trying to find the cause of problem, but so > far, had no luck. > > > > > Otis > > > > [snip] > > -- > Dogacan Güney > > -- Doğacan Güney ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
