Doğacan Güney wrote: > Hi everyone, > > Has anyone tried Fetcher2 from latest trunk? On our tests, Fetcher2 is > always slower (by a large margin) that Fetcher. > > For a segment with ~30000 urls, we ran Fetcher with 150 threads and > Fetcher2 with 50 threads. Fetcher finishes around 1 hour, while > Fetcher2 takes around 4 hours. We ran this test more than once and > got similar results. > > Are we running Fetcher2 with too few/too many threads? I was under the > impression that Fetcher2 doesn't need as many threads as Fetcher since > threads do not block.
Yes, that was the idea. Could you test it with the same number of threads? Is the configuration identical in all other aspects? Are you running the version with the fix from NUTCH-474? > > Any suggestions? > If you already have a setup to reproduce this, you could perhaps spend some time debugging this ... add some timing info, and queue info logging. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
