[ https://issues.apache.org/jira/browse/NUTCH-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741082#action_12741082 ]
Julien Nioche commented on NUTCH-721: ------------------------------------- I had another look at this issue after applying the patch from Nutch-719. I can easily reproduce the situation from the original post by setting fetcher.threads.per.host.by.ip to true. The nutch-site file sent by Rodger does not specify it so it would rely on this value by default. Once setting it to false all threads are active and the fetching is much faster. I have used the first 5K URLs from the fetchlist sent by Rodger and compared the perfs with by.ip set to false : OldFetcher : real 32m26.003s user 1m11.768s sys 0m10.337s OldFetcher : real 30m52.965s user 1m10.696s sys 0m10.425s Fetcher : real 31m21.924s user 1m12.725s sys 0m10.797s Fetcher : real 30m3.017s user 1m15.509s sys 0m10.909s I ran each step twice and as we can see the results are comparable. This explanation is also compliant with Steven's observation that we get 5-7 times the rate as we would hit the DNS cache for subsequent calls for URLs from non unique sites. The IP resolution is done by the QueueFeeder which explains why it is slowing down the number of URLs being available for fetching. I don't think that the oldFetcher allows to group URLs by IP for politeness in which case why not making fetcher.threads.per.host.by.ip default to false in the new fetcher? > Fetcher2 Slow > ------------- > > Key: NUTCH-721 > URL: https://issues.apache.org/jira/browse/NUTCH-721 > Project: Nutch > Issue Type: Bug > Components: fetcher > Affects Versions: 1.0.0 > Environment: Fedora Core r6, Kernel 2.6.22-14, jdk1.6.0_12 > Reporter: Roger Dunk > Attachments: crawl_generate.tar.gz, nutch-site.xml > > > Fetcher2 fetches far more slowly than Fetcher1. > Config options: > fetcher.threads.fetch = 80 > fetcher.threads.per.host = 80 > fetcher.server.delay = 0 > generate.max.per.host = 1 > With a queue size of ~40,000, the result is: > activeThreads=80, spinWaiting=79, fetchQueues.totalSize=0 > with maybe a download of 1 page per second. > Runing with -noParse makes little difference. > CPU load average is around 0.2. With Fetcher1 CPU load is around 2.0 - 3.0 > Hosts already cached by local caching NS appear to download quickly upon a > re-fetch, so possible issue relating to NS lookups, however all things being > equal Fetcher1 runs fast without pre-caching hosts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.