[
https://issues.apache.org/jira/browse/NUTCH-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897297#action_12897297
]
Doğacan Güney commented on NUTCH-884:
-------------------------------------
+1. Maybe we can make the documentation on command line option a bit more
verbose and say "default number of map tasks" instead of "mapred.map.tasks".
If I understood the code correctly, I think this part should be -all and not
-threads:
- if (crawlId.equals("-threads") || crawlId.equals("-resume") ||
crawlId.equals("-parse")) {
+ if (!crawlId.equals("-threads") || crawlId.startsWith("-")) {
FetcherJob.fetch makes much more sense with the rearranged arguments.
> FetcherJob should run more reduce tasks than default
> ----------------------------------------------------
>
> Key: NUTCH-884
> URL: https://issues.apache.org/jira/browse/NUTCH-884
> Project: Nutch
> Issue Type: Improvement
> Components: fetcher
> Affects Versions: 2.0
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Fix For: 2.0
>
> Attachments: NUTCH-884.patch
>
>
> FetcherJob now performs fetching in the reduce phase. This means that in a
> typical Hadoop setup there will be many fewer reduce tasks than map tasks,
> and consequently the max. total throughput of Fetcher will be proportionally
> reduced. I propose that FetcherJob should set the number of reduce tasks to
> the number of map tasks. This way the fetching will be more granular.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
