[
https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Kimball updated MAPREDUCE-434:
------------------------------------
Attachment: MAPREDUCE-434.2.patch
Submitting new patch sync'd with trunk.
When testing this patch, I uncovered a problematic interaction between
mapper/reducer tasks and the {{Merger}}. Specifically, {{MergeQueue}} uses a
static field {{Merger.includeFinalMerge}} which is set to true in {{MapTask}},
but is never cleared.
Now that both task types can occur in the same JVM, and a {{Merger}} may be
used in both (local reduce tasks did not actually instantiate a {{Merger}} in
the old codepath), this flag must be made a per-{{MergeQueue}} setting. I
removed the static field and pushed it into {{MergeQueue}} as a member field;
the {{Merger.merge()}} methods used as constructors for {{MergeQueue}} now take
a {{TaskType}} parameter which can be set to {{TaskType.MAP}} to achieve the
same effect.
> local map-reduce job limited to single reducer
> ----------------------------------------------
>
> Key: MAPREDUCE-434
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-434
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Environment: local job tracker
> Reporter: Yoram Arnon
> Assignee: Aaron Kimball
> Priority: Minor
> Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.patch
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is
> ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to
> partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
