[
https://issues.apache.org/jira/browse/HADOOP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489938
]
Runping Qi commented on HADOOP-206:
-----------------------------------
If LocalJobRunner does not force to set numReduceTasks to 1,
unit tests TestMapRed and TestRecordMR failed due to out of memory
in reduce:
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.j
ava:53)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:91)
at org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java
:1621)
at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey
(SequenceFile.java:2573)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.next(SequenceFile
.java:2358)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.getNext(ReduceTask
.java:207)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.ja
va:170)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceT
ask.java:245)
at org.apache.hadoop.mapred.TestMapRed$RandomCheckReducer.reduce(TestMap
Red.java:155)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
54)
Plus, LocalJobRunning calls only one reducer with partition equal 0.
That part has to be fixed too.
> local map-reduce job limited to single reducer
> ----------------------------------------------
>
> Key: HADOOP-206
> URL: https://issues.apache.org/jira/browse/HADOOP-206
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.0, 0.1.1, 0.2.0
> Environment: local job tracker
> Reporter: Yoram Arnon
> Assigned To: Runping Qi
> Priority: Minor
> Fix For: 0.13.0
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is
> ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to
> partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
