[ https://issues.apache.org/jira/browse/MAPREDUCE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins moved HADOOP-5340 to MAPREDUCE-2812: ------------------------------------------------ Affects Version/s: (was: 0.19.1) Key: MAPREDUCE-2812 (was: HADOOP-5340) Project: Hadoop Map/Reduce (was: Hadoop Common) > Combiner that aggregates all the mappers from a machine > ------------------------------------------------------- > > Key: MAPREDUCE-2812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2812 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Reporter: Nathan Marz > > From what I can tell, the Combiner just aggregates data from a single map > task. It would be useful, especially during map-only jobs, to have a combiner > that aggregates data from all the map tasks on a given machine. My use case > for this is to vertically partition a set of records which start out in the > same files. By doing this in a map-only task, way too many files are created > (About 50 files are created per input split). By pumping all the data through > a reducer, a lot of unnecessary overhead occurs. With the proposed feature, I > would get 50*number of machines files rather than 50*number of input splits > files for this use case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira