[
https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated MAPREDUCE-4502:
--------------------------------------
Attachment: MAPREDUCE-4525-pof.diff
I've prototyped proof of concept code of node-level aggregation. I made sure
that it works well, so I share it! Current situation is as follows:
* Node level aggregation in distributed mode is supported. I made sure of the
correctness of the result by using diff command.
* Fault tolerance is partially supported. At design level, it can be
supported(WIP).
* No test codes.
* No switch on/off feature.
Quick benchmark is as follows:
|Default Hadoop(sec)|Node level aggregation(sec)|
|105.0|82.0|
Experiment Environment is as follows:
* Input file is 1GB, generated by RandomTextWriter.
* 8 CPU/machines.
* 2 machines.
I'm appreciate if you review the design and code.
And, I have several questions to deal with:
1. How can I write tests such a complex case? Any ideas or references?
2. I've extended mr_protos.proto. Is it acceptable?
My next step is creating patch sets with tests and fully fault tolerance
support to merge it into trunk. I'm also planning to benchmark of node-level
aggregation. I'll show you here when I do it. Note that design note is
outdated, so I'm updating it now.
If you'd like to track changes my code changes, you can see it on [my
github|https://github.com/oza/hadoop-common/tree/MAPREDUCE-4525-Node-level-aggregation].
> Multi-level aggregation with combining the result of maps per node/rack
> -----------------------------------------------------------------------
>
> Key: MAPREDUCE-4502
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster, mrv2
> Reporter: Tsuyoshi OZAWA
> Assignee: Tsuyoshi OZAWA
> Attachments: design_v2.pdf, MAPREDUCE-4525-pof.diff,
> speculative_draft.pdf
>
>
> The shuffle costs is expensive in Hadoop in spite of the existence of
> combiner, because the scope of combining is limited within only one MapTask.
> To solve this problem, it's a good way to aggregate the result of maps per
> node/rack by launch combiner.
> This JIRA is to implement the multi-level aggregation infrastructure,
> including combining per container(MAPREDUCE-3902 is related), coordinating
> containers by application master without breaking fault tolerance of jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
