[
https://issues.apache.org/jira/browse/MAPREDUCE-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041781#comment-14041781
]
Todd Lipcon commented on MAPREDUCE-2841:
----------------------------------------
Hey Arun.
I agree that building a completely parallel C++ MR runtime is a much larger
project that should not be part of Hadoop. However, per my above comment, the
current goal for contribution is _only the MapOutputCollector_. That is to say,
there is no ABI concern, because the interface exposed to application
developers is really quite small (unlike something like pipes or a full task
SDK). I realize it's confusing because the github repo does mention these
larger goals of the project, but I agree with you that they are not worth the
cost-benefit within MR itself.
bq. I also noticed that the github has a bunch of code related to Pig, Hive
etc. - I think we'd all agree that they need to be in respective projects
eventually.
Fully agree. I raised the same concern offline to Sean and he is working on
making the "platform support" use a ServiceLoader instead of explicitly
registering all of the other frameworks inside the core code. (his comment
"Extract support for Hive/Pig/HBase/Mahout platforms to standalone jars, and
decouple the dependency with native task source code" is referring to this).
bq. Similarly, I am wary of importing a nearly 75K LOC codebase into a stable
project
I think the 75k you're counting may include the auto-generated shell scripts.
By my count, the non-test Java code is 3k lines, some of which is boilerplate
around the different platform implementations (which would move into those
projects). If you exclude the thirdparty dependencies which are bundled into
the repo, the non-test C++ source is 10kloc.
So, it's not a tiny import by any means, but for 2x improvement on terasort
wallclock, my opinion is that the maintenance burden is worth it.
As for importing to Tez, I don't think the community has generally agreed to
EOL MapReduce :) In other words, if you feel that Tez is the future, it makes
sense for you to not work on any further MR optimizations, but shouldn't
preclude others from doing so. There are lots of production apps on MR today
that do not want to switch to a new framework (regardless of compatibility
layers), so I think this would be valuable for the community.
> Task level native optimization
> ------------------------------
>
> Key: MAPREDUCE-2841
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2841
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Environment: x86-64 Linux/Unix
> Reporter: Binglin Chang
> Assignee: Sean Zhong
> Attachments: DESIGN.html, MAPREDUCE-2841.v1.patch,
> MAPREDUCE-2841.v2.patch, dualpivot-0.patch, dualpivotv20-0.patch,
> fb-shuffle.patch
>
>
> I'm recently working on native optimization for MapTask based on JNI.
> The basic idea is that, add a NativeMapOutputCollector to handle k/v pairs
> emitted by mapper, therefore sort, spill, IFile serialization can all be done
> in native code, preliminary test(on Xeon E5410, jdk6u24) showed promising
> results:
> 1. Sort is about 3x-10x as fast as java(only binary string compare is
> supported)
> 2. IFile serialization speed is about 3x of java, about 500MB/s, if hardware
> CRC32C is used, things can get much faster(1G/
> 3. Merge code is not completed yet, so the test use enough io.sort.mb to
> prevent mid-spill
> This leads to a total speed up of 2x~3x for the whole MapTask, if
> IdentityMapper(mapper does nothing) is used
> There are limitations of course, currently only Text and BytesWritable is
> supported, and I have not think through many things right now, such as how to
> support map side combine. I had some discussion with somebody familiar with
> hive, it seems that these limitations won't be much problem for Hive to
> benefit from those optimizations, at least. Advices or discussions about
> improving compatibility are most welcome:)
> Currently NativeMapOutputCollector has a static method called canEnable(),
> which checks if key/value type, comparator type, combiner are all compatible,
> then MapTask can choose to enable NativeMapOutputCollector.
> This is only a preliminary test, more work need to be done. I expect better
> final results, and I believe similar optimization can be adopt to reduce task
> and shuffle too.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
