[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447040#comment-13447040
]
Mariappan Asokan commented on MAPREDUCE-4049:
---------------------------------------------
{quote}
I don’t have conflict of interests with you. 4 months ago, I already welcomed
the watchers of this issue to help you commit your patch.
{quote}
I hope I didn't offend you by stating that "we seem to have a conflict of
interest here." I meant the conflict in our designs:)
{quote}
RDMA is not coupled with any merge and there is no such a thing "RDMA merge".
{quote}
Perhaps I should have stated "RDMA based merge" instead of "RDMA merge." From
my understanding of the pdf document attached to this Jira, there is a
{{NetMerger}} plugin component which does the merge on the reduce side. It
takes care of shuffling(small pieces of data using RDMA) and merge on the
reduce side in order to avoid multi-phase merges and associated disk IO.
If you can educate me and others on the relationship between {{NetMerger}} and
your {{ShuffleConsumerPlugin}}, perhaps we will understand more. To me
{{NetMerger}} implies it is doing a merge sort on the reduce side and hence it
is qualified to be a sort plugin on the reduce side.
{quote}
It is current Hadoop that couples shuffle with merge. You can be relaxed. I
don’t “want to retain that coupling”. My opinion is that your decoupling is
correct idea and I encourage it.
{quote}
I am glad that you like the idea of decoupling the merge and shuffle. In order
to achieve this decoupling, {{Shuffle}} class cannot return the key value
iterator. It should only do the shuffling of data and feed it to the code that
merges the map outputs. It is the responsibility of the merger to return the
key value iterator. I want to make that code pluggable and hence the need for
{{ReduceSortPlugin.}} If you want to make shuffle pluggable, as I suggested
before we can make
{{ShuffleRunner}}(which will be implemented by {{Shuffle}} class) pluggable.
{quote}
Your patch contains more than 7,000 rows, while my patch is only 400 rows. I
don’t want to wait till your patch passes Automatic QA, and code review, and
additional rounds.
{quote}
Since we are working on overlapping areas of the code, I just want to make sure
that we both get the design concepts right before looking at the number of
lines of code touched or which Jira is to be committed first and so on.
> plugin for generic shuffle service
> ----------------------------------
>
> Key: MAPREDUCE-4049
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance, task, tasktracker
> Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
> Reporter: Avner BenHanoch
> Labels: merge, plugin, rdma, shuffle
> Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Consumer Plugin
> TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml,
> mapreduce-4049.patch, mapreduce-4049.patch
>
>
> Support generic shuffle service as set of two plugins: ShuffleProvider &
> ShuffleConsumer.
> This will satisfy the following needs:
> # Better shuffle and merge performance. For example: we are working on
> shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
> or Infiniband) instead of using the current HTTP shuffle. Based on the fast
> RDMA shuffle, the plugin can also utilize a suitable merge approach during
> the intermediate merges. Hence, getting much better performance.
> # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
> dependency of NodeManager with a specific version of mapreduce shuffle
> (currently targeted to 0.24.0).
> References:
> # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
> from Auburn University with others,
> [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
> # I am attaching 2 documents with suggested Top Level Design for both plugins
> (currently, based on 1.0 branch)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
