[
https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367762#comment-14367762
]
Gopal V commented on TEZ-145:
-----------------------------
[~ozawa]: the CombineProcessor patch looks good.
This will help applications which do no in-memory aggregations, but you're
effectively moving the data over racks ~3x.
So this is a necessary part of the fix, but not the complete part as long as
the ShuffleVertexManager is being used to connect them up.
Because that vertex manager has no way to provide locality of tasks when
spinning up tasks (for rack-local or host-local).
> Support a combiner processor that can run non-local to map/reduce nodes
> -----------------------------------------------------------------------
>
> Key: TEZ-145
> URL: https://issues.apache.org/jira/browse/TEZ-145
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch
>
>
> For aggregate operators that can benefit by running in multi-level trees,
> support of being able to run a combiner in a non-local mode would allow
> performance efficiencies to be gained by running a combiner at a rack-level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
