[
https://issues.apache.org/jira/browse/TEZ-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988061#comment-15988061
]
Zhiyuan Yang commented on TEZ-3698:
-----------------------------------
Change looks correct, but unit tests is not verifying any events in case of no
final merge.
Regarding to performance, why is this favorable over pipeline shuffle? Does
this incur lots of memory pressure?
Nits:
1. sendPipelineEventForSpill should be renamed because now it's not just for
pipeline shuffle
2. This can be combined.
{code}
if (!pipelinedShuffle) {
if (isFinalMergeEnabled) {
return;
}
}
{code}
> UnorderedKV writer should be able to honor
> tez.runtime.enable.final-merge.in.output without pipelinedshuffle
> ------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-3698
> URL: https://issues.apache.org/jira/browse/TEZ-3698
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-3698.1.patch
>
>
> Final merge can be disabled with "tez.runtime.enable.final-merge.in.output"
> setting. Currently this works with UnorderedKV writer only with pipelined
> shuffle. It should be able to honor this parameter, without pipelined shuffle
> as well to avoid final merge.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
