[
https://issues.apache.org/jira/browse/TEZ-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001373#comment-16001373
]
Jonathan Eagles commented on TEZ-3709:
--------------------------------------
Thanks for the feedback, [~rajesh.balamohan]. For some background, this
improves all jobs with high number of segments, but is especially good for
auto-reduce parallelism case. For this case the total merge time is 33 seconds
with the patch applied and jumps to 50 seconds with the linked list suggestion
above. This also increases the the total time of the task by around 19 seconds,
since it is part of the final merge so there is no pipelining of this stage. I
know it adds some complexity but the benefits far outweigh I think in this
case. I am definitely open to feedback on ways to make the code clearer or
alternative approaches that have similar performance characteristics. Hopefully
the background on this case help to understand the patch in context.
> TezMerger is slow for high number of segments
> ---------------------------------------------
>
> Key: TEZ-3709
> URL: https://issues.apache.org/jira/browse/TEZ-3709
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Labels: performance
> Attachments: TEZ-3709.1.patch, TEZ-3709.2.patch, TEZ-3709.3.patch
>
>
> The below code is a bad performer at scale since it has to memcpy the whole
> list of segments for each item in the batch instead of of just once per batch.
> This is true for both computeBytesInMerges and getSegmentDescriptors.
> {code}
> for (int i = 0; i < batch; i++) {
> ArrayList#remove(0)
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
