[ https://issues.apache.org/jira/browse/TEZ-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065652#comment-16065652 ]
Zhiyuan Yang commented on TEZ-3769: ----------------------------------- +1 after fixing javac warning. One nit: it seems unnecessary to create a new ArrayList here: {code} SpillCallable spillCallable = new SpillCallable(new ArrayList<WrappedBuffer>(filledBuffers), {code} > Unordered: Fix wrong stats being sent out in the last event, when final merge > is disabled > ----------------------------------------------------------------------------------------- > > Key: TEZ-3769 > URL: https://issues.apache.org/jira/browse/TEZ-3769 > Project: Apache Tez > Issue Type: Bug > Reporter: Rajesh Balamohan > Attachments: TEZ-3769.1.patch, TEZ-3769.2.patch > > > When final merge is disabled (without pipelining), wrong stats was sent out > in the last event. > It was based on {{numRecordsPerPartition}} which contains the overall > partition data. It should be ideally be based on the spill result and its > buffers. > Also, {{finalSpill}} was unncessarily sending events when no data was present > (i.e, when currentBuffer didn't have any data). This can be optimized to > reduce the number of events being sent across. -- This message was sent by Atlassian JIRA (v6.4.14#64029)