[jira] [Commented] (TEZ-3303) Have ShuffleVertexManager consume more precise partition stats

[Ming Ma (JIRA)]

    [ 
https://issues.apache.org/jira/browse/TEZ-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359222#comment-15359222
 ] 

Ming Ma commented on TEZ-3303:
------------------------------

Thanks [~ozawa]. Yep, as [~sseth] said, for example you can provide the precise 
stats version of parsePartitionStats like this. I have verified that as part of 
the TEZ-3209 development.

{noformat}
  void parsePartitionStats(List<Integer> partitionStats) {
    for (int i=0; i<partitionStats.size(); i++) {
      stats[i] += partitionStats.get(i);
    }
  }
{noformat}

Note that this jira only covers the update of {{ShuffleVertexManager}}'s 
{{stats}} field based on precise partition stats in VME. 
{{ShuffleVertexManager}}'s {{stats}} currently isn't used to decide the auto 
reducer partition range; it is only used for scheduling order. But that is a 
separate issue and will be covered by TEZ-2962, or as part of TEZ-3209, which I 
have added for fair routing.

> Have ShuffleVertexManager consume more precise partition stats
> --------------------------------------------------------------
>
>                 Key: TEZ-3303
>                 URL: https://issues.apache.org/jira/browse/TEZ-3303
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Tsuyoshi Ozawa
>         Attachments: TEZ-3303.001.patch
>
>
> TEZ-3216 adds the support for more precise partition stats. 
> ShuffleVertexManager should be updated to consume the more precise partition 
> stats.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)