okumin commented on code in PR #317:
URL: https://github.com/apache/tez/pull/317#discussion_r1407735609
##########
tez-runtime-library/src/test/java/org/apache/tez/dag/library/vertexmanager/TestShuffleVertexManagerUtils.java:
##########
@@ -125,10 +126,10 @@ VertexManagerEvent getVertexManagerEvent(long[]
partitionSizes,
long uncompressedTotalSize, String vertexName, boolean
reportDetailedStats)
throws IOException {
ByteBuffer payload;
- long totalSize = 0;
+ final long totalSize;
// Use partition sizes to compute the total size.
if (partitionSizes != null) {
- totalSize = estimatedUncompressedSum(partitionSizes);
+ totalSize = Arrays.stream(partitionSizes).sum();
Review Comment:
This patch doesn't change the total size. That's because [the total size is
stored in a different field of Protbuf from partitions
stats](https://github.com/apache/tez/blob/rel/release-0.10.2/tez-runtime-library/src/main/java/org/apache/tez/runtime/library/common/shuffle/ShuffleUtils.java#L446-L447).
The design is valid since users have an option not to take partition stats at
all(`tez.runtime.report.partition.stats=none`).
TEZ-4521 would remove the possibility where partition stats contain the
compressed size. That's why I revised this file to prevent future users from
being confused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
