GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/3106
[FLINK-????] FIx inconsistent numBytesIn/Out metrics
This PR fixes _some_ jira issue (i can't find the correct issue ID right
now since JIRA is being stupid) regarding inconsistent numBytesIn/Out metrics.
Given 2 subsequent tasks A->B the numBytesOut count of A was lower than the
numBytesIn(Local/Remote) count of B by a huge margin, although they should be
(nearly) identical.
The problem is that A was counting how much bytes the serialized records
were using, whereas B was counting how large the ```Buffer```s were that it
received. A ```Buffer``` contains the following data:
```
|size_of_R1|R1|size_of_R2|R2|...
```
where as R1/R2 are serialized records and the sizes denote the serialized
length of their respective record.
So, while A was adding ```sizeOf(R1)```, B was adding sizeOf(size_of_R1) +
sizeOf(R1).
A was simply not accounting for the added bytes that size_of_RX were using,
which is what this PR is fixing by adding ```4``` after the serialization of
each record.
cc @uce
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink _metrics_incon
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3106.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3106
----
commit 5300a35ac6ee8815d07a739c9b370ae475f0d9a0
Author: zentol <[email protected]>
Date: 2017-01-12T13:59:22Z
[FLINK-????] FIx inconsistent numBytesIn/Out metrics
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
