Yun Gao created FLINK-10981:
-------------------------------
Summary: Add or modify metrics to show the maximum usage of
InputBufferPool/OutputBufferPool to help debugging back pressure
Key: FLINK-10981
URL: https://issues.apache.org/jira/browse/FLINK-10981
Project: Flink
Issue Type: Improvement
Components: Metrics, Network
Reporter: Yun Gao
Assignee: Yun Gao
Currently the network layer has provided two metrics items, namely
_InputBufferPoolUsageGauge_ and _OutputBufferPoolUsageGauge_ to show the usage
of input buffer pool and output buffer pool. When there are multiple
inputs(SingleInputGate) or __ outputs(ResultPartition), the two metrics items
show their average usage.
However, we found that the maximum usage of all the InputBufferPool or
OutputBufferPool is also useful in debugging back pressure. Suppose we have a
job with the following job graph:
{code:java}
F
\
\
_\/
A ---> B ----> C ---> D
\
\
\-> E
{code}
Besides, also suppose D is very slow and thus cause back pressure, but E is
very fast and F outputs few records, thus the usage of the corresponding
input/output buffer pool is almost 0.
Then the average input/output buffer usage of each task will be:
{code:java}
A(100%) --> (100%) B (50%) --> (50%) C (100%) --> (100%) D
{code}
But the maximum input/output buffer usage of each task will be:
{code:java}
A(100%) --> (100%) B (100%) --> (100%) C (100%) --> (100%) D
{code}
Users will be able to find the slowest task by finding the first task whose
input buffer usage is 100% but output usage is less than 100%.
If it is reasonable to show the maximum input/output buffer usage, I think
there may be three options:
# Modify the current computation logic of _InputBufferPoolUsageGauge_ and
_OutputBufferPoolUsageGauge._
# Add two __ new metrics items I_nputBufferPoolMaxUsageGauge and
OutputBufferPoolUsageGauge._
# Try to show distinct usage for each input/output buffer pool.
and I think maybe the second option is the most preferred.
How do you think about that?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
