[
https://issues.apache.org/jira/browse/FLINK-10981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736914#comment-16736914
]
Yun Gao commented on FLINK-10981:
---------------------------------
Hi [~pnowojski], sorry for responding so later. I totally agree with that the
multi-level metric mechanism will help a lot for users to debug jobs. The
aggregated values provide users with global view about the job status and free
users from massive details. I think we may achieve the aggregation either in
runtime or via a separate system like WebUI or monitor system.
For the back pressure metrics, I have a small concern that since users may not
know the total number of available buffers, they may not be able to tell if
back-pressure exists with the absolute queue length, and a percentage value may
be more intuitive.
For the original problem, now I think it may need to be considered together
with how we want to show back-pressure status to users in the future. If
finally we take methods other than the buffer usage, this problem may not
exist. :)
> Add or modify metrics to show the maximum usage of
> InputBufferPool/OutputBufferPool to help debugging back pressure
> -------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-10981
> URL: https://issues.apache.org/jira/browse/FLINK-10981
> Project: Flink
> Issue Type: Improvement
> Components: Metrics, Network
> Reporter: Yun Gao
> Assignee: Yun Gao
> Priority: Major
>
> Currently the network layer has provided two metrics items, namely
> _InputBufferPoolUsageGauge_ and _OutputBufferPoolUsageGauge_ to show the
> usage of input buffer pool and output buffer pool. When there are multiple
> inputs(SingleInputGate) or outputs(ResultPartition), the two metrics items
> show their average usage.
>
> However, we found that the maximum usage of all the InputBufferPool or
> OutputBufferPool is also useful in debugging back pressure. Suppose we have a
> job with the following job graph:
>
> {code:java}
> F
> \
> \
> _\/
> A ---> B ----> C ---> D
> \
> \
> \-> E
> {code}
> Besides, also suppose D is very slow and thus cause back pressure, but E is
> very fast and F outputs few records, thus the usage of the corresponding
> input/output buffer pool is almost 0.
>
> Then the average input/output buffer usage of each task will be:
>
> {code:java}
> A(100%) --> (100%) B (50%) --> (50%) C (100%) --> (100%) D
> {code}
>
>
> But the maximum input/output buffer usage of each task will be:
>
> {code:java}
> A(100%) --> (100%) B (100%) --> (100%) C (100%) --> (100%) D
> {code}
> Users will be able to find the slowest task by finding the first task whose
> input buffer usage is 100% but output usage is less than 100%.
>
>
> If it is reasonable to show the maximum input/output buffer usage, I think
> there may be three options:
> # Modify the current computation logic of _InputBufferPoolUsageGauge_ and
> _OutputBufferPoolUsageGauge._
> # Add two _new metrics items InputBufferPoolMaxUsageGauge and
> OutputBufferPoolMaxUsageGauge._
> # Try to show distinct usage for each input/output buffer pool.
> and I think maybe the second option is the most preferred.
>
> How do you think about that?
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
