[
https://issues.apache.org/jira/browse/FLINK-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249663#comment-17249663
]
Piotr Nowojski commented on FLINK-14712:
----------------------------------------
I'm planning to take over this issue. My plan is to (I'm investigating how to
achieve that):
# color the border of the frame/border of a task in the JobGraph based on the
backpressure status (currently it's blue. It could either be solid red if task
is backpressured, or a shade between blue and red based on backpressure ratio.
Also if task is causing backpressure it could be black)
# Display {{inPoolUsage}} and {{outPoolUsage}} metrics in the [detailed
backpressure
tab|https://ci.apache.org/projects/flink/flink-docs-master/ops/monitoring/back_pressure.html#back-pressure-status].
I think there is no need to clog this information on the job graph itself.
Once user finds the source of the backpressure, he can check the detailed stats
in this tab. This way also allows us to avoid the problem of aggregating pool
usage metrics from multiple subtasks and how to present the single aggregated
value for a task as a whole.
> Improve back-pressure reporting mechanism
> -----------------------------------------
>
> Key: FLINK-14712
> URL: https://issues.apache.org/jira/browse/FLINK-14712
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics, Runtime / Network, Runtime / REST
> Reporter: lining
> Assignee: Piotr Nowojski
> Priority: Major
> Attachments: image-2019-11-12-14-30-16-130.png
>
>
> h4. (1) The current monitor is heavy-weight.
> * Backpressure monitoring works by repeatedly taking stack trace samples
> of your running tasks.
> h4. (2) It is difficult to find out which vertex is the source of
> backpressure.
> * User need to know current and upstream's network metric to judge current
> whether is the source of backpressure. Now user has to record relevant
> information.
> h3. Proposed Changes
> 1. expose the new mechanism implemented in FLINK-14472 as a "is
> back-pressured" metric.
> 2. show the vertex that produces the backpressure source for the job.
> 3. expose network metric in IOMetricsInfo:
> * SubTask
> ** pool usage: outPoolUsage, inputExclusiveBuffersUsage,
> inputFloatingBuffersUsage.
> *** If the subtask is not back pressured, but it is causing backpressure
> (full input, empty output)
> *** By comparing exclusive/floating buffers usage, whether all channels are
> back-pressure or only some of them
> ** back-pressured for show whether it is back pressured.
> * Vertex
> ** pool usage: outPoolUsageAvg, inputExclusiveBuffersUsageAvg,
> inputFloatingBuffersUsageAvg
> ** back-pressured for show whether it is back pressured(merge all iths
> subtasks)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
