[ https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16735835#comment-16735835 ]
Nico Kruber commented on FLINK-11082: ------------------------------------- Nice find [~zjwang]. I guess the current behaviour comes from the two changes we were doing in parallel and that without the low-latency changes, whenever a buffer was added, it was ready-to-consume which is not that anymore. {{PipelinedSubpartition#add()}} always increases the backlog for any {{BufferConsumer}} which is initially empty and not ready for consumption - as noted. Instead, we can increase the backlog when adding a second {{BufferConsumer}} or when flushing, carefully considering not to increase too often. > Increase backlog only if it is available for consumption > -------------------------------------------------------- > > Key: FLINK-11082 > URL: https://issues.apache.org/jira/browse/FLINK-11082 > Project: Flink > Issue Type: Sub-task > Components: Network > Affects Versions: 1.8.0 > Reporter: zhijiang > Assignee: zhijiang > Priority: Minor > > The backlog should indicate how many buffers are available in subpartition > for downstream's consumption. The availability is considered from two > factors. One is {{BufferConsumer}} finished, and the other is flush triggered. > In current implementation, when the {{BufferConsumer}} is added into the > subpartition, then the backlog is increased as a result, but this > {{BufferConsumer}} is not yet available for network transport. > Furthermore, the backlog would affect requesting floating buffers on > downstream side. That means some floating buffers are fetched in advance but > not be used for long time, so the floating buffers are not made use of > efficiently. > We found this scenario extremely for rebalance selector on upstream side, so > we want to change when to increase backlog by finishing {{BufferConsumer}} or > flush triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)