[
https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880241#comment-16880241
]
Piotr Nowojski commented on FLINK-11082:
----------------------------------------
[~NicoK] reminded me that this might still not be completely solved. After this
fix, we do not eagerly assign floating buffers, but once we assign them they
will still linger on the unused/mostly idling channels. [~zjwang] what do you
think?
> Fix the calculation of backlog in PipelinedSubpartition
> -------------------------------------------------------
>
> Key: FLINK-11082
> URL: https://issues.apache.org/jira/browse/FLINK-11082
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.5.6, 1.7.1
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The backlog of subpartition should indicate how many buffers are consumable,
> then the consumer could feedback the corresponding credits for transporting
> these buffers. But in current PipelinedSubpartitionimplementation, the
> backlog is increased by 1 when a BufferConsumer is added into
> PipelinedSubpartition, and decreased by 1 when a BufferConsumer is removed
> from PipelinedSubpartition. So the backlog only reflects how many buffers are
> retained in PipelinedSubpartition, which is not always equivalent to the
> number of consumable buffers.
> The backlog inconsistency might result in floating buffers misdistribution on
> consumer side, because the consumer would request floating buffers based on
> backlog value, then one floating buffer might not be used in
> RemoteInputChannel long time after requesting.
> Considering the solution, the last buffer in PipelinedSubpartition could only
> be consumable in the case of flush triggered or partition finished. So we
> could calculate the backlog precisely based on partition flushed/finished
> conditions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
