[
https://issues.apache.org/jira/browse/FLINK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16897998#comment-16897998
]
zhijiang edited comment on FLINK-13493 at 8/1/19 12:10 PM:
-----------------------------------------------------------
Thanks for the clarification [~azagrebin]!
The set structure of readers in `BoundedBlockingSubpartition` make me think of
this concern. I agree it should be no problem now because of two things:
* We do not have multiple readers for one subpartition concurrently. The
current semantic of consuming multiple times for one subpartition is only
considering the failover case, that means next consumption attempt only comes
after the previous attempt canceled.
* The current blocking subpartition release would check whether reader set is
empty. As I proposed in FLINK-13478, it might no need to check that if the
ResultPartition already decides to release the whole partition, and this
decision might also come from JM RPC.
So the whole logic seems a bit weird that the subpartition notifies result
partition of consumed even though there are other unconsumed readers, and then
if the result partition triggers the whole release, that subpartition with
unconsumed readers actually would not be released.
was (Author: zjwang):
Thanks for the clarification [~azagrebin]!
The set structure of readers in `BoundedBlockingSubpartition` make me think of
this concern. I agree it should be no problem now because of two things:
* We do not have multiple readers for one subpartition concurrently. The
current semantic of consuming multiple times for one subpartition is only
considering the failover case, that means next consumption attempt only comes
after the previous attempt canceled.
* The current blocking subpartition release would check whether reader set is
empty. As I proposed in FLINK-13478, it might no need to check that if the
ResultPartition already decides to release the whole partition, and this
decision might also come from JM RPC.
So the whole logic seems a bit weird that the subpartition would notify result
partition consumed even though there are other unconsumed readers, and then if
the result partition triggers the whole release, the subpartition with
unconsumed readers actually would not be released.
> BoundedBlockingSubpartition only notifies onConsumedSubpartition when all the
> readers are empty
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-13493
> URL: https://issues.apache.org/jira/browse/FLINK-13493
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> In previous implementation, it would always notify the {{ResultPartition}} of
> consumed subpartition if any reader view is released. Based on the
> reference-counter release strategy it might cause problems if one blocking
> subpartition has multiple readers. That means the whole result partition
> might be released but there are still alive readers in some subpartitions.
> Although the default release strategy for blocking partition is based on
> JM/scheduler notification atm. But if we switch the option to based on
> consumption notification it would cause problems. And from the subpartition
> side it should has the right behavior no matter what is the specific release
> strategy in upper layer.
> In order to fix this bug, the {{BoundedBlockingSubpartition}} would only
> notify {{onConsumedSubpartition}} when all the readers are empty.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
