[jira] [Commented] (FLINK-13493) BoundedBlockingSubpartition only notifies onConsumedSubpartition when all the readers are empty
[
https://issues.apache.org/jira/browse/FLINK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897998#comment-16897998
]
zhijiang commented on FLINK-13493:
--
Thanks for the clarification [~azagrebin]!
I agree it should be no problem now because we do not have multiple readers for
one subpartition concurrently. The current semantic of consuming multiple times
for one subpartition is only considering the failover case, that means next
consumption attempt only comes after the previous attempt canceled. The set
structure of readers in `BoundedBlockingSubpartition` make me think of this
unnecessary concern.
> BoundedBlockingSubpartition only notifies onConsumedSubpartition when all the
> readers are empty
> ---
>
> Key: FLINK-13493
> URL: https://issues.apache.org/jira/browse/FLINK-13493
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> In previous implementation, it would always notify the {{ResultPartition}} of
> consumed subpartition if any reader view is released. Based on the
> reference-counter release strategy it might cause problems if one blocking
> subpartition has multiple readers. That means the whole result partition
> might be released but there are still alive readers in some subpartitions.
> Although the default release strategy for blocking partition is based on
> JM/scheduler notification atm. But if we switch the option to based on
> consumption notification it would cause problems. And from the subpartition
> side it should has the right behavior no matter what is the specific release
> strategy in upper layer.
> In order to fix this bug, the {{BoundedBlockingSubpartition}} would only
> notify {{onConsumedSubpartition}} when all the readers are empty.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-13493) BoundedBlockingSubpartition only notifies onConsumedSubpartition when all the readers are empty
[
https://issues.apache.org/jira/browse/FLINK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896834#comment-16896834
]
Andrey Zagrebin commented on FLINK-13493:
-
As I think, we also discussed this with [~StephanEwen]. Now it should not be a
problem because the existing connection should always end one way or another,
as we already discussed in FLINK-13478, and the existing readers should be all
closed, eventually triggering the file deletion. We also switched from counters
to boolean flags per subpartition to avoid counting of unconsumed subpartitions
by multiple reader confirmation from other subpartitions. Eventually, we do not
need the release on consumption for blocking partitions, it stays as a safety
net at the moment to switch to the previous behaviour. At the moment this issue
looks like for me the same as concerns described FLINK-13478.
> BoundedBlockingSubpartition only notifies onConsumedSubpartition when all the
> readers are empty
> ---
>
> Key: FLINK-13493
> URL: https://issues.apache.org/jira/browse/FLINK-13493
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>
> In previous implementation, it would always notify the {{ResultPartition}} of
> consumed subpartition if any reader view is released. Based on the
> reference-counter release strategy it might cause problems if one blocking
> subpartition has multiple readers. That means the whole result partition
> might be released but there are still alive readers in some subpartitions.
> Although the default release strategy for blocking partition is based on
> JM/scheduler notification atm. But if we switch the option to based on
> consumption notification it would cause problems. And from the subpartition
> side it should has the right behavior no matter what is the specific release
> strategy in upper layer.
> In order to fix this bug, the {{BoundedBlockingSubpartition}} would only
> notify {{onConsumedSubpartition}} when all the readers are empty.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
