[jira] [Commented] (FLINK-13435) Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898790#comment-16898790 ] Chesnay Schepler commented on FLINK-13435: -- [~azagrebin] Please do not add 1.10.0 as a fixVersion if it is also fixed for 1.9.0. > Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per > partition type > > > Key: FLINK-13435 > URL: https://issues.apache.org/jira/browse/FLINK-13435 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In a long term we do not need auto-release semantics for blocking > (persistent) partition. We expect them always to be released externally by JM > and assume they can be consumed multiple times. > The pipelined partitions have always only one consumer and one consumption > attempt. Afterwards they can be always released automatically. > ShuffleDescriptor.ReleaseType was introduced to make release semantics more > flexible but it is not needed in a long term. > FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be > able to fallback to 1.8 behaviour without the partition tracker and JM taking > care about blocking partition release. We can make this option specific for > NettyShuffleEnvironment which was the only existing shuffle service before. > If it is activated then the blocking partition is also auto-released on a > consumption attempt as it was before. The fine-grained recovery will just not > find the partition after the job restart in this case and will restart the > producer. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13435) Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898697#comment-16898697 ] zhijiang commented on FLINK-13435: -- Thanks for helping merge into 1.9, I forgot that. [~Zentol] > Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per > partition type > > > Key: FLINK-13435 > URL: https://issues.apache.org/jira/browse/FLINK-13435 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0, 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In a long term we do not need auto-release semantics for blocking > (persistent) partition. We expect them always to be released externally by JM > and assume they can be consumed multiple times. > The pipelined partitions have always only one consumer and one consumption > attempt. Afterwards they can be always released automatically. > ShuffleDescriptor.ReleaseType was introduced to make release semantics more > flexible but it is not needed in a long term. > FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be > able to fallback to 1.8 behaviour without the partition tracker and JM taking > care about blocking partition release. We can make this option specific for > NettyShuffleEnvironment which was the only existing shuffle service before. > If it is activated then the blocking partition is also auto-released on a > consumption attempt as it was before. The fine-grained recovery will just not > find the partition after the job restart in this case and will restart the > producer. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13435) Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898667#comment-16898667 ] Chesnay Schepler commented on FLINK-13435: -- 1.9: b4d7bff09a3d4c1816dc235d616753095f626243 > Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per > partition type > > > Key: FLINK-13435 > URL: https://issues.apache.org/jira/browse/FLINK-13435 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In a long term we do not need auto-release semantics for blocking > (persistent) partition. We expect them always to be released externally by JM > and assume they can be consumed multiple times. > The pipelined partitions have always only one consumer and one consumption > attempt. Afterwards they can be always released automatically. > ShuffleDescriptor.ReleaseType was introduced to make release semantics more > flexible but it is not needed in a long term. > FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be > able to fallback to 1.8 behaviour without the partition tracker and JM taking > care about blocking partition release. We can make this option specific for > NettyShuffleEnvironment which was the only existing shuffle service before. > If it is activated then the blocking partition is also auto-released on a > consumption attempt as it was before. The fine-grained recovery will just not > find the partition after the job restart in this case and will restart the > producer. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13435) Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898661#comment-16898661 ] Chesnay Schepler commented on FLINK-13435: -- [~zjwang] This was marked as fixed for 1.9, but only merged to master. > Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per > partition type > > > Key: FLINK-13435 > URL: https://issues.apache.org/jira/browse/FLINK-13435 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In a long term we do not need auto-release semantics for blocking > (persistent) partition. We expect them always to be released externally by JM > and assume they can be consumed multiple times. > The pipelined partitions have always only one consumer and one consumption > attempt. Afterwards they can be always released automatically. > ShuffleDescriptor.ReleaseType was introduced to make release semantics more > flexible but it is not needed in a long term. > FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be > able to fallback to 1.8 behaviour without the partition tracker and JM taking > care about blocking partition release. We can make this option specific for > NettyShuffleEnvironment which was the only existing shuffle service before. > If it is activated then the blocking partition is also auto-released on a > consumption attempt as it was before. The fine-grained recovery will just not > find the partition after the job restart in this case and will restart the > producer. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (FLINK-13435) Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type
[ https://issues.apache.org/jira/browse/FLINK-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16895881#comment-16895881 ] Andrey Zagrebin commented on FLINK-13435: - [~zjwang] [~Zentol] Could you review? > Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per > partition type > > > Key: FLINK-13435 > URL: https://issues.apache.org/jira/browse/FLINK-13435 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination, Runtime / Network >Affects Versions: 1.9.0 >Reporter: Andrey Zagrebin >Assignee: Andrey Zagrebin >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0, 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > In a long term we do not need auto-release semantics for blocking > (persistent) partition. We expect them always to be released externally by JM > and assume they can be consumed multiple times. > The pipelined partitions have always only one consumer and one consumption > attempt. Afterwards they can be always released automatically. > ShuffleDescriptor.ReleaseType was introduced to make release semantics more > flexible but it is not needed in a long term. > FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be > able to fallback to 1.8 behaviour without the partition tracker and JM taking > care about blocking partition release. We can make this option specific for > NettyShuffleEnvironment which was the only existing shuffle service before. > If it is activated then the blocking partition is also auto-released on a > consumption attempt as it was before. The fine-grained recovery will just not > find the partition after the job restart in this case and will restart the > producer. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
