[jira] [Commented] (FLINK-19907) Channel state (upstream) can be restored after emission of new elements (watermarks)
[ https://issues.apache.org/jira/browse/FLINK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225326#comment-17225326 ] Arvid Heise commented on FLINK-19907: - Also merged in release-1.11 as 8addf891ad97ce83bb22c559aef28411f3953014. Closing issue. > Channel state (upstream) can be restored after emission of new elements > (watermarks) > > > Key: FLINK-19907 > URL: https://issues.apache.org/jira/browse/FLINK-19907 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.12.0, 1.11.2 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Fix For: 1.12.0, 1.11.3 > > > In StreamTask.beforeInvoke: > 1. > operatorChain.initializeStateAndOpenOperators(createStreamTaskStateInitializer()); > 2. readRecoveredChannelState(); > But operatorChain.initializeStateAndOpenOperators can emit watermarks (or > potentially some other stream elements). > I've encountered this issue while adding an EndOfRecovery marker - in some > runs of in OverWindowITCase.testRowTimeBoundedPartitionedRangeOver the marker > was emitted after the watermark. > > cc: [~zjwang], [~pnowojski] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19907) Channel state (upstream) can be restored after emission of new elements (watermarks)
[ https://issues.apache.org/jira/browse/FLINK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224534#comment-17224534 ] Arvid Heise commented on FLINK-19907: - [~roman_khachatryan]'s PR fixes the second case while maintaining the first. The issue is that the watermarks are already emitted from a state that is after the in-flight data of the output and input. > Channel state (upstream) can be restored after emission of new elements > (watermarks) > > > Key: FLINK-19907 > URL: https://issues.apache.org/jira/browse/FLINK-19907 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.12.0, 1.11.2 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Fix For: 1.12.0 > > > In StreamTask.beforeInvoke: > 1. > operatorChain.initializeStateAndOpenOperators(createStreamTaskStateInitializer()); > 2. readRecoveredChannelState(); > But operatorChain.initializeStateAndOpenOperators can emit watermarks (or > potentially some other stream elements). > I've encountered this issue while adding an EndOfRecovery marker - in some > runs of in OverWindowITCase.testRowTimeBoundedPartitionedRangeOver the marker > was emitted after the watermark. > > cc: [~zjwang], [~pnowojski] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19907) Channel state (upstream) can be restored after emission of new elements (watermarks)
[ https://issues.apache.org/jira/browse/FLINK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224536#comment-17224536 ] Arvid Heise commented on FLINK-19907: - Merged into master as 3ab31a0473c01c27feb3b64c3f379ebe481a71ce. > Channel state (upstream) can be restored after emission of new elements > (watermarks) > > > Key: FLINK-19907 > URL: https://issues.apache.org/jira/browse/FLINK-19907 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.12.0, 1.11.2 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Fix For: 1.12.0 > > > In StreamTask.beforeInvoke: > 1. > operatorChain.initializeStateAndOpenOperators(createStreamTaskStateInitializer()); > 2. readRecoveredChannelState(); > But operatorChain.initializeStateAndOpenOperators can emit watermarks (or > potentially some other stream elements). > I've encountered this issue while adding an EndOfRecovery marker - in some > runs of in OverWindowITCase.testRowTimeBoundedPartitionedRangeOver the marker > was emitted after the watermark. > > cc: [~zjwang], [~pnowojski] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19907) Channel state (upstream) can be restored after emission of new elements (watermarks)
[ https://issues.apache.org/jira/browse/FLINK-19907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224507#comment-17224507 ] Zhijiang commented on FLINK-19907: -- I am not quite sure the actual semantic for the new emitted elements while initializing operator state here. But I think we should consider two issues for guaranteeing the precision. * Determinacy for repeated restore: That means the behavior should be consistent while executing the state restore multiple times. * Consistency with normal running: Assume the new emitted elements also exist while state snapshot, what is the sequence between them and in-flight channel state, then we should also obey the same sequence after restoring. > Channel state (upstream) can be restored after emission of new elements > (watermarks) > > > Key: FLINK-19907 > URL: https://issues.apache.org/jira/browse/FLINK-19907 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.12.0, 1.11.2 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Fix For: 1.12.0 > > > In StreamTask.beforeInvoke: > 1. > operatorChain.initializeStateAndOpenOperators(createStreamTaskStateInitializer()); > 2. readRecoveredChannelState(); > But operatorChain.initializeStateAndOpenOperators can emit watermarks (or > potentially some other stream elements). > I've encountered this issue while adding an EndOfRecovery marker - in some > runs of in OverWindowITCase.testRowTimeBoundedPartitionedRangeOver the marker > was emitted after the watermark. > > cc: [~zjwang], [~pnowojski] -- This message was sent by Atlassian Jira (v8.3.4#803005)
