[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=201858=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201858 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 21/Feb/19 10:32 Start Date: 21/Feb/19 10:32 Worklog Time Spent: 10m Work Description: mxm commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465947895 Thanks for merging! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 201858) Time Spent: 6h 20m (was: 6h 10m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 6h 20m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=201628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201628 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 20/Feb/19 21:36 Start Date: 20/Feb/19 21:36 Worklog Time Spent: 10m Work Description: aaltay commented on pull request #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 201628) Time Spent: 6h 10m (was: 6h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=201596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201596 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 20/Feb/19 20:44 Start Date: 20/Feb/19 20:44 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465746463 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 201596) Time Spent: 6h (was: 5h 50m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 6h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=201025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-201025 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 20/Feb/19 01:46 Start Date: 20/Feb/19 01:46 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465385561 There is still some missing containers (dataflow runner harness). Python containers are published. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 201025) Time Spent: 5h 50m (was: 5h 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5h 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200902 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 19/Feb/19 21:46 Start Date: 19/Feb/19 21:46 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465323737 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200902) Time Spent: 5h 40m (was: 5.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5h 40m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200766 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 19/Feb/19 16:52 Start Date: 19/Feb/19 16:52 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465216424 Dataflow containers for the release branch was not published. I did that yesterday, re-trying the test. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200766) Time Spent: 5.5h (was: 5h 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200765 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 19/Feb/19 16:51 Start Date: 19/Feb/19 16:51 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465215987 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200765) Time Spent: 5h 20m (was: 5h 10m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200457 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 19/Feb/19 10:27 Start Date: 19/Feb/19 10:27 Worklog Time Spent: 10m Work Description: mxm commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-465075520 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200457) Time Spent: 5h 10m (was: 5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200159 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 17:43 Start Date: 18/Feb/19 17:43 Worklog Time Spent: 10m Work Description: mxm commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-464824532 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200159) Time Spent: 5h (was: 4h 50m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200158 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 17:43 Start Date: 18/Feb/19 17:43 Worklog Time Spent: 10m Work Description: mxm commented on issue #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-6678 URL: https://github.com/apache/beam/pull/7874#issuecomment-464824507 Python tests seem to be flaky: https://builds.apache.org/job/beam_PreCommit_Python_Commit/4472/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200158) Time Spent: 4h 50m (was: 4h 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200136 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 16:28 Start Date: 18/Feb/19 16:28 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7874: [release-2.11.0] Backport for BEAM-6650 and BEAM-7849 URL: https://github.com/apache/beam/pull/7874 Backport to `release-2.11.0` from #7810 and #7849. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- See [.test-infra/jenkins/README](../.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200136) Time Spent: 4h 40m (was: 4.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Elements emitted during the
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200132 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 16:17 Start Date: 18/Feb/19 16:17 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200132) Time Spent: 4.5h (was: 4h 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200129 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 16:13 Start Date: 18/Feb/19 16:13 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-464793773 Thanks for taking a look! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200129) Time Spent: 4h 20m (was: 4h 10m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=200095=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200095 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 18/Feb/19 15:16 Start Date: 18/Feb/19 15:16 Worklog Time Spent: 10m Work Description: aljoscha commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-464769857 I think this looks good now! Plus, it removes quite some complexity. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 200095) Time Spent: 4h 10m (was: 4h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=199891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199891 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 17/Feb/19 22:30 Start Date: 17/Feb/19 22:30 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-464519090 @aljoscha I've replaced the `KeyGroupStateInternals` with `SplitStateInternals`. Historically there might have been a reason for the two ways of buffering elements during snapshotting, but as it stands now the split state is the way to go. We don't gain anything by distributing the buffered output elements onto key groups. The split state even leads to better distribution of the buffered state, since it will be distributed round-robin when restoring. I think this is the best way to fix the leak in the snapshotting when elements are emitted in `finalizeBundle`. Will follow-up with the `prepareSnapshotPreBarrier` solution for 1.6/1.7, but it would be great if we can get this in for the upcoming 2.11.0 release. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 199891) Time Spent: 3h 50m (was: 3h 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=199892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199892 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 17/Feb/19 22:30 Start Date: 17/Feb/19 22:30 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-464519122 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 199892) Time Spent: 4h (was: 3h 50m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=199205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199205 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 15/Feb/19 12:44 Start Date: 15/Feb/19 12:44 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-464037281 CC @tweise This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 199205) Time Spent: 3h 40m (was: 3.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198850 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 18:48 Start Date: 14/Feb/19 18:48 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463745076 @aljoscha Can you take another look? I think it should be good to go. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198850) Time Spent: 3.5h (was: 3h 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198746 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 15:44 Start Date: 14/Feb/19 15:44 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463676026 I was thinking we want to preserve the order of the elements emitted during `finalizeBundle`. I guess there is only ever one key set during finalization and the order of the bag state is guaranteed to be preserved. So we should be good. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198746) Time Spent: 3h 20m (was: 3h 10m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[
https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198739
]
ASF GitHub Bot logged work on BEAM-6650:
Author: ASF GitHub Bot
Created on: 14/Feb/19 15:31
Start Date: 14/Feb/19 15:31
Worklog Time Spent: 10m
Work Description: aljoscha commented on pull request #7810: [BEAM-6650]
Add bundle test with checkpointing for keyed processing
URL: https://github.com/apache/beam/pull/7810#discussion_r256883235
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
##
@@ -106,337 +80,150 @@ public K getKey() {
@Override
public T state(
final StateNamespace namespace, StateTag address, final
StateContext context) {
-
-return address.bind(
-new StateTag.StateBinder() {
-
- @Override
- public ValueState bindValue(StateTag>
address, Coder coder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
ValueState.class.getSimpleName()));
- }
-
- @Override
- public BagState bindBag(StateTag> address,
Coder elemCoder) {
-
-return new FlinkKeyGroupBagState<>(address, namespace, elemCoder);
- }
-
- @Override
- public SetState bindSet(StateTag> address,
Coder elemCoder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
SetState.class.getSimpleName()));
- }
-
- @Override
- public MapState bindMap(
- StateTag> spec,
- Coder mapKeyCoder,
- Coder mapValueCoder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
MapState.class.getSimpleName()));
- }
-
- @Override
- public
- CombiningState bindCombiningValue(
- StateTag> address,
- Coder accumCoder,
- Combine.CombineFn combineFn) {
-throw new UnsupportedOperationException("bindCombiningValue is not
supported.");
- }
-
- @Override
- public
- CombiningState
bindCombiningValueWithContext(
- StateTag> address,
- Coder accumCoder,
- CombineWithContext.CombineFnWithContext combineFn) {
-throw new UnsupportedOperationException(
-"bindCombiningValueWithContext is not supported.");
- }
-
- @Override
- public WatermarkHoldState bindWatermark(
- StateTag address, TimestampCombiner
timestampCombiner) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
CombiningState.class.getSimpleName()));
- }
-});
- }
-
- /**
- * Reference from {@link Combine.CombineFn}.
- *
- * Accumulators are stored in each KeyGroup, call addInput() when a
element comes, call
- * extractOutput() to produce the desired value when need to read data.
- */
- interface KeyGroupCombiner {
-
-/**
- * Returns a new, mutable accumulator value, representing the accumulation
of zero input values.
- */
-AccumT createAccumulator();
-
-/** Adds the given input value to the given accumulator, returning the new
accumulator value. */
-AccumT addInput(AccumT accumulator, InputT input);
-
-/**
- * Returns the output value that is the result of all accumulators from
KeyGroups that are
- * assigned to this operator.
- */
-OutputT extractOutput(Iterable accumulators);
- }
-
- private abstract class AbstractKeyGroupState {
-
-private String stateName;
-private String namespace;
-private Coder coder;
-private KeyGroupCombiner keyGroupCombiner;
-
-AbstractKeyGroupState(
-String stateName,
-String namespace,
-Coder coder,
-KeyGroupCombiner keyGroupCombiner) {
- this.stateName = stateName;
- this.namespace = namespace;
- this.coder = coder;
- this.keyGroupCombiner = keyGroupCombiner;
-}
-
-/** Choose keyGroup of input and addInput to accumulator. */
-void addInput(InputT input) {
- int keyGroupIdx = keyedStateBackend.getCurrentKeyGroupIndex();
- int localIdx = getIndexForKeyGroup(keyGroupIdx);
- Map, Map>> stateTable =
stateTables[localIdx];
- Tuple2, Map> tuple2 = stateTable.get(stateName);
- if (tuple2 == null) {
-tuple2 = new Tuple2<>();
-tuple2.f0 = coder;
-tuple2.f1 = new HashMap<>();
-stateTable.put(stateName, tuple2);
- }
- Map map = (Map) tuple2.f1;
- AccumT accumulator = map.get(namespace);
- if (accumulator == null) {
-
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[
https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198687
]
ASF GitHub Bot logged work on BEAM-6650:
Author: ASF GitHub Bot
Created on: 14/Feb/19 14:09
Start Date: 14/Feb/19 14:09
Worklog Time Spent: 10m
Work Description: mxm commented on pull request #7810: [BEAM-6650] Add
bundle test with checkpointing for keyed processing
URL: https://github.com/apache/beam/pull/7810#discussion_r256840389
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkKeyGroupStateInternals.java
##
@@ -106,337 +80,150 @@ public K getKey() {
@Override
public T state(
final StateNamespace namespace, StateTag address, final
StateContext context) {
-
-return address.bind(
-new StateTag.StateBinder() {
-
- @Override
- public ValueState bindValue(StateTag>
address, Coder coder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
ValueState.class.getSimpleName()));
- }
-
- @Override
- public BagState bindBag(StateTag> address,
Coder elemCoder) {
-
-return new FlinkKeyGroupBagState<>(address, namespace, elemCoder);
- }
-
- @Override
- public SetState bindSet(StateTag> address,
Coder elemCoder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
SetState.class.getSimpleName()));
- }
-
- @Override
- public MapState bindMap(
- StateTag> spec,
- Coder mapKeyCoder,
- Coder mapValueCoder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
MapState.class.getSimpleName()));
- }
-
- @Override
- public
- CombiningState bindCombiningValue(
- StateTag> address,
- Coder accumCoder,
- Combine.CombineFn combineFn) {
-throw new UnsupportedOperationException("bindCombiningValue is not
supported.");
- }
-
- @Override
- public
- CombiningState
bindCombiningValueWithContext(
- StateTag> address,
- Coder accumCoder,
- CombineWithContext.CombineFnWithContext combineFn) {
-throw new UnsupportedOperationException(
-"bindCombiningValueWithContext is not supported.");
- }
-
- @Override
- public WatermarkHoldState bindWatermark(
- StateTag address, TimestampCombiner
timestampCombiner) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported",
CombiningState.class.getSimpleName()));
- }
-});
- }
-
- /**
- * Reference from {@link Combine.CombineFn}.
- *
- * Accumulators are stored in each KeyGroup, call addInput() when a
element comes, call
- * extractOutput() to produce the desired value when need to read data.
- */
- interface KeyGroupCombiner {
-
-/**
- * Returns a new, mutable accumulator value, representing the accumulation
of zero input values.
- */
-AccumT createAccumulator();
-
-/** Adds the given input value to the given accumulator, returning the new
accumulator value. */
-AccumT addInput(AccumT accumulator, InputT input);
-
-/**
- * Returns the output value that is the result of all accumulators from
KeyGroups that are
- * assigned to this operator.
- */
-OutputT extractOutput(Iterable accumulators);
- }
-
- private abstract class AbstractKeyGroupState {
-
-private String stateName;
-private String namespace;
-private Coder coder;
-private KeyGroupCombiner keyGroupCombiner;
-
-AbstractKeyGroupState(
-String stateName,
-String namespace,
-Coder coder,
-KeyGroupCombiner keyGroupCombiner) {
- this.stateName = stateName;
- this.namespace = namespace;
- this.coder = coder;
- this.keyGroupCombiner = keyGroupCombiner;
-}
-
-/** Choose keyGroup of input and addInput to accumulator. */
-void addInput(InputT input) {
- int keyGroupIdx = keyedStateBackend.getCurrentKeyGroupIndex();
- int localIdx = getIndexForKeyGroup(keyGroupIdx);
- Map, Map>> stateTable =
stateTables[localIdx];
- Tuple2, Map> tuple2 = stateTable.get(stateName);
- if (tuple2 == null) {
-tuple2 = new Tuple2<>();
-tuple2.f0 = coder;
-tuple2.f1 = new HashMap<>();
-stateTable.put(stateName, tuple2);
- }
- Map map = (Map) tuple2.f1;
- AccumT accumulator = map.get(namespace);
- if (accumulator == null) {
-
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198688 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 14:09 Start Date: 14/Feb/19 14:09 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256838143 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java ## @@ -295,14 +295,10 @@ public void initializeState(StateInitializationContext context) throws Exception sideInputReader = NullSideInputReader.of(sideInputs); -// maybe init by initializeState Review comment: There is no other place where `nonKeyedStateInternals` is set or used. It is solely used for the `OutputManagerFactory` which is initialized below. Actually, your comment made me realize that we should move the initialization there. Have updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198688) Time Spent: 2h 40m (was: 2.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198689 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 14:10 Start Date: 14/Feb/19 14:10 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256847535 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java ## @@ -295,14 +295,10 @@ public void initializeState(StateInitializationContext context) throws Exception sideInputReader = NullSideInputReader.of(sideInputs); -// maybe init by initializeState Review comment: btw, your comment was here twice and I deleted the duplicate :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198689) Time Spent: 2h 50m (was: 2h 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198678 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 13:47 Start Date: 14/Feb/19 13:47 Worklog Time Spent: 10m Work Description: aljoscha commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256827198 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java ## @@ -295,14 +295,10 @@ public void initializeState(StateInitializationContext context) throws Exception sideInputReader = NullSideInputReader.of(sideInputs); -// maybe init by initializeState Review comment: Why'd you remove this check? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198678) Time Spent: 2.5h (was: 2h 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198666 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 13:21 Start Date: 14/Feb/19 13:21 Worklog Time Spent: 10m Work Description: aljoscha commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256827198 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java ## @@ -295,14 +295,10 @@ public void initializeState(StateInitializationContext context) throws Exception sideInputReader = NullSideInputReader.of(sideInputs); -// maybe init by initializeState Review comment: Why'd you remove this check? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198666) Time Spent: 2h 20m (was: 2h 10m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198663 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 13:21 Start Date: 14/Feb/19 13:21 Worklog Time Spent: 10m Work Description: aljoscha commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256823701 ## File path: runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/custom/AllKeyStateFunction.java ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.flink.translation.wrappers.streaming.state.custom; + +import java.util.ArrayDeque; +import java.util.Deque; +import org.apache.flink.api.common.state.ListState; +import org.apache.flink.runtime.state.KeyedStateFunction; + +/** Flink 1.6 specific KeyedStateFunction. */ Review comment: typo: 1.7 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198663) Time Spent: 1h 50m (was: 1h 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198665 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 13:21 Start Date: 14/Feb/19 13:21 Worklog Time Spent: 10m Work Description: aljoscha commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256827143 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java ## @@ -295,14 +295,10 @@ public void initializeState(StateInitializationContext context) throws Exception sideInputReader = NullSideInputReader.of(sideInputs); -// maybe init by initializeState Review comment: Why'd you remove this check? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198665) Time Spent: 2h 10m (was: 2h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198664 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 14/Feb/19 13:21 Start Date: 14/Feb/19 13:21 Worklog Time Spent: 10m Work Description: aljoscha commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#discussion_r256823817 ## File path: runners/flink/src/1.5/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/custom/AllKeyStateFunction.java ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.beam.runners.flink.translation.wrappers.streaming.state.custom; + +import java.util.ArrayDeque; +import java.util.Deque; +import org.apache.flink.api.common.state.ListState; +import org.apache.flink.runtime.state.KeyedStateFunction; + +/** Flink 1.6 specific KeyedStateFunction. */ Review comment: 1.5? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198664) Time Spent: 2h (was: 1h 50m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198263 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 18:33 Start Date: 13/Feb/19 18:33 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463313948 @aljoscha I'm assuming insertion order at the state backend at the moment. Not sure if that assumption holds. At least for the tests it works. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198263) Time Spent: 1h 40m (was: 1.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198249 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 18:06 Start Date: 13/Feb/19 18:06 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463304417 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198249) Time Spent: 1.5h (was: 1h 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198205 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 16:50 Start Date: 13/Feb/19 16:50 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463275855 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198205) Time Spent: 1h 10m (was: 1h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198204 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 16:50 Start Date: 13/Feb/19 16:50 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463275837 Batch execution fails for ValidatesRunner, due to memory issues on jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198204) Time Spent: 1h (was: 50m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198176 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 16:04 Start Date: 13/Feb/19 16:04 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463257230 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198176) Time Spent: 50m (was: 40m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198155 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 15:36 Start Date: 13/Feb/19 15:36 Worklog Time Spent: 10m Work Description: mxm commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463245737 Thanks for your input. That was very helpful. - For 1.6/1.7, `StreamOperator.prepareSnapshotPreBarrier()` is a great fix which allows us to get rid off a lot of code. We can remove `FlinkSplitStateInternals` and `FlinkKeyGroupStateInternals` because we can just emit pending elements in `prepareSnapshotPreBarrier`. I've just done the change and it was very straight-forward. However, I think I hold it back until we drop 1.5 support. In light of the upcoming release, I'd first fix this the 1.5 way. - For 1.5, `KeyedStateBackend.applyToAllKeys()` was a good idea, but unfortunately the `KeyedStateFunction` changed from an abstract class to an interface. The build structure now permits different code paths for 1.5/1.6/1.7, so I'm taking advantage of this feature for the implementation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198155) Time Spent: 40m (was: 0.5h) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=198011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198011 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 13/Feb/19 10:24 Start Date: 13/Feb/19 10:24 Worklog Time Spent: 10m Work Description: aljoscha commented on issue #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810#issuecomment-463144070 I think we can fix this for Flink 1.5 by using `KeyedStateBackend.applyToAllKeys()`, which was introduced in 1.5. This allows using a `KeyedStateBackend` and iterate over the state for all keys, which is needed to access all state, which wasn't possible before. That's also why before there was this (admittedly strange) solution with the key-group backend. I think the good fix for this is to use `StreamOperator.prepareSnapshotPreBarrier()`: https://github.com/apache/flink/blob/e1d1234477c731fe3f398c7f3f12123f73764242/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/StreamOperator.java#L114. I remember that I wanted this added to Flink so that we can use it for such cases in Beam. I.e. instead of checkpointing the data that we output from `finalize()`, we can simply forward it before the checkpoint happens. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 198011) Time Spent: 0.5h (was: 20m) > FlinkRunner fails to checkpoint elements emitted during finishBundle > > > Key: BEAM-6650 > URL: https://issues.apache.org/jira/browse/BEAM-6650 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Fix For: 2.11.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Elements emitted during the finalizeBundle call in snapshopState are lost > after the pipeline is restored. This only happens when the operator is keyed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6650) FlinkRunner fails to checkpoint elements emitted during finishBundle
[ https://issues.apache.org/jira/browse/BEAM-6650?focusedWorklogId=197143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-197143 ] ASF GitHub Bot logged work on BEAM-6650: Author: ASF GitHub Bot Created on: 11/Feb/19 20:22 Start Date: 11/Feb/19 20:22 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7810: [BEAM-6650] Add bundle test with checkpointing for keyed processing URL: https://github.com/apache/beam/pull/7810 The checkpointing does not work correctly when FlinkKeyGroupStateInternals are used, i.e. when the DoFn uses keyed KV input. Any elements emitted during finalizeBundle() will not be checkpointed. The test demonstrates the desired behavior. Trying to fix this across Flink 1.5 - 1.7 is a bit tricky due to the different way the timer state is treated. The fix in `DoFnOperator` only works for 1.5. @aljoscha Perhaps you can give some input on this? Couldn't we use Flink's keyed operator state for the buffer output? Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [