[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187919&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187919 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 22:22 Start Date: 21/Jan/19 22:22 Worklog Time Spent: 10m Work Description: mxm commented on issue #7584: [release][BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7584#issuecomment-456213613 There appears to be a problem with `:beam-sdks-python-precommit-dataflow:precommitIT` unrelated to the changes of this PR. The PR doesn't make changes to Dataflow or any of its components. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187919) Time Spent: 12h 10m (was: 12h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Fix For: 2.11.0 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187921&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187921 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 22:24 Start Date: 21/Jan/19 22:24 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7584: [release][BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7584 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187921) Time Spent: 12h 20m (was: 12h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Fix For: 2.11.0 > > Time Spent: 12h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187914&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187914 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 21:16 Start Date: 21/Jan/19 21:16 Worklog Time Spent: 10m Work Description: mxm commented on issue #7584: [release][BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7584#issuecomment-456201034 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187914) Time Spent: 12h (was: 11h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Fix For: 2.11.0 > > Time Spent: 12h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187902&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187902 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 20:35 Start Date: 21/Jan/19 20:35 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7584: [release][BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7584 Backported from master. CC @tweise Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187902) Time Spent: 11h 50m (was: 11h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Fix For: 2.11.0 > > Time Spent: 11h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pi
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187570 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 02:34 Start Date: 21/Jan/19 02:34 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187570) Time Spent: 11h 40m (was: 11.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 11h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187569&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187569
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 21/Jan/19 02:26
Start Date: 21/Jan/19 02:26
Worklog Time Spent: 10m
Work Description: mxm commented on issue #7362: [BEAM-5396] Assign
portable operator uids
URL: https://github.com/apache/beam/pull/7362#issuecomment-455929590
Some notes about the test status before merging:
1. Java PreCommit fails unrelated, Flink tests pass.
```
20:58:36 * What went wrong:
20:58:36 Execution failed for task
':beam-runners-direct-java:needsRunnerTests'.
20:58:36 > There were failing tests. See the report at:
file:///home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit@2/src/runners/direct-java/build/reports/tests/needsRunnerTests/index.html
```
2. Python Validates Runner `test_flattened_side_input` fails unrelated. This
test is a regression captured here:
https://issues.apache.org/jira/browse/BEAM-6473
```
20:46:15
==
20:46:15 ERROR: test_flattened_side_input (__main__.FlinkRunnerTest)
20:46:15
--
20:46:15 Traceback (most recent call last):
20:46:15 File "apache_beam/runners/portability/fn_api_runner_test.py",
line 205, in test_flattened_side_input
20:46:15 label='CheckFlattenOfSideInput')
20:46:15 File "apache_beam/pipeline.py", line 425, in __exit__
20:46:15 self.run().wait_until_finish()
20:46:15 File "apache_beam/runners/portability/portable_runner.py", line
349, in wait_until_finish
20:46:15 self._job_id, self._state, self._last_error_message()))
20:46:15 RuntimeError: Pipeline
test_flattened_side_input_1548035149.49_dc53d64a-ec8e-4fb5-bbf3-6b23c1e23c84
failed in state FAILED: java.lang.IllegalArgumentException: PCollectionNodes
[PCollectionNode{id=ref_PCollection_PCollection_12, PCollection=unique_name:
"17side3/Map(decode).None"
20:46:15 coder_id: "ref_Coder_BytesCoder_1"
20:46:15 is_bounded: BOUNDED
20:46:15 windowing_strategy_id: "ref_Windowing_Windowing_1"
20:46:15 }] were consumed but never produced
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 187569)
Time Spent: 11.5h (was: 11h 20m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187563 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 01:40 Start Date: 21/Jan/19 01:40 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455923841 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187563) Time Spent: 11h (was: 10h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 11h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187565 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 01:40 Start Date: 21/Jan/19 01:40 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455923868 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187565) Time Spent: 11h 20m (was: 11h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 11h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187564&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187564 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 01:40 Start Date: 21/Jan/19 01:40 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455923852 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187564) Time Spent: 11h 10m (was: 11h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 11h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187561&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187561 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 00:36 Start Date: 21/Jan/19 00:36 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455917760 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187561) Time Spent: 10h 50m (was: 10h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187560&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187560 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 00:36 Start Date: 21/Jan/19 00:36 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455917743 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187560) Time Spent: 10h 40m (was: 10.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187559 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 21/Jan/19 00:35 Start Date: 21/Jan/19 00:35 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455917703 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187559) Time Spent: 10.5h (was: 10h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187404&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187404 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 18:35 Start Date: 19/Jan/19 18:35 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455804396 Broken PreCommit tests: ``` org.apache.beam.runners.fnexecution.data.GrpcDataServiceTest.testMessageReceivedBySingleClientWhenThereAreMultipleClients org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187404) Time Spent: 10h 20m (was: 10h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187396&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187396 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:53 Start Date: 19/Jan/19 17:53 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455800987 Run Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187396) Time Spent: 10h 10m (was: 10h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187394&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187394 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:52 Start Date: 19/Jan/19 17:52 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455800894 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187394) Time Spent: 9h 50m (was: 9h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187395&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187395 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:52 Start Date: 19/Jan/19 17:52 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455800927 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187395) Time Spent: 10h (was: 9h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 10h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187387 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:33 Start Date: 19/Jan/19 17:33 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455799347 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187387) Time Spent: 9.5h (was: 9h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187388&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187388 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:33 Start Date: 19/Jan/19 17:33 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455799282 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187388) Time Spent: 9h 40m (was: 9.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187386&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187386 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:32 Start Date: 19/Jan/19 17:32 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455799282 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187386) Time Spent: 9h 20m (was: 9h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187385 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 17:28 Start Date: 19/Jan/19 17:28 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455798933 Broken PreCommit test: ```org.apache.beam.sdk.io.FileIOTest.testMatchWatchForNewFiles``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187385) Time Spent: 9h 10m (was: 9h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187355 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 15:59 Start Date: 19/Jan/19 15:59 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455791972 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187355) Time Spent: 9h (was: 8h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 9h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187354 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 15:58 Start Date: 19/Jan/19 15:58 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455791961 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187354) Time Spent: 8h 50m (was: 8h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187353&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187353 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 15:58 Start Date: 19/Jan/19 15:58 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455791949 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187353) Time Spent: 8h 40m (was: 8.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187241&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187241 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730934 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187241) Time Spent: 8.5h (was: 8h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187240 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730888 PreCommit failing due to #7575. https://builds.apache.org/job/beam_PreCommit_Java_Commit/3648/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187240) Time Spent: 8h 20m (was: 8h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187239&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187239 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 19/Jan/19 00:38 Start Date: 19/Jan/19 00:38 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455730888 PreCommit failing due to #7575. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187239) Time Spent: 8h 10m (was: 8h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187147&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187147 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 23:21 Start Date: 18/Jan/19 23:21 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455719046 Blocked on https://jira.apache.org/jira/browse/BEAM-6469 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187147) Time Spent: 8h (was: 7h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 8h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187118&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187118 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:45 Start Date: 18/Jan/19 22:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455712328 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187118) Time Spent: 7h 50m (was: 7h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187117 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:45 Start Date: 18/Jan/19 22:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455712294 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187117) Time Spent: 7h 40m (was: 7.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187114 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:39 Start Date: 18/Jan/19 22:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-45577 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187114) Time Spent: 7.5h (was: 7h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=187113&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-187113 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 18/Jan/19 22:39 Start Date: 18/Jan/19 22:39 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455711034 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 187113) Time Spent: 7h 20m (was: 7h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=186123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186123 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 17/Jan/19 02:14 Start Date: 17/Jan/19 02:14 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455016389 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186123) Time Spent: 7h (was: 6h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=186124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186124 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 17/Jan/19 02:14 Start Date: 17/Jan/19 02:14 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455016468 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186124) Time Spent: 7h 10m (was: 7h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 7h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=186121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186121 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 17/Jan/19 02:13 Start Date: 17/Jan/19 02:13 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455016372 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186121) Time Spent: 6h 40m (was: 6.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=186122&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-186122 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 17/Jan/19 02:13 Start Date: 17/Jan/19 02:13 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-455016389 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 186122) Time Spent: 6h 50m (was: 6h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=185876&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185876 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 16/Jan/19 16:55 Start Date: 16/Jan/19 16:55 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454854794 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185876) Time Spent: 6.5h (was: 6h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=185875&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185875 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 16/Jan/19 16:55 Start Date: 16/Jan/19 16:55 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454854676 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185875) Time Spent: 6h 20m (was: 6h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=185341&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185341 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 15/Jan/19 16:42 Start Date: 15/Jan/19 16:42 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454461428 Python Validates Runner is fixed now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185341) Time Spent: 6h 10m (was: 6h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=185340&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185340 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 15/Jan/19 16:42 Start Date: 15/Jan/19 16:42 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454461532 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185340) Time Spent: 6h (was: 5h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 6h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=185339&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185339 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 15/Jan/19 16:42 Start Date: 15/Jan/19 16:42 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454461428 Python Validates Runner is fixed now. Jenkins: Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185339) Time Spent: 5h 50m (was: 5h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184840 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 14/Jan/19 15:20 Start Date: 14/Jan/19 15:20 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454041577 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184840) Time Spent: 5h 40m (was: 5.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184829&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184829 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 14/Jan/19 14:45 Start Date: 14/Jan/19 14:45 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-454028672 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184829) Time Spent: 5.5h (was: 5h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184462 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 12/Jan/19 02:23 Start Date: 12/Jan/19 02:23 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453711696 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184462) Time Spent: 5h 20m (was: 5h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184460&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184460 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 12/Jan/19 02:02 Start Date: 12/Jan/19 02:02 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453710271 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184460) Time Spent: 5h 10m (was: 5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184273&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184273 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 11/Jan/19 17:16 Start Date: 11/Jan/19 17:16 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453590167 @mxm test failures appear repeatable: https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184273) Time Spent: 5h (was: 4h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184269 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 11/Jan/19 17:02 Start Date: 11/Jan/19 17:02 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453585875 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184269) Time Spent: 4h 50m (was: 4h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184267 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 11/Jan/19 17:00 Start Date: 11/Jan/19 17:00 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453585379 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184267) Time Spent: 4h 40m (was: 4.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184238&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184238 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 11/Jan/19 15:00 Start Date: 11/Jan/19 15:00 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453544519 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184238) Time Spent: 4.5h (was: 4h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=184010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184010 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 23:37 Start Date: 10/Jan/19 23:37 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453302620 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184010) Time Spent: 4h 20m (was: 4h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=183958&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183958 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 21:47 Start Date: 10/Jan/19 21:47 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453266406 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183958) Time Spent: 4h 10m (was: 4h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=183948&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183948 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 21:27 Start Date: 10/Jan/19 21:27 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453260382 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183948) Time Spent: 4h (was: 3h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 4h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=183942&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183942 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 21:25 Start Date: 10/Jan/19 21:25 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453259679 Tested with the original pipeline and the restore error is gone. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183942) Time Spent: 3h 50m (was: 3h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=183939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183939 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 21:20 Start Date: 10/Jan/19 21:20 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453258000 Run Python Flink ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183939) Time Spent: 3h 40m (was: 3.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=183934&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183934 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 10/Jan/19 21:09 Start Date: 10/Jan/19 21:09 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-453254528 @tweise Rebased to latest master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183934) Time Spent: 3.5h (was: 3h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=180793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180793 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 03/Jan/19 17:12 Start Date: 03/Jan/19 17:12 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-451211047 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180793) Time Spent: 3h 20m (was: 3h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179826&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179826 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Dec/18 19:24 Start Date: 30/Dec/18 19:24 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-450581317 That would be great. I rebased the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179826) Time Spent: 3h 10m (was: 3h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179822 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Dec/18 18:53 Start Date: 30/Dec/18 18:53 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-450579728 OK, I will test it with the original pipeline but it may take a bit before getting to it. Can we rebase the PR though? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179822) Time Spent: 3h (was: 2h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 3h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179536&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179536 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 28/Dec/18 19:06 Start Date: 28/Dec/18 19:06 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-450411071 >There is a commit from another PR included here, is that intentional? Yes, this is based on #7359 (see description) because the test wouldn't otherwise run. > Also, we you able to reproduce the restore error? As you wrote in the JIRA, it depends on the pipeline. We shouldn't see the restore error anymore because of the uid assigned to all stateful operators. Before it would depend on the internal operator ids which only works if the pipeline generation is deterministic. Perhaps you could re-run your pipeline with this PR to verify that the restore works? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179536) Time Spent: 2h 50m (was: 2h 40m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 2h 50m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179534 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 28/Dec/18 18:57 Start Date: 28/Dec/18 18:57 Worklog Time Spent: 10m Work Description: tweise commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-450409755 @mxm thank! There is a commit from another PR included here, is that intentional? Also, we you able to reproduce the restore error? If not, then I would need to do some more extensive testing for this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179534) Time Spent: 2h 40m (was: 2.5h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 2h 40m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179533&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179533 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 28/Dec/18 18:50 Start Date: 28/Dec/18 18:50 Worklog Time Spent: 10m Work Description: mxm commented on issue #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362#issuecomment-450408705 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179533) Time Spent: 2.5h (was: 2h 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 2.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=179172&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179172 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 27/Dec/18 18:33 Start Date: 27/Dec/18 18:33 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7362: [BEAM-5396] Assign portable operator uids URL: https://github.com/apache/beam/pull/7362 This adds operator uids to portable operators with state. A test verifies restoring from a Savepoint for both legacy and portable pipelines. CC @tweise Based on #7359. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 179172) Time Spent: 2h 20m (was: 2h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Maximilian Michels >Priority: Major > Labels: portability, portability-flink > Time Spent: 2h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171516
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 03/Dec/18 10:08
Start Date: 03/Dec/18 10:08
Worklog Time Spent: 10m
Work Description: mxm closed pull request #7169: [BEAM-5396] Savepoint
restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
index 9802defeb1a4..80a9ef860a4c 100644
---
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
+++
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
@@ -19,19 +19,31 @@
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Splitter;
+import java.net.URL;
+import java.util.Collections;
import java.util.List;
import javax.annotation.Nullable;
import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.JobExecutionResult;
import org.apache.flink.api.java.CollectionEnvironment;
import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.client.program.ClusterClient;
+import org.apache.flink.client.program.JobWithJars;
+import org.apache.flink.client.program.ProgramInvocationException;
+import org.apache.flink.client.program.StandaloneClusterClient;
+import org.apache.flink.client.program.rest.RestClusterClient;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.CoreOptions;
import org.apache.flink.configuration.GlobalConfiguration;
+import org.apache.flink.configuration.JobManagerOptions;
import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.jobgraph.SavepointRestoreSettings;
import org.apache.flink.runtime.state.StateBackend;
import org.apache.flink.streaming.api.TimeCharacteristic;
import
org.apache.flink.streaming.api.environment.CheckpointConfig.ExternalizedCheckpointCleanup;
+import org.apache.flink.streaming.api.environment.RemoteStreamEnvironment;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.graph.StreamGraph;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -128,7 +140,7 @@ static StreamExecutionEnvironment
createStreamExecutionEnvironment(
String masterUrl = options.getFlinkMaster();
Configuration flinkConfig = getFlinkConfiguration(confDir);
-StreamExecutionEnvironment flinkStreamEnv = null;
+final StreamExecutionEnvironment flinkStreamEnv;
// depending on the master, create the right environment.
if ("[local]".equals(masterUrl)) {
@@ -138,11 +150,22 @@ static StreamExecutionEnvironment
createStreamExecutionEnvironment(
} else if (masterUrl.matches(".*:\\d*")) {
List parts = Splitter.on(':').splitToList(masterUrl);
flinkConfig.setInteger(RestOptions.PORT, Integer.parseInt(parts.get(1)));
+
+ final SavepointRestoreSettings savepointRestoreSettings;
+ if (options.getSavepointPath() != null) {
+savepointRestoreSettings =
+SavepointRestoreSettings.forPath(
+options.getSavepointPath(),
options.getAllowNonRestoredState());
+ } else {
+savepointRestoreSettings = SavepointRestoreSettings.none();
+ }
+
flinkStreamEnv =
- StreamExecutionEnvironment.createRemoteEnvironment(
+ new BeamFlinkRemoteStreamEnvironment(
parts.get(0),
Integer.parseInt(parts.get(1)),
flinkConfig,
+ savepointRestoreSettings,
filesToStage.toArray(new String[filesToStage.size()]));
} else {
LOG.warn("Unrecognized Flink Master URL {}. Defaulting to [auto].",
masterUrl);
@@ -263,4 +286,82 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
+
+ /**
+ * Remote stream environment that supports job execution with restore from
savepoint.
+ *
+ * This class can be removed once Flink provides this functionality.
+ *
+ * TODO: https://issues.apache.org/jira/browse/BEAM-5396
+ */
+ private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
+private final SavepointRestoreSettings restoreSett
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171515 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 03/Dec/18 10:08 Start Date: 03/Dec/18 10:08 Worklog Time Spent: 10m Work Description: mxm commented on issue #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169#issuecomment-443655496 Very nice. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171515) Time Spent: 2h (was: 1h 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Time Spent: 2h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171134&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171134 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Nov/18 17:47 Start Date: 30/Nov/18 17:47 Worklog Time Spent: 10m Work Description: tweise commented on issue #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169#issuecomment-443283332 I have tested this with the synthetic source example: 1. Run example: `python -m apache_beam.examples.flink.flink_streaming_impulse --parallelism=1 --flink_master=localhost:8081` 2. Cancel with savepoint: `./bin/flink cancel -s /Users/tweise/flink/flink-1.5.5/savepoints/1 2be8be7dc874ec2bd88c6c232fefeb45` 3. Restore: `python -m apache_beam.examples.flink.flink_streaming_impulse --parallelism=1 --flink_master=localhost:8081 --savepoint_path=file:/Users/tweise/flink/flink-1.5.5/savepoints/1/savepoint-2be8be-bbf6cce24652` Then check the JM log to confirm the savepoint was used to resume. Testing this only works with "remote execution" and involves the Flink CLI. Given there is coverage in Flink for this, not sure it is worth the complexity and effort? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171134) Time Spent: 1h 20m (was: 1h 10m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Time Spent: 1h 20m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171145&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171145
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 17:52
Start Date: 30/Nov/18 17:52
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237947042
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -204,4 +204,17 @@
ExecutionMode getExecutionModeForBatch();
void setExecutionModeForBatch(ExecutionMode executionMode);
+
+ @Description("Savepoint restore path.")
+ String getSavepointPath();
Review comment:
Done!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171145)
Time Spent: 1h 50m (was: 1h 40m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171144&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171144
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 17:51
Start Date: 30/Nov/18 17:51
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237946939
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -138,11 +150,20 @@ static StreamExecutionEnvironment
createStreamExecutionEnvironment(
} else if (masterUrl.matches(".*:\\d*")) {
List parts = Splitter.on(':').splitToList(masterUrl);
flinkConfig.setInteger(RestOptions.PORT, Integer.parseInt(parts.get(1)));
+
+ SavepointRestoreSettings savepointRestoreSettings =
SavepointRestoreSettings.none();
+ if (options.getSavepointPath() != null) {
+savepointRestoreSettings =
+SavepointRestoreSettings.forPath(
+options.getSavepointPath(),
options.getAllowNonRestoredState());
+ }
Review comment:
It's probably more a matter of taste, I made the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171144)
Time Spent: 1h 40m (was: 1.5h)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171137&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171137
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 17:49
Start Date: 30/Nov/18 17:49
Worklog Time Spent: 10m
Work Description: tweise commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237946357
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -263,4 +284,81 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
+
+ /**
+ * TODO: https://issues.apache.org/jira/browse/BEAM-5396
+ *
+ * Remote stream environment that supports job execution with restore
from savepoint. This
+ * class can be removed once Flink provides this functionality.
+ */
+ private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
+private final SavepointRestoreSettings restoreSettings;
+
+public BeamFlinkRemoteStreamEnvironment(
+String host,
+int port,
+Configuration clientConfiguration,
+SavepointRestoreSettings restoreSettings,
+String... jarFiles) {
+ super(host, port, clientConfiguration, jarFiles, null);
+ this.restoreSettings = restoreSettings;
+}
+
+// copied from RemoteStreamEnvironment and augmented to pass savepoint
restore settings
+@Override
+protected JobExecutionResult executeRemotely(StreamGraph streamGraph,
List jarFiles)
+throws ProgramInvocationException {
+
+ List globalClasspaths = Collections.emptyList();
+ String host = super.getHost();
+ int port = super.getPort();
+
+ if (LOG.isInfoEnabled()) {
+LOG.info("Running remotely at {}:{}", host, port);
+ }
+
+ ClassLoader usercodeClassLoader =
+ JobWithJars.buildUserCodeClassLoader(
+ jarFiles, globalClasspaths, getClass().getClassLoader());
+
+ Configuration configuration = new Configuration();
+ configuration.addAll(super.getClientConfiguration());
+
+ configuration.setString(JobManagerOptions.ADDRESS, host);
+ configuration.setInteger(JobManagerOptions.PORT, port);
+
+ configuration.setInteger(RestOptions.PORT, port);
+
+ final ClusterClient client;
+ try {
+if
(CoreOptions.LEGACY_MODE.equals(configuration.getString(CoreOptions.MODE))) {
+ client = new StandaloneClusterClient(configuration);
+} else {
+ client = new RestClusterClient<>(configuration,
"RemoteStreamEnvironment");
+}
+ } catch (Exception e) {
+throw new ProgramInvocationException(
+"Cannot establish connection to JobManager: " + e.getMessage(), e);
+ }
+
+
client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
+
+ try {
+return client
+.run(streamGraph, jarFiles, globalClasspaths, usercodeClassLoader,
restoreSettings)
+.getJobExecutionResult();
+ } catch (ProgramInvocationException e) {
+throw e;
+ } catch (Exception e) {
+String term = e.getMessage() == null ? "." : (": " + e.getMessage());
+throw new ProgramInvocationException("The program execution failed" +
term, e);
Review comment:
This is unchanged copied, intentionally. I want to keep it as is, and in the
future replace with utility that lives in Flink.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171137)
Time Spent: 1.5h (was: 1h 20m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171109&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171109 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Nov/18 16:05 Start Date: 30/Nov/18 16:05 Worklog Time Spent: 10m Work Description: tweise commented on issue #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169#issuecomment-443251414 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171109) Time Spent: 1h 10m (was: 1h) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Time Spent: 1h 10m > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171035&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171035
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 10:19
Start Date: 30/Nov/18 10:19
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237808455
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -204,4 +204,17 @@
ExecutionMode getExecutionModeForBatch();
void setExecutionModeForBatch(ExecutionMode executionMode);
+
+ @Description("Savepoint restore path.")
+ String getSavepointPath();
Review comment:
Would be nice to test the default in `PipelineOptionstest#testdefaults`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171035)
Time Spent: 0.5h (was: 20m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171040&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171040 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Nov/18 10:23 Start Date: 30/Nov/18 10:23 Worklog Time Spent: 10m Work Description: mxm commented on issue #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169#issuecomment-443159214 As a side note, would it make sense to add a test for basic savepoint/restore functionality of a Beam pipeline? Have you tested restoring with the current PR? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171040) Time Spent: 1h (was: 50m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Time Spent: 1h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171038&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171038
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 10:19
Start Date: 30/Nov/18 10:19
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237808234
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -204,4 +204,17 @@
ExecutionMode getExecutionModeForBatch();
void setExecutionModeForBatch(ExecutionMode executionMode);
+
+ @Description("Savepoint restore path.")
+ String getSavepointPath();
Review comment:
Maybe extend the description a bit, also default set to `""`?
If specified, attempts to restore the streaming pipeline from the
provided Savepoint path.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171038)
Time Spent: 40m (was: 0.5h)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171039&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171039
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 10:19
Start Date: 30/Nov/18 10:19
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237808499
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineOptions.java
##
@@ -204,4 +204,17 @@
ExecutionMode getExecutionModeForBatch();
void setExecutionModeForBatch(ExecutionMode executionMode);
+
+ @Description("Savepoint restore path.")
+ String getSavepointPath();
+
+ void setSavepointPath(String path);
+
+ @Description(
+ "Flag indicating whether non restored state is allowed if the savepoint "
+ + "contains state for an operator that is no longer part of the
pipeline.")
+ @Default.Boolean(false)
Review comment:
Would be nice to test the default in `PipelineOptionstest#testdefaults`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171039)
Time Spent: 50m (was: 40m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171037&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171037
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 10:19
Start Date: 30/Nov/18 10:19
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237810589
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -263,4 +284,81 @@ private static void applyLatencyTrackingInterval(
long latencyTrackingInterval = options.getLatencyTrackingInterval();
config.setLatencyTrackingInterval(latencyTrackingInterval);
}
+
+ /**
+ * TODO: https://issues.apache.org/jira/browse/BEAM-5396
+ *
+ * Remote stream environment that supports job execution with restore
from savepoint. This
+ * class can be removed once Flink provides this functionality.
+ */
+ private static class BeamFlinkRemoteStreamEnvironment extends
RemoteStreamEnvironment {
+private final SavepointRestoreSettings restoreSettings;
+
+public BeamFlinkRemoteStreamEnvironment(
+String host,
+int port,
+Configuration clientConfiguration,
+SavepointRestoreSettings restoreSettings,
+String... jarFiles) {
+ super(host, port, clientConfiguration, jarFiles, null);
+ this.restoreSettings = restoreSettings;
+}
+
+// copied from RemoteStreamEnvironment and augmented to pass savepoint
restore settings
+@Override
+protected JobExecutionResult executeRemotely(StreamGraph streamGraph,
List jarFiles)
+throws ProgramInvocationException {
+
+ List globalClasspaths = Collections.emptyList();
+ String host = super.getHost();
+ int port = super.getPort();
+
+ if (LOG.isInfoEnabled()) {
+LOG.info("Running remotely at {}:{}", host, port);
+ }
+
+ ClassLoader usercodeClassLoader =
+ JobWithJars.buildUserCodeClassLoader(
+ jarFiles, globalClasspaths, getClass().getClassLoader());
+
+ Configuration configuration = new Configuration();
+ configuration.addAll(super.getClientConfiguration());
+
+ configuration.setString(JobManagerOptions.ADDRESS, host);
+ configuration.setInteger(JobManagerOptions.PORT, port);
+
+ configuration.setInteger(RestOptions.PORT, port);
+
+ final ClusterClient client;
+ try {
+if
(CoreOptions.LEGACY_MODE.equals(configuration.getString(CoreOptions.MODE))) {
+ client = new StandaloneClusterClient(configuration);
+} else {
+ client = new RestClusterClient<>(configuration,
"RemoteStreamEnvironment");
+}
+ } catch (Exception e) {
+throw new ProgramInvocationException(
+"Cannot establish connection to JobManager: " + e.getMessage(), e);
+ }
+
+
client.setPrintStatusDuringExecution(getConfig().isSysoutLoggingEnabled());
+
+ try {
+return client
+.run(streamGraph, jarFiles, globalClasspaths, usercodeClassLoader,
restoreSettings)
+.getJobExecutionResult();
+ } catch (ProgramInvocationException e) {
+throw e;
+ } catch (Exception e) {
+String term = e.getMessage() == null ? "." : (": " + e.getMessage());
+throw new ProgramInvocationException("The program execution failed" +
term, e);
Review comment:
Not sure about the `term` string. I think we can drop it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171037)
Time Spent: 40m (was: 0.5h)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evo
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[
https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171034
]
ASF GitHub Bot logged work on BEAM-5396:
Author: ASF GitHub Bot
Created on: 30/Nov/18 10:19
Start Date: 30/Nov/18 10:19
Worklog Time Spent: 10m
Work Description: mxm commented on a change in pull request #7169:
[BEAM-5396] Savepoint restore option in Flink runner
URL: https://github.com/apache/beam/pull/7169#discussion_r237809921
##
File path:
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java
##
@@ -138,11 +150,20 @@ static StreamExecutionEnvironment
createStreamExecutionEnvironment(
} else if (masterUrl.matches(".*:\\d*")) {
List parts = Splitter.on(':').splitToList(masterUrl);
flinkConfig.setInteger(RestOptions.PORT, Integer.parseInt(parts.get(1)));
+
+ SavepointRestoreSettings savepointRestoreSettings =
SavepointRestoreSettings.none();
+ if (options.getSavepointPath() != null) {
+savepointRestoreSettings =
+SavepointRestoreSettings.forPath(
+options.getSavepointPath(),
options.getAllowNonRestoredState());
+ }
Review comment:
Maybe it's just me but I prefer this pattern:
```java
final SavepointRestoreSettings savepointRestoreSettings;
if (options.getSavepointPath() != null) {
savepointRestoreSettings =
SavepointRestoreSettings.forPath(
options.getSavepointPath(),
options.getAllowNonRestoredState());
} else {
savepointRestoreSettings = SavepointRestoreSettings.none();
}
```
It's slightly more verbose but also more robust.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 171034)
Time Spent: 20m (was: 10m)
> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Labels: portability, portability-flink
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The portable Flink runner needs to support Flink savepoints for production
> use. It should be possible to upgrade a stateful portable Beam pipeline that
> runs on Flink, which involves taking a savepoint and then starting the new
> version of the pipeline from that savepoint. The potential issues with
> pipeline evolution and migration are similar to those when using the Flink
> DataStream API (schema / name changes etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171036&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171036 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Nov/18 10:19 Start Date: 30/Nov/18 10:19 Worklog Time Spent: 10m Work Description: mxm commented on a change in pull request #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169#discussion_r237810211 ## File path: runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java ## @@ -263,4 +284,81 @@ private static void applyLatencyTrackingInterval( long latencyTrackingInterval = options.getLatencyTrackingInterval(); config.setLatencyTrackingInterval(latencyTrackingInterval); } + + /** + * TODO: https://issues.apache.org/jira/browse/BEAM-5396 Review comment: Can we move this after the class description? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171036) Time Spent: 0.5h (was: 20m) > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/jira/browse/BEAM-5396 > Project: Beam > Issue Type: Improvement > Components: runner-flink >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: portability, portability-flink > Time Spent: 0.5h > Remaining Estimate: 0h > > The portable Flink runner needs to support Flink savepoints for production > use. It should be possible to upgrade a stateful portable Beam pipeline that > runs on Flink, which involves taking a savepoint and then starting the new > version of the pipeline from that savepoint. The potential issues with > pipeline evolution and migration are similar to those when using the Flink > DataStream API (schema / name changes etc.). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5396) Flink portable runner savepoint / upgrade support
[ https://issues.apache.org/jira/browse/BEAM-5396?focusedWorklogId=171002&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-171002 ] ASF GitHub Bot logged work on BEAM-5396: Author: ASF GitHub Bot Created on: 30/Nov/18 06:34 Start Date: 30/Nov/18 06:34 Worklog Time Spent: 10m Work Description: tweise opened a new pull request #7169: [BEAM-5396] Savepoint restore option in Flink runner URL: https://github.com/apache/beam/pull/7169 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 171002) Time Spent: 10m Remaining Estimate: 0h > Flink portable runner savepoint / upgrade support > - > > Key: BEAM-5396 > URL: https://issues.apache.org/ji
