[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154078&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154078 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 13/Oct/18 01:29 Start Date: 13/Oct/18 01:29 Worklog Time Spent: 10m Work Description: HuangLED edited a comment on issue #6680: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6680#issuecomment-429468108 R: @herohde cc: @boyuanzz @pabloem Addressed. Also, option definition moved to WorkerOptions per Pablo's suggestion. Thanks to Boyuan for pointing out the right place for error message. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154078) Time Spent: 2.5h (was: 2h 20m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154043 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 21:52 Start Date: 12/Oct/18 21:52 Worklog Time Spent: 10m Work Description: HuangLED edited a comment on issue #6680: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6680#issuecomment-429468108 R: @herohde cc: @boyuanzz @pabloem Addressed. Also, option definition moved to WorkerOptions. Thanks to Boyuan for pointing out the right place for error message. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154043) Time Spent: 2h 20m (was: 2h 10m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154036&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154036 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 21:30 Start Date: 12/Oct/18 21:30 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6680: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6680#issuecomment-429468108 R: @boyuanzz @herohde @pabloem Addressed. Also, option definition moved to WorkerOptions. Thanks to Boyuan for pointing out the right place for error message. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154036) Time Spent: 2h 10m (was: 2h) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154027&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154027 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 20:59 Start Date: 12/Oct/18 20:59 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6680: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6680#issuecomment-429460451 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154027) Time Spent: 2h (was: 1h 50m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154024&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154024 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 20:47 Start Date: 12/Oct/18 20:47 Worklog Time Spent: 10m Work Description: HuangLED opened a new pull request #6680: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6680 Python support for customer worker jar (as a staged file). Tested positive and negative case by starting actual jobs. PreCommit pass locally. Follow this checklist to help us incorporate your contribution quickly and easily: - [X ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154024) Time Spent: 1h 50m (was: 1h 40m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Iss
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154021&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154021 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 20:44 Start Date: 12/Oct/18 20:44 Worklog Time Spent: 10m Work Description: HuangLED commented on a change in pull request #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#discussion_r224910055 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -674,7 +674,12 @@ def _add_argparse_args(cls, parser): 'job submission, the files will be staged in the staging area ' '(--staging_location option) and the workers will install them in ' 'same order they were specified on the command line.')) - +parser.add_argument( +'--dataflow_worker_jar', +dest='dataflow_worker_jar', +type=str, +help='Dataflow worker jar.' +) Review comment: Thanks! Issue addressed but lost the status in this PR due to my sub-optional git operations. Opening another PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154021) Time Spent: 1h 40m (was: 1.5h) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=154020&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-154020 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 20:42 Start Date: 12/Oct/18 20:42 Worklog Time Spent: 10m Work Description: HuangLED closed pull request #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/sdks/python/apache_beam/options/pipeline_options.py b/sdks/python/apache_beam/options/pipeline_options.py index a172535b100..2c061e0ec52 100644 --- a/sdks/python/apache_beam/options/pipeline_options.py +++ b/sdks/python/apache_beam/options/pipeline_options.py @@ -674,7 +674,12 @@ def _add_argparse_args(cls, parser): 'job submission, the files will be staged in the staging area ' '(--staging_location option) and the workers will install them in ' 'same order they were specified on the command line.')) - +parser.add_argument( +'--dataflow_worker_jar', +dest='dataflow_worker_jar', +type=str, +help='Dataflow worker jar.' +) class PortableOptions(PipelineOptions): diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py index 1acd3488524..5be60bd701b 100644 --- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py +++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py @@ -381,6 +381,12 @@ def run_pipeline(self, pipeline): self.dataflow_client = apiclient.DataflowApplicationClient( pipeline._options) +if setup_options.dataflow_worker_jar: + experiments = ["use_staged_dataflow_worker_jar"] + if debug_options.experiments is not None: +experiments = list(set(experiments + debug_options.experiments)) + debug_options.experiments = experiments + # Create the job description and send a request to the service. The result # can be None if there is no need to send a request to the service (e.g. # template creation). If a request was sent and failed then the call will diff --git a/sdks/python/apache_beam/runners/portability/stager.py b/sdks/python/apache_beam/runners/portability/stager.py index ef7401ac6aa..e336fd3f9b9 100644 --- a/sdks/python/apache_beam/runners/portability/stager.py +++ b/sdks/python/apache_beam/runners/portability/stager.py @@ -123,8 +123,7 @@ def stage_job_resources(self, Returns: A list of file names (no paths) for the resources staged. All the - files - are assumed to be staged at staging_location. + files are assumed to be staged at staging_location. Raises: RuntimeError: If files specified are not found or error encountered @@ -256,6 +255,13 @@ def stage_job_resources(self, 'The file "%s" cannot be found. Its location was specified by ' 'the --sdk_location command-line option.' % sdk_path) +if hasattr(setup_options, 'dataflow_worker_jar') and \ +setup_options.dataflow_worker_jar: + jar_staged_filename = 'dataflow-worker.jar' + staged_path = FileSystems.join(staging_location, jar_staged_filename) + self.stage_artifact(setup_options.dataflow_worker_jar, staged_path) + resources.append(jar_staged_filename) + # Delete all temp files created while staging job resources. shutil.rmtree(temp_dir) retrieval_token = self.commit_manifest() This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 154020) Time Spent: 1.5h (was: 1h 20m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https:
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153974&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153974 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 18:20 Start Date: 12/Oct/18 18:20 Worklog Time Spent: 10m Work Description: pabloem commented on a change in pull request #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#discussion_r224874588 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -674,7 +674,12 @@ def _add_argparse_args(cls, parser): 'job submission, the files will be staged in the staging area ' '(--staging_location option) and the workers will install them in ' 'same order they were specified on the command line.')) - +parser.add_argument( +'--dataflow_worker_jar', +dest='dataflow_worker_jar', +type=str, +help='Dataflow worker jar.' +) Review comment: I'm thinking that the option would be better in `WorkerOptions`, or some other options class related to Dataflow+Portability. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153974) Time Spent: 1h 20m (was: 1h 10m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153771&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153771 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 06:00 Start Date: 12/Oct/18 06:00 Worklog Time Spent: 10m Work Description: HuangLED removed a comment on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429161109 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153771) Time Spent: 1h 10m (was: 1h) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153769&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153769 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 06:00 Start Date: 12/Oct/18 06:00 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429213487 Run Python PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153769) Time Spent: 50m (was: 40m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153770&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153770 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 06:00 Start Date: 12/Oct/18 06:00 Worklog Time Spent: 10m Work Description: HuangLED removed a comment on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429199020 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153770) Time Spent: 1h (was: 50m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153761&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153761 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 04:15 Start Date: 12/Oct/18 04:15 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429199020 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153761) Time Spent: 40m (was: 0.5h) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153745&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153745 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 12/Oct/18 00:03 Start Date: 12/Oct/18 00:03 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429161109 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153745) Time Spent: 0.5h (was: 20m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153733 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 11/Oct/18 23:22 Start Date: 11/Oct/18 23:22 Worklog Time Spent: 10m Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667#issuecomment-429154014 R: @herohde @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153733) Time Spent: 20m (was: 10m) > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > One of the slightly subtle aspects is that we would need to ignore one of the > staged jars for portable Python jobs. That requires a change to the Python > boot code: > https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar
[ https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153718&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153718 ] ASF GitHub Bot logged work on BEAM-5637: Author: ASF GitHub Bot Created on: 11/Oct/18 22:29 Start Date: 11/Oct/18 22:29 Worklog Time Spent: 10m Work Description: HuangLED opened a new pull request #6667: [BEAM-5637] Python support for custom dataflow worker jar URL: https://github.com/apache/beam/pull/6667 Follow this checklist to help us incorporate your contribution quickly and easily: - [ X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 153718) Time Spent: 10m Remaining Estimate: 0h > Python support for custom dataflow worker jar > - > > Key: BEAM-5637 > URL: https://issues.apache.org/jira/browse/BEAM-5637 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Henning Rohde >Assignee: Ruoyun Huang >
