[
https://issues.apache.org/jira/browse/BEAM-3418?focusedWorklogId=82473&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-82473
]
ASF GitHub Bot logged work on BEAM-3418:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Mar/18 20:39
Start Date: 20/Mar/18 20:39
Worklog Time Spent: 10m
Work Description: aaltay commented on a change in pull request #4587:
[BEAM-3418] Send worker_id in all grpc channels to runner harness
URL: https://github.com/apache/beam/pull/4587#discussion_r175914311
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -177,6 +177,14 @@ def __init__(self, packages, options,
environment_version, pipeline_url):
if self.debug_options.experiments:
for experiment in self.debug_options.experiments:
self.proto.experiments.append(experiment)
+ # Add use_multiple_sdk_containers flag if its not already present. Do not
+ # add the flag if 'no_use_multiple_sdk_containers' is present.
+ # TODO: Cleanup use_multiple_sdk_containers once we deprecate Python SDK
+ # till version 2.4.
+ if (job_type.startswith('FNAPI_') and
+ 'use_multiple_sdk_containers' not in self.proto.experiments and
+ 'no_use_multiple_sdk_containers' not in self.proto.experiments):
+ self.proto.experiments.append('use_multiple_sdk_containers')
Review comment:
It is preferable to modify debug_options.experiments (as done above for
`runner_harness_override`). This also properly helps with updating the user
visible pipeline options in the UI, and it will auto added to the proto by the
loop above.
It would also help combine things related to `if
job_type.startswith('FNAPI_'):` in a single place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 82473)
Time Spent: 1h 50m (was: 1h 40m)
> Python Fnapi - Support Multiple SDK workers on a single VM
> ----------------------------------------------------------
>
> Key: BEAM-3418
> URL: https://issues.apache.org/jira/browse/BEAM-3418
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-harness
> Reporter: Ankur Goenka
> Assignee: Ankur Goenka
> Priority: Major
> Labels: performance, portability
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Support multiple python SDK process on a VM to fully utilize a machine.
> Each SDK Process will work in isolation and interact with Runner Harness
> independently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
