[
https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=369444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369444
]
ASF GitHub Bot logged work on BEAM-7861:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Jan/20 22:52
Start Date: 09/Jan/20 22:52
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10536: [BEAM-7861] Add
direct_running_mode option for direct runners to switch between multi_threading
and multi_processing easily
URL: https://github.com/apache/beam/pull/10536#discussion_r364999362
##########
File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
##########
@@ -450,6 +450,19 @@ def run_pipeline(self,
pipeline_options.DirectOptions).direct_runner_bundle_repeat
self._num_workers = options.view_as(
pipeline_options.DirectOptions).direct_num_workers or self._num_workers
+
+ # set direct workers running mode if it is defined with pipeline options.
+ running_mode = \
+ options.view_as(pipeline_options.DirectOptions).direct_running_mode
+ if running_mode == 'multi_threading':
+ self._default_environment = environments.EmbeddedPythonGrpcEnvironment()
+ elif running_mode == 'multi_processing':
+ command_bytes = b'%s -m apache_beam.runners.worker.sdk_worker_main' \
+ % sys.executable.encode('ascii')
Review comment:
Why not utf-8?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 369444)
Time Spent: 2h 20m (was: 2h 10m)
> Make it easy to change between multi-process and multi-thread mode for Python
> Direct runners
> --------------------------------------------------------------------------------------------
>
> Key: BEAM-7861
> URL: https://issues.apache.org/jira/browse/BEAM-7861
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Hannah Jiang
> Assignee: Hannah Jiang
> Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> BEAM-3645 makes it possible to run a map task parallel.
> However, users need to change runner when switch between multithreading and
> multiprocessing mode.
> We want to add a flag (ex: --use-multiprocess) to make the switch easy
> without changing the runner each time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
