[
https://issues.apache.org/jira/browse/BEAM-7657?focusedWorklogId=324565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-324565
]
ASF GitHub Bot logged work on BEAM-7657:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Oct/19 19:16
Start Date: 07/Oct/19 19:16
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #9709: [BEAM-7657]
fix misleading sdk_worker_parallelism description
URL: https://github.com/apache/beam/pull/9709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 324565)
Time Spent: 20m (was: 10m)
> sdk worker parallelism comments are misleading
> ----------------------------------------------
>
> Key: BEAM-7657
> URL: https://issues.apache.org/jira/browse/BEAM-7657
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The SDK worker parallelism arg is set two places, in pipeline options [1] [2]
> and the job server driver [3].
>
> {noformat}
> if pipeline.sdk_worker_parallelism > 0:
> pipeline.sdk_worker_parallelism is used.
> elif pipeline.sdk_worker_parallelism == 0:
> if jobServerDriver.sdkWorkerParallelism > 0:
> jobServerDriver.sdkWorkerParallelism is used.
> elif jobServerDriver.sdkWorkerParallelism == 0:
> the runner chooses parallelism based on cores available.
> {noformat}
> Somewhat confusingly, the default is 0 for python pipelines, but 1 for java
> pipelines. But anyway, jobServerDriver.sdkWorkerParallelism defaults to 1, so
> the comment "If 0, it will be automatically set by looking at different
> parameters.." is misleading, and actually only true if
> jobServerDriver.sdkWorkerParallelism was explicitly set to 0 as well.
> [1]
> [https://github.com/apache/beam/blob/8b379b475a3c838eb12e9b7809ebd8f386095962/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L69-L74]
> [2]
> [https://github.com/apache/beam/blob/37b76b67b5d0cbd92e6a3fadee67f9fcf93cbc5d/sdks/python/apache_beam/options/pipeline_options.py#L805-L810]
> [3]
> [https://github.com/apache/beam/blob/f3623e8ba2257f7659ccb312dc2574f862ef41b5/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java#L97-L103]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
