[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319940 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 28/Sep/19 02:58 Start Date: 28/Sep/19 02:58 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319940) Time Spent: 1h 10m (was: 1h) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319925 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 28/Sep/19 01:06 Start Date: 28/Sep/19 01:06 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675#issuecomment-536137736 Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319925) Time Spent: 1h (was: 50m) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319923 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 28/Sep/19 00:58 Start Date: 28/Sep/19 00:58 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675#issuecomment-536136805 Thanks. @angoenka PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319923) Time Spent: 50m (was: 40m) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319922 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 28/Sep/19 00:57 Start Date: 28/Sep/19 00:57 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675#discussion_r329289253 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -680,6 +680,16 @@ def _add_argparse_args(cls, parser): 'enabled with this flag. Please sync with the owners of the runner ' 'before enabling any experiments.')) +parser.add_argument( +'--number_of_worker_threads', Review comment: Done. And also updated documentation about limited availability of the option (currently only when experiment 'use_unified_worker' is set). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319922) Time Spent: 40m (was: 0.5h) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319854=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319854 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 27/Sep/19 22:01 Start Date: 27/Sep/19 22:01 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675#discussion_r329265698 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -680,6 +680,16 @@ def _add_argparse_args(cls, parser): 'enabled with this flag. Please sync with the owners of the runner ' 'before enabling any experiments.')) +parser.add_argument( +'--number_of_worker_threads', Review comment: It will be good to keep the name same as in JRH `getNumberOfWorkerHarnessThreads` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319854) Time Spent: 0.5h (was: 20m) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319676 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 27/Sep/19 16:56 Start Date: 27/Sep/19 16:56 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675#issuecomment-536018265 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 319676) Time Spent: 20m (was: 10m) > Add a num_threads_per_worker pipeline option to Python SDK. > --- > > Key: BEAM-8318 > URL: https://issues.apache.org/jira/browse/BEAM-8318 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Similar to what we have here for Java: > [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8318) Add a num_threads_per_worker pipeline option to Python SDK.
[ https://issues.apache.org/jira/browse/BEAM-8318?focusedWorklogId=319321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-319321 ] ASF GitHub Bot logged work on BEAM-8318: Author: ASF GitHub Bot Created on: 27/Sep/19 02:43 Start Date: 27/Sep/19 02:43 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker. URL: https://github.com/apache/beam/pull/9675 This will be similar to following already available for Java SDK. https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineDebugOptions.java#L178 Currently, only works for DataflowRunner on Fn API path. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[