[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=229951&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229951 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 18/Apr/19 23:14 Start Date: 18/Apr/19 23:14 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229951) Time Spent: 4.5h (was: 4h 20m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4.5h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=229950&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229950 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 18/Apr/19 23:14 Start Date: 18/Apr/19 23:14 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-484719498 Thanks @tvalentyn Updated the description and Merging the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229950) Time Spent: 4h 20m (was: 4h 10m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4h 20m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=229909&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229909 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 18/Apr/19 20:46 Start Date: 18/Apr/19 20:46 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-484684088 Ping @tvalentyn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229909) Time Spent: 4h 10m (was: 4h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=229271&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229271 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 17/Apr/19 18:05 Start Date: 17/Apr/19 18:05 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-484199851 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229271) Time Spent: 4h (was: 3h 50m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=229270&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-229270 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 17/Apr/19 18:04 Start Date: 17/Apr/19 18:04 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-484199793 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 229270) Time Spent: 3h 50m (was: 3h 40m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228675&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228675 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 20:49 Start Date: 16/Apr/19 20:49 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275986294 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -780,7 +780,9 @@ def _add_argparse_args(cls, parser): class PortableOptions(PipelineOptions): - + """Portable options are common options expected to be understood by most of Review comment: I think PortableOptions is understandable by developers but might be less user-friendly, since users might not know what portability refers to. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228675) Time Spent: 3h 40m (was: 3.5h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228661&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228661 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 20:44 Start Date: 16/Apr/19 20:44 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275984317 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -780,7 +780,9 @@ def _add_argparse_args(cls, parser): class PortableOptions(PipelineOptions): - + """Portable options are common options expected to be understood by most of Review comment: I agree that we can remove the RunnerOptions as its not used anywhere. I would like to stick with the current convention of not adding "Runner" to PortableOptions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228661) Time Spent: 3.5h (was: 3h 20m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228653 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 20:37 Start Date: 16/Apr/19 20:37 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275981582 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -801,7 +803,7 @@ def _add_argparse_args(cls, parser): '""} }. All fields in the json are optional except ' 'command.')) parser.add_argument( -'--sdk_worker_parallelism', default=None, +'--sdk_worker_parallelism', default=1, Review comment: I have seen machine lock down when running python processes equal to the number of cores. We can still go with 0 here as long as we set the default to be 1 in JobServerDriver.java. And I think this is reasonable. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228653) Time Spent: 3h 10m (was: 3h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228654&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228654
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 16/Apr/19 20:37
Start Date: 16/Apr/19 20:37
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #8286: [BEAM-6853]
Make sdkWorkerParallelism option consistent
URL: https://github.com/apache/beam/pull/8286#discussion_r275981679
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
##
@@ -96,8 +96,11 @@ protected InMemoryJobService createJobService() throws
IOException {
@Option(
name = "--sdk-worker-parallelism",
-usage = "Default parallelism for SDK worker processes (see portable
pipeline options)")
-private Long sdkWorkerParallelism = 1L;
+usage =
+"Default parallelism for SDK worker processes (see portable
pipeline options). "
Review comment:
Makes sense. I will update the wording.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228654)
Time Spent: 3h 20m (was: 3h 10m)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228267 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 09:50 Start Date: 16/Apr/19 09:50 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275718110 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -801,7 +803,7 @@ def _add_argparse_args(cls, parser): '""} }. All fields in the json are optional except ' 'command.')) parser.add_argument( -'--sdk_worker_parallelism', default=None, +'--sdk_worker_parallelism', default=1, Review comment: I agree, we probably want `0`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228267) Time Spent: 3h (was: 2h 50m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228266 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 09:49 Start Date: 16/Apr/19 09:49 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275715708 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -801,7 +803,7 @@ def _add_argparse_args(cls, parser): '""} }. All fields in the json are optional except ' 'command.')) parser.add_argument( -'--sdk_worker_parallelism', default=None, +'--sdk_worker_parallelism', default=1, Review comment: From PortablePipelineOptions.java: ``` @Description( "Sets the number of sdk worker processes that will run on each worker node. Default is 1. If" + " 0, it will be automatically set by the runner by looking at different parameters " + "(e.g. number of CPU cores on the worker machine).")``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228266) Time Spent: 2h 50m (was: 2h 40m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228255&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228255 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 09:44 Start Date: 16/Apr/19 09:44 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275715708 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -801,7 +803,7 @@ def _add_argparse_args(cls, parser): '""} }. All fields in the json are optional except ' 'command.')) parser.add_argument( -'--sdk_worker_parallelism', default=None, +'--sdk_worker_parallelism', default=1, Review comment: From PortablePipelineOptions.java: ``` @Description( "Sets the number of sdk worker processes that will run on each worker node. Default is 1. If" + " 0, it will be automatically set by the runner by looking at different parameters " + "(e.g. number of CPU cores on the worker machine).")``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228255) Time Spent: 2h 40m (was: 2.5h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=228254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228254 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 16/Apr/19 09:44 Start Date: 16/Apr/19 09:44 Worklog Time Spent: 10m Work Description: mxm commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275715481 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -780,7 +780,9 @@ def _add_argparse_args(cls, parser): class PortableOptions(PipelineOptions): - + """Portable options are common options expected to be understood by most of Review comment: Fine with me to remove RunnerOptions because it is just a stub. Not sure about PortableOptions => PortableRunnerOptions because we currently do not include Runner in any option name, e.g. `FlinkPipelineOptions`, not `FlinkRunnerPipelineOptions`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228254) Time Spent: 2.5h (was: 2h 20m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227976&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227976 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 15/Apr/19 21:22 Start Date: 15/Apr/19 21:22 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275549927 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -801,7 +803,7 @@ def _add_argparse_args(cls, parser): '""} }. All fields in the json are optional except ' 'command.')) parser.add_argument( -'--sdk_worker_parallelism', default=None, +'--sdk_worker_parallelism', default=1, Review comment: Do we have strong reasons not to make 0 be a default value? Folks working on portability probably have an informed opinion on this. cc: @mxm, @robertwb. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227976) Time Spent: 2h 20m (was: 2h 10m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227974&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227974 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 15/Apr/19 21:22 Start Date: 15/Apr/19 21:22 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#discussion_r275533621 ## File path: sdks/python/apache_beam/options/pipeline_options.py ## @@ -780,7 +780,9 @@ def _add_argparse_args(cls, parser): class PortableOptions(PipelineOptions): - + """Portable options are common options expected to be understood by most of Review comment: @angoenka What do you think about removing current RunnerOptions in Python SDK, and renaming PortableOptions to either RunnerOptions or PortableRunnerOptions in all SDKs? See also: https://github.com/apache/beam/pull/8225#discussion_r274984602 cc: @mxm, @robertwb . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227974) Time Spent: 2h (was: 1h 50m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227975&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227975
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 15/Apr/19 21:22
Start Date: 15/Apr/19 21:22
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8286: [BEAM-6853]
Make sdkWorkerParallelism option consistent
URL: https://github.com/apache/beam/pull/8286#discussion_r275545246
##
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobServerDriver.java
##
@@ -96,8 +96,11 @@ protected InMemoryJobService createJobService() throws
IOException {
@Option(
name = "--sdk-worker-parallelism",
-usage = "Default parallelism for SDK worker processes (see portable
pipeline options)")
-private Long sdkWorkerParallelism = 1L;
+usage =
+"Default parallelism for SDK worker processes (see portable
pipeline options). "
Review comment:
Consider slightly different order: `Parallelism level for SDK worker
processes.
This value is used only when the pipeline option sdkWorkerParallelism is set
to 0, otherwise the value provided in pipeline options takes precedence.
Default is 0, which means worker parallelism will be dynamically decided by a
runner. See also: sdkWorkerParallelism Pipeline Option`. Also, should last
sentence be a Javadoc reference?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227975)
Time Spent: 2h 10m (was: 2h)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227970 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 15/Apr/19 21:16 Start Date: 15/Apr/19 21:16 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-483422211 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227970) Time Spent: 1h 50m (was: 1h 40m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227080 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 13/Apr/19 01:30 Start Date: 13/Apr/19 01:30 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-482764982 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227080) Time Spent: 1h 40m (was: 1.5h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=227077&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227077 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 13/Apr/19 01:20 Start Date: 13/Apr/19 01:20 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8286: [BEAM-6853] Make sdkWorkerParallelism option consistent URL: https://github.com/apache/beam/pull/8286#issuecomment-482764219 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227077) Time Spent: 1.5h (was: 1h 20m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=226361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226361
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 11/Apr/19 21:20
Start Date: 11/Apr/19 21:20
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #8082: [BEAM-6853]
Add sdk-worker-parallelism and environment-cache-millis to python sdk
URL: https://github.com/apache/beam/pull/8082#discussion_r274670172
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -737,6 +737,16 @@ def _add_argparse_args(cls, parser):
'"", "env":{"": '
'""} }. All fields in the json are optional except '
'command.'))
+parser.add_argument(
+'--sdk-worker-parallelism', default=None,
+help=('Sets the number of sdk worker processes that will run on each '
+ 'worker node. Default is 1. If 0, it will be automatically set '
+ 'by the runner by looking at different parameters (e.g. number '
+ 'of CPU cores on the worker machine).'))
+parser.add_argument(
+'--environment-cache-millis', default=0,
Review comment:
0 should be fine. We should update the test to set this value if the test
fails.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226361)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=226360&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226360
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 11/Apr/19 21:20
Start Date: 11/Apr/19 21:20
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #8082: [BEAM-6853]
Add sdk-worker-parallelism and environment-cache-millis to python sdk
URL: https://github.com/apache/beam/pull/8082#discussion_r274670006
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -737,6 +737,15 @@ def _add_argparse_args(cls, parser):
'"", "env":{"": '
'""} }. All fields in the json are optional except '
'command.'))
+parser.add_argument(
+'--sdk-worker-parallelism', default=None,
+help=('Sets the number of sdk worker processes that will run on each '
+ 'worker node. Default is 1. If 0, it will be automatically set '
Review comment:
I kept this null to match the behavior in java
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L74
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226360)
Time Spent: 1h 20m (was: 1h 10m)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=226116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226116 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 11/Apr/19 16:17 Start Date: 11/Apr/19 16:17 Worklog Time Spent: 10m Work Description: tvalentyn commented on issue #8082: [BEAM-6853] Add sdk-worker-parallelism and environment-cache-millis to python sdk URL: https://github.com/apache/beam/pull/8082#issuecomment-482182040 Why are these options needed in PortableRunner? Do we expect all runners that support FnAPI to support these options? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 226116) Time Spent: 1h 10m (was: 1h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=226114&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226114
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 11/Apr/19 16:16
Start Date: 11/Apr/19 16:16
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8082: [BEAM-6853]
Add sdk-worker-parallelism and environment-cache-millis to python sdk
URL: https://github.com/apache/beam/pull/8082#discussion_r274505023
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -737,6 +737,16 @@ def _add_argparse_args(cls, parser):
'"", "env":{"": '
'""} }. All fields in the json are optional except '
'command.'))
+parser.add_argument(
+'--sdk-worker-parallelism', default=None,
+help=('Sets the number of sdk worker processes that will run on each '
+ 'worker node. Default is 1. If 0, it will be automatically set '
+ 'by the runner by looking at different parameters (e.g. number '
+ 'of CPU cores on the worker machine).'))
+parser.add_argument(
+'--environment-cache-millis', default=0,
Review comment:
Is 0 a reasonable default? It causes Portable wordcount tests to fail,
unless we set a different value. In our gradle files we set 1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226114)
Time Spent: 1h (was: 50m)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=226110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-226110
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 11/Apr/19 16:14
Start Date: 11/Apr/19 16:14
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #8082: [BEAM-6853]
Add sdk-worker-parallelism and environment-cache-millis to python sdk
URL: https://github.com/apache/beam/pull/8082#discussion_r274504237
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -737,6 +737,15 @@ def _add_argparse_args(cls, parser):
'"", "env":{"": '
'""} }. All fields in the json are optional except '
'command.'))
+parser.add_argument(
+'--sdk-worker-parallelism', default=None,
+help=('Sets the number of sdk worker processes that will run on each '
+ 'worker node. Default is 1. If 0, it will be automatically set '
Review comment:
@angoenka Why does helpstring say "Default is 1" but in parser instruction
we have `default=None`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 226110)
Time Spent: 50m (was: 40m)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=220685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-220685 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 29/Mar/19 19:38 Start Date: 29/Mar/19 19:38 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #8082: [BEAM-6853] Add sdk-worker-parallelism and environment-cache-millis to python sdk URL: https://github.com/apache/beam/pull/8082 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 220685) Time Spent: 40m (was: 0.5h) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[
https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=215383&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215383
]
ASF GitHub Bot logged work on BEAM-6853:
Author: ASF GitHub Bot
Created on: 19/Mar/19 11:21
Start Date: 19/Mar/19 11:21
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #8082: [BEAM-6853]
Add sdk-worker-parallelism and environment-cache-millis to python sdk
URL: https://github.com/apache/beam/pull/8082#discussion_r266838809
##
File path: sdks/python/apache_beam/options/pipeline_options.py
##
@@ -737,6 +737,15 @@ def _add_argparse_args(cls, parser):
'"", "env":{"": '
'""} }. All fields in the json are optional except '
'command.'))
+parser.add_argument(
+'--sdk-worker-parallelism', default=None,
+help=('Sets the number of sdk worker processes that will run on each '
+ 'worker node. Default is 1. If 0, it will be automatically set '
Review comment:
I'd leave 0 a bit unspecified, e.g. it will be chose by the runner (e.g. by
looking at the number of CPU cores on the worker machine).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 215383)
Time Spent: 0.5h (was: 20m)
> Make Java and python portable options same
> --
>
> Key: BEAM-6853
> URL: https://issues.apache.org/jira/browse/BEAM-6853
> Project: Beam
> Issue Type: Task
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Java
> [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80]
> and [Python Portable
> options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719]
> don't have the same values limiting the use of sdk-worker-parallelism and
> environment-cache-millis in python sdk.
>
> Add these options to the python sdk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=215060&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215060 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 18/Mar/19 21:41 Start Date: 18/Mar/19 21:41 Worklog Time Spent: 10m Work Description: angoenka commented on issue #8082: [BEAM-6853] Add sdk-worker-parallelism and environment-cache-millis to python sdk URL: https://github.com/apache/beam/pull/8082#issuecomment-474112733 R: @robertwb @charlesccychen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 215060) Time Spent: 20m (was: 10m) > Make Java and python portable options same > -- > > Key: BEAM-6853 > URL: https://issues.apache.org/jira/browse/BEAM-6853 > Project: Beam > Issue Type: Task > Components: sdk-py-core >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Java > [PortableRunnerOptions|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PortablePipelineOptions.java#L80] > and [Python Portable > options|https://github.com/apache/beam/blob/f21cfaefd54afb798103dc90ab57290739e81e81/sdks/python/apache_beam/options/pipeline_options.py#L719] > don't have the same values limiting the use of sdk-worker-parallelism and > environment-cache-millis in python sdk. > > Add these options to the python sdk. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6853) Make Java and python portable options same
[ https://issues.apache.org/jira/browse/BEAM-6853?focusedWorklogId=215057&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-215057 ] ASF GitHub Bot logged work on BEAM-6853: Author: ASF GitHub Bot Created on: 18/Mar/19 21:41 Start Date: 18/Mar/19 21:41 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #8082: [BEAM-6853] Add sdk-worker-parallelism and environment-cache-millis to python sdk URL: https://github.com/apache/beam/pull/8082 These options are already added in the java. We need to add them to python sdk to use them in python. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | --- See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs. --
