[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2019-02-15 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=199421&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199421
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 15/Feb/19 20:40
Start Date: 15/Feb/19 20:40
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 199421)
Time Spent: 1.5h  (was: 1h 20m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Labels: triaged
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2019-02-15 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=199419&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199419
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 15/Feb/19 20:35
Start Date: 15/Feb/19 20:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#issuecomment-464190116
 
 
   R: @aaltay I typo'd a function in a previous commit, breaking top.  I've 
filed https://issues.apache.org/jira/browse/BEAM-6682 to improve this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 199419)
Time Spent: 1h 10m  (was: 1h)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Labels: triaged
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2019-02-15 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=199420&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199420
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 15/Feb/19 20:35
Start Date: 15/Feb/19 20:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#issuecomment-464190116
 
 
   R: @aaltay I typo'd a function in a previous commit, breaking top.  I've 
filed https://issues.apache.org/jira/browse/BEAM-6682 to improve this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 199420)
Time Spent: 1h 20m  (was: 1h 10m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Labels: triaged
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2019-02-15 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=199411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-199411
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 15/Feb/19 20:25
Start Date: 15/Feb/19 20:25
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#issuecomment-464187444
 
 
   I didn't see this until today. Thanks for the contribution!
   LGTM.
   
   R: @aaltay Please merge.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 199411)
Time Spent: 1h  (was: 50m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Labels: triaged
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2019-02-14 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=198790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-198790
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 14/Feb/19 17:13
Start Date: 14/Feb/19 17:13
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#issuecomment-463710905
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 198790)
Time Spent: 50m  (was: 40m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Labels: triaged
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2018-12-16 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=175833&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175833
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 16/Dec/18 16:31
Start Date: 16/Dec/18 16:31
Worklog Time Spent: 10m 
  Work Description: bramp commented on issue #7149: [BEAM-6144] Add support 
for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#issuecomment-447656645
 
 
   Please take another look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 175833)
Time Spent: 40m  (was: 0.5h)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2018-12-16 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=175832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175832
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 16/Dec/18 16:29
Start Date: 16/Dec/18 16:29
Worklog Time Spent: 10m 
  Work Description: bramp commented on a change in pull request #7149: 
[BEAM-6144] Add support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#discussion_r241986912
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
 ##
 @@ -140,6 +144,18 @@ func Translate(p *pb.Pipeline, opts *JobOptions, 
workerURL, jarURL, modelURL str
if opts.NumWorkers > 0 {
job.Environment.WorkerPools[0].NumWorkers = opts.NumWorkers
}
+   if opts.Algorithm != "" {
+   settings := &df.AutoscalingSettings{
+   Algorithm: map[string]string{
+   "NONE": 
"AUTOSCALING_ALGORITHM_NONE",
+   "THROUGHPUT_BASED": 
"AUTOSCALING_ALGORITHM_BASIC",
+   }[opts.Algorithm],
+   }
+   if opts.MaxNumWorkers > 0 {
 
 Review comment:
   Refactored so that can be set independently.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 175832)
Time Spent: 0.5h  (was: 20m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2018-12-05 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=172450&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-172450
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 05/Dec/18 18:58
Start Date: 05/Dec/18 18:58
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #7149: 
[BEAM-6144] Add support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149#discussion_r239194436
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
 ##
 @@ -140,6 +144,18 @@ func Translate(p *pb.Pipeline, opts *JobOptions, 
workerURL, jarURL, modelURL str
if opts.NumWorkers > 0 {
job.Environment.WorkerPools[0].NumWorkers = opts.NumWorkers
}
+   if opts.Algorithm != "" {
+   settings := &df.AutoscalingSettings{
+   Algorithm: map[string]string{
+   "NONE": 
"AUTOSCALING_ALGORITHM_NONE",
+   "THROUGHPUT_BASED": 
"AUTOSCALING_ALGORITHM_BASIC",
+   }[opts.Algorithm],
+   }
+   if opts.MaxNumWorkers > 0 {
 
 Review comment:
   `MaxNumWorkers` could be set even if `opts.Algorithm` is not set. If 
Algorithm is not set, Dataflow will choose the appropriate autoscaling 
algortihm and max num workers would still be used for that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 172450)
Time Spent: 20m  (was: 10m)

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-6144) Add support for the autoscalingAlgorithm flag

2018-11-28 Thread ASF GitHub Bot (JIRA) 


 [ 
https://issues.apache.org/jira/browse/BEAM-6144?focusedWorklogId=170272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-170272
 ]

ASF GitHub Bot logged work on BEAM-6144:


Author: ASF GitHub Bot
Created on: 28/Nov/18 15:55
Start Date: 28/Nov/18 15:55
Worklog Time Spent: 10m 
  Work Description: bramp opened a new pull request #7149: [BEAM-6144] Add 
support for the autoscaling flags to the GO SDK.
URL: https://github.com/apache/beam/pull/7149
 
 
   Add support for the autoscaling flags to the GO SDK.
   
   --autoscaling_algorithm
   --max_num_workers
   
   Inline with what's documented here: 
https://cloud.google.com/dataflow/docs/guides/specifying-exec-params
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 170272)
Time Spent: 10m
Remaining Estimate: 0h

> Add support for the autoscalingAlgorithm flag
> -
>
> Key: BEAM-6144
> URL: https://issues.apache.org/jira/browse/BEAM-6144
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Andrew Brampton
>Assignee: Robert Burke
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The go SDK does not support the --autoscalingAlgorithm and --maxNumWorkers 
> flags, like the Java and Python SDKs. Add support for this.



-