Build failed in Jenkins: beam_PostCommit_Python_Verify #4617

[Apache Jenkins Server]

See 


--
[...truncated 1.04 MB...]
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/test_pipeline.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying 

[jira] [Created] (BEAM-4031) Add missing dataflow customization options for Go SDK

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-4031:
---

 Summary: Add missing dataflow customization options for Go SDK
 Key: BEAM-4031
 URL: https://issues.apache.org/jira/browse/BEAM-4031
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Henning Rohde
Assignee: Henning Rohde


We're missing at least:

zone
temp_location
worker_machine_type



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4616

[Apache Jenkins Server]

See 


Changes:

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 1.04 MB...]
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying 

[jira] [Work logged] (BEAM-3973) Allow to disable batch API in SpannerIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3973?focusedWorklogId=88687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88687
 ]

ASF GitHub Bot logged work on BEAM-3973:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:24
Start Date: 07/Apr/18 02:24
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4946: [BEAM-3973] Adds 
a parameter to the Cloud Spanner read connector that can disable batch API
URL: https://github.com/apache/beam/pull/4946#issuecomment-379425746
 
 
   Please let me know when the comments are addressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88687)
Time Spent: 2h 50m  (was: 2h 40m)

> Allow to disable batch API in SpannerIO
> ---
>
> Key: BEAM-3973
> URL: https://issues.apache.org/jira/browse/BEAM-3973
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Mairbek Khadikov
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API 
> provides abstractions to scale out reads from Spanner, but it requires the 
> query to be root-partitionable. The root-partitionable queries cover majority 
> of the use cases, however there are examples when running arbitrary query is 
> useful. For example, reading all the table names from the 
> information_schema.* and reading the content of those tables in the next 
> step. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88675
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r178998423
 
 

 ##
 File path: build.gradle
 ##
 @@ -175,3 +175,74 @@ task goPreCommit() {
   dependsOn ":rat"
   dependsOn ":sdks:go:test"
 }
+
+// This task runs PerfKitBenchmarker, which does benchmarking of the IO ITs.
+// The arguments passed to it allows it to invoke gradle again with the 
desired benchmark.
+//
+// To invoke this, run:
+//
+// ./gradlew performanceTest \
+//  -DpkbLocation=""
+//  -DintegrationTestPipelineOptions='["--numberOfRecords=1000", ""]' \
+//  -DintegrationTest= \
+//  -DitModule= \
+//  -DintegrationTestRunner=direct/dataflow/
 
 Review comment:
   Can you give the exact value that should be used for DataflowRunner here 
(without ) ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88675)
Time Spent: 2h 10m  (was: 2h)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88673
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905570
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // Add runners needed to run integration tests on
+  task packageIntegrationTests(type: Jar) {
+if (gradle.startParameter.taskNames.contains('integrationTest')) {
+
+  def runner = configuration.runner
+  dependencies {
+if (runner.contains('dataflow')) {
 
 Review comment:
   Why do we have to set runner-specific dependencies here? Can't this be set 
by the user command that invokes the Gradle task to run the test ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88673)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88683
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905413
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
 
 Review comment:
   Do all ITs have to have to same timeout 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88683)
Time Spent: 3h 10m  (was: 3h)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88678
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905431
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
 
 Review comment:
   Is this something we can fix at PerfKit side (and not specify empty strings 
here) ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88678)
Time Spent: 2.5h  (was: 2h 20m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88672
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905373
 
 

 ##
 File path: build.gradle
 ##
 @@ -22,6 +22,7 @@
 // tasks.
 
 apply from: project(":").file("build_rules.gradle")
+createPerformanceTestHarness()
 
 Review comment:
   Let's add a comment here explaining why we need this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88672)
Time Spent: 2h  (was: 1h 50m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88680
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905587
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // Add runners needed to run integration tests on
+  task packageIntegrationTests(type: Jar) {
+if (gradle.startParameter.taskNames.contains('integrationTest')) {
+
+  def runner = configuration.runner
+  dependencies {
+if (runner.contains('dataflow')) {
+  testCompile project(path: ":runners:google-cloud-dataflow-java", 
configuration: 'shadowTest')
+} else {
+  testCompile project(path: ":runners:direct-java", configuration: 
'shadowTest')
+}
+  }
+}
+  }
+
+  // Task for running integration tests
+  task integrationTest(type: Test) {
+include "**/*IT.class"
+systemProperties.beamTestPipelineOptions = 
configuration.integrationTestPipelineOptions
+  }
+}
+
+ext.createPerformanceTestHarness = {
+  println "createPerformanceTestHarness with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
 
 Review comment:
   Ditto. Please explain what 'it' parameter is.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88680)
Time Spent: 2h 50m  (was: 2h 40m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This 

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88676
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905477
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
 
 Review comment:
   implicit 'it' parameter.
   
   BTW who sets this parameter ?
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88676)
Time Spent: 2h 20m  (was: 2h 10m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88682=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88682
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905678
 
 

 ##
 File path: 
sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
 ##
 @@ -47,18 +47,16 @@
  * This test requires a running instance of Postgres. Pass in connection 
information using
  * PipelineOptions:
  * 
- *  mvn -e -Pio-it verify -pl sdks/java/io/jdbc 
-DintegrationTestPipelineOptions='[
+ *  ./gradlew integrationTest -p sdks/java/io/jdbc 
-DintegrationTestPipelineOptions='[
  *  "--postgresServerName=1.2.3.4",
  *  "--postgresUsername=postgres",
  *  "--postgresDatabaseName=myfancydb",
  *  "--postgresPassword=mypass",
  *  "--postgresSsl=false",
  *  "--numberOfRecords=1000" ]'
+ *  --tests org.apache.beam.sdk.io.jdbc.JdbcIOIT
  * 
  *
- * If you want to run this with a runner besides directrunner, there are 
profiles for dataflow
- * and spark in the jdbc pom. You'll want to activate those in addition to the 
normal test runner
 
 Review comment:
   Does this restriction not apply anymore ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88682)
Time Spent: 3h  (was: 2h 50m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88677
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905401
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
 
 Review comment:
   Could you add single lines of comments explaining what these properties are 
? For example, I'm not sure what 'official' is used for.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88677)
Time Spent: 2h 20m  (was: 2h 10m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88681=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88681
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905624
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // Add runners needed to run integration tests on
+  task packageIntegrationTests(type: Jar) {
+if (gradle.startParameter.taskNames.contains('integrationTest')) {
+
+  def runner = configuration.runner
+  dependencies {
+if (runner.contains('dataflow')) {
+  testCompile project(path: ":runners:google-cloud-dataflow-java", 
configuration: 'shadowTest')
+} else {
+  testCompile project(path: ":runners:direct-java", configuration: 
'shadowTest')
+}
+  }
+}
+  }
+
+  // Task for running integration tests
+  task integrationTest(type: Test) {
+include "**/*IT.class"
+systemProperties.beamTestPipelineOptions = 
configuration.integrationTestPipelineOptions
+  }
+}
+
+ext.createPerformanceTestHarness = {
+  println "createPerformanceTestHarness with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // This task runs PerfKitBenchmarker, which does benchmarking of the IO ITs.
+  // The arguments passed to it allows it to invoke gradle again with the 
desired benchmark.
+  //
+  // To invoke this, run:
+  //
+  // ./gradlew performanceTest \
+  //  -DpkbLocation=""
+  //  -DintegrationTestPipelineOptions='["--numberOfRecords=1000", ""]' \
+  //  -DintegrationTest= \
+  //  -DitModule= \
+  //  -DintegrationTestRunner=direct/dataflow/
+  //
+  // There are more options with default values that can be tweaked if needed 
(see below).
+  task performanceTest(type: Exec) {
+
+// Perfkit needs to work in the Beam's root directory,
 
 Review comment:
   You mean, test has to be invoked from the root directory ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: 

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88684
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905518
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // Add runners needed to run integration tests on
+  task packageIntegrationTests(type: Jar) {
+if (gradle.startParameter.taskNames.contains('integrationTest')) {
 
 Review comment:
   So, tasks names starting with 'integrationTest' is a requirement ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88684)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88674
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:15
Start Date: 07/Apr/18 02:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5003: [BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905443
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
 
 Review comment:
   Ditto.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88674)
Time Spent: 2h 10m  (was: 2h)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4615

[Apache Jenkins Server]

See 


--
[...truncated 1.04 MB...]
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88669
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:10
Start Date: 07/Apr/18 02:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5003: 
[BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905569
 
 

 ##
 File path: sdks/java/io/cassandra/build.gradle
 ##
 @@ -18,6 +18,7 @@
 
 apply from: project(":").file("build_rules.gradle")
 applyJavaNature(artifactId: "beam-sdks-java-io-cassandra")
+enableJavaPerformanceTesting()
 
 Review comment:
   What does this line do? Does it enable for all build?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88669)
Time Spent: 1h 50m  (was: 1h 40m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88671
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:10
Start Date: 07/Apr/18 02:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5003: 
[BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905554
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
+  String benchmarks = System.getProperty('benchmarks', 
'beam_integration_benchmark')
+
+  String beamPrebuilt = System.getProperty('beamPrebuilt', 'true')
+  String beamSdk = System.getProperty('beamSdk', 'java')
+
+  String timeout = System.getProperty('itTimeout', '1200')
+
+  String kubeconfig = System.getProperty('kubeconfig', 
System.getProperty('user.home') + '/.kube/config')
+  String kubectl = System.getProperty('kubectl', 'kubectl')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no scripts are expected.
+  String kubernetesScripts = System.getProperty('kubernetesScripts', '')
+
+  String integrationTestPipelineOptions = 
System.getProperty('integrationTestPipelineOptions')
+
+  // PerfKit will have trouble reading 'null' path. It expects empty string if 
no config file is expected.
+  String optionsConfigFile = System.getProperty('beamITOptions', '')
+
+  String integrationTest = System.getProperty('integrationTest')
+  String itModule = System.getProperty('itModule')
+
+  String runner = System.getProperty('integrationTestRunner', 'direct')
+
+  String extraProperties = System.getProperty('beamExtraProperties', '')
+}
+
+// Configures a project with a set of tasks needed for running performance 
tests
+ext.enableJavaPerformanceTesting = {
+  println "enableJavaPerformanceTesting with ${it ? "$it" : "default 
configuration"} for project ${project.name}"
+
+  // Use the implicit it parameter of the closure to handle zero argument or 
one argument map calls.
+  JavaPerformanceTestConfiguration configuration = it ? it as 
JavaPerformanceTestConfiguration : new JavaPerformanceTestConfiguration()
+
+  // Add runners needed to run integration tests on
+  task packageIntegrationTests(type: Jar) {
+if (gradle.startParameter.taskNames.contains('integrationTest')) {
+
+  def runner = configuration.runner
+  dependencies {
+if (runner.contains('dataflow')) {
 
 Review comment:
   Do we need to add hard coded branch for each possible runner here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88671)
Time Spent: 1h 50m  (was: 1h 40m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88670
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:10
Start Date: 07/Apr/18 02:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5003: 
[BEAM-3942] Update performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#discussion_r179905523
 
 

 ##
 File path: build_rules.gradle
 ##
 @@ -593,6 +593,118 @@ ext.applyJavaNature = {
   }
 }
 
+// Reads and contains all necessary performance test parameters
+class JavaPerformanceTestConfiguration {
+  String pkbLocation = System.getProperty('pkbLocation')
+
+  String logLevel = System.getProperty('logLevel', 'INFO')
+  String gradleBinary = System.getProperty('gradleBinary', './gradlew')
+  String isOfficial = System.getProperty('official', 'true')
 
 Review comment:
   Do we want to default to `true` here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88670)
Time Spent: 1h 50m  (was: 1h 40m)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=88667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88667
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 07/Apr/18 02:01
Start Date: 07/Apr/18 02:01
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5048: [BEAM-3255] Updates to 
support the release process via Gradle
URL: https://github.com/apache/beam/pull/5048#issuecomment-379424432
 
 
   cc: @robertwb 
   
   Robert any clue about the following error:
   
   ```
   Traceback (most recent call last):
 File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in 
_bootstrap
   self.run()
 File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
   self._target(*self._args, **self._kwargs)
 File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/gen_protos.py",
 line 148, in _install_grpcio_tools_and_generate_proto_files
   shutil.rmtree(build_path)
 File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
   onerror(os.listdir, path, sys.exc_info())
 File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
   names = os.listdir(path)
   OSError: [Errno 2] No such file or directory: 
'/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/.eggs/grpcio-wheels-build'
   Traceback (most recent call last):
 File "setup.py", line 234, in 
   'test': generate_protos_first(test),
 File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/setuptools/__init__.py",
 line 129, in setup
   return distutils.core.setup(**attrs)
 File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
   dist.run_commands()
 File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
   self.run_command(cmd)
 File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
   cmd_obj.run()
 File "setup.py", line 142, in run
   gen_protos.generate_proto_files()
 File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/gen_protos.py",
 line 96, in generate_proto_files
   raise ValueError("Proto generation failed (see log for details).")
   ValueError: Proto generation failed (see log for details).
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88667)
Time Spent: 40m  (was: 0.5h)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #6401

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4614

[Apache Jenkins Server]

See 


--
[...truncated 1.04 MB...]
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers_test.py -> 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6400

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #7

[Apache Jenkins Server]

See 


Changes:

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 1.71 MB...]
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy3.processTestClass(Unknown Source)
at 

Build failed in Jenkins: beam_PerformanceTests_Spark #1559

[Apache Jenkins Server]

See 


Changes:

[wcn] Add float support for the SDK.

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

[wcn] Add diagnostics for sinks.

[wcn] Add hooks to the Go SDK.

[lcwik] [BEAM-3250] Add JUnit test reporting to Jenkins.

[altay] Migrate JUnit test archiving to Python Gradle build

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 88.64 KB...]
'apache-beam-testing:bqjob_r2a0955745cdcd015_01629d8305fb_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-07 00:30:34,527 f1126adb MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-07 00:31:01,781 f1126adb MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-07 00:31:04,006 f1126adb MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r390a56f0bc64874d_01629d83799f_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r390a56f0bc64874d_01629d83799f_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r390a56f0bc64874d_01629d83799f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-07 00:31:04,006 f1126adb MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-07 00:31:24,869 f1126adb MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-07 00:31:27,298 f1126adb MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r683b7ed549481cb6_01629d83d49c_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r683b7ed549481cb6_01629d83d49c_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r683b7ed549481cb6_01629d83d49c_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-07 00:31:27,299 f1126adb MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-07 00:31:53,174 f1126adb MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-07 00:31:55,330 f1126adb MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r4fb88dfbfce54044_01629d84424f_1 ... (0s) Current status: 
RUNNING   

[jira] [Created] (BEAM-4030) Add CombineFn.compact, similar to Java

[Robert Bradshaw (JIRA)]

Robert Bradshaw created BEAM-4030:
-

 Summary: Add CombineFn.compact, similar to Java
 Key: BEAM-4030
 URL: https://issues.apache.org/jira/browse/BEAM-4030
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Robert Bradshaw
Assignee: Ahmet Altay


Some CombineFns buffer elements in their add_inputs because a combining 
operation cost can be effectively amortized across many elements. However, this 
introduces the extra (possibly higher) cost of potentially serializing more 
expensive buffers through shuffle. We should add a CombineFn.compact(self, 
accumulator) method (defaulting to the identity) similar to what the Java SDK 
provides which is called when flushing an element from the PGBKCV table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #113

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #17

[Apache Jenkins Server]

See 


Changes:

[wcn] Add float support for the SDK.

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

[wcn] Add diagnostics for sinks.

[wcn] Add hooks to the Go SDK.

[lcwik] [BEAM-3250] Add JUnit test reporting to Jenkins.

[altay] Migrate JUnit test archiving to Python Gradle build

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 95.08 KB...]
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2447)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2335)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:623)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy60.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy61.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:248)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:235)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException):
 Cannot create 
file/TEXTIO_IT__1523060455051/.temp-beam-2018-04-07_00-20-55-0/f62dd96a-d7ca-4cf8-bca3-8f0ac5e8294b.
 Name node is in safe mode.
The reported blocks 31 has reached the threshold 0.9990 of total blocks 31. The 
number of live datanodes 1 has reached the minimum number 0. In safe mode 
extension. Safe mode will be turned off automatically in 27 seconds.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2447)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #19

[Apache Jenkins Server]

See 


Changes:

[wcn] Add float support for the SDK.

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

[wcn] Add diagnostics for sinks.

[wcn] Add hooks to the Go SDK.

[lcwik] [BEAM-3250] Add JUnit test reporting to Jenkins.

[altay] Migrate JUnit test archiving to Python Gradle build

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 47.36 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.224.126.213:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:181)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:160)
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:381)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:353)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:284)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting for a 
server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.224.126.213:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #421

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #342

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #24

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #360

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #110

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1115

[Apache Jenkins Server]

See 


Changes:

[wcn] Add float support for the SDK.

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

[wcn] Add diagnostics for sinks.

[wcn] Add hooks to the Go SDK.

[lcwik] [BEAM-3250] Add JUnit test reporting to Jenkins.

[altay] Migrate JUnit test archiving to Python Gradle build

[herohde] Use WorkerBinary override in Go Dataflow

--
[...truncated 61.72 KB...]
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.019s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.079s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  3.206 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  2.759 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.150 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  9.768 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.330 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.979 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.162 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 30.558 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #18

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-2927) Python SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2927?focusedWorklogId=88651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88651
 ]

ASF GitHub Bot logged work on BEAM-2927:


Author: ASF GitHub Bot
Created on: 07/Apr/18 00:12
Start Date: 07/Apr/18 00:12
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4983: [BEAM-2927] 
Re-enable side inputs for Fn API on Dataflow
URL: https://github.com/apache/beam/pull/4983#issuecomment-379416476
 
 
   jenkins: retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88651)
Time Spent: 2.5h  (was: 2h 20m)

> Python SDK support for portable side input
> --
>
> Key: BEAM-2927
> URL: https://issues.apache.org/jira/browse/BEAM-2927
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
> Fix For: 2.5.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #344

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-2927) Python SDK support for portable side input

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-2927:
--
Fix Version/s: 2.5.0

> Python SDK support for portable side input
> --
>
> Key: BEAM-2927
> URL: https://issues.apache.org/jira/browse/BEAM-2927
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
> Fix For: 2.5.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #18

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #346

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #24

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88650
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 07/Apr/18 00:03
Start Date: 07/Apr/18 00:03
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379415200
 
 
   Jenkins retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88650)
Time Spent: 1h 10m  (was: 1h)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4613

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

--
[...truncated 1.04 MB...]
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/opcounters_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operation_specs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/operations.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_main_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sdk_worker_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/sideinputs_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_fast.pyx -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_slow.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/statesampler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/worker_id_interceptor_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/testing/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing
copying apache_beam/testing/pipeline_verifiers.py -> 
apache-beam-2.5.0.dev0/apache_beam/testing

[beam] branch master updated (7d8461a -> 316d667)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7d8461a  Merge pull request #5035 from 
charlesccychen/fix-dr-attributeerror
 add a90f7bc  Use WorkerBinary override in Go Dataflow
 new 316d667  Merge pull request #5050: Use WorkerBinary override in Go 
Dataflow

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/runners/dataflow/dataflow.go | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5050: Use WorkerBinary override in Go Dataflow

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 316d667682fe9c422b482940f9ac4f4b5ee1f4bc
Merge: 7d8461a a90f7bc
Author: Thomas Groh 
AuthorDate: Fri Apr 6 16:54:35 2018 -0700

Merge pull request #5050: Use WorkerBinary override in Go Dataflow

 sdks/go/pkg/beam/runners/dataflow/dataflow.go | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=88649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88649
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:48
Start Date: 06/Apr/18 23:48
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5051: [BEAM-3995] Add 
basic DirectRunner Gradle run task to Nexmark
URL: https://github.com/apache/beam/pull/5051#issuecomment-379413203
 
 
   I've gone through a few iterations. First, hardcoded some configs. Second, 
building a fat jar - I still like it best but something about our shadow config 
is already screwed up ;_;. Last, using JavaExec to do like Maven `exec:java`. 
You have to plumb the args as a string then split but otherwise the same. Not 
as bad as I was worried about.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88649)
Time Spent: 50m  (was: 40m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88648
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:45
Start Date: 06/Apr/18 23:45
Worklog Time Spent: 10m 
  Work Description: shoyer commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379412756
 
 
   Commits are squashed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88648)
Time Spent: 6.5h  (was: 6h 20m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2927) Python SDK support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2927?focusedWorklogId=88647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88647
 ]

ASF GitHub Bot logged work on BEAM-2927:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:44
Start Date: 06/Apr/18 23:44
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4983: [BEAM-2927] 
Re-enable side inputs for Fn API on Dataflow
URL: https://github.com/apache/beam/pull/4983#issuecomment-379412603
 
 
   run python postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88647)
Time Spent: 2h 20m  (was: 2h 10m)

> Python SDK support for portable side input
> --
>
> Key: BEAM-2927
> URL: https://issues.apache.org/jira/browse/BEAM-2927
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88646
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:42
Start Date: 06/Apr/18 23:42
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379412336
 
 
   Yes, please squash your commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88646)
Time Spent: 6h 20m  (was: 6h 10m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=88642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88642
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:28
Start Date: 06/Apr/18 23:28
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5048: [BEAM-3255] Updates 
to support the release process via Gradle
URL: https://github.com/apache/beam/pull/5048#issuecomment-379410215
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88642)
Time Spent: 0.5h  (was: 20m)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=88640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88640
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 06/Apr/18 23:18
Start Date: 06/Apr/18 23:18
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5048: [BEAM-3255] Updates 
to support the release process via Gradle
URL: https://github.com/apache/beam/pull/5048#issuecomment-379408687
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88640)
Time Spent: 20m  (was: 10m)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #6

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

--
[...truncated 1.70 MB...]
at 
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy3.processTestClass(Unknown Source)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #23

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

--
[...truncated 108.92 MB...]
INFO: 04/06/2018 22:37:33   PAssert$166/GroupGlobally/GroupDummyAndContents 
-> PAssert$166/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$166/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$166/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$166/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$166/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude)(1/1)
 switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/06/2018 22:37:33 PAssert$166/GroupGlobally/GroupDummyAndContents -> 
PAssert$166/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$166/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$166/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$166/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$166/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude)(1/1)
 switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 10:37:33 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem (1/1) (3822c0798c6ba02e5c896312c0500a95) switched from 
RUNNING to FINISHED.
Apr 06, 2018 10:37:33 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map/ParMultiDo(Anonymous)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem (1/1) (93a9e5e516ccd99678b2faf280e9be89) switched from 
RUNNING to FINISHED.
Apr 06, 2018 10:37:33 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (be92d05099b04f2601808dad2ccd39e5) switched from RUNNING 
to FINISHED.
Apr 06, 2018 10:37:33 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/06/2018 22:37:33   
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/06/2018 22:37:33 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 10:37:33 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/06/2018 22:37:33   
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map/ParMultiDo(Anonymous)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/06/2018 22:37:33 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map/ParMultiDo(Anonymous)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) 

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88630
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 22:29
Start Date: 06/Apr/18 22:29
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379401511
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88630)
Time Spent: 1h  (was: 50m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88628
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 22:22
Start Date: 06/Apr/18 22:22
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5035: [BEAM-4024] Fix 
AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379400320
 
 
   I will merge this based on passed gradle python precommit tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88628)
Time Spent: 1.5h  (was: 1h 20m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> 

[beam] 01/01: Merge pull request #5035 from charlesccychen/fix-dr-attributeerror

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7d8461a86c8dd32453f57ef85f345f7962010b1b
Merge: 510aeca 1b2b747
Author: Ahmet Altay 
AuthorDate: Fri Apr 6 15:22:35 2018 -0700

Merge pull request #5035 from charlesccychen/fix-dr-attributeerror

[BEAM-4024] Fix AttributeError in BundleBasedDirectRunner

 sdks/python/apache_beam/runners/direct/executor.py | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88629=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88629
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 22:22
Start Date: 06/Apr/18 22:22
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5035: [BEAM-4024] Fix 
AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/runners/direct/executor.py 
b/sdks/python/apache_beam/runners/direct/executor.py
index 107f00764a2..8ab30b4046c 100644
--- a/sdks/python/apache_beam/runners/direct/executor.py
+++ b/sdks/python/apache_beam/runners/direct/executor.py
@@ -274,6 +274,7 @@ def __init__(self, transform_evaluator_registry, 
evaluation_context,
 self._input_bundle = input_bundle
 # For non-empty bundles, store the window of the max EOW.
 # TODO(mariagh): Move to class _Bundle's inner _StackedWindowedValues
+self._latest_main_input_window = None
 if input_bundle.has_elements():
   self._latest_main_input_window = input_bundle._elements[0].windows[0]
   for elem in input_bundle.get_elements_iterable():


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88629)
Time Spent: 1h 40m  (was: 1.5h)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 

[beam] branch master updated (510aeca -> 7d8461a)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 510aeca  [BEAM-3257] Migrate JUnit test archiving to Python Gradle 
build
 add 1b2b747  [BEAM-4024] Fix AttributeError in BundleBasedDirectRunner
 new 7d8461a  Merge pull request #5035 from 
charlesccychen/fix-dr-attributeerror

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/direct/executor.py | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4014?focusedWorklogId=88619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88619
 ]

ASF GitHub Bot logged work on BEAM-4014:


Author: ASF GitHub Bot
Created on: 06/Apr/18 21:52
Start Date: 06/Apr/18 21:52
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5047: 
[BEAM-4014] Replace the Java maven based PostCommit with a Gradle based version.
URL: https://github.com/apache/beam/pull/5047#discussion_r179884471
 
 

 ##
 File path: .test-infra/jenkins/job_beam_PostCommit_Java_GradleBuild.groovy
 ##
 @@ -20,46 +20,50 @@ import common_job_properties
 
 // This job runs the Java postcommit tests, including the suite of integration
 // tests.
-mavenJob('beam_PostCommit_Java_MavenInstall') {
-  description('Runs postcommit tests on the Java SDK.')
-
+job('beam_PostCommit_Java_GradleBuild') {
 
 Review comment:
   As it exists, this is a Dataflow Java PostCommit, rather than a 
Gradle-target. Perhaps a name change is in order?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88619)
Time Spent: 40m  (was: 0.5h)

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4014?focusedWorklogId=88614=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88614
 ]

ASF GitHub Bot logged work on BEAM-4014:


Author: ASF GitHub Bot
Created on: 06/Apr/18 21:12
Start Date: 06/Apr/18 21:12
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5047: [BEAM-4014] Replace 
the Java maven based PostCommit with a Gradle based version.
URL: https://github.com/apache/beam/pull/5047#issuecomment-379381052
 
 
   If #5048 is merged, the project names need to be updated.
   
   CC: @swegner 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88614)
Time Spent: 0.5h  (was: 20m)

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=88613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88613
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 06/Apr/18 21:07
Start Date: 06/Apr/18 21:07
Worklog Time Spent: 10m 
  Work Description: swegner opened a new pull request #5048: [BEAM-3255] 
Updates to support the release process via Gradle
URL: https://github.com/apache/beam/pull/5048
 
 
   This merges many changes from our Gradle fixit week for publishing and 
validating releases from Gradle.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88613)
Time Spent: 10m
Remaining Estimate: 0h

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88609=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88609
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:57
Start Date: 06/Apr/18 20:57
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5035: [BEAM-4024] 
Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379376088
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88609)
Time Spent: 1h 20m  (was: 1h 10m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88608
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:56
Start Date: 06/Apr/18 20:56
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379375704
 
 
   @udim You're probably right about the `./gradlew clean` being unnecessary. I 
only had it there because it was mentioned in the original code with `mvn clean 
verify`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88608)
Time Spent: 1h 40m  (was: 1.5h)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4014?focusedWorklogId=88607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88607
 ]

ASF GitHub Bot logged work on BEAM-4014:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:52
Start Date: 06/Apr/18 20:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5047: [BEAM-4014] Replace 
the Java maven based PostCommit with a Gradle based version.
URL: https://github.com/apache/beam/pull/5047#issuecomment-379374480
 
 
   CC: @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88607)
Time Spent: 20m  (was: 10m)

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4014?focusedWorklogId=88606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88606
 ]

ASF GitHub Bot logged work on BEAM-4014:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:51
Start Date: 06/Apr/18 20:51
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #5047: [BEAM-4014] 
Replace the Java maven based PostCommit with a Gradle based version.
URL: https://github.com/apache/beam/pull/5047
 
 
   
https://builds.apache.org/view/A-D/view/Beam/job/beam_PreCommit_Java_MavenInstall/19300/testReport/api/json
   shows the list of ITs that were executed. Note that its all the 
examples/java ITs and the sdks/java/io/google-cloud-platform ITs.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88606)
Time Spent: 10m
Remaining Estimate: 0h

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-4014:
---

Assignee: Luke Cwik  (was: Thomas Groh)

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88603
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:41
Start Date: 06/Apr/18 20:41
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #412: 
[BEAM-3985] Document Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#discussion_r179870676
 
 

 ##
 File path: src/contribute/contribution-guide.md
 ##
 @@ -244,16 +244,28 @@ All code should have appropriate unit testing coverage. 
New code should have
 new tests in the same contribution. Bug fixes should include a regression test
 to prevent the issue from reoccurring.
 
- Java SDK
+The entire set of unit tests can be run with this command at the root of the
+git repository.
 
-For contributions to the Java code, run unit tests locally via Maven.
+$ ./gradlew test
 
 Review comment:
   Perhaps adding some instructions for calling unit tests on certain languages 
may be useful. This is slightly different than the precommit info down below 
because it is only unit tests.
   
   Examples:
   ```
   $ ./gradlew sdks:java:test
   $ ./gradlew sdks:python:test
   $ ./gradlew sdks:go:test
   ```
   
   Also may be worth using the "check" task instead of "test". According to the 
gradle docs:
   
   > It is common for all verification tasks, including tests and linting, to 
be executed using the check task. 
([Source](https://docs.gradle.org/current/userguide/command_line_interface.html#_running_all_checks))
   
   I don't know if Beam is doing this at the moment, but I feel like this area 
of the code should mention using check for more intense checks (i.e. for 
replacing mvn verify), while test is better for basic unit tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88603)
Time Spent: 1.5h  (was: 1h 20m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88597
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:28
Start Date: 06/Apr/18 20:28
Worklog Time Spent: 10m 
  Work Description: youngoli commented on a change in pull request #412: 
[BEAM-3985] Document Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#discussion_r179867936
 
 

 ##
 File path: src/contribute/contribution-guide.md
 ##
 @@ -244,16 +244,28 @@ All code should have appropriate unit testing coverage. 
New code should have
 new tests in the same contribution. Bug fixes should include a regression test
 to prevent the issue from reoccurring.
 
- Java SDK
+The entire set of unit tests can be run with this command at the root of the
+git repository.
 
-For contributions to the Java code, run unit tests locally via Maven.
+$ ./gradlew test
 
-$ mvn clean verify
+Run this command to test for lint errors.
+
+$ ./gradlew lint
 
 Review comment:
   I don't think this needs to be mentioned here since the lint task only lints 
the Python code right now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88597)
Time Spent: 1h 20m  (was: 1h 10m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88596=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88596
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:21
Start Date: 06/Apr/18 20:21
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379366509
 
 
   Sorry for not coordinating with you @youngoli, let's merge #414 first and 
I'll deal with the conflicts in mine.
   BTW, what do you think of my suggestions for running tests and lint? I don't 
believe that running `clean` first is necessary, but I might be wrong.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88596)
Time Spent: 1h 10m  (was: 1h)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88594
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:10
Start Date: 06/Apr/18 20:10
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379363801
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88594)
Time Spent: 50m  (was: 40m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88593=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88593
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 20:10
Start Date: 06/Apr/18 20:10
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88593)
Time Spent: 40m  (was: 0.5h)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #5

[Apache Jenkins Server]

See 


Changes:

[altay] Migrate JUnit test archiving to Python Gradle build

--
[...truncated 1.70 MB...]
at org.apache.spark.SparkContext.(SparkContext.scala:457)
at 
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)

[jira] [Commented] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428857#comment-16428857
 ] 

Ahmet Altay commented on BEAM-3956:
---

Great, thank you!

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88578=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88578
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:29
Start Date: 06/Apr/18 19:29
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5045: [BEAM-3257] Migrate 
JUnit test archiving to Python Gradle build
URL: https://github.com/apache/beam/pull/5045
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
index d2fdef72ad3..255277d471a 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
@@ -41,6 +41,12 @@ job('beam_PreCommit_Python_GradleBuild') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins. Note that Nose documentation
+  // specifically mentions that it produces JUnit compatible test results.
+  publishers {
+archiveJunit('**/nosetests.xml')
+  }
+
   def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + ' 
:pythonPreCommit'
   // Sets that this is a PreCommit job.
   common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run 
Python Gradle PreCommit')
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
deleted file mode 100644
index f0429e4aa18..000
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import common_job_properties
-
-// This is the Python precommit which runs a maven install, and the current set
-// of precommit tests.
-mavenJob('beam_PreCommit_Python_MavenInstall') {
-  description('Runs an install of the current GitHub Pull Request.')
-
-  previousNames('beam_PreCommit_MavenVerify')
-
-  // Execute concurrent builds if necessary.
-  concurrentBuild()
-
-  // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(
-delegate,
-'master',
-150)
-
-  // Set Maven parameters.
-  common_job_properties.setMavenConfig(delegate)
-
-  // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/python -am -amd', 'Run Python PreCommit')
-
-  // Maven modules for this job: The Python SDK, its dependencies, and things 
that depend on it,
-  // excluding the container.
-  goals([
-'--batch-mode',
-'--errors',
-'--activate-profiles release',
-'--projects sdks/python,!sdks/python/container',
-'--also-make',
-'--also-make-dependents',
-'-D pullRequest=$ghprbPullId',
-'help:effective-settings',
-'clean',
-'install',
-  ].join(' '))
-}
diff --git a/sdks/python/setup.cfg b/sdks/python/setup.cfg
index 0f6acea2c09..041d6fa55f4 100644
--- a/sdks/python/setup.cfg
+++ b/sdks/python/setup.cfg
@@ -26,6 +26,9 @@ verbosity=2
 # fast_coders_test and typecoders_test.
 exclude=fast_coders_test|typecoders_test
 
+# Creates an xml file compatible with standard XUnit XML format.
+with-xunit=1
+
 # Configurations to control coverage.py.
 [coverage:run]
 branch = True
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index ff88ac42fa8..3cc16823684 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -50,7 +50,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests
   {toxinidir}/run_tox_cleanup.sh
 
 [testenv:py27-cython]
@@ -64,7 +64,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py 

[beam] branch master updated (16232c2 -> 510aeca)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 16232c2  [BEAM-3355] Diagnostic interfaces
 add b83fb93  Migrate JUnit test archiving to Python Gradle build
 new 510aeca  [BEAM-3257] Migrate JUnit test archiving to Python Gradle 
build

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../job_beam_PreCommit_Python_GradleBuild.groovy   |  6 +++
 .../job_beam_PreCommit_Python_MavenInstall.groovy  | 57 --
 sdks/python/setup.cfg  |  3 ++
 sdks/python/tox.ini|  6 +--
 4 files changed, 12 insertions(+), 60 deletions(-)
 delete mode 100644 
.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: [BEAM-3257] Migrate JUnit test archiving to Python Gradle build

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 510aeca108781ae466817238e59973393ba31ab1
Merge: 16232c2 b83fb93
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 15:29:12 2018 -0400

[BEAM-3257] Migrate JUnit test archiving to Python Gradle build

 .../job_beam_PreCommit_Python_GradleBuild.groovy   |  6 +++
 .../job_beam_PreCommit_Python_MavenInstall.groovy  | 57 --
 sdks/python/setup.cfg  |  3 ++
 sdks/python/tox.ini|  6 +--
 4 files changed, 12 insertions(+), 60 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88577
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:26
Start Date: 06/Apr/18 19:26
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5042: [BEAM-3257] Publish 
Python precommit test results to Jenkins, remove Maven based Python precommit.
URL: https://github.com/apache/beam/pull/5042
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
index d2fdef72ad3..255277d471a 100644
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
+++ b/.test-infra/jenkins/job_beam_PreCommit_Python_GradleBuild.groovy
@@ -41,6 +41,12 @@ job('beam_PreCommit_Python_GradleBuild') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins. Note that Nose documentation
+  // specifically mentions that it produces JUnit compatible test results.
+  publishers {
+archiveJunit('**/nosetests.xml')
+  }
+
   def gradle_command_line = './gradlew ' + gradle_switches.join(' ') + ' 
:pythonPreCommit'
   // Sets that this is a PreCommit job.
   common_job_properties.setPreCommit(delegate, gradle_command_line, 'Run 
Python Gradle PreCommit')
diff --git a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy 
b/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
deleted file mode 100644
index f0429e4aa18..000
--- a/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import common_job_properties
-
-// This is the Python precommit which runs a maven install, and the current set
-// of precommit tests.
-mavenJob('beam_PreCommit_Python_MavenInstall') {
-  description('Runs an install of the current GitHub Pull Request.')
-
-  previousNames('beam_PreCommit_MavenVerify')
-
-  // Execute concurrent builds if necessary.
-  concurrentBuild()
-
-  // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(
-delegate,
-'master',
-150)
-
-  // Set Maven parameters.
-  common_job_properties.setMavenConfig(delegate)
-
-  // Sets that this is a PreCommit job.
-  common_job_properties.setPreCommit(delegate, 'mvn clean install -pl 
sdks/python -am -amd', 'Run Python PreCommit')
-
-  // Maven modules for this job: The Python SDK, its dependencies, and things 
that depend on it,
-  // excluding the container.
-  goals([
-'--batch-mode',
-'--errors',
-'--activate-profiles release',
-'--projects sdks/python,!sdks/python/container',
-'--also-make',
-'--also-make-dependents',
-'-D pullRequest=$ghprbPullId',
-'help:effective-settings',
-'clean',
-'install',
-  ].join(' '))
-}
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index ff88ac42fa8..41611384a80 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -50,7 +50,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests --with-xunit
   {toxinidir}/run_tox_cleanup.sh
 
 [testenv:py27-cython]
@@ -64,7 +64,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests --with-xunit
   {toxinidir}/run_tox_cleanup.sh
 
 [testenv:py27-gcp]
@@ -74,7 +74,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests --with-xunit
   {toxinidir}/run_tox_cleanup.sh
 
 [testenv:py27-lint]


 

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88576
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:26
Start Date: 06/Apr/18 19:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5042: [BEAM-3257] Publish 
Python precommit test results to Jenkins, remove Maven based Python precommit.
URL: https://github.com/apache/beam/pull/5042#issuecomment-379353336
 
 
   Superseeded by #5045


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88576)
Time Spent: 3h 50m  (was: 3h 40m)

> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Code is here: 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88575
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:25
Start Date: 06/Apr/18 19:25
Worklog Time Spent: 10m 
  Work Description: aaltay opened a new pull request #5045: [BEAM-3257] 
Migrate JUnit test archiving to Python Gradle build
URL: https://github.com/apache/beam/pull/5045
 
 
   Publish python test results to Jenkins.
   
   Delete Maven based Python precommit.
   
   Note that this change supersedes #4689 #5042
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88575)
Time Spent: 3h 40m  (was: 3.5h)

> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Code is here: 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428811#comment-16428811
 ] 

Ahmet Altay commented on BEAM-3956:
---

[~shoyer] Would you be able to fix the Direct Runner related issues introduce 
in #4376?

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88573=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88573
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:16
Start Date: 06/Apr/18 19:16
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5035: [BEAM-4024] Fix 
AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379350953
 
 
   Thank you. I will merge once tests here pass.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88573)
Time Spent: 1h 10m  (was: 1h)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #21

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88571
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 19:05
Start Date: 06/Apr/18 19:05
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5042: 
[BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven 
based Python precommit.
URL: https://github.com/apache/beam/pull/5042#discussion_r179849434
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -50,7 +50,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests --with-xunit
 
 Review comment:
   I believe it will since they will overwrite each other. You would only get 
the test results from the last run.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88571)
Time Spent: 3.5h  (was: 3h 20m)

> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Code is here: 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada updated BEAM-4028:

Description: 
Steps can have different names depending on the runner (stage, step, user, 
system name...). 

Depending on the needs of different components (operations, logging, metrics, 
statesampling) these step names are passed around without a specific order.

Instead, SDK should rely on `NameContext` objects that carry all the naming 
information for a single step.

  was:
Steps can have different steps depending on the runner (stage, step, user, 
system name...). 

Depending on the needs of different components (operations, logging, metrics, 
statesampling) these step names are passed around without a specific order.

Instead, SDK should rely on `NameContext` objects that carry all the naming 
information for a single step.


> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88565
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:45
Start Date: 06/Apr/18 18:45
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5035: [BEAM-4024] 
Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379342317
 
 
   @aaltay: I have filed https://issues.apache.org/jira/browse/BEAM-4029 for 
adding additional tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88565)
Time Spent: 1h  (was: 50m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> 

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88564
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:44
Start Date: 06/Apr/18 18:44
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5035: [BEAM-4024] 
Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379342182
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88564)
Time Spent: 50m  (was: 40m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in 

[jira] [Commented] (BEAM-3991) Update dependency 'google-api-services-storage' to latest version

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428763#comment-16428763
 ] 

Chamikara Jayalath commented on BEAM-3991:
--

Due to conflicting transitive dependencies we have to update all of the 
following dependencies.

google-api-services-bigquery
google-api-services-cloudresourcemanager
google-api-services-pubsub
google-api-services-storage
google-api-services-dataflow
google-api-services-clouddebugger
google-api-client

 

Please note that GCS dependency update is a blocker due to following service 
discontinuation announcement.

https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html

> Update dependency 'google-api-services-storage' to latest version
> -
>
> Key: BEAM-3991
> URL: https://issues.apache.org/jira/browse/BEAM-3991
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.5.0
>
>
> Currently we use version 'v1-rev71-1.22.0' which is deprecated and about two 
> years old.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88561=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88561
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:39
Start Date: 06/Apr/18 18:39
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379340982
 
 
   So I didn't realize you were working on this. I made some overlapping 
changes in my PR #414, so one of us will probably need to merge the changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88561)
Time Spent: 1h  (was: 50m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=88558=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88558
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:38
Start Date: 06/Apr/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #4941: BEAM-3910: Add 
float support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/coder.go b/sdks/go/pkg/beam/coder.go
index c7a2d4cabc5..b7f62186135 100644
--- a/sdks/go/pkg/beam/coder.go
+++ b/sdks/go/pkg/beam/coder.go
@@ -107,6 +107,13 @@ func inferCoder(t FullType) (*coder.Coder, error) {
return nil, err
}
return {Kind: coder.Custom, T: t, Custom: 
c}, nil
+   case reflectx.Float32, reflectx.Float64:
+   c, err := coderx.NewFloat(t.Type())
+   if err != nil {
+   return nil, err
+   }
+   return {Kind: coder.Custom, T: t, Custom: 
c}, nil
+
case reflectx.String, reflectx.ByteSlice:
// TODO(BEAM-3580): we should stop encoding string 
using the bytecoder. It forces
// conversions at runtime in inconvenient places.
diff --git a/sdks/go/pkg/beam/core/runtime/coderx/float.go 
b/sdks/go/pkg/beam/core/runtime/coderx/float.go
new file mode 100644
index 000..403cd30ac1d
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/coderx/float.go
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package coderx
+
+import (
+   "fmt"
+   "math"
+   "math/bits"
+   "reflect"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/graph/coder"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/reflectx"
+)
+
+func init() {
+   runtime.RegisterFunction(encFloat)
+   runtime.RegisterFunction(decFloat)
+}
+
+func encFloat(v typex.T) []byte {
+   var val float64
+   switch n := v.(type) {
+   case float32:
+   val = float64(n)
+   case float64:
+   val = n
+   default:
+   panic(fmt.Sprintf("received unknown value type: want a float, 
got %T", n))
+   }
+
+   return encVarUintZ(bits.ReverseBytes64(math.Float64bits(val)))
+}
+
+func decFloat(t reflect.Type, data []byte) (typex.T, error) {
+   uval, err := decVarUintZ(reflectx.Uint64, data)
+   if err != nil {
+   return nil, fmt.Errorf("invalid float encoding for: %v", data)
+   }
+
+   n := math.Float64frombits(bits.ReverseBytes64(uval.(uint64)))
+   switch t.Kind() {
+   case reflect.Float64:
+   return n, nil
+   case reflect.Float32:
+   return float32(n), nil
+   default:
+   panic(fmt.Sprintf("unreachable statement: expected a float, got 
%v", t))
+   }
+}
+
+// NewFloat returns a coder for the given float type. It uses the same
+// encoding scheme as the gob package.
+func NewFloat(t reflect.Type) (*coder.CustomCoder, error) {
+   switch t.Kind() {
+   case reflect.Float32, reflect.Float64:
+   return coder.NewCustomCoder("float", t, encFloat, decFloat)
+   default:
+   return nil, fmt.Errorf("not a float type: %v", t)
+   }
+}
diff --git a/sdks/go/pkg/beam/core/runtime/coderx/float_test.go 
b/sdks/go/pkg/beam/core/runtime/coderx/float_test.go
new file mode 100644
index 000..460057e900b
--- /dev/null
+++ b/sdks/go/pkg/beam/core/runtime/coderx/float_test.go
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more

[jira] [Work logged] (BEAM-3252) Update contributors guide to discuss Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3252?focusedWorklogId=88559=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88559
 ]

ASF GitHub Bot logged work on BEAM-3252:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:38
Start Date: 06/Apr/18 18:38
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #414: [BEAM-3252] Updating 
contributor's guide to reference Gradle.
URL: https://github.com/apache/beam-site/pull/414#issuecomment-379340670
 
 
   Note: I was making this PR unaware that @udim was working on an overlapping 
change in #412. The overlapping changes will probably need to be merged by 
someone.
   
   Also note: This change and other Maven->Gradle changes should probably go in 
a separate branch so the site doesn't mention Maven in some places and Gradle 
in others.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88559)
Time Spent: 20m  (was: 10m)

> Update contributors guide to discuss Gradle
> ---
>
> Key: BEAM-3252
> URL: https://issues.apache.org/jira/browse/BEAM-3252
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Luke Cwik
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88560
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:38
Start Date: 06/Apr/18 18:38
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #4311: [BEAM-3355] 
Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/harness/harness.go 
b/sdks/go/pkg/beam/core/runtime/harness/harness.go
index 5e545c550f9..f8efffece71 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/harness.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/harness.go
@@ -17,16 +17,14 @@
 package harness
 
 import (
-   "bytes"
"context"
"fmt"
"io"
-   "io/ioutil"
-   "runtime/pprof"
"sync"
"time"
 
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime/exec"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/hooks"
"github.com/apache/beam/sdks/go/pkg/beam/log"
fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
"github.com/apache/beam/sdks/go/pkg/beam/util/grpcx"
@@ -37,12 +35,13 @@ import (
 // TODO(herohde) 2/8/2017: for now, assume we stage a full binary (not a 
plugin).
 
 // Main is the main entrypoint for the Go harness. It runs at "runtime" -- not
-// "pipeline-construction time" -- on each worker. It is a Fn API client and
+// "pipeline-construction time" -- on each worker. It is a FnAPI client and
 // ultimately responsible for correctly executing user code.
 func Main(ctx context.Context, loggingEndpoint, controlEndpoint string) error {
-   setupRemoteLogging(ctx, loggingEndpoint)
-   setupDiagnosticRecording()
+   hooks.DeserializeHooksFromOptions()
 
+   hooks.RunInitHooks(ctx)
+   setupRemoteLogging(ctx, loggingEndpoint)
recordHeader()
 
// Connect to FnAPI control server. Receive and execute work.
@@ -87,8 +86,6 @@ func Main(ctx context.Context, loggingEndpoint, 
controlEndpoint string) error {
data:   {},
}
 
-   var cpuProfBuf bytes.Buffer
-
// gRPC requires all readers of a stream be the same goroutine, so this 
goroutine
// is responsible for managing the network data. All it does is pull 
data from
// the stream, and hand off the message to a goroutine to actually be 
handled,
@@ -112,18 +109,10 @@ func Main(ctx context.Context, loggingEndpoint, 
controlEndpoint string) error {
log.Debugf(ctx, "RECV: %v", 
proto.MarshalTextString(req))
recordInstructionRequest(req)
 
-   if isEnabled("cpu_profiling") {
-   cpuProfBuf.Reset()
-   pprof.StartCPUProfile()
-   }
+   hooks.RunRequestHooks(ctx, req)
resp := ctrl.handleInstruction(ctx, req)
 
-   if isEnabled("cpu_profiling") {
-   pprof.StopCPUProfile()
-   if err := 
ioutil.WriteFile(fmt.Sprintf("%s/cpu_prof%s", storagePath, req.InstructionId), 
cpuProfBuf.Bytes(), 0644); err != nil {
-   log.Warnf(ctx, "Failed to write CPU 
profile for instruction %s: %v", req.InstructionId, err)
-   }
-   }
+   hooks.RunResponseHooks(ctx, req, resp)
 
recordInstructionResponse(resp)
if resp != nil {
diff --git a/sdks/go/pkg/beam/core/runtime/harness/session.go 
b/sdks/go/pkg/beam/core/runtime/harness/session.go
index 67f577f5035..69a1d992786 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/session.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/session.go
@@ -16,13 +16,14 @@
 package harness
 
 import (
+   "context"
"fmt"
-   "os"
+   "io"
"sync"
-   "time"
 
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
"github.com/apache/beam/sdks/go/pkg/beam/core/runtime/harness/session"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/hooks"
pb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
"github.com/golang/protobuf/proto"
 )
@@ -35,12 +36,13 @@ const (
dataSend
 )
 
+// capture is set by the capture hook below.
+var capture io.WriteCloser
+
 var (
selectedOptions = make(map[string]bool)
-   // TODO(wcn): add a buffered writer around capture and use it.
- 

[beam] 01/01: [BEAM-3355] Diagnostic interfaces

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 16232c2a8964e294662bbfa30a12aff1a17cf8fa
Merge: c75a1d9 dea8f6e
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 14:38:53 2018 -0400

[BEAM-3355] Diagnostic interfaces

 sdks/go/pkg/beam/core/runtime/harness/harness.go |  25 +--
 sdks/go/pkg/beam/core/runtime/harness/session.go |  97 +++-
 sdks/go/pkg/beam/core/util/hooks/hooks.go| 186 +++
 sdks/go/pkg/beam/runners/dataflow/dataflow.go|  40 -
 sdks/go/pkg/beam/runners/dataflow/messages.go|   6 +-
 sdks/go/pkg/beam/runners/session/session.go  |   6 +-
 sdks/go/pkg/beam/util/grpcx/hook.go  |  86 +++
 sdks/go/pkg/beam/x/hooks/perf/perf.go| 170 +
 8 files changed, 550 insertions(+), 66 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88556
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:37
Start Date: 06/Apr/18 18:37
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88556)
Time Spent: 0.5h  (was: 20m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Steps can have different steps depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: BEAM-3910: Add float support for the Go SDK.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c75a1d919ba11826a1fd89851569dace7ba0f95d
Merge: 6df71d2 0d12851
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 14:38:17 2018 -0400

BEAM-3910: Add float support for the Go SDK.

 sdks/go/pkg/beam/coder.go  |   7 +
 sdks/go/pkg/beam/core/runtime/coderx/float.go  |  75 ++
 sdks/go/pkg/beam/core/runtime/coderx/float_test.go |  50 +++
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  |   8 +
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go   | 165 +++--
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto   |   2 +
 6 files changed, 228 insertions(+), 79 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (c75a1d9 -> 16232c2)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c75a1d9  BEAM-3910: Add float support for the Go SDK.
 add dea8f6e  Add hooks to the Go SDK.
 new 16232c2  [BEAM-3355] Diagnostic interfaces

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/harness/harness.go |  25 +--
 sdks/go/pkg/beam/core/runtime/harness/session.go |  97 +++-
 sdks/go/pkg/beam/core/util/hooks/hooks.go| 186 +++
 sdks/go/pkg/beam/runners/dataflow/dataflow.go|  40 -
 sdks/go/pkg/beam/runners/dataflow/messages.go|   6 +-
 sdks/go/pkg/beam/runners/session/session.go  |   6 +-
 sdks/go/pkg/beam/util/grpcx/hook.go  |  86 +++
 sdks/go/pkg/beam/x/hooks/perf/perf.go| 170 +
 8 files changed, 550 insertions(+), 66 deletions(-)
 create mode 100644 sdks/go/pkg/beam/core/util/hooks/hooks.go
 create mode 100644 sdks/go/pkg/beam/util/grpcx/hook.go
 create mode 100644 sdks/go/pkg/beam/x/hooks/perf/perf.go

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88557
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:37
Start Date: 06/Apr/18 18:37
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5042: 
[BEAM-3257] Publish Python precommit test results to Jenkins, remove Maven 
based Python precommit.
URL: https://github.com/apache/beam/pull/5042#discussion_r179842426
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -50,7 +50,7 @@ commands =
   pip --version
   {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
-  python setup.py test
+  python setup.py nosetests --with-xunit
 
 Review comment:
   Will each run's `nosetests.xml` output file overwrite the others'?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88557)
Time Spent: 3h 20m  (was: 3h 10m)

> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Code is here: 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (6df71d2 -> c75a1d9)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6df71d2  Add diagnostics for sinks.
 add 0d12851  Add float support for the SDK.
 new c75a1d9  BEAM-3910: Add float support for the Go SDK.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/coder.go  |   7 +
 sdks/go/pkg/beam/core/runtime/coderx/float.go  |  75 ++
 .../coderx/{varint_test.go => float_test.go}   |  48 +-
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  |   8 +
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go   | 165 +++--
 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto   |   2 +
 6 files changed, 185 insertions(+), 120 deletions(-)
 create mode 100644 sdks/go/pkg/beam/core/runtime/coderx/float.go
 copy sdks/go/pkg/beam/core/runtime/coderx/{varint_test.go => float_test.go} 
(62%)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (ddd0c5c -> 6df71d2)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from ddd0c5c  [BEAM-3250] Add JUnit test reporting to Jenkins.
 add 071718a  Add diagnostics for sinks.
 new 6df71d2  Add diagnostics for sinks.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/exec/datasink.go   | 13 +++--
 sdks/go/pkg/beam/core/runtime/exec/datasource.go |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.