[beam] 01/01: Add diagnostics for sinks.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6df71d2cf257120d424098d5cc84369561fdd788
Merge: ddd0c5c 071718a
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 14:37:16 2018 -0400

Add diagnostics for sinks.

 sdks/go/pkg/beam/core/runtime/exec/datasink.go   | 13 +++--
 sdks/go/pkg/beam/core/runtime/exec/datasource.go |  2 +-
 2 files changed, 12 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3252) Update contributors guide to discuss Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3252?focusedWorklogId=88555=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88555
 ]

ASF GitHub Bot logged work on BEAM-3252:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:36
Start Date: 06/Apr/18 18:36
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #414: [BEAM-3252] 
Updating contributor's guide to reference Gradle.
URL: https://github.com/apache/beam-site/pull/414
 
 
   Removed as many references to maven or pom files under contribute
   as I could. There's still a few references in the Eclipse Tips file
   since I have no clue how to use Eclipse with Gradle and have yet to
   find someone who does.
   
   Also references in maturity-model in the dependency tree. It's not
   really possible to create a Gradle dependency tree that isn't hundreds
   of thousands of lines though, because each distinct configuration in
   each project has it's own dependencies so you end up creating hundreds
   of dependency trees, many of which are hundreds of lines long by
   themselves.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88555)
Time Spent: 10m
Remaining Estimate: 0h

> Update contributors guide to discuss Gradle
> ---
>
> Key: BEAM-3252
> URL: https://issues.apache.org/jira/browse/BEAM-3252
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Luke Cwik
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=88553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88553
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:35
Start Date: 06/Apr/18 18:35
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5041: [BEAM-3250] Add 
JUnit test reporting to Jenkins.
URL: https://github.com/apache/beam/pull/5041
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy
index 512cfa976a1..f1c7e18c066 100644
--- a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy
@@ -36,6 +36,11 @@ job('beam_PostCommit_Java_ValidatesRunner_Apex_Gradle') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
   // Sets that this is a PostCommit job.
   common_job_properties.setPostCommit(delegate)
 
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 318c834957f..5159f8afae2 100644
--- 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++ 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -37,6 +37,11 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
   // Sets that this is a PostCommit job.
   common_job_properties.setPostCommit(delegate)
 
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy
index af9855adc89..6daed0bf95b 100644
--- a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy
@@ -34,6 +34,11 @@ job('beam_PostCommit_Java_ValidatesRunner_Flink_Gradle') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
   // Sets that this is a PostCommit job.
   common_job_properties.setPostCommit(delegate)
 
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy
index 8ba0a71dc59..ef6b34ccdf8 100644
--- 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy
+++ 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy
@@ -39,6 +39,11 @@ job('beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
   // Sets that this is a PostCommit job.
   // 0 5 31 2 * will run on Feb 31 (i.e. never) according to job properties.
   // In post-commit this job triggers only on SCM changes.
diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy
index 33c0d3f8f4d..7824e91e8d2 100644
--- a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy
+++ b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy
@@ -36,6 +36,11 @@ job('beam_PostCommit_Java_ValidatesRunner_Spark_Gradle') {
 '--rerun-tasks',
   ]
 
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
   // Sets that this is a PostCommit job.
   common_job_properties.setPostCommit(delegate)
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88553)
Time Spent: 4h  (was: 3h 50m)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
>  

[jira] [Work logged] (BEAM-3257) Migrate Python Jenkins PreCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3257?focusedWorklogId=88554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88554
 ]

ASF GitHub Bot logged work on BEAM-3257:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:35
Start Date: 06/Apr/18 18:35
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5042: [BEAM-3257] Publish 
Python precommit test results to Jenkins, remove Maven based Python precommit.
URL: https://github.com/apache/beam/pull/5042#issuecomment-379339872
 
 
   CC: @udim @robertwb 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88554)
Time Spent: 3h 10m  (was: 3h)

> Migrate Python Jenkins PreCommits to Gradle
> ---
>
> Key: BEAM-3257
> URL: https://issues.apache.org/jira/browse/BEAM-3257
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Code is here: 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Python_MavenInstall.groovy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[Stephan Hoyer (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428749#comment-16428749
 ] 

Stephan Hoyer commented on BEAM-3956:
-

Indeed, it looks losing stack-traces for the direct runner was introduced 
inadvertently by [https://github.com/apache/beam/pull/4376]

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-3250] Add JUnit test reporting to Jenkins.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ddd0c5ce35cf1c408a970d73dd415d3d57f06396
Merge: 15af0a4 1d8d326
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 14:35:27 2018 -0400

[BEAM-3250] Add JUnit test reporting to Jenkins.

 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy| 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 +
 5 files changed, 25 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (15af0a4 -> ddd0c5c)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 15af0a4  Merge pull request #5038: Performance improvement for inputs
 add 1d8d326  [BEAM-3250] Add JUnit test reporting to Jenkins.
 new ddd0c5c  [BEAM-3250] Add JUnit test reporting to Jenkins.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Apex.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Flink.groovy| 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Gearpump.groovy | 5 +
 .../jenkins/job_beam_PostCommit_Java_ValidatesRunner_Spark.groovy| 5 +
 5 files changed, 25 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=88552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88552
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:31
Start Date: 06/Apr/18 18:31
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5041: 
[BEAM-3250] Add JUnit test reporting to Jenkins.
URL: https://github.com/apache/beam/pull/5041#discussion_r179840994
 
 

 ##
 File path: 
.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
 ##
 @@ -37,7 +37,12 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') 
{
 '--rerun-tasks',
   ]
 
-  // Sets that this is a PostCommit job.
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
+   // Sets that this is a PostCommit job.
 
 Review comment:
   Wow, annoying. Will fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88552)
Time Spent: 3h 50m  (was: 3h 40m)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4029) Test ValidatesRunner tests for BundleBasedDirectRunner

[Charles Chen (JIRA)]

Charles Chen created BEAM-4029:
--

 Summary: Test ValidatesRunner tests for BundleBasedDirectRunner
 Key: BEAM-4029
 URL: https://issues.apache.org/jira/browse/BEAM-4029
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


We currently only run tests for the BundleBasedDirectRunner for streaming 
tests.  We should also run them for ValidatesRunner tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88551=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88551
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:30
Start Date: 06/Apr/18 18:30
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379338408
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88551)
Time Spent: 50m  (was: 40m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1558

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 65.32 KB...]
2018-04-06 18:20:09,398 e9cabf12 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 18:20:36,533 e9cabf12 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 18:20:40,089 e9cabf12 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r330c9edad39f865e_01629c3058ff_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r330c9edad39f865e_01629c3058ff_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r330c9edad39f865e_01629c3058ff_1 ... (0s) Current status: DONE   
2018-04-06 18:20:40,089 e9cabf12 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 18:21:06,122 e9cabf12 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 18:21:09,496 e9cabf12 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r74791a8a2845a559_01629c30cc9e_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r74791a8a2845a559_01629c30cc9e_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r74791a8a2845a559_01629c30cc9e_1 ... (0s) Current status: DONE   
2018-04-06 18:21:09,497 e9cabf12 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 18:21:25,016 e9cabf12 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 18:21:28,498 e9cabf12 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_rcbecdd3d9f5b8fb_01629c31166d_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_rcbecdd3d9f5b8fb_01629c31166d_1 
... (0s) Current status: RUNNING
 Waiting on 
bqjob_rcbecdd3d9f5b8fb_01629c31166d_1 ... (0s) Current status: DONE   
2018-04-06 18:21:28,499 e9cabf12 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 18:21:47,683 e9cabf12 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 18:21:51,091 e9cabf12 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r184897204e59f67_01629c316f3a_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r184897204e59f67_01629c316f3a_1 
... (0s) Current status: RUNNING
 Waiting on 
bqjob_r184897204e59f67_01629c316f3a_1 ... (0s) Current status: DONE   
2018-04-06 18:21:51,092 e9cabf12 MainThread INFO

[jira] [Work logged] (BEAM-4026) Go SDK branch link in ongoing projects page is 404ing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4026?focusedWorklogId=88548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88548
 ]

ASF GitHub Bot logged work on BEAM-4026:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:27
Start Date: 06/Apr/18 18:27
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #413: [BEAM-4026] Remove 
link to deleted Go SDK branch
URL: https://github.com/apache/beam-site/pull/413
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/contribute/work-in-progress.md 
b/src/contribute/work-in-progress.md
index 1c8eb5f6a..b766b7be9 100644
--- a/src/contribute/work-in-progress.md
+++ b/src/contribute/work-in-progress.md
@@ -29,5 +29,5 @@ Current branches include:
 | [JStorm Runner]({{ site.baseurl }}/documentation/runners/jstorm) | 
[jstorm-runner](https://github.com/apache/beam/tree/jstorm-runner) | 
[runner-jstorm](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-jstorm)
 | [BEAM-1899](https://issues.apache.org/jira/browse/BEAM-1899) |
 | MapReduce Runner | 
[mr-runner](https://github.com/apache/beam/tree/mr-runner) | 
[runner-mapreduce](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-mapreduce)
 | [BEAM-165](https://issues.apache.org/jira/browse/BEAM-165) |
 | Tez Runner | [tez-runner](https://github.com/apache/beam/tree/tez-runner) | 
[runner-tez](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-tez)
 | [BEAM-2709](https://issues.apache.org/jira/browse/BEAM-2709) |
-| Go SDK | [go-sdk](https://github.com/apache/beam/tree/go-sdk) | 
[sdk-go](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20sdk-go)
 | [BEAM-2083](https://issues.apache.org/jira/browse/BEAM-2083) |
+| Go SDK | master | 
[sdk-go](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20sdk-go)
 | [BEAM-2083](https://issues.apache.org/jira/browse/BEAM-2083) |
 {:.table}


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88548)
Time Spent: 40m  (was: 0.5h)

> Go SDK branch link in ongoing projects page is 404ing
> -
>
> Key: BEAM-4026
> URL: https://issues.apache.org/jira/browse/BEAM-4026
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88550
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:27
Start Date: 06/Apr/18 18:27
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5043: [BEAM-4028] Adding 
NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-379337759
 
 
   Run Python PostCommits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88550)
Time Spent: 20m  (was: 10m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Steps can have different steps depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=88549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88549
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:27
Start Date: 06/Apr/18 18:27
Worklog Time Spent: 10m 
  Work Description: pabloem opened a new pull request #5043: [BEAM-4028] 
Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043
 
 
   Steps can have different names depending on the runner (stage, step, user, 
system name...). Depending on the needs of different components (operations, 
logging, metrics, statesampling) these step names are passed around without a 
specific order.
   
   This change starts by creating name context classes and passing them into 
operations. Next steps are:
   
   - Migrating operations away from using their operation names, and removing 
them from the object definition in `operations.pxd`, and clean all the 
operation creating logic.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88549)
Time Spent: 10m
Remaining Estimate: 0h

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Steps can have different steps depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #3

[Apache Jenkins Server]

See 


--
[...truncated 1.71 MB...]
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.(SparkContext.scala:457)
at 
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at 

[beam-site] branch asf-site updated (934f2bd -> cc24caa)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 934f2bd  Prepare repository for deployment.
 add ea4881a  [BEAM-4026] Remove link to deleted Go SDK branch
 add a49c7b1  This closes #413
 new cc24caa  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/work-in-progress/index.html | 2 +-
 src/contribute/work-in-progress.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88547
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:25
Start Date: 06/Apr/18 18:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4311: [BEAM-3355] 
Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#issuecomment-379337077
 
 
   Thanks.
   
   I meant a simple example pipeline I can run on both Dataflow and Flink and 
that illustrates how the hook mechanisms should be used. I'm fine if that being 
added separately and not have it block this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88547)
Time Spent: 9h 40m  (was: 9.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=88546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88546
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:23
Start Date: 06/Apr/18 18:23
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5041: 
[BEAM-3250] Add JUnit test reporting to Jenkins.
URL: https://github.com/apache/beam/pull/5041#discussion_r179838688
 
 

 ##
 File path: 
.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
 ##
 @@ -37,7 +37,12 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') 
{
 '--rerun-tasks',
   ]
 
-  // Sets that this is a PostCommit job.
+  // Publish all test results to Jenkins
+  publishers {
+archiveJunit('**/build/test-results/**/*.xml')
+  }
+
+   // Sets that this is a PostCommit job.
 
 Review comment:
   I _believe_ there is an extra space before `//` and in other files as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88546)
Time Spent: 3h 40m  (was: 3.5h)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[Josh Burkart (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428710#comment-16428710
 ] 

Josh Burkart commented on BEAM-3956:


Also, this used to work with the DirectRunner in 2.3.0, but no longer works in 
2.4.0.

2.3.0 example output:
{noformat}
WARNING:root:A task failed with exception: 'unicode' object does not support 
item assignment [while running 'ModifyTimestamps']
Traceback (most recent call last):

<...normal Python traceback...>

TypeError: 'unicode' object does not support item assignment [while running 
'ModifyTimestamps']
{noformat}
2.4.0 example output:
{noformat}
TypeError: 'unicode' object does not support item assignment [while running 
'ModifyTimestamps']
{noformat}

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #20

[Apache Jenkins Server]

See 


--
[...truncated 109.70 MB...]

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/06/2018 18:19:07 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map/ParMultiDo(Anonymous)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (d9d35c63a6522eaed8048e0ce7f2ccd7) switched from RUNNING 
to FINISHED.
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (d9d35c63a6522eaed8048e0ce7f2ccd7).
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (d9d35c63a6522eaed8048e0ce7f2ccd7) [FINISHED]
Apr 06, 2018 6:19:07 PM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (d9d35c63a6522eaed8048e0ce7f2ccd7)
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (a9d5d2fca5fc7fe37acb8fc3f7d02683) switched from 
RUNNING to FINISHED.
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (a9d5d2fca5fc7fe37acb8fc3f7d02683).
Apr 06, 2018 6:19:07 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (a9d5d2fca5fc7fe37acb8fc3f7d02683) [FINISHED]
Apr 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88545
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:16
Start Date: 06/Apr/18 18:16
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179836738
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
 
 Review comment:
   I'm just saying that having 2 close layers both use the name Hook is 
somewhat confusing terminology. But I'm fine leaving that for now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88545)
Time Spent: 9.5h  (was: 9h 20m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=88544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88544
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:15
Start Date: 06/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #5041: [BEAM-3250] 
Add JUnit test reporting to Jenkins.
URL: https://github.com/apache/beam/pull/5041
 
 
   This helps debug post commit failures because we get to see the test failure 
appear part of the Jenkins UI.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88544)
Time Spent: 3.5h  (was: 3h 20m)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/02: [BEAM-4026] Remove link to deleted Go SDK branch

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ea4881a577daf9b406ab6c4be231f7cf2b168b96
Author: melissa 
AuthorDate: Fri Apr 6 10:09:26 2018 -0700

[BEAM-4026] Remove link to deleted Go SDK branch
---
 src/contribute/work-in-progress.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/contribute/work-in-progress.md 
b/src/contribute/work-in-progress.md
index 1c8eb5f..b766b7b 100644
--- a/src/contribute/work-in-progress.md
+++ b/src/contribute/work-in-progress.md
@@ -29,5 +29,5 @@ Current branches include:
 | [JStorm Runner]({{ site.baseurl }}/documentation/runners/jstorm) | 
[jstorm-runner](https://github.com/apache/beam/tree/jstorm-runner) | 
[runner-jstorm](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-jstorm)
 | [BEAM-1899](https://issues.apache.org/jira/browse/BEAM-1899) |
 | MapReduce Runner | 
[mr-runner](https://github.com/apache/beam/tree/mr-runner) | 
[runner-mapreduce](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-mapreduce)
 | [BEAM-165](https://issues.apache.org/jira/browse/BEAM-165) |
 | Tez Runner | [tez-runner](https://github.com/apache/beam/tree/tez-runner) | 
[runner-tez](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-tez)
 | [BEAM-2709](https://issues.apache.org/jira/browse/BEAM-2709) |
-| Go SDK | [go-sdk](https://github.com/apache/beam/tree/go-sdk) | 
[sdk-go](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20sdk-go)
 | [BEAM-2083](https://issues.apache.org/jira/browse/BEAM-2083) |
+| Go SDK | master | 
[sdk-go](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20sdk-go)
 | [BEAM-2083](https://issues.apache.org/jira/browse/BEAM-2083) |
 {:.table}

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #413

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit a49c7b151bb53b043a9f38e13e48f7aabaea613c
Merge: 934f2bd ea4881a
Author: Mergebot 
AuthorDate: Fri Apr 6 11:15:08 2018 -0700

This closes #413

 src/contribute/work-in-progress.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (8a1d7b6 -> a49c7b1)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 8a1d7b6  This closes #407
 add 934f2bd  Prepare repository for deployment.
 new ea4881a  [BEAM-4026] Remove link to deleted Go SDK branch
 new a49c7b1  This closes #413

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/css/site.css   |  39 +++-
 content/documentation/execution-model/index.html   |  16 +-
 content/documentation/index.html   |  16 +-
 content/documentation/io/authoring-java/index.html |  16 +-
 .../documentation/io/authoring-overview/index.html |  16 +-
 .../documentation/io/authoring-python/index.html   |  16 +-
 .../documentation/io/built-in/hadoop/index.html|  16 +-
 content/documentation/io/built-in/index.html   |  16 +-
 content/documentation/io/contributing/index.html   |  16 +-
 content/documentation/io/io-toc/index.html |  16 +-
 content/documentation/io/testing/index.html|  16 +-
 .../pipelines/create-your-pipeline/index.html  |  16 +-
 .../pipelines/design-your-pipeline/index.html  |  16 +-
 .../pipelines/test-your-pipeline/index.html|  16 +-
 content/documentation/programming-guide/index.html |  16 +-
 content/documentation/resources/index.html |  16 +-
 content/documentation/runners/jstorm/index.html|  16 +-
 content/js/section-nav.js  | 234 +
 src/contribute/work-in-progress.md |   2 +-
 19 files changed, 310 insertions(+), 221 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-4026) Go SDK branch link in ongoing projects page is 404ing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4026?focusedWorklogId=88543=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88543
 ]

ASF GitHub Bot logged work on BEAM-4026:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:14
Start Date: 06/Apr/18 18:14
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #413: [BEAM-4026] Remove link 
to deleted Go SDK branch
URL: https://github.com/apache/beam-site/pull/413#issuecomment-379334083
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88543)
Time Spent: 0.5h  (was: 20m)

> Go SDK branch link in ongoing projects page is 404ing
> -
>
> Key: BEAM-4026
> URL: https://issues.apache.org/jira/browse/BEAM-4026
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #16

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 42.58 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding 

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88541
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:08
Start Date: 06/Apr/18 18:08
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379332465
 
 
   Dead link error is
   
   ```
   - ./.testcontent/contribute/work-in-progress/index.html
 *  External link https://github.com/apache/beam/tree/go-sdk failed: 404 No 
error
   ```
   
   @melap does it need to be fixed in this PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88541)
Time Spent: 40m  (was: 0.5h)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #23

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 42.61 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #109

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 31.67 KB...]
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 3.19
[INFO] os.detected.version.major: 3
[INFO] os.detected.version.minor: 19
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 14.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-x86_64
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: IO :: Hadoop Input Format 
2.5.0-SNAPSHOT
[INFO] 
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-common/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 2.9 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-common/2.5.0-SNAPSHOT/maven-metadata.xml
 (2.9 kB at 2.8 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-common/2.5.0-SNAPSHOT/beam-sdks-java-io-hadoop-common-2.5.0-20180406.073309-38.pom
Progress (1): 3.1 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-common/2.5.0-SNAPSHOT/beam-sdks-java-io-hadoop-common-2.5.0-20180406.073309-38.pom
 (3.1 kB at 20 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-parent/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 618 B   Downloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-parent/2.5.0-SNAPSHOT/maven-metadata.xml
 (618 B at 2.8 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-parent/2.5.0-SNAPSHOT/beam-sdks-java-io-parent-2.5.0-20180406.071108-35.pom
Progress (1): 4.1/5.0 kBProgress (1): 5.0 kBDownloaded 
from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-parent/2.5.0-SNAPSHOT/beam-sdks-java-io-parent-2.5.0-20180406.071108-35.pom
 (5.0 kB at 31 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-parent/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 615 B   Downloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-parent/2.5.0-SNAPSHOT/maven-metadata.xml
 (615 B at 3.6 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-parent/2.5.0-SNAPSHOT/beam-sdks-java-parent-2.5.0-20180406.070405-35.pom
Progress (1): 2.0 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-parent/2.5.0-SNAPSHOT/beam-sdks-java-parent-2.5.0-20180406.070405-35.pom
 (2.0 kB at 13 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-parent/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 610 B   Downloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-parent/2.5.0-SNAPSHOT/maven-metadata.xml
 (610 B at 3.3 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-parent/2.5.0-SNAPSHOT/beam-sdks-parent-2.5.0-20180406.070330-35.pom
Progress (1): 2.0 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-parent/2.5.0-SNAPSHOT/beam-sdks-parent-2.5.0-20180406.070330-35.pom
 (2.0 kB at 13 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 819 B   Downloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/maven-metadata.xml
 (819 B at 5.1 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/beam-parent-2.5.0-20180406.070135-35.pom
Progress (1): 4.1/81 kBProgress (1): 5.3/81 kBProgress (1): 9.4/81 kBProgress 
(1): 12/81 kB Progress (1): 17/81 kBProgress (1): 21/81 kBProgress (1): 25/81 
kBProgress (1): 29/81 kBProgress (1): 33/81 kBProgress (1): 37/81 kBProgress 
(1): 41/81 kBProgress (1): 45/81 kBProgress (1): 49/81 kBProgress (1): 53/81 
kBProgress (1): 57/81 kBProgress (1): 61/81 kBProgress (1): 65/81 kBProgress 
(1): 69/81 kBProgress (1): 73/81 kBProgress (1): 77/81 kBProgress (1): 81 kB
  Downloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/beam-parent-2.5.0-20180406.070135-35.pom
 (81 kB at 348 

Build failed in Jenkins: beam_PerformanceTests_Python #1114

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 59.62 KB...]
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.031s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.101s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  4.059 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  2.854 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.143 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [ 10.561 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.323 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.597 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.208 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 31.344 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED
[INFO] Apache Beam :: Runners :: Local Java Core .. SKIPPED
[INFO] Apache Beam :: Runners :: Direct Java .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SKIPPED
[INFO] 

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88540
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:06
Start Date: 06/Apr/18 18:06
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #412: 
[BEAM-3985] Document Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#discussion_r179834306
 
 

 ##
 File path: src/contribute/contribution-guide.md
 ##
 @@ -264,18 +276,9 @@ You can use following command to run a single test method.
 
 $ python setup.py test -s ..
 
-To Check for lint errors locally, install "tox" package and run following
-command.
-
-$ pip install tox
-$ tox -e lint_py2,lint_py3
-
-
-Beam supports running Python SDK tests using Maven. For this, navigate to root
-directory of your Apache Beam clone and execute following command. Currently
-this cannot be run from a virtual environment.
+To Check for lint errors locally, run the following command.
 
-$ mvn clean verify -pl sdks/python
+$ ../../gradlew lint
 
 Review comment:
   Make it clear that ../.. is here because we assume it is being run from the 
sdks/python directory.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88540)
Time Spent: 0.5h  (was: 20m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #341

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 27.60 KB...]
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 

Build failed in Jenkins: beam_PerformanceTests_JDBC #420

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 35.25 KB...]
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #112

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 29.74 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #17

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 43.48 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #359

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 28.65 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #17

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 43.92 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #18

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 37.98 KB...]
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #345

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 29.09 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88538
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:03
Start Date: 06/Apr/18 18:03
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179833473
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
+
+// CaptureHookFactory produces a CaptureHook from the supplied
+// options.
+type CaptureHookFactory func([]string) CaptureHook
+
+var captureHookRegistry = make(map[string]CaptureHookFactory)
+var enabledCaptureHook string
+
+func init() {
+   hf := func(opts []string) hooks.Hook {
+   return hooks.Hook{
+   Init: func(_ context.Context) error {
+   if len(opts) > 0 {
+   name, opts := hooks.Decode(opts[0])
+   capture = 
captureHookRegistry[name](opts)
+   }
+   return nil
+   },
+   }
+   }
+
+   hooks.RegisterHook("session", hf)
+}
+
+// RegisterCaptureHook registers a CaptureHookFactory for the
+// supplied identifier.
+func RegisterCaptureHook(name string, c CaptureHookFactory) {
+   if _, exists := captureHookRegistry[name]; exists {
+   panic(fmt.Sprintf("RegisterSessionCaptureHook: %s registered 
twice", name))
+   }
+   captureHookRegistry[name] = c
+}
+
+// EnableCaptureHook is called to request the use of a hook in a pipeline.
+// It updates the supplied pipelines to capture this request.
+func EnableCaptureHook(name string, opts []string) {
+   if _, exists := captureHookRegistry[name]; !exists {
+   panic(fmt.Sprintf("EnableHook: %s not registered", name))
+   }
+   if enabledCaptureHook != "" {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88538)
Time Spent: 9h 10m  (was: 9h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88539
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:03
Start Date: 06/Apr/18 18:03
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #4311: [BEAM-3355] Diagnostic 
interfaces
URL: https://github.com/apache/beam/pull/4311#issuecomment-379331297
 
 
   PTAL all comments addressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88539)
Time Spent: 9h 20m  (was: 9h 10m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #343

[Apache Jenkins Server]

See 


Changes:

[coheigea] Put String literals first when comparing using String.equals

[wcn] BEAM- Performance

--
[...truncated 29.40 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 

[jira] [Created] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[Pablo Estrada (JIRA)]

Pablo Estrada created BEAM-4028:
---

 Summary: Step / Operation naming should rely on a NameContext class
 Key: BEAM-4028
 URL: https://issues.apache.org/jira/browse/BEAM-4028
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Pablo Estrada


Steps can have different steps depending on the runner (stage, step, user, 
system name...). 

Depending on the needs of different components (operations, logging, metrics, 
statesampling) these step names are passed around without a specific order.

Instead, SDK should rely on `NameContext` objects that carry all the naming 
information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88537
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 18:01
Start Date: 06/Apr/18 18:01
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179833157
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
+
+// CaptureHookFactory produces a CaptureHook from the supplied
+// options.
+type CaptureHookFactory func([]string) CaptureHook
+
+var captureHookRegistry = make(map[string]CaptureHookFactory)
+var enabledCaptureHook string
+
+func init() {
+   hf := func(opts []string) hooks.Hook {
+   return hooks.Hook{
+   Init: func(_ context.Context) error {
+   if len(opts) > 0 {
+   name, opts := hooks.Decode(opts[0])
+   capture = 
captureHookRegistry[name](opts)
+   }
+   return nil
+   },
+   }
+   }
+
+   hooks.RegisterHook("session", hf)
+}
+
+// RegisterCaptureHook registers a CaptureHookFactory for the
+// supplied identifier.
+func RegisterCaptureHook(name string, c CaptureHookFactory) {
+   if _, exists := captureHookRegistry[name]; exists {
+   panic(fmt.Sprintf("RegisterSessionCaptureHook: %s registered 
twice", name))
+   }
+   captureHookRegistry[name] = c
+}
+
+// EnableCaptureHook is called to request the use of a hook in a pipeline.
+// It updates the supplied pipelines to capture this request.
+func EnableCaptureHook(name string, opts []string) {
+   if _, exists := captureHookRegistry[name]; !exists {
+   panic(fmt.Sprintf("EnableHook: %s not registered", name))
+   }
+   if enabledCaptureHook != "" {
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88537)
Time Spent: 9h  (was: 8h 50m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3250.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4015) GCS artifact proxy is breaking the build when credentials are not available

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-4015.
-
   Resolution: Workaround
Fix Version/s: 2.5.0

Tentatively resolving, but please feel free to reopen if the suggestions don't 
work for you.

> GCS artifact proxy is breaking the build when credentials are not available
> ---
>
> Key: BEAM-4015
> URL: https://issues.apache.org/jira/browse/BEAM-4015
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ismaël Mejía
>Assignee: Henning Rohde
>Priority: Blocker
> Fix For: 2.5.0
>
>
> When running the maven build in a machine without the valid auth credentials 
> the module breaks like this:
> {code:bash}
> {{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
> beam-runners-gcp-gcsproxy ---}}
> {{[INFO] Prepared command line : bin/go build -buildmode=default -o 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
> github.com/apache/beam/cmd/gcsproxy}}
> {{[ERROR] }}
> {{[ERROR] -Exec.Err-}}
> {{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
> {{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
> undefined: option.WithoutAuthentication}}
> {{[ERROR] }}
> {{}}
> {{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact 
> proxy FAILURE [  1.038 s]}}
> {{}}
> {{[INFO] BUILD FAILURE}}
> {{}}
> {{[ERROR] Failed to execute goal 
> com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
> beam-runners-gcp-gcsproxy: Can't find generated target file : 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
> {{[ERROR] }}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri resolved BEAM-4021.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 180, in _multicall}}
> {{ res = hook_impl.function(*args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 452, in tox_testenv_install_deps}}
> {{ venv._install(deps, action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 331, in _install}}
> {{ action=action)}}
> {{ File 
> 

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=88535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88535
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:56
Start Date: 06/Apr/18 17:56
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5024: [BEAM-4011] Unify Python 
IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#issuecomment-379329439
 
 
   This code is working. I've opened BEAM-4027 for the precommit failure with 
cython.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88535)
Time Spent: 40m  (was: 0.5h)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=88536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88536
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:56
Start Date: 06/Apr/18 17:56
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5024: [BEAM-4011] Unify Python 
IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#issuecomment-379329516
 
 
   R: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88536)
Time Spent: 50m  (was: 40m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4027) python precommit fails in py27-cython

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-4027:

Attachment: precommit-cython-log

> python precommit fails in py27-cython
> -
>
> Key: BEAM-4027
> URL: https://issues.apache.org/jira/browse/BEAM-4027
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Ahmet Altay
>Priority: Major
> Attachments: precommit-cython-log
>
>
> Steps:
> - pip install cython==0.26.1
> - ../../gradlew clean
> - nosetests apache_beam.examples.wordcount_debugging_test:WordCountTest 
> apache_beam.examples.cookbook.multiple_output_pardo_test:MultipleOutputParDo
> Result: 2 errors (see attached logs)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4027) python precommit fails in py27-cython

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-4027:
---

 Summary: python precommit fails in py27-cython
 Key: BEAM-4027
 URL: https://issues.apache.org/jira/browse/BEAM-4027
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Ahmet Altay


Steps:

- pip install cython==0.26.1

- ../../gradlew clean

- nosetests apache_beam.examples.wordcount_debugging_test:WordCountTest 
apache_beam.examples.cookbook.multiple_output_pardo_test:MultipleOutputParDo

Result: 2 errors (see attached logs)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88533=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88533
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:44
Start Date: 06/Apr/18 17:44
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179828479
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, req, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88530
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:40
Start Date: 06/Apr/18 17:40
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #4311: [BEAM-3355] Diagnostic 
interfaces
URL: https://github.com/apache/beam/pull/4311#issuecomment-379325307
 
 
   There's an example of hooking profiling in the Dataflow runner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88530)
Time Spent: 8h 40m  (was: 8.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88529=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88529
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:39
Start Date: 06/Apr/18 17:39
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179827403
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
 
 Review comment:
   It's a hook that happens to be a writer. It's the hook into the capture 
facility, and all the APIs are talking about 'hooks' not 'writers' so this 
suggestion feels incorrectly motivated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88529)
Time Spent: 8.5h  (was: 8h 20m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88528=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88528
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:37
Start Date: 06/Apr/18 17:37
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179826856
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88528)
Time Spent: 8h 20m  (was: 8h 10m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88527
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:36
Start Date: 06/Apr/18 17:36
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179826558
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
 
 Review comment:
   OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88527)
Time Spent: 8h 10m  (was: 8h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88526
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:36
Start Date: 06/Apr/18 17:36
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179826460
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, req, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88525
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:35
Start Date: 06/Apr/18 17:35
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179826372
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, req, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88524
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:34
Start Date: 06/Apr/18 17:34
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179826078
 
 

 ##
 File path: sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
 ##
 @@ -924,7 +924,7 @@ type PTransform struct {
//
Outputs map[string]string `protobuf:"bytes,4,rep,name=outputs" 
json:"outputs,omitempty" protobuf_key:"bytes,1,opt,name=key" 
protobuf_val:"bytes,2,opt,name=value"`
// (Optional) Static display data for this PTransform application. If
-   // there is none, or it is not relevant (such as use by the Fn API)
 
 Review comment:
   Reverted.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88524)
Time Spent: 7h 40m  (was: 7.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88520=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88520
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:28
Start Date: 06/Apr/18 17:28
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179824381
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/harness.go
 ##
 @@ -37,12 +35,13 @@ import (
 // TODO(herohde) 2/8/2017: for now, assume we stage a full binary (not a 
plugin).
 
 // Main is the main entrypoint for the Go harness. It runs at "runtime" -- not
-// "pipeline-construction time" -- on each worker. It is a Fn API client and
+// "pipeline-construction time" -- on each worker. It is a FnAPI client and
 // ultimately responsible for correctly executing user code.
 func Main(ctx context.Context, loggingEndpoint, controlEndpoint string) error {
-   setupRemoteLogging(ctx, loggingEndpoint)
-   setupDiagnosticRecording()
+   hooks.DeserializeHooks()
 
+   hooks.RunInitHooks(ctx)
+   setupRemoteLogging(ctx, loggingEndpoint)
recordHeader()
 
 Review comment:
   No, that needs more work. BEAM-4017


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88520)
Time Spent: 7.5h  (was: 7h 20m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88518
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:27
Start Date: 06/Apr/18 17:27
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179824037
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -90,15 +94,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
}
 
if *cpuProfiling != "" {
-   beam.PipelineOptions.Set("cpu_profiling", "true")
-   beam.PipelineOptions.Set("storage_path", 
"/var/opt/google/traces")
+   perf.EnableProfCaptureHook("gcs_profile_writer", 
[]string{*cpuProfiling})
}
 
if *sessionRecording != "" {
-   beam.PipelineOptions.Set("session_recording", "true")
-   beam.PipelineOptions.Set("storage_path", 
"/var/opt/google/traces")
+   // TODO(wcn): implement this.
 
 Review comment:
   I actually did, I didn't fix the comment. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88518)
Time Spent: 7h 20m  (was: 7h 10m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88508=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88508
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179815311
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
 
 Review comment:
   nit: rename it CaptureWriter? I was confused when reading the code by it 
also called a hook.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88508)
Time Spent: 6h 10m  (was: 6h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88507
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179815875
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/session.go
 ##
 @@ -213,3 +186,59 @@ func recordFooter() error {
},
})
 }
+
+// CaptureHook writes the messaging content consumed and
+// produced by the worker, allowing the data to be used as
+// an input for the session runner. Since workers can exist
+// in a variety of environments, this allows the runner
+// to tailor the behavior best for its particular needs.
+type CaptureHook io.WriteCloser
+
+// CaptureHookFactory produces a CaptureHook from the supplied
+// options.
+type CaptureHookFactory func([]string) CaptureHook
+
+var captureHookRegistry = make(map[string]CaptureHookFactory)
+var enabledCaptureHook string
+
+func init() {
+   hf := func(opts []string) hooks.Hook {
+   return hooks.Hook{
+   Init: func(_ context.Context) error {
+   if len(opts) > 0 {
+   name, opts := hooks.Decode(opts[0])
+   capture = 
captureHookRegistry[name](opts)
+   }
+   return nil
+   },
+   }
+   }
+
+   hooks.RegisterHook("session", hf)
+}
+
+// RegisterCaptureHook registers a CaptureHookFactory for the
+// supplied identifier.
+func RegisterCaptureHook(name string, c CaptureHookFactory) {
+   if _, exists := captureHookRegistry[name]; exists {
+   panic(fmt.Sprintf("RegisterSessionCaptureHook: %s registered 
twice", name))
+   }
+   captureHookRegistry[name] = c
+}
+
+// EnableCaptureHook is called to request the use of a hook in a pipeline.
+// It updates the supplied pipelines to capture this request.
+func EnableCaptureHook(name string, opts []string) {
+   if _, exists := captureHookRegistry[name]; !exists {
+   panic(fmt.Sprintf("EnableHook: %s not registered", name))
+   }
+   if enabledCaptureHook != "" {
 
 Review comment:
   Instead of keeping track if it manually, perhaps add an IsEnabled function 
in hooks to allow this check? Other hooks will have the same requirement of 
only allowing one instance.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88507)
Time Spent: 6h  (was: 5h 50m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88511
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179819467
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88512
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179823479
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88514
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179819164
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
 
 Review comment:
   I don't think the TODO is needed. We should do what we have here. Hooks are 
already "unsafe" IMO and we may want to allow modifications/filtering at some 
later point.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88514)
Time Spent: 6h 40m  (was: 6.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88510
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179817281
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
 
 Review comment:
   Perhaps mention explicitly that this is called before we set up logging or 
connect to the server.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88510)
Time Spent: 6.5h  (was: 6h 20m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88509
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179816955
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/harness/harness.go
 ##
 @@ -37,12 +35,13 @@ import (
 // TODO(herohde) 2/8/2017: for now, assume we stage a full binary (not a 
plugin).
 
 // Main is the main entrypoint for the Go harness. It runs at "runtime" -- not
-// "pipeline-construction time" -- on each worker. It is a Fn API client and
+// "pipeline-construction time" -- on each worker. It is a FnAPI client and
 // ultimately responsible for correctly executing user code.
 func Main(ctx context.Context, loggingEndpoint, controlEndpoint string) error {
-   setupRemoteLogging(ctx, loggingEndpoint)
-   setupDiagnosticRecording()
+   hooks.DeserializeHooks()
 
+   hooks.RunInitHooks(ctx)
+   setupRemoteLogging(ctx, loggingEndpoint)
recordHeader()
 
 Review comment:
   Question: is session not being converted to a hook at this time?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88509)
Time Spent: 6h 20m  (was: 6h 10m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88517
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179819835
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,182 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package hooks allows runners to tailor execution of the worker harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a FnAPI instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a FnAPI instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {
+   if err := h.Resp(ctx, 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88513
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179820369
 
 

 ##
 File path: sdks/go/pkg/beam/model/pipeline_v1/beam_runner_api.pb.go
 ##
 @@ -924,7 +924,7 @@ type PTransform struct {
//
Outputs map[string]string `protobuf:"bytes,4,rep,name=outputs" 
json:"outputs,omitempty" protobuf_key:"bytes,1,opt,name=key" 
protobuf_val:"bytes,2,opt,name=value"`
// (Optional) Static display data for this PTransform application. If
-   // there is none, or it is not relevant (such as use by the Fn API)
 
 Review comment:
   Is this from a new go generate or a search'n'replace? If the latter, we 
should make the change to the proto file or revert.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88513)
Time Spent: 6h 40m  (was: 6.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88515
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179821103
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -90,15 +94,20 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
}
 
if *cpuProfiling != "" {
-   beam.PipelineOptions.Set("cpu_profiling", "true")
-   beam.PipelineOptions.Set("storage_path", 
"/var/opt/google/traces")
+   perf.EnableProfCaptureHook("gcs_profile_writer", 
[]string{*cpuProfiling})
}
 
if *sessionRecording != "" {
-   beam.PipelineOptions.Set("session_recording", "true")
-   beam.PipelineOptions.Set("storage_path", 
"/var/opt/google/traces")
+   // TODO(wcn): implement this.
 
 Review comment:
   Please open a JIRA.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88515)
Time Spent: 6h 50m  (was: 6h 40m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88516
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:25
Start Date: 06/Apr/18 17:25
Worklog Time Spent: 10m 
  Work Description: herohde commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179820829
 
 

 ##
 File path: sdks/go/pkg/beam/runners/dataflow/dataflow.go
 ##
 @@ -147,9 +156,9 @@ func Execute(ctx context.Context, p *beam.Pipeline) error {
SdkPipelineOptions: newMsg(pipelineOptions{
DisplayData: findPipelineFlags(),
Options: dataflowOptions{
-   Options: options.Options,
PipelineURL: modelURL,
},
+   GoOptions: options,
 
 Review comment:
   Clever! Thanks for figuring out how to make this work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88516)
Time Spent: 7h  (was: 6h 50m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4026) Go SDK branch link in ongoing projects page is 404ing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4026?focusedWorklogId=88497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88497
 ]

ASF GitHub Bot logged work on BEAM-4026:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:12
Start Date: 06/Apr/18 17:12
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #413: [BEAM-4026] Remove link 
to deleted Go SDK branch
URL: https://github.com/apache/beam-site/pull/413#issuecomment-379317610
 
 
   R: @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88497)
Time Spent: 20m  (was: 10m)

> Go SDK branch link in ongoing projects page is 404ing
> -
>
> Key: BEAM-4026
> URL: https://issues.apache.org/jira/browse/BEAM-4026
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4026) Go SDK branch link in ongoing projects page is 404ing

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4026?focusedWorklogId=88496=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88496
 ]

ASF GitHub Bot logged work on BEAM-4026:


Author: ASF GitHub Bot
Created on: 06/Apr/18 17:12
Start Date: 06/Apr/18 17:12
Worklog Time Spent: 10m 
  Work Description: melap opened a new pull request #413: [BEAM-4026] 
Remove link to deleted Go SDK branch
URL: https://github.com/apache/beam-site/pull/413
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88496)
Time Spent: 10m
Remaining Estimate: 0h

> Go SDK branch link in ongoing projects page is 404ing
> -
>
> Key: BEAM-4026
> URL: https://issues.apache.org/jira/browse/BEAM-4026
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4026) Go SDK branch link in ongoing projects page is 404ing

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-4026:
--

 Summary: Go SDK branch link in ongoing projects page is 404ing
 Key: BEAM-4026
 URL: https://issues.apache.org/jira/browse/BEAM-4026
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Melissa Pashniak
Assignee: Melissa Pashniak






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #2

[Apache Jenkins Server]

See 


Changes:

[wcn] BEAM- Performance

--
[...truncated 1.70 MB...]
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.(SparkContext.scala:457)
at 
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 

[jira] [Closed] (BEAM-3782) Do not ignore FileSystem errors and document expected behavior

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri closed BEAM-3782.
---
   Resolution: Won't Do
Fix Version/s: Not applicable

> Do not ignore FileSystem errors and document expected behavior
> --
>
> Key: BEAM-3782
> URL: https://issues.apache.org/jira/browse/BEAM-3782
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>
> Cloned from BEAM-3600:
> copy/rename:
>  * it should be an error if the dst file exists
>  * it should be an error if the src file doesn't exist
> delete:
>  * it should be an error if the file/dir doesn't exist
> FileBasedSink.finalize_write:
>  * should check (src, dst) pairs for existence:
>  ** src only - regular rename
>  ** dst only - skip rename
>  ** both src and dst - if files don't match on metadata (checksum or size), 
> delete dst then rename, otherwise delete src and skip rename
>  ** neither exist - return error: bad state
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3782) Do not ignore FileSystem errors and document expected behavior

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428544#comment-16428544
 ] 

Udi Meiri commented on BEAM-3782:
-

Not sure why I cloned the original. Closing

> Do not ignore FileSystem errors and document expected behavior
> --
>
> Key: BEAM-3782
> URL: https://issues.apache.org/jira/browse/BEAM-3782
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> Cloned from BEAM-3600:
> copy/rename:
>  * it should be an error if the dst file exists
>  * it should be an error if the src file doesn't exist
> delete:
>  * it should be an error if the file/dir doesn't exist
> FileBasedSink.finalize_write:
>  * should check (src, dst) pairs for existence:
>  ** src only - regular rename
>  ** dst only - skip rename
>  ** both src and dst - if files don't match on metadata (checksum or size), 
> delete dst then rename, otherwise delete src and skip rename
>  ** neither exist - return error: bad state
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2517) Document how to build Python SDK from BEAM head in contribution guide.

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri resolved BEAM-2517.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Document how to build Python SDK from BEAM head in contribution guide.
> --
>
> Key: BEAM-2517
> URL: https://issues.apache.org/jira/browse/BEAM-2517
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We should add instructions how to build Python SDK from BEAM head to BEAM 
> contributor guide[1] .
> The commands can be as follows:
> cd ./beam/sdks/python
> python setup.py sdist
> SDK tarball will appear in ./dist/
> [1]: https://beam.apache.org/contribute/contribution-guide



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3965) HDFS read broken in python

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri resolved BEAM-3965.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> HDFS read broken in python
> --
>
> Key: BEAM-3965
> URL: https://issues.apache.org/jira/browse/BEAM-3965
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> When running a command like:
> {noformat}
> python setup.py sdist > /dev/null && python -m apache_beam.examples.wordcount 
> --output gs://.../py-wordcount-output \
>   --hdfs_host ... --hdfs_port 50070 --hdfs_user ehudm --runner DataflowRunner 
> --project ... \
>   --temp_location gs://.../temp-hdfs-int --staging_location 
> gs://.../staging-hdfs-int \
>   --sdk_location dist/apache-beam-2.5.0.dev0.tar.gz --input 
> hdfs://kinglear.txt
> {noformat}
> I get:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
>   File 
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/wordcount.py",
>  line 136, in 
> run()
>   File 
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/examples/wordcount.py",
>  line 90, in run
> lines = p | 'read' >> ReadFromText(known_args.input)
>   File "apache_beam/io/textio.py", line 522, in __init__
> skip_header_lines=skip_header_lines)
>   File "apache_beam/io/textio.py", line 117, in __init__
> validate=validate)
>   File "apache_beam/io/filebasedsource.py", line 119, in __init__
> self._validate()
>   File "apache_beam/options/value_provider.py", line 124, in _f
> return fnc(self, *args, **kwargs)
>   File "apache_beam/io/filebasedsource.py", line 176, in _validate
> match_result = FileSystems.match([pattern], limits=[1])[0]
>   File "apache_beam/io/filesystems.py", line 159, in match
> return filesystem.match(patterns, limits)
>   File "apache_beam/io/hadoopfilesystem.py", line 221, in match
> raise BeamIOError('Match operation failed', exceptions)
> apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions 
> {'hdfs://kinglear.txt': KeyError('name',)}
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #19

[Apache Jenkins Server]

See 


Changes:

[wcn] BEAM- Performance

--
[...truncated 67.24 MB...]

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/06/2018 16:25:34   ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/06/2018 16:25:34 ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (048c30dcf840d4ef6b2792c01fdc12d3) switched from RUNNING 
to FINISHED.
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (048c30dcf840d4ef6b2792c01fdc12d3).
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (048c30dcf840d4ef6b2792c01fdc12d3) [FINISHED]
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: PAssert$166/GroupGlobally/GroupDummyAndContents -> 
PAssert$166/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$166/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$166/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$166/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$166/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(1/1) (274755a74747220b0a8ea64f06935c7d) switched from RUNNING to FINISHED.
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: Combine.perKey(TestCombineFnWithContext) -> 
PAssert$167/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$167/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (ee63179bfe662b66b2ed5424fab74b03) switched from 
RUNNING to FINISHED.
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
PAssert$166/GroupGlobally/GroupDummyAndContents -> 
PAssert$166/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$166/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$166/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$166/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$166/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(1/1) (274755a74747220b0a8ea64f06935c7d).
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (329d8cae685de3c99fcbd607cd14db13) switched from 
RUNNING to FINISHED.
Apr 06, 2018 4:25:34 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for Combine.perKey(TestCombineFnWithContext) 
-> PAssert$167/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$167/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> 

[beam] branch master updated (e58c693 -> 15af0a4)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e58c693  Put String literals first when comparing using String.equals
 add b9971f5  BEAM- Performance
 new 15af0a4  Merge pull request #5038: Performance improvement for inputs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/exec/input.go | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5038: Performance improvement for inputs

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 15af0a45474b34f7e9d602f2956f0ba3eba0784c
Merge: e58c693 b9971f5
Author: Thomas Groh 
AuthorDate: Fri Apr 6 09:12:05 2018 -0700

Merge pull request #5038: Performance improvement for inputs

 sdks/go/pkg/beam/core/runtime/exec/input.go | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1

[Apache Jenkins Server]

See 


--
[...truncated 1.71 MB...]
at 
org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.SparkContext.(SparkContext.scala:457)
at 
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:103)
at 
org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:68)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.testUnboundedSourceMetrics(StreamingSourceMetricsTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 

[beam] 01/01: Put String literals first when comparing using String.equals

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e58c693f76baf829160b8fd5b7a08be08a620c36
Merge: 77207c3 e004f2e
Author: Lukasz Cwik 
AuthorDate: Fri Apr 6 11:26:23 2018 -0400

Put String literals first when comparing using String.equals

 .../src/main/java/org/apache/beam/examples/complete/TfIdf.java |  8 
 .../java/org/apache/beam/examples/cookbook/TriggerExample.java |  2 +-
 .../java/org/apache/beam/runners/apex/ApexYarnLauncher.java|  2 +-
 .../runners/apex/translation/SideInputTranslationTest.java |  2 +-
 .../beam/runners/flink/FlinkPipelineExecutionEnvironment.java  | 10 +-
 .../apache/beam/runners/flink/streaming/DoFnOperatorTest.java  |  4 ++--
 .../java/org/apache/beam/runners/dataflow/DataflowMetrics.java |  2 +-
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java  |  2 +-
 .../org/apache/beam/runners/dataflow/TestDataflowRunner.java   |  3 +--
 .../beam/runners/dataflow/DataflowPipelineTranslatorTest.java  |  2 +-
 .../beam/runners/fnexecution/control/SdkHarnessClientTest.java |  4 ++--
 .../apache/beam/runners/spark/SparkNativePipelineVisitor.java  |  2 +-
 .../core/src/main/java/org/apache/beam/sdk/util/ClassPath.java |  6 +++---
 .../core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/values/TupleTag.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/DistinctTest.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/ParDoTest.java|  4 ++--
 .../beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java   |  2 +-
 .../beam/sdk/extensions/gcp/storage/GcsPathValidator.java  |  2 +-
 .../org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java |  4 ++--
 .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java |  6 +++---
 .../apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java  |  2 +-
 .../java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java   |  4 ++--
 .../org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchema.java  |  2 +-
 .../org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java|  8 
 .../jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java |  2 +-
 .../src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java|  2 +-
 .../org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java   |  2 +-
 29 files changed, 48 insertions(+), 49 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (77207c3 -> e58c693)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 77207c3  [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle
 add e004f2e  Put String literals first when comparing using String.equals
 new e58c693  Put String literals first when comparing using String.equals

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/java/org/apache/beam/examples/complete/TfIdf.java |  8 
 .../java/org/apache/beam/examples/cookbook/TriggerExample.java |  2 +-
 .../java/org/apache/beam/runners/apex/ApexYarnLauncher.java|  2 +-
 .../runners/apex/translation/SideInputTranslationTest.java |  2 +-
 .../beam/runners/flink/FlinkPipelineExecutionEnvironment.java  | 10 +-
 .../apache/beam/runners/flink/streaming/DoFnOperatorTest.java  |  4 ++--
 .../java/org/apache/beam/runners/dataflow/DataflowMetrics.java |  2 +-
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java  |  2 +-
 .../org/apache/beam/runners/dataflow/TestDataflowRunner.java   |  3 +--
 .../beam/runners/dataflow/DataflowPipelineTranslatorTest.java  |  2 +-
 .../beam/runners/fnexecution/control/SdkHarnessClientTest.java |  4 ++--
 .../apache/beam/runners/spark/SparkNativePipelineVisitor.java  |  2 +-
 .../core/src/main/java/org/apache/beam/sdk/util/ClassPath.java |  6 +++---
 .../core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/values/TupleTag.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/DistinctTest.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/ParDoTest.java|  4 ++--
 .../beam/sdk/transforms/reflect/DoFnSignaturesTestUtils.java   |  2 +-
 .../beam/sdk/extensions/gcp/storage/GcsPathValidator.java  |  2 +-
 .../org/apache/beam/sdk/io/cassandra/CassandraServiceImpl.java |  4 ++--
 .../org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java |  6 +++---
 .../apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java  |  2 +-
 .../java/org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java   |  4 ++--
 .../org/apache/beam/sdk/io/gcp/spanner/ReadSpannerSchema.java  |  2 +-
 .../org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java|  8 
 .../jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java |  2 +-
 .../src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOTest.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java|  2 +-
 .../org/apache/beam/sdk/io/kinesis/ShardReadersPoolTest.java   |  2 +-
 29 files changed, 48 insertions(+), 49 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-3910) Support floating point values in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3910?focusedWorklogId=88448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88448
 ]

ASF GitHub Bot logged work on BEAM-3910:


Author: ASF GitHub Bot
Created on: 06/Apr/18 15:11
Start Date: 06/Apr/18 15:11
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #4941: BEAM-3910: Add float 
support for the Go SDK.
URL: https://github.com/apache/beam/pull/4941#issuecomment-379283848
 
 
   @lukecwik please merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88448)
Time Spent: 1h 20m  (was: 1h 10m)
Remaining Estimate: 22h 40m  (was: 22h 50m)

> Support floating point values in Go SDK
> ---
>
> Key: BEAM-3910
> URL: https://issues.apache.org/jira/browse/BEAM-3910
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Bill Neubauer
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 1h 20m
>  Remaining Estimate: 22h 40m
>
> The Go SDK supports all the integer types of the language, but does not 
> support floats.
> My plan for coding is to use the same technique the gob package uses, which 
> results in a compact encoding for simple values.
> [https://golang.org/src/encoding/gob/encode.go?#L210|https://golang.org/src/encoding/gob/encode.go#L210]
>  with rationale explained in 
> https://golang.org/pkg/encoding/gob/#hdr-Encoding_Details
> The resulting uint is then encoded using the existing coders in coderx.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5292

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88421
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 06/Apr/18 12:41
Start Date: 06/Apr/18 12:41
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on issue #4990: [BEAM-3981] 
[WIP] Futurize and fix python 2 compatibility for coders subpackage
URL: https://github.com/apache/beam/pull/4990#issuecomment-379232283
 
 
   I've added some changes. Most notable:
   
   - Replace bytes with memoryview in stream cython files. This also works with 
subtypes of bytes like the future.builtins bytes type.
   
   - Revert isinstance checks to typechecks for performance with the use of 
past.builtins.
   
   I've also added the applied strategy to the [Python 3 proposal 
document](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit).
 It would be great to get some feedback on this so we can start moving forward 
with the other subpackages.
   
   @aaltay @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88421)
Time Spent: 5h 40m  (was: 5.5h)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1557

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 90.54 KB...]
'apache-beam-testing:bqjob_r2b6a7ab9ca9c3430_01629ae56cee_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r2b6a7ab9ca9c3430_01629ae56cee_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r2b6a7ab9ca9c3430_01629ae56cee_1 ... (0s) Current status: DONE   
2018-04-06 12:19:11,586 0edd048f MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 12:19:40,068 0edd048f MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 12:19:42,239 0edd048f MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r102c23fa88121b40_01629ae5e530_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r102c23fa88121b40_01629ae5e530_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r102c23fa88121b40_01629ae5e530_1 ... (0s) Current status: DONE   
2018-04-06 12:19:42,240 0edd048f MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 12:20:10,684 0edd048f MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 12:20:12,976 0edd048f MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r77f4db16871466f_01629ae65cc6_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r77f4db16871466f_01629ae65cc6_1 ... (0s) 
Current status: RUNNING 
Waiting on 
bqjob_r77f4db16871466f_01629ae65cc6_1 ... (0s) Current status: DONE   
2018-04-06 12:20:12,976 0edd048f MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 12:20:38,068 0edd048f MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 12:20:40,157 0edd048f MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing 

Build failed in Jenkins: beam_PerformanceTests_Python #1113

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 1.64 KB...]
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8278656900538688900.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4284664975817978160.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2918022323359320116.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins205198673370429089.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7567963834475646218.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r 

[jira] [Work logged] (BEAM-3942) Update performance testing framework to use Gradle.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?focusedWorklogId=88410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88410
 ]

ASF GitHub Bot logged work on BEAM-3942:


Author: ASF GitHub Bot
Created on: 06/Apr/18 12:07
Start Date: 06/Apr/18 12:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5003: [BEAM-3942] Update 
performance testing framework to use Gradle
URL: https://github.com/apache/beam/pull/5003#issuecomment-379233885
 
 
   R: @udim 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88410)
Time Spent: 1h 40m  (was: 1.5h)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #15

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 77207c3deec8648cde741b6984c6fbfbbe063008 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77207c3deec8648cde741b6984c6fbfbbe063008
Commit message: "[BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle"
 > git rev-list --no-walk 5056fd915e4d90a9ca0fcf716191469d96b7077b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4442624536004254157.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5086648304600127932.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins125823683312007494.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1523008864391
namespace "filebasedioithdfs-1523008864391" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4735116346572113700.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1523008864391
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7350676351820718925.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins34177832182622075.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins444927881457005711.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7196819135552081169.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6492195773344439283.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6608199318696247050.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #22

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 63.18 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #108

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 24.40 KB...]
--beam_options_config_file=
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-04-06 12:02:15,940 3348fa82 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-04-06 12:02:15,941 3348fa82 MainThread INFO Initializing the edw 
service decoder
2018-04-06 12:02:16,397 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Provisioning resources for benchmark beam_integration_benchmark
2018-04-06 12:02:16,401 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Preparing benchmark beam_integration_benchmark
2018-04-06 12:02:16,402 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: git clone https://github.com/apache/beam.git
2018-04-06 12:02:38,493 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 create -f 

2018-04-06 12:02:40,617 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running benchmark beam_integration_benchmark
2018-04-06 12:02:40,637 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:02:51,105 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:01,505 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:11,878 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:22,345 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:32,767 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:43,065 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:03:53,329 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:04:03,637 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:04:13,893 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 
--kubeconfig=
 get svc postgres-for-dev -ojsonpath={.status.loadBalancer.ingress[0].ip}
2018-04-06 12:04:24,129 3348fa82 MainThread beam_integration_benchmark(1/1) 
INFO Running: kubectl 

Build failed in Jenkins: beam_PerformanceTests_JDBC #419

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 45.78 KB...]
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #111

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 26.53 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #340

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 27.59 KB...]
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #17

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3250] Migrate Dataflow ValidatesRunner test to Gradle

[herohde] CR: use 4 forks in Dataflow test

[herohde] [BEAM-3250] Migrate Spark ValidatesRunner tests to Gradle

--
[...truncated 35.79 KB...]
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding