Build failed in Jenkins: beam_PerformanceTests_Spark #1516

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

[aljoscha.krettek] [BEAM-622] Add checkpointing tests for DoFnOperator and

[aljoscha.krettek] [BEAM-3087] Make reader state update and element emission 
atomic

[aljoscha.krettek] [BEAM-2393] Make BoundedSource fault-tolerant

--
[...truncated 66.46 KB...]
2018-03-27 06:29:33,786 4d54b4d0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 06:30:01,566 4d54b4d0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 06:30:05,819 4d54b4d0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r5baaf5432524ea8b_0162662634b5_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r5baaf5432524ea8b_0162662634b5_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r5baaf5432524ea8b_0162662634b5_1 ... (0s) Current status: DONE   
2018-03-27 06:30:05,820 4d54b4d0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 06:30:25,634 4d54b4d0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 06:30:31,183 4d54b4d0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r35ac0573eba4bcfd_01626626931f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r35ac0573eba4bcfd_01626626931f_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r35ac0573eba4bcfd_01626626931f_1 ... (1s) Current status: RUNNING 

 Waiting on bqjob_r35ac0573eba4bcfd_01626626931f_1 ... (1s) Current status: 
DONE   
2018-03-27 06:30:31,184 4d54b4d0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 06:30:46,962 4d54b4d0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 06:30:50,839 4d54b4d0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4c8ed562ecf6d262_01626626e52d_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r4c8ed562ecf6d262_01626626e52d_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r4c8ed562ecf6d262_01626626e52d_1 ... (0s) Current status: DONE   
2018-03-27 06:30:50,839 4d54b4d0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 06:31:06,715 4d54b4d0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 06:31:10,954 4d54b4d0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r15304ac94b85c41_0162662733c1_1': Invalid schema
update. Field timestamp has changed typ

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #6300

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #71

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #317

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #299

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

[aljoscha.krettek] [BEAM-622] Add checkpointing tests for DoFnOperator and

[aljoscha.krettek] [BEAM-3087] Make reader state update and element emission 
atomic

[aljoscha.krettek] [BEAM-2393] Make BoundedSource fault-tolerant

--
[...truncated 691.50 KB...]
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:ne

Build failed in Jenkins: beam_PerformanceTests_JDBC #379

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

[aljoscha.krettek] [BEAM-622] Add checkpointing tests for DoFnOperator and

[aljoscha.krettek] [BEAM-3087] Make reader state update and element emission 
atomic

[aljoscha.krettek] [BEAM-2393] Make BoundedSource fault-tolerant

--
[...truncated 32.98 KB...]
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excludi

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #68

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1072

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

[aljoscha.krettek] [BEAM-622] Add checkpointing tests for DoFnOperator and

[aljoscha.krettek] [BEAM-3087] Make reader state update and element emission 
atomic

[aljoscha.krettek] [BEAM-2393] Make BoundedSource fault-tolerant

--
[...truncated 62.82 KB...]
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.053s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.094s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [ 12.697 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  5.457 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.101 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [ 10.798 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  4.025 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.559 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.188 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 31.067 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3781

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #301

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #303

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84727&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84727
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 05:35
Start Date: 27/Mar/18 05:35
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376402683
 
 
   +R: @aaltay 
   Major changes:
   1. change directory of python automation shell scripts, 
release/src/main/groovy -> release/src/main/python-release
   2. Functionalize each steps of quickstart and mobile-gaming validation 
in purpose of reusing code for wheel validations.
   3. Add validations for wheel files. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84727)
Time Spent: 6h 50m  (was: 6h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3780

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84723
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 05:19
Start Date: 27/Mar/18 05:19
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177310475
 
 

 ##
 File path: release/src/main/groovy/mobilegaming-java-dataflow.groovy
 ##
 @@ -0,0 +1,106 @@
+#!groovy
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import MobileGamingJavaUtils
+
+t = new TestScripts(args)
+
+/*
+ * Run the mobile game examples on Dataflow.
+ * https://beam.apache.org/get-started/mobile-gaming-example/
+ */
+
+t.describe 'Run Apache Beam Java SDK Mobile Gaming Examples - Dataflow'
+
+QuickstartArchetype.generate(t)
+
+t.intent 'Running the Mobile-Gaming Code with DataflowRunner'
+
+def runner = "DataflowRunner"
+
+/**
+ *  Run the UserScore example on DataflowRunner
+ * */
+
+t.intent("Running: UserScore example on DataflowRunner")
 
 Review comment:
   Using intents to document intentions of each example run. Using "describe()" 
at very begging of the code to document the purpose of the script.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84723)
Time Spent: 78h 40m  (was: 78.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 78h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84720&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84720
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 05:14
Start Date: 27/Mar/18 05:14
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177310007
 
 

 ##
 File path: release/src/main/groovy/mobilegaming-java-dataflow.groovy
 ##
 @@ -0,0 +1,106 @@
+#!groovy
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import MobileGamingJavaUtils
+
+t = new TestScripts(args)
+
+/*
+ * Run the mobile game examples on Dataflow.
+ * https://beam.apache.org/get-started/mobile-gaming-example/
+ */
+
+t.describe 'Run Apache Beam Java SDK Mobile Gaming Examples - Dataflow'
+
+QuickstartArchetype.generate(t)
+
+t.intent 'Running the Mobile-Gaming Code with DataflowRunner'
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84720)
Time Spent: 78.5h  (was: 78h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 78.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84719&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84719
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 05:14
Start Date: 27/Mar/18 05:14
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177309978
 
 

 ##
 File path: release/src/main/groovy/mobilegaming-java-dataflow.groovy
 ##
 @@ -0,0 +1,106 @@
+#!groovy
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import MobileGamingJavaUtils
+
+t = new TestScripts(args)
+
+/*
+ * Run the mobile game examples on Dataflow.
+ * https://beam.apache.org/get-started/mobile-gaming-example/
+ */
+
+t.describe 'Run Apache Beam Java SDK Mobile Gaming Examples - Dataflow'
+
+QuickstartArchetype.generate(t)
+
+t.intent 'Running the Mobile-Gaming Code with DataflowRunner'
+
+def runner = "DataflowRunner"
+
+/**
+ *  Run the UserScore example on DataflowRunner
+ * */
+
+t.intent("Running: UserScore example on DataflowRunner")
+t.run(MobileGamingJavaUtils.createExampleExecutionCommand("UserScore", runner, 
t))
+t.run "gsutil cat 
gs://${t.gcsBucket()}/${MobileGamingJavaUtils.getUserScoreOutputName(runner)}* 
| grep user19_BananaWallaby"
+t.see "total_score: 231, user: user19_BananaWallaby"
+t.intent("SUCCEED: UserScore successfully run on DataflowRunner.")
 
 Review comment:
   Made a new method success() in TestScripts


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84719)
Time Spent: 78h 20m  (was: 78h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 78h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-3087) Extend lock scope in Flink UnboundedSourceWrapper

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-3087.
--
   Resolution: Fixed
Fix Version/s: 2.5.0

> Extend lock scope in Flink UnboundedSourceWrapper
> -
>
> Key: BEAM-3087
> URL: https://issues.apache.org/jira/browse/BEAM-3087
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Grzegorz Kołakowski
>Priority: Major
> Fix For: 2.5.0
>
>
> In {{UnboundedSourceWrapper}} the lock scope is not big enough: we 
> synchronise in {{emitElement()}} but should instead synchronise inside the 
> reader loop in {{run()}} because the {{Source}} interface does not allow 
> concurrent calls.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2393?focusedWorklogId=84714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84714
 ]

ASF GitHub Bot logged work on BEAM-2393:


Author: ASF GitHub Bot
Created on: 27/Mar/18 04:50
Start Date: 27/Mar/18 04:50
Worklog Time Spent: 10m 
  Work Description: aljoscha closed pull request #4895: [BEAM-2393] Make 
BoundedSource fault-tolerant in FlinkRunner Streaming mode
URL: https://github.com/apache/beam/pull/4895
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index 09acc82352f..88119d13915 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -169,8 +169,11 @@ public Reader createReader(PipelineOptions options, 
Checkpoint checkpoint)
   return new CheckpointCoder<>(boundedSource.getDefaultOutputCoder());
 }
 
+/**
+ * A marker representing the progress and state of an {@link 
BoundedToUnboundedSourceAdapter}.
+ */
 @VisibleForTesting
-static class Checkpoint implements UnboundedSource.CheckpointMark {
+public static class Checkpoint implements 
UnboundedSource.CheckpointMark {
   private final @Nullable List> residualElements;
   private final @Nullable BoundedSource residualSource;
 
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 8f954a5cf0e..146ec9fb918 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -41,6 +41,7 @@
 import org.apache.beam.runners.core.construction.ReadTranslation;
 import org.apache.beam.runners.core.construction.SplittableParDo;
 import 
org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar;
+import 
org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
 import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
@@ -50,7 +51,6 @@
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SplittableDoFnOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
-import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.BoundedSourceWrapper;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper;
 import org.apache.beam.sdk.coders.Coder;
@@ -86,18 +86,27 @@
 import org.apache.flink.api.common.functions.FlatMapFunction;
 import org.apache.flink.api.common.functions.MapFunction;
 import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.StoppableFunction;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.api.java.functions.KeySelector;
 import org.apache.flink.api.java.tuple.Tuple2;
 import org.apache.flink.api.java.typeutils.GenericTypeInfo;
 import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.CheckpointListener;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
 import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.datastream.DataStreamSource;
 import org.apache.flink.streaming.api.datastream.KeyedStream;
 import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import 
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
 import org.apache.flink.streaming.api.operators.OneInputStrea

[jira] [Closed] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-2393.
--
   Resolution: Fixed
Fix Version/s: 2.5.0

> BoundedSource is not fault-tolerant in FlinkRunner Streaming mode
> -
>
> Key: BEAM-2393
> URL: https://issues.apache.org/jira/browse/BEAM-2393
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Grzegorz Kołakowski
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> {{BoundedSourceWrapper}} does not implement snapshot() and restore(), when 
> the failure to restart, it will send duplicate data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2393) BoundedSource is not fault-tolerant in FlinkRunner Streaming mode

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2393?focusedWorklogId=84713&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84713
 ]

ASF GitHub Bot logged work on BEAM-2393:


Author: ASF GitHub Bot
Created on: 27/Mar/18 04:50
Start Date: 27/Mar/18 04:50
Worklog Time Spent: 10m 
  Work Description: aljoscha commented on issue #4895: [BEAM-2393] Make 
BoundedSource fault-tolerant in FlinkRunner Streaming mode
URL: https://github.com/apache/beam/pull/4895#issuecomment-376396275
 
 
   Thanks a lot, @grzegorz8! 😄 I merged this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84713)
Time Spent: 50m  (was: 40m)

> BoundedSource is not fault-tolerant in FlinkRunner Streaming mode
> -
>
> Key: BEAM-2393
> URL: https://issues.apache.org/jira/browse/BEAM-2393
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Jingsong Lee
>Assignee: Grzegorz Kołakowski
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> {{BoundedSourceWrapper}} does not implement snapshot() and restore(), when 
> the failure to restart, it will send duplicate data.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (302b247 -> f93a332)

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 302b247  Merge pull request #4937: [BEAM-622] Add checkpointing tests 
for DoFnOperator and WindowDoFnOperator
 new fef8f54  [BEAM-3087] Make reader state update and element emission 
atomic
 new 0e44feb  [BEAM-2393] Make BoundedSource fault-tolerant
 new f93a332  Merge pull request #4895: [BEAM-2393] Make BoundedSource 
fault-tolerant in FlinkRunner Streaming mode

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../UnboundedReadFromBoundedSource.java|   5 +-
 .../flink/FlinkStreamingTransformTranslators.java  | 138 ++-
 .../streaming/io/BoundedSourceWrapper.java | 259 -
 .../streaming/io/UnboundedSourceWrapper.java   |  67 +++---
 .../flink/streaming/BoundedSourceRestoreTest.java  | 236 +++
 5 files changed, 404 insertions(+), 301 deletions(-)
 delete mode 100644 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BoundedSourceWrapper.java
 create mode 100644 
runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/BoundedSourceRestoreTest.java

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 02/03: [BEAM-2393] Make BoundedSource fault-tolerant

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0e44feb120a73c36116b2bd6b89e4f1676d7266f
Author: Grzegorz Kołakowski 
AuthorDate: Wed Feb 21 11:11:53 2018 +0100

[BEAM-2393] Make BoundedSource fault-tolerant
---
 .../UnboundedReadFromBoundedSource.java|   5 +-
 .../flink/FlinkStreamingTransformTranslators.java  | 138 ++-
 .../streaming/io/BoundedSourceWrapper.java | 259 -
 .../flink/streaming/BoundedSourceRestoreTest.java  | 236 +++
 4 files changed, 367 insertions(+), 271 deletions(-)

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index 09acc82..88119d1 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -169,8 +169,11 @@ public class UnboundedReadFromBoundedSource extends 
PTransform(boundedSource.getDefaultOutputCoder());
 }
 
+/**
+ * A marker representing the progress and state of an {@link 
BoundedToUnboundedSourceAdapter}.
+ */
 @VisibleForTesting
-static class Checkpoint implements UnboundedSource.CheckpointMark {
+public static class Checkpoint implements 
UnboundedSource.CheckpointMark {
   private final @Nullable List> residualElements;
   private final @Nullable BoundedSource residualSource;
 
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
index 970ece1..74ca168 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java
@@ -42,6 +42,7 @@ import 
org.apache.beam.runners.core.construction.ParDoTranslation;
 import org.apache.beam.runners.core.construction.ReadTranslation;
 import org.apache.beam.runners.core.construction.SplittableParDo;
 import 
org.apache.beam.runners.core.construction.TransformPayloadTranslatorRegistrar;
+import 
org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
 import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
@@ -51,7 +52,6 @@ import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKey
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SplittableDoFnOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
-import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.BoundedSourceWrapper;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper;
 import org.apache.beam.sdk.coders.Coder;
@@ -87,18 +87,27 @@ import org.apache.beam.sdk.values.WindowingStrategy;
 import org.apache.flink.api.common.functions.FlatMapFunction;
 import org.apache.flink.api.common.functions.MapFunction;
 import org.apache.flink.api.common.functions.RichFlatMapFunction;
+import org.apache.flink.api.common.functions.StoppableFunction;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
 import org.apache.flink.api.java.functions.KeySelector;
 import org.apache.flink.api.java.tuple.Tuple2;
 import org.apache.flink.api.java.typeutils.GenericTypeInfo;
 import org.apache.flink.api.java.typeutils.ResultTypeQueryable;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.state.CheckpointListener;
+import org.apache.flink.runtime.state.FunctionInitializationContext;
+import org.apache.flink.runtime.state.FunctionSnapshotContext;
+import org.apache.flink.streaming.api.checkpoint.CheckpointedFunction;
 import org.apache.flink.streaming.api.datastream.DataStream;
 import org.apache.flink.streaming.api.datastream.DataStreamSource;
 import org.apache.flink.streaming.api.datastream.KeyedStream;
 import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import 
org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction;
 import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
 import org.apache.

[beam] 03/03: Merge pull request #4895: [BEAM-2393] Make BoundedSource fault-tolerant in FlinkRunner Streaming mode

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f93a332a20857f425fa1c1b69d2423f3c2e7880d
Merge: 302b247 0e44feb
Author: Aljoscha Krettek 
AuthorDate: Tue Mar 27 06:49:21 2018 +0200

Merge pull request #4895: [BEAM-2393] Make BoundedSource fault-tolerant in 
FlinkRunner Streaming mode

 .../UnboundedReadFromBoundedSource.java|   5 +-
 .../flink/FlinkStreamingTransformTranslators.java  | 138 ++-
 .../streaming/io/BoundedSourceWrapper.java | 259 -
 .../streaming/io/UnboundedSourceWrapper.java   |  67 +++---
 .../flink/streaming/BoundedSourceRestoreTest.java  | 236 +++
 5 files changed, 404 insertions(+), 301 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
aljos...@apache.org.

[beam] 01/03: [BEAM-3087] Make reader state update and element emission atomic

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fef8f54be417be88730ebf2d0e8db8c55fbd013f
Author: Grzegorz Kołakowski 
AuthorDate: Wed Feb 21 10:31:53 2018 +0100

[BEAM-3087] Make reader state update and element emission atomic

Reader advancement should be considered as reader state update too.
Therefore, the reader's advancement and element emission are in the
same synchronized section.
---
 .../streaming/io/UnboundedSourceWrapper.java   | 67 --
 1 file changed, 37 insertions(+), 30 deletions(-)

diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
index fc23c01..3f04b6c 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSourceWrapper.java
@@ -186,7 +186,7 @@ public class UnboundedSourceWrapper<
 
 if (isRestored) {
   // restore the splitSources from the checkpoint to ensure consistent 
ordering
-  for (KV, 
CheckpointMarkT> restored:
+  for (KV, 
CheckpointMarkT> restored :
   stateForCheckpoint.get()) {
 localSplitSources.add(restored.getKey());
 localReaders.add(restored.getKey().createReader(
@@ -229,19 +229,25 @@ public class UnboundedSourceWrapper<
   // the easy case, we just read from one reader
   UnboundedSource.UnboundedReader reader = localReaders.get(0);
 
-  boolean dataAvailable = readerInvoker.invokeStart(reader);
-  if (dataAvailable) {
-emitElement(ctx, reader);
+  synchronized (ctx.getCheckpointLock()) {
+boolean dataAvailable = readerInvoker.invokeStart(reader);
+if (dataAvailable) {
+  emitElement(ctx, reader);
+}
   }
 
   setNextWatermarkTimer(this.runtimeContext);
 
   while (isRunning) {
-dataAvailable = readerInvoker.invokeAdvance(reader);
+boolean dataAvailable;
+synchronized (ctx.getCheckpointLock()) {
+  dataAvailable = readerInvoker.invokeAdvance(reader);
 
-if (dataAvailable)  {
-  emitElement(ctx, reader);
-} else {
+  if (dataAvailable) {
+emitElement(ctx, reader);
+  }
+}
+if (!dataAvailable) {
   Thread.sleep(50);
 }
   }
@@ -254,9 +260,11 @@ public class UnboundedSourceWrapper<
 
   // start each reader and emit data if immediately available
   for (UnboundedSource.UnboundedReader reader : localReaders) {
-boolean dataAvailable = readerInvoker.invokeStart(reader);
-if (dataAvailable) {
-  emitElement(ctx, reader);
+synchronized (ctx.getCheckpointLock()) {
+  boolean dataAvailable = readerInvoker.invokeStart(reader);
+  if (dataAvailable) {
+emitElement(ctx, reader);
+  }
 }
   }
 
@@ -267,11 +275,13 @@ public class UnboundedSourceWrapper<
   boolean hadData = false;
   while (isRunning) {
 UnboundedSource.UnboundedReader reader = 
localReaders.get(currentReader);
-boolean dataAvailable = readerInvoker.invokeAdvance(reader);
 
-if (dataAvailable) {
-  emitElement(ctx, reader);
-  hadData = true;
+synchronized (ctx.getCheckpointLock()) {
+  boolean dataAvailable = readerInvoker.invokeAdvance(reader);
+  if (dataAvailable) {
+emitElement(ctx, reader);
+hadData = true;
+  }
 }
 
 currentReader = (currentReader + 1) % numReaders;
@@ -321,24 +331,21 @@ public class UnboundedSourceWrapper<
   UnboundedSource.UnboundedReader reader) {
 // make sure that reader state update and element emission are atomic
 // with respect to snapshots
-synchronized (ctx.getCheckpointLock()) {
-
-  OutputT item = reader.getCurrent();
-  byte[] recordId = reader.getCurrentRecordId();
-  Instant timestamp = reader.getCurrentTimestamp();
-
-  WindowedValue> windowedValue =
-  WindowedValue.of(new ValueWithRecordId<>(item, recordId), timestamp,
-  GlobalWindow.INSTANCE, PaneInfo.NO_FIRING);
-  ctx.collectWithTimestamp(windowedValue, timestamp.getMillis());
-}
+OutputT item = reader.getCurrent();
+byte[] recordId = reader.getCurrentRecordId();
+Instant timestamp = reader.getCurrentTimestamp();
+
+WindowedValue> windowedValue =
+WindowedValue.of(new ValueWithRecordId<>(item, recordId), timestamp,
+GlobalWindow.INSTANCE, PaneInfo.NO_FIRING);
+ctx.collectWithTimestamp(windowedValue, timestamp.getMillis());
   }
 
 

[jira] [Work logged] (BEAM-622) Add checkpointing tests for DoFnOperator and WindowDoFnOperator

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-622?focusedWorklogId=84711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84711
 ]

ASF GitHub Bot logged work on BEAM-622:
---

Author: ASF GitHub Bot
Created on: 27/Mar/18 04:38
Start Date: 27/Mar/18 04:38
Worklog Time Spent: 10m 
  Work Description: aljoscha closed pull request #4937: [BEAM-622] Add 
checkpointing tests for DoFnOperator and WindowDoFnOpe…
URL: https://github.com/apache/beam/pull/4937
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
index 0b376e9ddd1..73be0ef09ce 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
@@ -17,13 +17,11 @@
  */
 package org.apache.beam.runners.flink.streaming;
 
+import static 
org.apache.beam.runners.flink.streaming.StreamRecordStripper.stripStreamRecordFromWindowedValue;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
 import static org.junit.Assert.assertThat;
 
-import com.google.common.base.Function;
-import com.google.common.collect.FluentIterable;
 import java.nio.ByteBuffer;
-import javax.annotation.Nullable;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.ValueWithRecordId;
@@ -64,7 +62,7 @@ public void testDeduping() throws Exception {
 WindowedValue.valueInGlobalWindow(new ValueWithRecordId<>(key1, 
key1.getBytes();
 
 assertThat(
-this.stripStreamRecordFromWindowedValue(harness.getOutput()),
+stripStreamRecordFromWindowedValue(harness.getOutput()),
 contains(WindowedValue.valueInGlobalWindow(key1),
 WindowedValue.valueInGlobalWindow(key2)));
 
@@ -86,7 +84,7 @@ public void testDeduping() throws Exception {
 WindowedValue.valueInGlobalWindow(new ValueWithRecordId<>(key3, 
key3.getBytes();
 
 assertThat(
-this.stripStreamRecordFromWindowedValue(harness.getOutput()),
+stripStreamRecordFromWindowedValue(harness.getOutput()),
 contains(WindowedValue.valueInGlobalWindow(key3)));
 
 harness.close();
@@ -102,26 +100,4 @@ public void testDeduping() throws Exception {
 value -> ByteBuffer.wrap(value.getValue().getId()),
 TypeInformation.of(ByteBuffer.class));
   }
-
-  private  Iterable> stripStreamRecordFromWindowedValue(
-  Iterable input) {
-
-return FluentIterable.from(input)
-.filter(
-o ->
-o instanceof StreamRecord && ((StreamRecord) o).getValue() 
instanceof WindowedValue)
-.transform(
-new Function>() {
-  @Nullable
-  @Override
-  @SuppressWarnings({"unchecked", "rawtypes"})
-  public WindowedValue apply(@Nullable Object o) {
-if (o instanceof StreamRecord
-&& ((StreamRecord) o).getValue() instanceof WindowedValue) 
{
-  return (WindowedValue) ((StreamRecord) o).getValue();
-}
-throw new RuntimeException("unreachable");
-  }
-});
-  }
 }
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
index 73a0a08f29c..4d6fca62fb5 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.flink.streaming;
 
+import static 
org.apache.beam.runners.flink.streaming.StreamRecordStripper.stripStreamRecordFromWindowedValue;
 import static org.hamcrest.Matchers.emptyIterable;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
 import static org.junit.Assert.assertEquals;
@@ -28,6 +29,7 @@
 import com.google.common.collect.ImmutableMap;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.Optional;
 import javax.annotation.Nullable;
 import org.apache.beam.runners.core.StatefulDoFnRunner;
 import org.apache.beam.runners.flink.FlinkPipelineOptions;
@@ -39,6 +41,7 @@
 import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 i

[jira] [Work logged] (BEAM-622) Add checkpointing tests for DoFnOperator and WindowDoFnOperator

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-622?focusedWorklogId=84710&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84710
 ]

ASF GitHub Bot logged work on BEAM-622:
---

Author: ASF GitHub Bot
Created on: 27/Mar/18 04:38
Start Date: 27/Mar/18 04:38
Worklog Time Spent: 10m 
  Work Description: aljoscha commented on issue #4937: [BEAM-622] Add 
checkpointing tests for DoFnOperator and WindowDoFnOpe…
URL: https://github.com/apache/beam/pull/4937#issuecomment-376394698
 
 
   Thanks again!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84710)
Time Spent: 1h 10m  (was: 1h)

> Add checkpointing tests for DoFnOperator and WindowDoFnOperator 
> 
>
> Key: BEAM-622
> URL: https://issues.apache.org/jira/browse/BEAM-622
> Project: Beam
>  Issue Type: Test
>  Components: runner-flink
>Affects Versions: 0.3.0-incubating
>Reporter: Maximilian Michels
>Assignee: Grzegorz Kołakowski
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Tests which test the correct snapshotting of these two operators are missing. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: [BEAM-622] Add checkpointing tests for DoFnOperator and WindowDoFnOperator

[aljoscha]

This is an automated email from the ASF dual-hosted git repository.

aljoscha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new ebc0fb0  [BEAM-622] Add checkpointing tests for DoFnOperator and 
WindowDoFnOperator
 new 302b247  Merge pull request #4937: [BEAM-622] Add checkpointing tests 
for DoFnOperator and WindowDoFnOperator
ebc0fb0 is described below

commit ebc0fb0e3d0f7340400fe07d7d508907da351424
Author: Grzegorz Kołakowski 
AuthorDate: Thu Mar 22 09:51:44 2018 +0100

[BEAM-622] Add checkpointing tests for DoFnOperator and WindowDoFnOperator

Timers restore test for DoFnOperator already exists.
---
 .../flink/streaming/DedupingOperatorTest.java  |  30 +--
 .../runners/flink/streaming/DoFnOperatorTest.java  | 177 ++--
 .../flink/streaming/StreamRecordStripper.java  |  49 +
 .../flink/streaming/WindowDoFnOperatorTest.java| 227 +
 4 files changed, 397 insertions(+), 86 deletions(-)

diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
index 0b376e9..73be0ef 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DedupingOperatorTest.java
@@ -17,13 +17,11 @@
  */
 package org.apache.beam.runners.flink.streaming;
 
+import static 
org.apache.beam.runners.flink.streaming.StreamRecordStripper.stripStreamRecordFromWindowedValue;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
 import static org.junit.Assert.assertThat;
 
-import com.google.common.base.Function;
-import com.google.common.collect.FluentIterable;
 import java.nio.ByteBuffer;
-import javax.annotation.Nullable;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.DedupingOperator;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.ValueWithRecordId;
@@ -64,7 +62,7 @@ public class DedupingOperatorTest {
 WindowedValue.valueInGlobalWindow(new ValueWithRecordId<>(key1, 
key1.getBytes();
 
 assertThat(
-this.stripStreamRecordFromWindowedValue(harness.getOutput()),
+stripStreamRecordFromWindowedValue(harness.getOutput()),
 contains(WindowedValue.valueInGlobalWindow(key1),
 WindowedValue.valueInGlobalWindow(key2)));
 
@@ -86,7 +84,7 @@ public class DedupingOperatorTest {
 WindowedValue.valueInGlobalWindow(new ValueWithRecordId<>(key3, 
key3.getBytes();
 
 assertThat(
-this.stripStreamRecordFromWindowedValue(harness.getOutput()),
+stripStreamRecordFromWindowedValue(harness.getOutput()),
 contains(WindowedValue.valueInGlobalWindow(key3)));
 
 harness.close();
@@ -102,26 +100,4 @@ public class DedupingOperatorTest {
 value -> ByteBuffer.wrap(value.getValue().getId()),
 TypeInformation.of(ByteBuffer.class));
   }
-
-  private  Iterable> stripStreamRecordFromWindowedValue(
-  Iterable input) {
-
-return FluentIterable.from(input)
-.filter(
-o ->
-o instanceof StreamRecord && ((StreamRecord) o).getValue() 
instanceof WindowedValue)
-.transform(
-new Function>() {
-  @Nullable
-  @Override
-  @SuppressWarnings({"unchecked", "rawtypes"})
-  public WindowedValue apply(@Nullable Object o) {
-if (o instanceof StreamRecord
-&& ((StreamRecord) o).getValue() instanceof WindowedValue) 
{
-  return (WindowedValue) ((StreamRecord) o).getValue();
-}
-throw new RuntimeException("unreachable");
-  }
-});
-  }
 }
diff --git 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
index 73a0a08..4d6fca6 100644
--- 
a/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
+++ 
b/runners/flink/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.flink.streaming;
 
+import static 
org.apache.beam.runners.flink.streaming.StreamRecordStripper.stripStreamRecordFromWindowedValue;
 import static org.hamcrest.Matchers.emptyIterable;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
 import static org.junit.Assert.assertEquals;
@@ -28,6 +29,7 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.Optional;
 import javax.annotation.Nullabl

Build failed in Jenkins: beam_PostCommit_Python_Verify #4512

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

--
[...truncated 290.19 KB...]
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-27 04:36:26.076255
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-27 04:36:26.641964
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 04:36:26.802871
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 04:36:26.818487
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1191

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix test_pre_finalize_error to test exceptions.

--
[...truncated 779.49 KB...]
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-26_21_14_59-1408162820328540407]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-26_21_14_59-1408162820328540407?project=apache-beam-testing
root: INFO: Job 2018-03-26_21_14_59-1408162820328540407 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-27T04:14:59.909Z: JOB_MESSAGE_WARNING: Job 
2018-03-26_21_14_59-1408162820328540407 might autoscale up to 1000 workers.
root: INFO: 2018-03-27T04:14:59.937Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-26_21_14_59-1408162820328540407. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-27T04:14:59.953Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-26_21_14_59-1408162820328540407.
root: INFO: 2018-03-27T04:15:02.630Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-27T04:15:02.912Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-27T04:15:03.662Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-27T04:15:03.696Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-27T04:15:03.721Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-27T04:15:03.860Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-27T04:15:03.898Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-27T04:15:03.952Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-27T04:15:03.979Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-27T04:15:04.003Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-27T04:15:04.028Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-27T04:15:04.050Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-27T04:15:04.081Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-27T04:15:04.102Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-27T04:15:04.135Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11-u13 for input s12-reify-value0-c11
root: INFO: 2018-03-27T04:15:04.171Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Write, through flatten s11-u13, into 
producer assert_that/Group/GroupByKey/Reify
root: INFO: 2018-03-27T04:15:04.201Z: JOB_MESSAGE_DETAILED: Fusing con

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84697&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84697
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 04:15
Start Date: 27/Mar/18 04:15
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177303918
 
 

 ##
 File path: release/src/main/groovy/TestScripts.groovy
 ##
 @@ -95,16 +116,31 @@ class TestScripts {
  }
}
 
-   // Check for expected results in stdout of the last command
+   // Check for expected results in stdout of the last command, if fails, log 
errors then exit.
public void see(String expected) {
  if (!var.lastText.contains(expected)) {
var.startDir.deleteDir()
println "Cannot find ${expected} in ${var.lastText}"
-   _error("Cannot find expected text")
+   error("Cannot find expected text")
  }
  println "Verified $expected"
}
 
+   public boolean seeOneOf(List expected) {
+ for (String expect: expected) {
+   if(var.lastText.contains(expect)) {
+ println "Verified $expect"
+ return true
+   }
+ }
+ println "Cannot find ${expected} in text"
+ return false
+   }
+
+   public boolean seeExact(String expected) {
 
 Review comment:
   yes, we do need it while checking whether tables are created/delete from 
dataset. 
   Renamed method being seeSubstring()


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84697)
Time Spent: 78h 10m  (was: 78h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 78h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4953 from udim/pre-finalize-test-fix

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8ac31c855a6c71c0ce031e0ed3ddfbeb5b83c03d
Merge: 749a93b a9177a3
Author: Ahmet Altay 
AuthorDate: Mon Mar 26 20:51:49 2018 -0700

Merge pull request #4953 from udim/pre-finalize-test-fix

Fix test_pre_finalize_error to test exception raising.

 sdks/python/apache_beam/io/filebasedsink_test.py | 22 +-
 1 file changed, 9 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (749a93b -> 8ac31c8)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 749a93b  Merge pull request #4945 from apache/aaltay-patch-1
 add a9177a3  Fix test_pre_finalize_error to test exceptions.
 new 8ac31c8  Merge pull request #4953 from udim/pre-finalize-test-fix

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/io/filebasedsink_test.py | 22 +-
 1 file changed, 9 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1190

[Apache Jenkins Server]

See 


--
[...truncated 1.18 MB...]
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "Map()/MapToVoidKey1.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "Map()/MapToVoidKey1"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-26_20_28_20-1071160417201048090]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-26_20_28_20-1071160417201048090?project=apache-beam-testing
root: INFO: Job 2018-03-26_20_28_20-1071160417201048090 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-27T03:28:20.553Z: JOB_MESSAGE_WARNING: Job 
2018-03-26_20_28_20-1071160417201048090 might autoscale up to 1000 workers.
root: INFO: 2018-03-27T03:28:20.583Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-26_20_28_20-1071160417201048090. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-27T03:28:20.604Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-26_20_28_20-1071160417201048090.
root: INFO: 2018-03-27T03:28:23.218Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-27T03:28:23.390Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-27T03:28:24.148Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-27T03:28:24.175Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-27T03:28:24.201Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-27T03:28:24.230Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-27T03:28:24.256Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-27T03:28:24.298Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-27T03:28:24.328Z: JOB_MESSAGE_DETAILED: Fusing consumer 
Map()/MapToVoidKey0 into side list/Read
root: INFO: 2018-03-27T03:28:24.360Z: JOB_MESSAGE_DETAILED: Fusing consumer 
Map()/MapToVoidKey1 into side list/Read
root: INFO: 2018-03-27T03:28:24.392Z: JOB_MESSAGE_DETAILED: Fusing consumer 
Map()/MapToVoidKey0 into side list/Read
root: INFO: 2018-03-27T03:28:24.415Z: JOB_MESSAGE_DETAILED: Fusing consumer 
Map()/MapToVoidKey1 into side list/Read
root: INFO: 2018-03-27T03:28:24.448Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-27T03:28:24.481Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-27T03:28:24.513Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-27T03:28:24.546Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/Write into assert_that/Group/GroupByKey/Reify
root: INFO: 2018-03-27T0

Build failed in Jenkins: beam_PostCommit_Python_Verify #4511

[Apache Jenkins Server]

See 


--
[...truncated 290.17 KB...]
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-27 03:29:31.906515
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-27 03:29:32.473091
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 03:29:32.650853
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 03:29:32.667921
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84691
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:28
Start Date: 27/Mar/18 03:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376385106
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84691)
Time Spent: 6h 40m  (was: 6.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84685&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84685
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:11
Start Date: 27/Mar/18 03:11
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177295914
 
 

 ##
 File path: release/src/main/groovy/TestScripts.groovy
 ##
 @@ -95,16 +116,31 @@ class TestScripts {
  }
}
 
-   // Check for expected results in stdout of the last command
+   // Check for expected results in stdout of the last command, if fails, log 
errors then exit.
public void see(String expected) {
  if (!var.lastText.contains(expected)) {
var.startDir.deleteDir()
println "Cannot find ${expected} in ${var.lastText}"
-   _error("Cannot find expected text")
+   error("Cannot find expected text")
  }
  println "Verified $expected"
}
 
+   public boolean seeOneOf(List expected) {
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84685)
Time Spent: 78h  (was: 77h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 78h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84683&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84683
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:09
Start Date: 27/Mar/18 03:09
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177295643
 
 

 ##
 File path: release/src/main/groovy/MoblieGamingJavaUtils.groovy
 ##
 @@ -0,0 +1,148 @@
+#!groovy
+import java.text.SimpleDateFormat
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+class MobileGamingJavaUtils {
+
+  public static final RUNNERS = [DirectRunner: "direct-runner",
+DataflowRunner: "dataflow-runner",
+SparkRunner: "spark-runner",
+ApexRunner: "apex-runner",
+FlinkRunner: "flink-runner"]
+
+  public static final EXECUTION_TIMEOUT = 20
+
+  // Lists used to verify team names generated in the LeaderBoard example.
+  // This list should be kept sync with COLORS in 
org.apache.beam.examples.complete.game.injector.Injector.
+  public static final COLORS = new ArrayList<>(Arrays.asList(
+"Magenta",
+"AliceBlue",
+"Almond",
+"Amaranth",
+"Amber",
+"Amethyst",
+"AndroidGreen",
+"AntiqueBrass",
+"Fuchsia",
+"Ruby",
+"AppleGreen",
+"Apricot",
+"Aqua",
+"ArmyGreen",
+"Asparagus",
+"Auburn",
+"Azure",
+"Banana",
+"Beige",
+"Bisque",
+"BarnRed",
+"BattleshipGrey"))
+
+  private static final USERSCORE_OUTPUT_PREFIX = "java-userscore-result-"
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84683)
Time Spent: 77h 40m  (was: 77.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 77h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84684&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84684
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:09
Start Date: 27/Mar/18 03:09
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177295655
 
 

 ##
 File path: release/src/main/groovy/MoblieGamingJavaUtils.groovy
 ##
 @@ -0,0 +1,148 @@
+#!groovy
+import java.text.SimpleDateFormat
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+class MobileGamingJavaUtils {
+
+  public static final RUNNERS = [DirectRunner: "direct-runner",
+DataflowRunner: "dataflow-runner",
+SparkRunner: "spark-runner",
+ApexRunner: "apex-runner",
+FlinkRunner: "flink-runner"]
+
+  public static final EXECUTION_TIMEOUT = 20
+
+  // Lists used to verify team names generated in the LeaderBoard example.
+  // This list should be kept sync with COLORS in 
org.apache.beam.examples.complete.game.injector.Injector.
+  public static final COLORS = new ArrayList<>(Arrays.asList(
+"Magenta",
+"AliceBlue",
+"Almond",
+"Amaranth",
+"Amber",
+"Amethyst",
+"AndroidGreen",
+"AntiqueBrass",
+"Fuchsia",
+"Ruby",
+"AppleGreen",
+"Apricot",
+"Aqua",
+"ArmyGreen",
+"Asparagus",
+"Auburn",
+"Azure",
+"Banana",
+"Beige",
+"Bisque",
+"BarnRed",
+"BattleshipGrey"))
+
+  private static final USERSCORE_OUTPUT_PREFIX = "java-userscore-result-"
+
+  private static final HOURLYTEAMSCORE_OUTPUT_PREFIX = 
"java-hourlyteamscore-result-"
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84684)
Time Spent: 77h 50m  (was: 77h 40m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 77h 50m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84682&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84682
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:08
Start Date: 27/Mar/18 03:08
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177295512
 
 

 ##
 File path: release/src/main/groovy/MoblieGamingJavaUtils.groovy
 ##
 @@ -0,0 +1,148 @@
+#!groovy
+import java.text.SimpleDateFormat
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+class MobileGamingJavaUtils {
+
+  public static final RUNNERS = [DirectRunner: "direct-runner",
+DataflowRunner: "dataflow-runner",
+SparkRunner: "spark-runner",
+ApexRunner: "apex-runner",
+FlinkRunner: "flink-runner"]
+
+  public static final EXECUTION_TIMEOUT = 20
 
 Review comment:
   EXECUTION_TIMEOUT -> EXECUTION_TIMEOUT_IN_MINUTES


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84682)
Time Spent: 77.5h  (was: 77h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 77.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=84681&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84681
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 27/Mar/18 03:08
Start Date: 27/Mar/18 03:08
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on a change in pull request #4788: 
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#discussion_r177295512
 
 

 ##
 File path: release/src/main/groovy/MoblieGamingJavaUtils.groovy
 ##
 @@ -0,0 +1,148 @@
+#!groovy
+import java.text.SimpleDateFormat
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+class MobileGamingJavaUtils {
+
+  public static final RUNNERS = [DirectRunner: "direct-runner",
+DataflowRunner: "dataflow-runner",
+SparkRunner: "spark-runner",
+ApexRunner: "apex-runner",
+FlinkRunner: "flink-runner"]
+
+  public static final EXECUTION_TIMEOUT = 20
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84681)
Time Spent: 77h 20m  (was: 77h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 77h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84675&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84675
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 02:47
Start Date: 27/Mar/18 02:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376378468
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84675)
Time Spent: 6.5h  (was: 6h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84672&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84672
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 02:43
Start Date: 27/Mar/18 02:43
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376377834
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84672)
Time Spent: 6h 20m  (was: 6h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #6299

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=84671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84671
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 27/Mar/18 02:29
Start Date: 27/Mar/18 02:29
Worklog Time Spent: 10m 
  Work Description: mariapython commented on issue #4949: [BEAM-3818] WIP: 
Add support for streaming side inputs in the DirectRunner (part 2: unblock 
tasks as the _SideInputsContainer gets updated) 
URL: https://github.com/apache/beam/pull/4949#issuecomment-376375686
 
 
   R: @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84671)
Time Spent: 5h  (was: 4h 50m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84663&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84663
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:34
Start Date: 27/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376366429
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84663)
Time Spent: 6h 10m  (was: 6h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84662
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:30
Start Date: 27/Mar/18 01:30
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376365658
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84662)
Time Spent: 6h  (was: 5h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84658&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84658
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:23
Start Date: 27/Mar/18 01:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376364532
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84658)
Time Spent: 5h 50m  (was: 5h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1515

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 108.48 KB...]
2018-03-27 01:14:02,041 96d1ad9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 01:14:22,591 96d1ad9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 01:14:27,019 96d1ad9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r29b61b6750275f84_0162650537f7_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r29b61b6750275f84_0162650537f7_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r29b61b6750275f84_0162650537f7_1 ... (0s) Current status: DONE   
2018-03-27 01:14:27,020 96d1ad9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 01:14:46,067 96d1ad9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 01:14:50,279 96d1ad9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r35cb53b817a05b01_016265059429_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r35cb53b817a05b01_016265059429_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r35cb53b817a05b01_016265059429_1 ... (0s) Current status: DONE   
2018-03-27 01:14:50,280 96d1ad9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 01:15:05,303 96d1ad9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 01:15:11,154 96d1ad9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3a70ece037cc870c_01626505e164_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r3a70ece037cc870c_01626505e164_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r3a70ece037cc870c_01626505e164_1 ... (1s) Current status: RUNNING 

 Waiting on bqjob_r3a70ece037cc870c_01626505e164_1 ... (1s) Current status: 
DONE   
2018-03-27 01:15:11,154 96d1ad9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-27 01:15:33,251 96d1ad9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-27 01:15:39,829 96d1ad9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r59597783e305d65e_0162650652ea_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r59597783e305d65e_0162650652ea_1 
... (0s) Current status: RUNNING
  Waiting

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84656
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:18
Start Date: 27/Mar/18 01:18
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281326
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/bundle/bundle.go
 ##
 @@ -0,0 +1,36 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package bundle
+
+import "context"
+
+type key string
+
+const bundleKey key = "beam:bundle"
+
+// SetID sets the id of the current Bundle.
+func SetID(ctx context.Context, id string) context.Context {
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84656)
Time Spent: 7h 40m  (was: 7.5h)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84655&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84655
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:17
Start Date: 27/Mar/18 01:17
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376363633
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84655)
Time Spent: 5h 40m  (was: 5.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84652
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:17
Start Date: 27/Mar/18 01:17
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281229
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84652)
Time Spent: 7h 20m  (was: 7h 10m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84653
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:17
Start Date: 27/Mar/18 01:17
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281280
 
 

 ##
 File path: sdks/go/pkg/beam/metrics_test.go
 ##
 @@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package beam_test
+
+import (
+   "context"
+   "regexp"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/metrics"
+)
+
+func ctxWithPtransformID(id string) context.Context {
+   ctx := context.Background()
+   ctx = ptransform.SetID(ctx, id)
+   ctx = bundle.SetID(ctx, "exampleBundle")
+   return ctx
+}
+
+func dumpAndClearMetrics() {
+   metrics.DumpToOut()
+   metrics.Clear()
+}
+
+var (
+   wordRE = regexp.MustCompile(`[a-zA-Z]+('[a-z])?`)
+)
+
+func Example_metricsDeclaredAnywhere() {
+
+   // Metrics can be declared both inside or outside DoFns.
+   outside := beam.GetCounter("example.namespace", "count")
+
+   extractWordsDofn := func(ctx context.Context, line string, emit 
func(string)) {
+   inside := beam.GetDistribution("example.namespace", 
"characters")
+   for _, word := range wordRE.FindAllString(line, -1) {
+   emit(word)
+   outside.Inc(ctx, 1)
+   inside.Update(ctx, int64(len(word)))
+   }
+   }
+   ctx := ctxWithPtransformID("example")
+   extractWordsDofn(ctx, "this has six words in it", func(string) {})
+   extractWordsDofn(ctx, "this has seven words in it, see?", func(string) 
{})
+
+   dumpAndClearMetrics()
+   // Output: Bundle: "exampleBundle" - PTransformID: "example"
+   //  example.namespace.characters - count: 13 sum: 43 min: 2 max: 5
+   //  example.namespace.count - value: 13
+}
+
+func Example_metricsReusable() {
+
+   // Metrics can be used in multiple DoFns
+   c := beam.GetCounter("example.reusable", "count")
+
+   extractWordsDofn := func(ctx context.Context, line string, emit 
func(string)) {
+   for _, word := range wordRE.FindAllString(line, -1) {
+   emit(word)
+   c.Inc(ctx, 1)
+   }
+   }
+
+   extractRunesDofn := func(ctx context.Context, line string, emit 
func(rune)) {
+   for _, r := range line {
+   emit(r)
+   c.Inc(ctx, 1)
+   }
+   }
+   extractWordsDofn(ctxWithPtransformID("extract1"), "this has six words 
in it", func(string) {})
+
+   extractRunesDofn(ctxWithPtransformID("extract2"), "seven thousand", 
func(rune) {})
+
+   dumpAndClearMetrics()
+   // Output: Bundle: "exampleBundle" - PTransformID: "extract1"
+   //  example.reusable.count - value: 6
+   // Bundle: "exampleBundle" - PTransformID: "extract2"
+   //  example.reusable.count - value: 14
+}
+
+var ctx = context.Background()
 
 Review comment:
   Yup! Added a comment to that effect, and moved it to the top.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84653)
Time Spent: 7.5h  (was: 7h 20m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84650
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:16
Start Date: 27/Mar/18 01:16
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281109
 
 

 ##
 File path: sdks/go/examples/wordcount/wordcount.go
 ##
 @@ -104,9 +104,11 @@ var (
 var wordRE = regexp.MustCompile(`[a-zA-Z]+('[a-z])?`)
 
 // extractFn is a DoFn that emits the words in a given line.
-func extractFn(line string, emit func(string)) {
+func extractFn(ctx context.Context, line string, emit func(string)) {
+   c := beam.GetCounter("extract", "count")
 
 Review comment:
   Done. I'm going to defer updating the other examples to match java to 
another PR to avoid too much churn.
   
   As a rule we do want to declare metrics globally. It was to 
demonstrate/sanity check that local definitions with the same namespace.name 
are "the same" counter. This is now handled in the declareAnywhere example.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84650)
Time Spent: 7h  (was: 6h 50m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 7h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84651
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:16
Start Date: 27/Mar/18 01:16
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281181
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84646&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84646
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:15
Start Date: 27/Mar/18 01:15
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280985
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84648&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84648
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:15
Start Date: 27/Mar/18 01:15
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281046
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84647&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84647
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:15
Start Date: 27/Mar/18 01:15
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177281022
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84645&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84645
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:15
Start Date: 27/Mar/18 01:15
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280837
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84642&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84642
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280778
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84642)
Time Spent: 5h 50m  (was: 5h 40m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84639
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280721
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84639)
Time Spent: 5h 20m  (was: 5h 10m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84644&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84644
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280803
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
+
+// Counter is a simple counter for incrementing and decrementing a value.
+ty

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84643
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280792
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84643)
Time Spent: 6h  (was: 5h 50m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84640&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84640
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280743
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
 
 Review comment:
   Good point! I dropped the "ker" suffix on the type.
   
   We can always work around not having the private interface. It certainly 
doesn't make sense to make it public at this point. Having the interface at all 
saves us from having N different metric container typed maps for each bundleID 
and PtransformID segregated storage.
   I feel it's a reasonable use of a private interface, since it's purely for 
the convenience of this package specifically, and since 3 distinct counter 
containers needed a way to convert to the proto type for exporting.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84640)
Time Spent: 5.5h  (was: 5h 20m)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84641&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84641
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:14
Start Date: 27/Mar/18 01:14
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280762
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
+// and the PTransform context in which they are used. Further, they are
+// reported as a delta against the bundle being processed, so that overcounting
+// doesn't occur if a bundle needs to be retried.
+//
+// Metric containers aren't initialized until their first mutation, which
+// follows from the Beam model design, where metrics are only sent for a bundle
+// if they have changed. This is particularly convenient for distributions 
which
+// means their min and max fields can be set to the first value on creation
+// rather than have some marker of uninitialized state, which would otherwise
+// need to be checked for on every update.
+//
+// Metric values are implemented as lightweight proxies of the user provided
+// namespace and name. This allows them to be declared globally, and used in
+// any ParDo. To handle reporting deltas on the metrics by bundle, metrics
+// are keyed by bundleID,PTransformID,namespace, and name, so metrics that
+// are identical except for bundles are treated as distinct, effectively
+// providing per bundle deltas, since a new value container is used per bundle.
+package metrics
+
+import (
+   "context"
+   "fmt"
+   "sort"
+   "sync"
+   "time"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/bundle"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/util/ptransform"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+   "github.com/golang/protobuf/ptypes"
+)
+
+// userMetricker knows how to convert it's value to a Metrics_User proto.
+type userMetricker interface {
+   toProto() *fnexecution_v1.Metrics_User
+}
+
+type metricName struct {
+   namespace, name string
+}
+
+func (n metricName) String() string {
+   return fmt.Sprintf("%s.%s", n.namespace, n.name)
+}
+
+func validateName(mn metricName) {
+   if len(mn.name) == 0 || len(mn.namespace) == 0 {
+   panic(fmt.Sprintf("namespace and name are required to be 
non-empty, got %q and %q", mn.namespace, mn.name))
+   }
+}
+
+type metricKey struct {
+   name   metricName
+   bundle, ptransform string
+}
+
+var (
+   metricMu sync.RWMutex
+   // metricStorage is a map of BundleIds to PtransformIds to metrics.
+   metricStorage = make(map[string]map[string]map[metricName]userMetricker)
+   counters  = sync.Map{}
+   distributions = sync.Map{}
+   gauges= sync.Map{}
+)
+
+type metricType uint8
+
+const (
+   counterType metricType = iota
+   distributionType
+   gaugeType
+)
+
+func (t metricType) String() string {
+   switch t {
+   case counterType:
+   return "Counter"
+   case distributionType:
+   return "Distribution"
+   case gaugeType:
+   return "Gauge"
+   default:
+   panic(fmt.Sprintf("Unknown metric type value: %v", uint8(t)))
+   }
+}
+
+// Clocker is an interface to access the current time.
+// TODO(lostluck): 2018/03/05 Use a common internal beam clock instead, once 
that exists.
+type Clocker interface {
+   Now() time.Time
+}
+
+type defaultclock struct{}
+
+func (defaultclock) Now() time.Time {
+   return time.Now()
+}
+
+var clock Clocker = defaultclock{}
 
 Review comment:
   Done.

---

[jira] [Work logged] (BEAM-3545) Fn API metrics in Go SDK harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?focusedWorklogId=84638&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84638
 ]

ASF GitHub Bot logged work on BEAM-3545:


Author: ASF GitHub Bot
Created on: 27/Mar/18 01:13
Start Date: 27/Mar/18 01:13
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #4899: 
[BEAM-3545] Go SDK UserCounters
URL: https://github.com/apache/beam/pull/4899#discussion_r177280688
 
 

 ##
 File path: sdks/go/pkg/beam/core/metrics/metrics.go
 ##
 @@ -0,0 +1,447 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package metrics handles Beam metrics.
+//
+// Metrics in the Beam model are uniquely identified by a namespace, a name,
 
 Review comment:
   I couldn't find one on the beam site that wasn't JavaDoc or PyDoc. Ideally 
we'd refer to one of the design docs, or a spec, but such things don't exist as 
far as I could find. One will probably turn up, but I don't believe it's worth 
blocking this PR over.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84638)
Time Spent: 5h 10m  (was: 5h)

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #67

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 126.99 KB...]
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.5.Final from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.0 from the shaded 
jar.
[INFO] Excluding org.objenesis:objenesis:jar:1.0 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.5.0-SNAPSHOT
 from the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded ja

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1189

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 829.59 KB...]
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Unkey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Unkey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s15", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_equal"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Match.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Match"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"n

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #298

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 26.30 KB...]
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the s

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #70

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 25.02 KB...]
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84634&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84634
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Mar/18 00:48
Start Date: 27/Mar/18 00:48
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376358568
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84634)
Time Spent: 5.5h  (was: 5h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #316

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 25.01 KB...]
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-pro

Build failed in Jenkins: beam_PerformanceTests_JDBC #378

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 42.56 KB...]
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-ut

Build failed in Jenkins: beam_PerformanceTests_Python #1071

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 59.71 KB...]
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.017s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.110s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  3.353 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.657 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.117 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  9.090 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.262 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.963 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.185 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 26.349 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED
[INFO] Apache Beam :: Runners :: Local Java Core .. SKIPPED
[INFO] Apache Beam :: Runners :: Direct Java .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SKIPPED
[INFO] Apache Beam :: SDKs ::

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #300

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 677.75 KB...]
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #302

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 704.11 KB...]
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:j

Build failed in Jenkins: beam_PostCommit_Python_Verify #4510

[Apache Jenkins Server]

See 


Changes:

[github] Updated to Ubuntu 16 version of python 2

--
[...truncated 290.19 KB...]
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-27 00:37:26.752944
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-27 00:37:27.323411
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 00:37:27.497816
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-27 00:37:27.512230
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=84627&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84627
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 27/Mar/18 00:26
Start Date: 27/Mar/18 00:26
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4901: [BEAM-3744] 
Expand Pubsub read API for Python.
URL: https://github.com/apache/beam/pull/4901#issuecomment-376355285
 
 
   LGTM. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84627)
Time Spent: 8.5h  (was: 8h 20m)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=84613&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84613
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 26/Mar/18 23:56
Start Date: 26/Mar/18 23:56
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #4901: [BEAM-3744] 
Expand Pubsub read API for Python.
URL: https://github.com/apache/beam/pull/4901#issuecomment-376349971
 
 
   Yeah, that was an invalid diff. Please ignore those comments. I'm 
re-reviewing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84613)
Time Spent: 8h 20m  (was: 8h 10m)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3944) Convert beam_PerformanceTests_Python to use Gradle

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-3944:
-
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-3249

> Convert beam_PerformanceTests_Python to use Gradle
> --
>
> Key: BEAM-3944
> URL: https://issues.apache.org/jira/browse/BEAM-3944
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3942) Update performance testing framework to use Gradle.

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-3942:
-
Issue Type: Sub-task  (was: Task)
Parent: BEAM-3249

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3943) Convert existing Java performance tests to use Gradle

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-3943:
-
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-3249

> Convert existing Java performance tests to use Gradle
> -
>
> Key: BEAM-3943
> URL: https://issues.apache.org/jira/browse/BEAM-3943
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Chamikara Jayalath
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3744) Support full PubsubMessages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3744?focusedWorklogId=84610&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84610
 ]

ASF GitHub Bot logged work on BEAM-3744:


Author: ASF GitHub Bot
Created on: 26/Mar/18 23:50
Start Date: 26/Mar/18 23:50
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #4901: [BEAM-3744] Expand 
Pubsub read API for Python.
URL: https://github.com/apache/beam/pull/4901#issuecomment-376348824
 
 
   @chamikaramj , I believe your comments are referring to a different PR?
   retest this please
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84610)
Time Spent: 8h 10m  (was: 8h)

> Support full PubsubMessages
> ---
>
> Key: BEAM-3744
> URL: https://issues.apache.org/jira/browse/BEAM-3744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> Tracking changes to Pubsub support in Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=84611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84611
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 26/Mar/18 23:50
Start Date: 26/Mar/18 23:50
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-376348902
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84611)
Time Spent: 5h 20m  (was: 5h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3942) Update performance testing framework to use Gradle.

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-3942:
-
Issue Type: Task  (was: Improvement)

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3942) Update performance testing framework to use Gradle.

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414785#comment-16414785
 ] 

Chamikara Jayalath commented on BEAM-3942:
--

Assigning to Łukasz who has a PoC for this.

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Chamikara Jayalath
>Priority: Major
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3942) Update performance testing framework to use Gradle.

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3942:


Assignee: Łukasz Gajowy

> Update performance testing framework to use Gradle.
> ---
>
> Key: BEAM-3942
> URL: https://issues.apache.org/jira/browse/BEAM-3942
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Chamikara Jayalath
>Assignee: Łukasz Gajowy
>Priority: Major
>
> This requires performing updates to PerfKitBenchmarker and Beam so that we 
> can execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3251) Migrate Performance Tests Jenkins Jobs to use Gradle

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414783#comment-16414783
 ] 

Chamikara Jayalath commented on BEAM-3251:
--

Subtasks:

https://issues.apache.org/jira/browse/BEAM-3942

https://issues.apache.org/jira/browse/BEAM-3943

https://issues.apache.org/jira/browse/BEAM-3944

 

> Migrate Performance Tests Jenkins Jobs to use Gradle
> 
>
> Key: BEAM-3251
> URL: https://issues.apache.org/jira/browse/BEAM-3251
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Priority: Major
>
> Jenkins performance test jobs are defined using job-dsl here: 
> https://github.com/apache/beam/search?l=Groovy&q=PerfKit&type=&utf8=%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (597e3f9 -> 749a93b)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 597e3f9  Merge pull request #4918 from 
echauchot/BEAM-3892-json-serializable-metric-results
 add 4ed629b  Updated to Ubuntu 16 version of python 2
 new 749a93b  Merge pull request #4945 from apache/aaltay-patch-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/container/Dockerfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4945 from apache/aaltay-patch-1

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 749a93bc17a075a9e2d6c1f061d9dafcaf4af795
Merge: 597e3f9 4ed629b
Author: Ahmet Altay 
AuthorDate: Mon Mar 26 16:41:14 2018 -0700

Merge pull request #4945 from apache/aaltay-patch-1

Updated to Ubuntu 16 version of python 2

 sdks/python/container/Dockerfile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Created] (BEAM-3944) Convert beam_PerformanceTests_Python to use Gradle

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3944:


 Summary: Convert beam_PerformanceTests_Python to use Gradle
 Key: BEAM-3944
 URL: https://issues.apache.org/jira/browse/BEAM-3944
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3943) Convert existing Java performance tests to use Gradle

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3943:


 Summary: Convert existing Java performance tests to use Gradle
 Key: BEAM-3943
 URL: https://issues.apache.org/jira/browse/BEAM-3943
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Chamikara Jayalath






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3942) Update performance testing framework to use Gradle.

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3942:


 Summary: Update performance testing framework to use Gradle.
 Key: BEAM-3942
 URL: https://issues.apache.org/jira/browse/BEAM-3942
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Chamikara Jayalath


This requires performing updates to PerfKitBenchmarker and Beam so that we can 
execute performance tests using Gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3255) Update release process to use Gradle

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414771#comment-16414771
 ] 

Alan Myrvold commented on BEAM-3255:


Added more sub-tasks to complete this:

https://issues.apache.org/jira/browse/BEAM-3938

https://issues.apache.org/jira/browse/BEAM-3939

https://issues.apache.org/jira/browse/BEAM-3940

https://issues.apache.org/jira/browse/BEAM-3941

 

 

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3941) Remove job_beam_Release_NightlySnapshot.groovy and add a nightly gradle jenkins job

[Alan Myrvold (JIRA)]

Alan Myrvold created BEAM-3941:
--

 Summary: Remove job_beam_Release_NightlySnapshot.groovy and add a 
nightly gradle jenkins job
 Key: BEAM-3941
 URL: https://issues.apache.org/jira/browse/BEAM-3941
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Alan Myrvold


The maven job, 
[job_beam_Release_NightlySnapshot.groovy|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy]
 should either be deleted and replaced by a gradle jobm or changed to use 
gradle and not maven.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3940) Convert to gradle, mvn release:branch, release:prepare and release:perform, from the Beam Release Guide

[Alan Myrvold (JIRA)]

Alan Myrvold created BEAM-3940:
--

 Summary: Convert to gradle, mvn release:branch, release:prepare 
and release:perform, from the Beam Release Guide
 Key: BEAM-3940
 URL: https://issues.apache.org/jira/browse/BEAM-3940
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Alan Myrvold


[https://beam.apache.org/contribute/release-guide/] describes using mvn for 
release:branch, release:prepare, and release:perform.

These should all be converted to gradle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3938) Gradle publish task should authenticate when run from jenkins

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414762#comment-16414762
 ] 

Luke Cwik commented on BEAM-3938:
-

https://cwiki.apache.org/confluence/display/INFRA/Gradle+Installations

Says that you can use `asfNexusUsername` and `asfNexusPassword` properties.

> Gradle publish task should authenticate when run from jenkins
> -
>
> Key: BEAM-3938
> URL: https://issues.apache.org/jira/browse/BEAM-3938
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Alan Myrvold
>Priority: Major
>
> ./gradlew publish should be able to write to 
> [https://repository.apache.org/content/repositories/snapshots] when run from 
> jenkins, as the maven 
> [job_beam_Release_NightlySnapshot.groovy|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy]
>  does.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3201) ElasticsearchIO should allow the user to optionally pass id, type and index per document

[Chet Aldrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414759#comment-16414759
 ] 

Chet Aldrich commented on BEAM-3201:


Oh ok, great, good to hear!



> ElasticsearchIO should allow the user to optionally pass id, type and index 
> per document
> 
>
> Key: BEAM-3201
> URL: https://issues.apache.org/jira/browse/BEAM-3201
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-elasticsearch
>Reporter: Etienne Chauchot
>Assignee: Chet Aldrich
>Priority: Major
>
> *Dynamic documents id*: Today the ESIO only inserts the payload of the ES 
> documents. Elasticsearch generates a document id for each record inserted. So 
> each new insertion is considered as a new document. Users want to be able to 
> update documents using the IO. So, for the write part of the IO, users should 
> be able to provide a document id so that they could update already stored 
> documents. Providing an id for the documents could also help the user on 
> indempotency.
> *Dynamic ES type and ES index*: In some cases (streaming pipeline with high 
> throughput) partitioning the PCollection to allow to plug to different ESIO 
> instances (pointing to different index/type) is not very practical, the users 
> would like to be able to set ES index/type per document.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3939) Update release guide to use gradle and not maven

[Alan Myrvold (JIRA)]

Alan Myrvold created BEAM-3939:
--

 Summary: Update release guide to use gradle and not maven
 Key: BEAM-3939
 URL: https://issues.apache.org/jira/browse/BEAM-3939
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Alan Myrvold


[https://beam.apache.org/contribute/release-guide] describes using maven. All 
the steps should be doable by gradle and the release guide updated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3938) Gradle publish task should authenticate when run from jenkins

[Alan Myrvold (JIRA)]

Alan Myrvold created BEAM-3938:
--

 Summary: Gradle publish task should authenticate when run from 
jenkins
 Key: BEAM-3938
 URL: https://issues.apache.org/jira/browse/BEAM-3938
 Project: Beam
  Issue Type: Sub-task
  Components: build-system
Reporter: Alan Myrvold
Assignee: Davor Bonaci


./gradlew publish should be able to write to 
[https://repository.apache.org/content/repositories/snapshots] when run from 
jenkins, as the maven 
[job_beam_Release_NightlySnapshot.groovy|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy]
 does.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3938) Gradle publish task should authenticate when run from jenkins

[Alan Myrvold (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Myrvold reassigned BEAM-3938:
--

Assignee: (was: Davor Bonaci)

> Gradle publish task should authenticate when run from jenkins
> -
>
> Key: BEAM-3938
> URL: https://issues.apache.org/jira/browse/BEAM-3938
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Alan Myrvold
>Priority: Major
>
> ./gradlew publish should be able to write to 
> [https://repository.apache.org/content/repositories/snapshots] when run from 
> jenkins, as the maven 
> [job_beam_Release_NightlySnapshot.groovy|https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy]
>  does.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)