Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #3989

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2403) Provide BigQueryIO with method to indicate Priority (INTERACTIVE/BATCH)

[Andre (JIRA)]

Andre created BEAM-2403:
---

 Summary: Provide BigQueryIO with method to indicate Priority 
(INTERACTIVE/BATCH)
 Key: BEAM-2403
 URL: https://issues.apache.org/jira/browse/BEAM-2403
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-gcp
Affects Versions: 2.0.0
Reporter: Andre
Assignee: Stephen Sisk
Priority: Minor


Currently every BigQuery job runs in BATCH mode when issued from Beam which 
could delay the execution of a scheduled job unnecessarily. By allowing to 
provide the priority INTERACTIVE with BigQueryIO this could be resolved.

See 
https://stackoverflow.com/questions/44198891/change-google-cloud-dataflow-bigquery-priority/44223123



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2404) BigQueryIO reading stalls if no data is returned by query

[Andre (JIRA)]

Andre created BEAM-2404:
---

 Summary: BigQueryIO reading stalls if no data is returned by query
 Key: BEAM-2404
 URL: https://issues.apache.org/jira/browse/BEAM-2404
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.0.0
Reporter: Andre
Assignee: Stephen Sisk


When running a BigQueryIO query that doesn't return any rows (e.g. nothing has 
changed in a delta job) the job seems to stall and nothing happens as no temp 
files are being written which I think might be what it is waiting for. Just 
adding one row to the source table will make the job run through successfully.

Code:
{code:java}
PCollection  rows = p.apply("ReadFromBQ",
 BigQueryIO.read()
 .fromQuery("SELECT * FROM `myproject.dataset.table`")
 .withoutResultFlattening().usingStandardSql());
{code}

Log:
{code:java} 
Jun 02, 2017 9:00:36 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_job_batch-query, projectId=my-project}.
bq show -j --format=prettyjson --project_id=my-project beam_job_batch-query
Jun 02, 2017 9:03:11 AM org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase 
executeExtract
INFO: Starting BigQuery extract job: beam_job_batch-extract
Jun 02, 2017 9:03:12 AM 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl startJob
INFO: Started BigQuery job: {jobId=beam_job_batch-extract, 
projectId=my-project}.
bq show -j --format=prettyjson --project_id=my-project beam_job_batch-extract
Jun 02, 2017 9:04:06 AM org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase 
executeExtract
INFO: BigQuery extract job completed: beam_job_batch-extract
Jun 02, 2017 9:04:08 AM org.apache.beam.sdk.io.FileBasedSource expandFilePattern
INFO: Matched 1 files for pattern 
gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
Jun 02, 2017 9:04:09 AM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern 
gs://my-bucket/tmp/BigQueryExtractTemp/ff594d003c6440a1ad84b9e02858b5c6/.avro
 matched 1 files with total size 9750
{code}  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2328) Introduce Apache Tika Input component

[Sergey Beryozkin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034412#comment-16034412
 ] 

Sergey Beryozkin commented on BEAM-2328:


Hi JB, Tim

re org.json dependencies, FYI, at the moment the only strong Tika dependency is 
tika-core. tika-parsers is a test dependency, it is not needed to compile, the 
current expectation is that the users of the future Tika Input component will 
add a tika-parsers dependency and as such Tika Parsers (including those that 
may depend on org.json) will not make it into the Beam distro. I reckon that 
can make it easier to align with the Tika 2.0-SNAPSHOT effort where a number of 
mainstream parsers (PDF, etc) is represented by individual modules. 

I guess an option to ship all of the tika-bundle with tika-io can also be 
considered but for a start having only a tika-core dependency seems workable to 
me...In this (current) case if the tika-core itself  is org.json free then it 
should not be an issue.



  



> Introduce Apache Tika Input component
> -
>
> Key: BEAM-2328
> URL: https://issues.apache.org/jira/browse/BEAM-2328
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas, sdk-java-extensions
>Reporter: Sergey Beryozkin
>Assignee: Sergey Beryozkin
> Fix For: 2.1.0
>
>
> Apache Tika is a popular project that offers an extensive support for parsing 
> the variety of file formats. It is used in many projects including Lucene and 
> Elastic Search. 
> Supporting a Tika Input (Read) at the Beam level would be of major interest 
> to many users.
> PR is to follow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #2242

[Apache Jenkins Server]

See 


--
[...truncated 68.06 KB...]
2017-06-02T12:02:12.693 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/beanshell/beanshell/2.0b4/beanshell-2.0b4.pom
 (2 KB at 53.0 KB/sec)
2017-06-02T12:02:12.695 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-i18n/1.0-beta-6/plexus-i18n-1.0-beta-6.pom
2017-06-02T12:02:12.720 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-i18n/1.0-beta-6/plexus-i18n-1.0-beta-6.pom
 (771 B at 30.1 KB/sec)
2017-06-02T12:02:12.722 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.4/plexus-components-1.1.4.pom
2017-06-02T12:02:12.748 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.4/plexus-components-1.1.4.pom
 (3 KB at 77.8 KB/sec)
2017-06-02T12:02:12.750 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-8/plexus-container-default-1.0-alpha-8.pom
2017-06-02T12:02:12.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-8/plexus-container-default-1.0-alpha-8.pom
 (8 KB at 262.7 KB/sec)
2017-06-02T12:02:12.779 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-testing/maven-plugin-testing-harness/1.3/maven-plugin-testing-harness-1.3.pom
2017-06-02T12:02:12.806 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-testing/maven-plugin-testing-harness/1.3/maven-plugin-testing-harness-1.3.pom
 (4 KB at 124.7 KB/sec)
2017-06-02T12:02:12.808 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-testing/maven-plugin-testing/1.3/maven-plugin-testing-1.3.pom
2017-06-02T12:02:12.833 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-testing/maven-plugin-testing/1.3/maven-plugin-testing-1.3.pom
 (10 KB at 358.0 KB/sec)
2017-06-02T12:02:12.837 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.2/plexus-archiver-2.2.pom
2017-06-02T12:02:12.862 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.2/plexus-archiver-2.2.pom
 (4 KB at 125.5 KB/sec)
2017-06-02T12:02:12.864 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.20/plexus-components-1.1.20.pom
2017-06-02T12:02:12.890 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.20/plexus-components-1.1.20.pom
 (3 KB at 113.3 KB/sec)
2017-06-02T12:02:12.892 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.1/plexus-3.1.pom
2017-06-02T12:02:12.919 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.1/plexus-3.1.pom
 (19 KB at 673.3 KB/sec)
2017-06-02T12:02:12.923 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.4/plexus-io-2.0.4.pom
2017-06-02T12:02:12.949 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-io/2.0.4/plexus-io-2.0.4.pom
 (2 KB at 65.1 KB/sec)
2017-06-02T12:02:12.951 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.19/plexus-components-1.1.19.pom
2017-06-02T12:02:12.976 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-components/1.1.19/plexus-components-1.1.19.pom
 (3 KB at 109.8 KB/sec)
2017-06-02T12:02:12.977 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.0.1/plexus-3.0.1.pom
2017-06-02T12:02:13.005 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus/3.0.1/plexus-3.0.1.pom
 (19 KB at 649.2 KB/sec)
2017-06-02T12:02:13.016 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/mojo/extra-enforcer-rules/1.0-beta-6/extra-enforcer-rules-1.0-beta-6.jar
2017-06-02T12:02:13.016 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
2017-06-02T12:02:13.017 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-dependency-tree/2.2/maven-dependency-tree-2.2.jar
2017-06-02T12:02:13.017 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar
2017-06-02T12:02:13.018 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-container-default/1.0-alpha-9/plexus-container-default-1.0-alpha-9.jar
2017-06-02T12:02:13.053 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model/2.0.9/maven-model-2.0.9.jar
 (86 KB at 2304.7 KB/sec)
2017-06-02T12:02:13.054 [INFO] Downloading: 
https://repo.maven.apache

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #84

[Apache Jenkins Server]

See 


--
[...truncated 351.89 KB...]
2017-06-02T12:02:07.178 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
2017-06-02T12:02:07.187 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.pom
 (2 KB at 177.3 KB/sec)
2017-06-02T12:02:07.191 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
2017-06-02T12:02:07.201 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_parent/2.0.15/error_prone_parent-2.0.15.pom
 (6 KB at 556.2 KB/sec)
2017-06-02T12:02:07.205 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
2017-06-02T12:02:07.213 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.pom
 (5 KB at 523.2 KB/sec)
2017-06-02T12:02:07.217 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
2017-06-02T12:02:07.226 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.pom
 (2 KB at 189.7 KB/sec)
2017-06-02T12:02:07.229 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
2017-06-02T12:02:07.242 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.pom
 (2 KB at 112.5 KB/sec)
2017-06-02T12:02:07.246 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
2017-06-02T12:02:07.256 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.pom
 (3 KB at 259.3 KB/sec)
2017-06-02T12:02:07.261 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
2017-06-02T12:02:07.270 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java-util/3.2.0/protobuf-java-util-3.2.0.pom
 (5 KB at 461.0 KB/sec)
2017-06-02T12:02:07.275 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom
2017-06-02T12:02:07.283 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.7/gson-2.7.pom 
(2 KB at 176.4 KB/sec)
2017-06-02T12:02:07.287 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
2017-06-02T12:02:07.297 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/gson/gson-parent/2.7/gson-parent-2.7.pom
 (4 KB at 349.6 KB/sec)
2017-06-02T12:02:07.302 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
2017-06-02T12:02:07.313 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf-lite/1.2.0/grpc-protobuf-lite-1.2.0.pom
 (3 KB at 188.6 KB/sec)
2017-06-02T12:02:07.318 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
2017-06-02T12:02:07.329 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-stub/1.2.0/grpc-stub-1.2.0.pom
 (3 KB at 184.5 KB/sec)
2017-06-02T12:02:07.340 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/protobuf/protobuf-java/3.2.0/protobuf-java-3.2.0.jar
2017-06-02T12:02:07.340 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-core/1.2.0/grpc-core-1.2.0.jar
2017-06-02T12:02:07.340 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
2017-06-02T12:02:07.342 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.1/jsr305-3.0.1.jar
2017-06-02T12:02:07.342 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
2017-06-02T12:02:07.356 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/errorprone/error_prone_annotations/2.0.15/error_prone_annotations-2.0.15.jar
 (11 KB at 624.2 KB/sec)
2017-06-02T12:02:07.356 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/instrumentation/instrumentation-api/0.3.0/instrumentation-api-0.3.0.jar
2017-06-02T12:02:07.358 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-context/1.2.0/grpc-context-1.2.0.jar
 (20 KB at 1097.1 KB/sec)
2017-06-02T12:02:07.358 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/grpc/grpc-protobuf/1.2.0/grpc-protobuf-1.2.0.jar
2017-06-02T12:02:07.361 [INFO] Downloa

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #34

[Apache Jenkins Server]

See 


--
[...truncated 893.98 KB...]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(6953fd2258fd57e2): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
a

Jenkins build is back to normal : beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #34

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3990

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-1688) Getting listener timeout after waiting for [10000] ms from ElasticsearchIO

[Etienne Chauchot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Etienne Chauchot reassigned BEAM-1688:
--

Assignee: Etienne Chauchot  (was: Jean-Baptiste Onofré)

> Getting listener timeout after waiting for [1] ms from ElasticsearchIO
> --
>
> Key: BEAM-1688
> URL: https://issues.apache.org/jira/browse/BEAM-1688
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.5.0
> Environment: Linux,  ElasticSearch 5.2.1
>Reporter: Sathish Jayaraman
>Assignee: Etienne Chauchot
>Priority: Minor
>
> I am trying to test a simple Apache Beam code with source as Elasticsearch. I 
> modified Beam's MinimalWordCount example to include source as Elasticsearch 
> instead of TextIO. 
> When I run the code,  I get below error, 
> An exception occured while executing the Java class. null: 
> InvocationTargetException: java.io.IOException: listener timeout after 
> waiting for [1] ms
> I debugged ElasticsearchIO.java and could see everything works fine, the 
> Elasicsearch client is built & data from index is being retrieved by the 
> code. But ParDo function after read transform does not get executed at all. 
> The Elasticsearch client keeps waiting and finally gets a timeout error.
> Any idea about the issue? 
> Have raised issue here too 
> http://stackoverflow.com/questions/42720857/getting-listener-timeout-after-waiting-for-1-ms-from-elasticsearchio



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: [BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

[lcwik]

[BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

This closes #3271


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7a075cc3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7a075cc3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7a075cc3

Branch: refs/heads/master
Commit: 7a075cc349e1c96f44d058583e803cd353fe40ec
Parents: 217f085 fa23fa2
Author: Lukasz Cwik 
Authored: Fri Jun 2 06:55:43 2017 -0700
Committer: Lukasz Cwik 
Committed: Fri Jun 2 06:55:43 2017 -0700

--
 sdks/common/fn-api/pom.xml  |  5 
 .../fn-api/src/main/proto/beam_fn_api.proto | 25 
 sdks/java/core/pom.xml  |  7 ++
 3 files changed, 33 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 217f085f2 -> 7a075cc34


[BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

This is towards removing the Fn API constructs once code has been migrated.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fa23fa2b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fa23fa2b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fa23fa2b

Branch: refs/heads/master
Commit: fa23fa2b0080461d0d860dc219283a9f90ebdb23
Parents: 217f085
Author: Luke Cwik 
Authored: Wed May 31 10:10:06 2017 -0700
Committer: Lukasz Cwik 
Committed: Thu Jun 1 21:58:55 2017 -0700

--
 sdks/common/fn-api/pom.xml  |  5 
 .../fn-api/src/main/proto/beam_fn_api.proto | 25 
 sdks/java/core/pom.xml  |  7 ++
 3 files changed, 33 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fa23fa2b/sdks/common/fn-api/pom.xml
--
diff --git a/sdks/common/fn-api/pom.xml b/sdks/common/fn-api/pom.xml
index ded9559..77a9ba5 100644
--- a/sdks/common/fn-api/pom.xml
+++ b/sdks/common/fn-api/pom.xml
@@ -82,6 +82,11 @@
 
   
 
+  org.apache.beam
+  beam-sdks-common-runner-api
+
+
+
   com.google.protobuf
   protobuf-java
 

http://git-wip-us.apache.org/repos/asf/beam/blob/fa23fa2b/sdks/common/fn-api/src/main/proto/beam_fn_api.proto
--
diff --git a/sdks/common/fn-api/src/main/proto/beam_fn_api.proto 
b/sdks/common/fn-api/src/main/proto/beam_fn_api.proto
index 79e1872..9fe2b2f 100644
--- a/sdks/common/fn-api/src/main/proto/beam_fn_api.proto
+++ b/sdks/common/fn-api/src/main/proto/beam_fn_api.proto
@@ -37,6 +37,7 @@ package org.apache.beam.fn.v1;
 option java_package = "org.apache.beam.fn.v1";
 option java_outer_classname = "BeamFnApi";
 
+import "beam_runner_api.proto";
 import "google/protobuf/any.proto";
 import "google/protobuf/timestamp.proto";
 
@@ -264,19 +265,35 @@ message RegisterRequest {
 message RegisterResponse {
 }
 
-// A descriptor of references used when processing a bundle.
-// Stable
+// Definitions that should be used to construct the bundle processing graph.
 message ProcessBundleDescriptor {
   // (Required) A pipeline level unique id which can be used as a reference to
   // refer to this.
   string id = 1;
 
-  // (Required) A list of primitive transforms that should
+  // (Deprecated) A list of primitive transforms that should
   // be used to construct the bundle processing graph.
   repeated PrimitiveTransform primitive_transform = 2;
 
-  // (Required) The set of all coders referenced in this bundle.
+  // (Deprecated) The set of all coders referenced in this bundle.
   repeated Coder coders = 4;
+
+  // (Required) A map from pipeline-scoped id to PTransform.
+  map transforms = 5;
+
+  // (Required) A map from pipeline-scoped id to PCollection.
+  map pcollections = 6;
+
+  // (Required) A map from pipeline-scoped id to WindowingStrategy.
+  map 
windowing_strategies = 7;
+
+  // (Required) A map from pipeline-scoped id to Coder.
+  // TODO: Rename to "coders" once deprecated coders field is removed. Unique
+  // name is choosen to make it an easy search/replace
+  map codersyyy = 8;
+
+  // (Required) A map from pipeline-scoped id to Environment.
+  map environments = 9;
 }
 
 // A request to process a given bundle.

http://git-wip-us.apache.org/repos/asf/beam/blob/fa23fa2b/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index 3c68c96..11b68e6 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -305,6 +305,13 @@
   beam-sdks-common-fn-api
   test-jar
   test
+  
+
+
+  org.apache.beam
+  beam-sdks-common-runner-api
+
+  
 
 
 

[GitHub] beam pull request #3271: [BEAM-1348] Add Runner API constructs to ProcessBun...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1348) Model the Fn Api

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034726#comment-16034726
 ] 

ASF GitHub Bot commented on BEAM-1348:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3271


> Model the Fn Api
> 
>
> Key: BEAM-1348
> URL: https://issues.apache.org/jira/browse/BEAM-1348
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-fn-api
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Create a proto representation of the services and data types required to 
> execute the Fn Api.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3285: Fix race condition when outputting pushed-back elem...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/beam/pull/3285

Fix race condition when outputting pushed-back elements in Flink Runner

This affected the Flink Streaming Runner DoFnOperator. The recent fix of
emitting pushed-back data when receiving a watermark on the first input
put the emission at the end of the method. This can cause the emitted
data to become late. The fix is to move the pushed-back element emission
to the start of the method.

`CombineTest.testSessionsCombineWithContext()` was flaky, because of this. 

R: @JingsongLi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/beam fix-pushed-back-values-flakyness

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3285.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3285


commit 61404c878fa8ccd23e2b3f1c8d2541593f01ad51
Author: Aljoscha Krettek 
Date:   2017-06-02T13:57:01Z

Fix race condition when outputting pushed-back elements in Flink Runner

This affected the Flink Streaming Runner DoFnOperator. The recent fix of
emitting pushed-back data when receiving a watermark on the first input
put the emission at the end of the method. This can cause the emitted
data to become late. The fix is to move the pushed-back element emission
to the start of the method.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #2243

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3256: [BEAM-2314] Add ValidatesRunner test for merging cu...

[echauchot]

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3256


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2314) Add ValidatesRunner test for custom merging windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034857#comment-16034857
 ] 

ASF GitHub Bot commented on BEAM-2314:
--

Github user echauchot closed the pull request at:

https://github.com/apache/beam/pull/3256


> Add ValidatesRunner test for custom merging windows
> ---
>
> Key: BEAM-2314
> URL: https://issues.apache.org/jira/browse/BEAM-2314
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Aljoscha Krettek
>Assignee: Etienne Chauchot
>
> While working on BEAM-160 it was discovered that not all Runners supported 
> custom merging {{WindowFns}}. This is now fixed but we should still add a 
> {{ValidatesRunner}} test for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3286: [BEAM-2314] Add ValidatesRunner test for merging cu...

[echauchot]

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3286

[BEAM-2314] Add ValidatesRunner test for merging custom windows

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
@aljoscha I reopened a new PR to test that jenkins integration is working 
again as requested
R: @aljoscha
CC: @jbonofre

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam BEAM-2314_merging_windows_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3286


commit 3be3911e15c58b4020ca0f32770dd99730ae6637
Author: Etienne Chauchot 
Date:   2017-05-29T10:05:51Z

[BEAM-2314] Add ValidatesRunner test for merging custom windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2314) Add ValidatesRunner test for custom merging windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034874#comment-16034874
 ] 

ASF GitHub Bot commented on BEAM-2314:
--

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3286

[BEAM-2314] Add ValidatesRunner test for merging custom windows

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
@aljoscha I reopened a new PR to test that jenkins integration is working 
again as requested
R: @aljoscha
CC: @jbonofre

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam BEAM-2314_merging_windows_test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3286.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3286


commit 3be3911e15c58b4020ca0f32770dd99730ae6637
Author: Etienne Chauchot 
Date:   2017-05-29T10:05:51Z

[BEAM-2314] Add ValidatesRunner test for merging custom windows




> Add ValidatesRunner test for custom merging windows
> ---
>
> Key: BEAM-2314
> URL: https://issues.apache.org/jira/browse/BEAM-2314
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Aljoscha Krettek
>Assignee: Etienne Chauchot
>
> While working on BEAM-160 it was discovered that not all Runners supported 
> custom merging {{WindowFns}}. This is now fixed but we should still add a 
> {{ValidatesRunner}} test for this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2375) upgrade beam-version to 2.0.0

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-2375:
--

Assignee: Jean-Baptiste Onofré  (was: James Xu)

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2375) upgrade beam-version to 2.0.0

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034898#comment-16034898
 ] 

Jean-Baptiste Onofré commented on BEAM-2375:


I'm proposing a PR with a complete rebase, including code change.

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3008

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3287: [BEAM-2375] Rebase SQL DSL branch with master

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam/pull/3287

[BEAM-2375] Rebase SQL DSL branch with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam SQL_REBASE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3287






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2375) upgrade beam-version to 2.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034970#comment-16034970
 ] 

ASF GitHub Bot commented on BEAM-2375:
--

GitHub user jbonofre opened a pull request:

https://github.com/apache/beam/pull/3287

[BEAM-2375] Rebase SQL DSL branch with master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`.
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/beam SQL_REBASE

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3287






> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[5/5] beam git commit: Add known window serialization for Java.

[robertwb]

Add known window serialization for Java.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b490e84e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b490e84e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b490e84e

Branch: refs/heads/master
Commit: b490e84ef0b4e56cabc091cfe2cc42f8f1e69caa
Parents: de75786
Author: Robert Bradshaw 
Authored: Thu May 25 13:26:02 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 2 09:16:43 2017 -0700

--
 .../WindowingStrategyTranslation.java   | 105 +++
 .../WindowingStrategyTranslationTest.java   |   9 ++
 .../src/main/proto/beam_known_payloads.proto|  53 --
 .../src/main/proto/standard_window_fns.proto|  53 ++
 sdks/python/apache_beam/transforms/window.py|  14 +--
 5 files changed, 154 insertions(+), 80 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b490e84e/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
index a226624..718efe7 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
@@ -30,7 +30,7 @@ import 
org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.OutputTime;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
-import org.apache.beam.sdk.common.runner.v1.RunnerApiPayloads;
+import org.apache.beam.sdk.common.runner.v1.StandardWindowFns;
 import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.Sessions;
@@ -165,6 +165,9 @@ public class WindowingStrategyTranslation implements 
Serializable {
   // This URN says that the WindowFn is just a UDF blob the Java SDK 
understands
   // TODO: standardize such things
   public static final String SERIALIZED_JAVA_WINDOWFN_URN = 
"beam:windowfn:javasdk:v0.1";
+  public static final String OLD_SERIALIZED_JAVA_WINDOWFN_URN = 
"urn:beam:windowfn:javasdk:0.1";
+  // Remove this once the dataflow worker understands all the above formats.
+  private static final boolean USE_OLD_SERIALIZED_JAVA_WINDOWFN_URN = true;
 
   /**
* Converts a {@link WindowFn} into a {@link 
RunnerApi.MessageWithComponents} where {@link
@@ -173,19 +176,80 @@ public class WindowingStrategyTranslation implements 
Serializable {
*/
   public static SdkFunctionSpec toProto(
   WindowFn windowFn, @SuppressWarnings("unused") SdkComponents 
components) {
-return SdkFunctionSpec.newBuilder()
-// TODO: Set environment ID
-.setSpec(
-FunctionSpec.newBuilder()
-.setUrn(SERIALIZED_JAVA_WINDOWFN_URN)
-.setParameter(
-Any.pack(
-BytesValue.newBuilder()
-.setValue(
-ByteString.copyFrom(
-
SerializableUtils.serializeToByteArray(windowFn)))
-.build(
-.build();
+// TODO: Set environment IDs
+if (USE_OLD_SERIALIZED_JAVA_WINDOWFN_URN) {
+  return SdkFunctionSpec.newBuilder()
+  .setSpec(
+  FunctionSpec.newBuilder()
+  .setUrn(OLD_SERIALIZED_JAVA_WINDOWFN_URN)
+  .setParameter(
+  Any.pack(
+  BytesValue.newBuilder()
+  .setValue(
+  ByteString.copyFrom(
+  
SerializableUtils.serializeToByteArray(windowFn)))
+  .build(
+  .build();
+} else if (windowFn instanceof GlobalWindows) {
+  return SdkFunctionSpec.newBuilder()
+  .setSpec(FunctionSpec.newBuilder().setUrn(GLOBAL_WINDOWS_FN))
+  .build();
+} else if (windowFn instanceof FixedWindows) {
+  return SdkFunctionSpec.newBuilder()
+  .setSpec(
+  FunctionSpec.newBuilder()
+  .setUrn(FIXED_WINDOWS_FN)
+  .setParameter(
+  Any.pack(
+  StandardWindowFn

[2/5] beam git commit: Ignore all proto generated files.

[robertwb]

Ignore all proto generated files.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4c20bfa9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4c20bfa9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4c20bfa9

Branch: refs/heads/master
Commit: 4c20bfa9929664031f3fce67a723d09fc629ad3a
Parents: b490e84
Author: Robert Bradshaw 
Authored: Wed May 31 15:53:47 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 2 09:16:43 2017 -0700

--
 sdks/python/run_pylint.sh | 7 +--
 1 file changed, 1 insertion(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4c20bfa9/sdks/python/run_pylint.sh
--
diff --git a/sdks/python/run_pylint.sh b/sdks/python/run_pylint.sh
index 4ef3e7f..7808136 100755
--- a/sdks/python/run_pylint.sh
+++ b/sdks/python/run_pylint.sh
@@ -46,12 +46,7 @@ EXCLUDED_GENERATED_FILES=(
 "apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py"
 "apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py"
 "apache_beam/coders/proto2_coder_test_messages_pb2.py"
-"apache_beam/runners/api/beam_fn_api_pb2.py"
-"apache_beam/runners/api/beam_fn_api_pb2_grpc.py"
-"apache_beam/runners/api/beam_runner_api_pb2.py"
-"apache_beam/runners/api/beam_runner_api_pb2_grpc.py"
-"apache_beam/runners/api/beam_known_payloads_pb2.py"
-"apache_beam/runners/api/beam_known_payloads_pb2_grpc.py"
+apache_beam/runners/api/*pb2*.py
 )
 
 FILES_TO_IGNORE=""

[3/5] beam git commit: Increase dataflow worker version.

[robertwb]

Increase dataflow worker version.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/99907b94
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/99907b94
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/99907b94

Branch: refs/heads/master
Commit: 99907b94c074d4dc2ac3911b690da6790056412a
Parents: 4c20bfa
Author: Robert Bradshaw 
Authored: Thu Jun 1 16:01:56 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 2 09:16:43 2017 -0700

--
 sdks/python/apache_beam/runners/dataflow/internal/dependency.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/99907b94/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
index e69c8d7..3a0ff46 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
@@ -73,7 +73,7 @@ from apache_beam.options.pipeline_options import SetupOptions
 # Update this version to the next version whenever there is a change that will
 # require changes to the execution environment.
 # This should be in the beam-[version]-[date] format, date is optional.
-BEAM_CONTAINER_VERSION = 'beam-2.1.0-20170518'
+BEAM_CONTAINER_VERSION = 'beam-2.1.0-20170601'
 
 # Standard file names used for staging files.
 WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'

[4/5] beam git commit: Closes #3222

[robertwb]

Closes #3222


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f9428c3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f9428c3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f9428c3

Branch: refs/heads/master
Commit: 2f9428c3e5e648d147fa68afdd13db7f6bfbcb0b
Parents: 7a075cc 99907b9
Author: Robert Bradshaw 
Authored: Fri Jun 2 09:16:43 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 2 09:16:43 2017 -0700

--
 pom.xml |   6 +
 runners/core-construction-java/pom.xml  |   5 +
 .../WindowingStrategyTranslation.java   | 149 +++
 .../WindowingStrategyTranslationTest.java   |   9 ++
 .../src/main/proto/standard_window_fns.proto|  53 +++
 .../runners/dataflow/dataflow_runner.py |  39 -
 .../runners/dataflow/dataflow_runner_test.py|  11 ++
 .../runners/dataflow/internal/dependency.py |   2 +-
 sdks/python/apache_beam/transforms/window.py|  57 ---
 sdks/python/apache_beam/utils/proto_utils.py|   6 +
 sdks/python/apache_beam/utils/urns.py   |  10 +-
 sdks/python/run_pylint.sh   |   5 +-
 12 files changed, 292 insertions(+), 60 deletions(-)
--

[1/5] beam git commit: Unify Java and Python WindowingStrategy representations.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 7a075cc34 -> 2f9428c3e


Unify Java and Python WindowingStrategy representations.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de757860
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de757860
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de757860

Branch: refs/heads/master
Commit: de757860945d5966a51173c54d29d0a733e66686
Parents: 7a075cc
Author: Robert Bradshaw 
Authored: Wed May 24 17:23:31 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Jun 2 09:16:42 2017 -0700

--
 pom.xml |  6 ++
 runners/core-construction-java/pom.xml  |  5 ++
 .../WindowingStrategyTranslation.java   | 60 ++--
 .../src/main/proto/beam_known_payloads.proto| 53 +
 .../runners/dataflow/dataflow_runner.py | 39 -
 .../runners/dataflow/dataflow_runner_test.py| 11 
 sdks/python/apache_beam/transforms/window.py| 57 ---
 sdks/python/apache_beam/utils/proto_utils.py|  6 ++
 sdks/python/apache_beam/utils/urns.py   | 10 ++--
 sdks/python/run_pylint.sh   |  2 +
 10 files changed, 206 insertions(+), 43 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/de757860/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3e302e7..805a8d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -945,6 +945,12 @@
   
 
   
+com.google.protobuf
+protobuf-java-util
+${protobuf.version}
+  
+
+  
 com.google.api.grpc
 grpc-google-common-protos
 ${grpc-google-common-protos.version}

http://git-wip-us.apache.org/repos/asf/beam/blob/de757860/runners/core-construction-java/pom.xml
--
diff --git a/runners/core-construction-java/pom.xml 
b/runners/core-construction-java/pom.xml
index 7eaa6f3..67951e9 100644
--- a/runners/core-construction-java/pom.xml
+++ b/runners/core-construction-java/pom.xml
@@ -70,6 +70,11 @@
 
 
 
+  com.google.protobuf
+  protobuf-java-util
+
+
+
   com.google.code.findbugs
   jsr305
 

http://git-wip-us.apache.org/repos/asf/beam/blob/de757860/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
index e92565f..a226624 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WindowingStrategyTranslation.java
@@ -17,12 +17,12 @@
  */
 package org.apache.beam.runners.core.construction;
 
-import static com.google.common.base.Preconditions.checkArgument;
-
 import com.google.protobuf.Any;
 import com.google.protobuf.ByteString;
 import com.google.protobuf.BytesValue;
 import com.google.protobuf.InvalidProtocolBufferException;
+import com.google.protobuf.util.Durations;
+import com.google.protobuf.util.Timestamps;
 import java.io.IOException;
 import java.io.Serializable;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi;
@@ -30,6 +30,11 @@ import 
org.apache.beam.sdk.common.runner.v1.RunnerApi.Components;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.FunctionSpec;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.OutputTime;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.SdkFunctionSpec;
+import org.apache.beam.sdk.common.runner.v1.RunnerApiPayloads;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Sessions;
+import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
 import org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
 import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
@@ -153,9 +158,13 @@ public class WindowingStrategyTranslation implements 
Serializable {
 }
   }
 
-  // This URN says that the WindowFn is just a UDF blob the indicated SDK 
understands
+  public static final String GLOBAL_WINDOWS_FN = 
"beam:windowfn:global_windows:v0.1";
+  public static final String FIXED_WINDOWS_FN = 
"beam:windowfn:fixed_windows:v0.1";
+  public static final String SLIDI

[GitHub] beam pull request #3222: [BEAM-115] Unify Java and Python WindowingStrategy ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3222


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034974#comment-16034974
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3222


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3009

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #3267

[davor]

This closes #3267


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f1386c1c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f1386c1c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f1386c1c

Branch: refs/heads/master
Commit: f1386c1cb2bc39b571ee6e94acc1612d22ae69b6
Parents: 2f9428c c5918b2
Author: Davor Bonaci 
Authored: Fri Jun 2 10:01:00 2017 -0700
Committer: Davor Bonaci 
Committed: Fri Jun 2 10:01:00 2017 -0700

--
 .../beam/sdk/extensions/joinlibrary/Join.java   |  65 ++-
 .../joinlibrary/OuterFullJoinTest.java  | 179 +++
 2 files changed, 243 insertions(+), 1 deletion(-)
--

[1/2] beam git commit: [BEAM-2378] support FULL OUTER JOIN

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 2f9428c3e -> f1386c1cb


[BEAM-2378] support FULL OUTER JOIN


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c5918b2f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c5918b2f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c5918b2f

Branch: refs/heads/master
Commit: c5918b2f7ce36c755e2a285c42cc6b628b9ee319
Parents: 2f9428c
Author: James Xu 
Authored: Wed May 31 10:28:55 2017 +0800
Committer: Davor Bonaci 
Committed: Fri Jun 2 10:00:57 2017 -0700

--
 .../beam/sdk/extensions/joinlibrary/Join.java   |  65 ++-
 .../joinlibrary/OuterFullJoinTest.java  | 179 +++
 2 files changed, 243 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c5918b2f/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
--
diff --git 
a/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
 
b/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
index f4e6ccb..9acb048 100644
--- 
a/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
+++ 
b/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java
@@ -141,7 +141,7 @@ public class Join {
* @param  Type of the values for the left collection.
* @param  Type of the values for the right collection.
* @return A joined collection of KV where Key is the key and value is a
-   * KV where Key is of type V1 and Value is type V2. Keys that
+   * KV where Key is of type V1 and Value is type V2. Values that
* should be null or empty is replaced with nullValue.
*/
   public static  PCollection>> rightOuterJoin(
@@ -184,4 +184,67 @@ public class Join {
KvCoder.of(((KvCoder) 
leftCollection.getCoder()).getValueCoder(),
   ((KvCoder) 
rightCollection.getCoder()).getValueCoder(;
   }
+
+  /**
+   * Full Outer Join of two collections of KV elements.
+   * @param leftCollection Left side collection to join.
+   * @param rightCollection Right side collection to join.
+   * @param leftNullValue Value to use as null value when left side do not 
match right side.
+   * @param rightNullValue Value to use as null value when right side do not 
match right side.
+   * @param  Type of the key for both collections
+   * @param  Type of the values for the left collection.
+   * @param  Type of the values for the right collection.
+   * @return A joined collection of KV where Key is the key and value is a
+   * KV where Key is of type V1 and Value is type V2. Values that
+   * should be null or empty is replaced with 
leftNullValue/rightNullValue.
+   */
+  public static  PCollection>> fullOuterJoin(
+  final PCollection> leftCollection,
+  final PCollection> rightCollection,
+  final V1 leftNullValue, final V2 rightNullValue) {
+checkNotNull(leftCollection);
+checkNotNull(rightCollection);
+checkNotNull(leftNullValue);
+checkNotNull(rightNullValue);
+
+final TupleTag v1Tuple = new TupleTag<>();
+final TupleTag v2Tuple = new TupleTag<>();
+
+PCollection> coGbkResultCollection =
+KeyedPCollectionTuple.of(v1Tuple, leftCollection)
+.and(v2Tuple, rightCollection)
+.apply(CoGroupByKey.create());
+
+return coGbkResultCollection.apply(ParDo.of(
+new DoFn, KV>>() {
+  @ProcessElement
+  public void processElement(ProcessContext c) {
+KV e = c.element();
+
+Iterable leftValuesIterable = e.getValue().getAll(v1Tuple);
+Iterable rightValuesIterable = e.getValue().getAll(v2Tuple);
+if (leftValuesIterable.iterator().hasNext()
+&& rightValuesIterable.iterator().hasNext()) {
+  for (V2 rightValue : rightValuesIterable) {
+for (V1 leftValue : leftValuesIterable) {
+  c.output(KV.of(e.getKey(), KV.of(leftValue, rightValue)));
+}
+  }
+} else if (leftValuesIterable.iterator().hasNext()
+&& !rightValuesIterable.iterator().hasNext()) {
+  for (V1 leftValue : leftValuesIterable) {
+c.output(KV.of(e.getKey(), KV.of(leftValue, rightNullValue)));
+  }
+} else if (!leftValuesIterable.iterator().hasNext()
+&& rightValuesIterable.iterator().hasNext()) {
+  for (V2 rightValue : rightValuesIterable) {
+c.output(KV.of(e.getKey(), KV.of(leftNullV

[GitHub] beam pull request #3267: [BEAM-2378] support FULL OUTER JOIN

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3267


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2378) extend join-library to support FULL OUTER JOIN

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035046#comment-16035046
 ] 

ASF GitHub Bot commented on BEAM-2378:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3267


> extend join-library to support FULL OUTER JOIN
> --
>
> Key: BEAM-2378
> URL: https://issues.apache.org/jira/browse/BEAM-2378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: James Xu
>Assignee: James Xu
>
> Currently INNER, LEFT OUTER, RIGHT OUTER are supported, would be nice to 
> support FULL OUTER JOIN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3010

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #3011

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #35

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

[robertwb] Unify Java and Python WindowingStrategy representations.

[robertwb] Add known window serialization for Java.

[robertwb] Ignore all proto generated files.

[robertwb] Increase dataflow worker version.

[davor] [BEAM-2378] support FULL OUTER JOIN

--
[...truncated 902.03 KB...]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(89af2474d71a1466): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #35

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

[robertwb] Unify Java and Python WindowingStrategy representations.

[robertwb] Add known window serialization for Java.

[robertwb] Ignore all proto generated files.

[robertwb] Increase dataflow worker version.

[davor] [BEAM-2378] support FULL OUTER JOIN

--
[...truncated 900.12 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(5a9adb8f877ed779): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.co

[1/3] beam git commit: Removes unused PropertyNames

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master f1386c1cb -> eee6726aa


Removes unused PropertyNames


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e201c37c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e201c37c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e201c37c

Branch: refs/heads/master
Commit: e201c37cbbb2c1f979d3e202b8e0d2d6564f3d2d
Parents: c8fce2f
Author: Eugene Kirpichov 
Authored: Fri May 26 16:11:20 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jun 2 10:54:26 2017 -0700

--
 .../runners/dataflow/util/PropertyNames.java| 46 
 1 file changed, 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e201c37c/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PropertyNames.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PropertyNames.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PropertyNames.java
index c8c9903..f82f1f1 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PropertyNames.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PropertyNames.java
@@ -21,67 +21,31 @@ package org.apache.beam.runners.dataflow.util;
  * Constant property names used by the SDK in CloudWorkflow specifications.
  */
 public class PropertyNames {
-  public static final String ALLOWED_ENCODINGS = "allowed_encodings";
-  public static final String APPEND_TRAILING_NEWLINES = 
"append_trailing_newlines";
-  public static final String BIGQUERY_CREATE_DISPOSITION = 
"create_disposition";
-  public static final String BIGQUERY_DATASET = "dataset";
-  public static final String BIGQUERY_PROJECT = "project";
-  public static final String BIGQUERY_SCHEMA = "schema";
-  public static final String BIGQUERY_TABLE = "table";
-  public static final String BIGQUERY_QUERY = "bigquery_query";
-  public static final String BIGQUERY_FLATTEN_RESULTS = 
"bigquery_flatten_results";
-  public static final String BIGQUERY_USE_LEGACY_SQL = 
"bigquery_use_legacy_sql";
-  public static final String BIGQUERY_WRITE_DISPOSITION = "write_disposition";
-  public static final String BIGQUERY_EXPORT_FORMAT = "bigquery_export_format";
-  public static final String BIGQUERY_EXPORT_SCHEMA = "bigquery_export_schema";
   public static final String CO_GBK_RESULT_SCHEMA = "co_gbk_result_schema";
-  public static final String COMBINE_FN = "combine_fn";
   public static final String COMPONENT_ENCODINGS = "component_encodings";
-  public static final String COMPRESSION_TYPE = "compression_type";
   public static final String CUSTOM_SOURCE_FORMAT = "custom_source";
-  public static final String CONCAT_SOURCE_SOURCES = "sources";
-  public static final String CONCAT_SOURCE_BASE_SPECS = "base_specs";
   public static final String SOURCE_STEP_INPUT = "custom_source_step_input";
   public static final String SOURCE_SPEC = "spec";
   public static final String SOURCE_METADATA = "metadata";
   public static final String SOURCE_DOES_NOT_NEED_SPLITTING = 
"does_not_need_splitting";
-  public static final String SOURCE_PRODUCES_SORTED_KEYS = 
"produces_sorted_keys";
   public static final String SOURCE_IS_INFINITE = "is_infinite";
   public static final String SOURCE_ESTIMATED_SIZE_BYTES = 
"estimated_size_bytes";
-  public static final String ELEMENT = "element";
-  public static final String ELEMENTS = "elements";
   public static final String ENCODING = "encoding";
-  public static final String ENCODING_ID = "encoding_id";
-  public static final String END_INDEX = "end_index";
-  public static final String END_OFFSET = "end_offset";
-  public static final String END_SHUFFLE_POSITION = "end_shuffle_position";
   public static final String ENVIRONMENT_VERSION_JOB_TYPE_KEY = "job_type";
   public static final String ENVIRONMENT_VERSION_MAJOR_KEY = "major";
-  public static final String FILENAME = "filename";
-  public static final String FILENAME_PREFIX = "filename_prefix";
-  public static final String FILENAME_SUFFIX = "filename_suffix";
-  public static final String FILEPATTERN = "filepattern";
-  public static final String FOOTER = "footer";
   public static final String FORMAT = "format";
-  public static final String HEADER = "header";
   public static final String INPUTS = "inputs";
-  public static final String INPUT_CODER = "input_coder";
-  public static final String IS_GENERATED = "is_generated";
   public static final String IS_MERGING_WINDOW_FN = "is_merging_window_fn";
   public static final String IS_PAIR_LIKE = "is_pair_like";
   public static final String IS_STREAM_LIKE =

[3/3] beam git commit: This closes #3265: Property names cleanup

[jkff]

This closes #3265: Property names cleanup


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eee6726a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eee6726a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eee6726a

Branch: refs/heads/master
Commit: eee6726aa0af9cd876eca34a8136977d074e9477
Parents: f1386c1 e201c37
Author: Eugene Kirpichov 
Authored: Fri Jun 2 10:54:41 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jun 2 10:54:41 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |  2 +-
 .../runners/dataflow/util/PropertyNames.java| 46 
 2 files changed, 1 insertion(+), 47 deletions(-)
--

[GitHub] beam pull request #3265: Property names cleanup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3265


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: Bumps dataflow container version to 0530

[jkff]

Bumps dataflow container version to 0530


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c8fce2fa
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c8fce2fa
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c8fce2fa

Branch: refs/heads/master
Commit: c8fce2faf4ca380530f563adf4992c8bf3a5f3a5
Parents: f1386c1
Author: Eugene Kirpichov 
Authored: Tue May 30 18:04:36 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jun 2 10:54:26 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c8fce2fa/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 16e1266..92c94a8 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170519
+
beam-master-20170530
 
1
 
6
   

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #85

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-1348] Add Runner API constructs to ProcessBundleDescriptor.

[robertwb] Unify Java and Python WindowingStrategy representations.

[robertwb] Add known window serialization for Java.

[robertwb] Ignore all proto generated files.

[robertwb] Increase dataflow worker version.

[davor] [BEAM-2378] support FULL OUTER JOIN

--
[...truncated 2.53 MB...]
2017-06-02T18:33:38.401 [INFO] 
2017-06-02T18:33:38.401 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-02T18:33:38.406 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-02T18:33:38.433 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 53.6 KB/sec)
2017-06-02T18:33:38.437 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-02T18:33:38.462 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 450.2 KB/sec)
2017-06-02T18:33:38.467 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-02T18:33:38.494 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 475.3 KB/sec)
2017-06-02T18:33:38.499 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-02T18:33:38.523 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 90.5 KB/sec)
2017-06-02T18:33:38.529 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-02T18:33:38.554 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 62.9 KB/sec)
2017-06-02T18:33:38.559 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-02T18:33:38.584 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 210.0 KB/sec)
2017-06-02T18:33:38.588 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-02T18:33:38.613 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 99.8 KB/sec)
2017-06-02T18:33:38.617 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-02T18:33:38.643 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 131.8 KB/sec)
2017-06-02T18:33:38.648 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-02T18:33:38.675 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 37.1 KB/sec)
2017-06-02T18:33:38.680 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-02T18:33:38.707 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 62.8 KB/sec)
2017-06-02T18:33:38.712 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-02T18:33:38.741 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 60.8 KB/sec)
2017-06-02T18:33:38.746 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-02T18:33:38.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 85.6 KB/sec)
2017-06-02T18:33:38.783 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-02T18:33:38.813 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 587.2 KB/sec)
2017-06-02T18:33:38.820 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-02T18:33:38.846 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 362.6 KB/sec)
2017-06-02T18:33:38.851 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3994

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3995

[Apache Jenkins Server]

See 

[1/2] beam git commit: Dead-letter support for BigQuery. Allow users to specify a retry policy, and get failed inserts back.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master eee6726aa -> 53c9bf4cd


Dead-letter support for BigQuery. Allow users to specify a retry policy, and 
get failed inserts back.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a57ff0ee
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a57ff0ee
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a57ff0ee

Branch: refs/heads/master
Commit: a57ff0eef1b7166d02243e19c4632f8924a014f0
Parents: eee6726
Author: Reuven Lax 
Authored: Thu May 11 06:40:47 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jun 2 13:15:01 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java|  8 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 22 -
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  8 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 39 +
 .../sdk/io/gcp/bigquery/InsertRetryPolicy.java  | 86 +++
 .../sdk/io/gcp/bigquery/StreamingInserts.java   | 35 ++--
 .../sdk/io/gcp/bigquery/StreamingWriteFn.java   | 50 ---
 .../io/gcp/bigquery/StreamingWriteTables.java   | 26 --
 .../beam/sdk/io/gcp/bigquery/WriteResult.java   | 27 --
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 56 +
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 88 +---
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   | 11 ++-
 .../sdk/io/gcp/bigquery/FakeDatasetService.java | 75 -
 .../io/gcp/bigquery/InsertRetryPolicyTest.java  | 79 ++
 14 files changed, 540 insertions(+), 70 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a57ff0ee/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
index 0abd469..3686f99 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java
@@ -59,6 +59,7 @@ import org.apache.beam.sdk.values.PCollectionTuple;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
 import org.apache.beam.sdk.values.TupleTagList;
+import org.apache.beam.sdk.values.TypeDescriptor;
 
 /** PTransform that uses BigQuery batch-load jobs to write a PCollection to 
BigQuery. */
 class BatchLoads
@@ -248,7 +249,8 @@ class BatchLoads
 // This transform will look at the set of files written for each table, 
and if any table has
 // too many files or bytes, will partition that table's files into 
multiple partitions for
 // loading.
-PCollection singleton = p.apply(Create.of((Void) 
null).withCoder(VoidCoder.of()));
+PCollection singleton = p.apply("singleton",
+Create.of((Void) null).withCoder(VoidCoder.of()));
 PCollectionTuple partitions =
 singleton.apply(
 "WritePartition",
@@ -333,6 +335,8 @@ class BatchLoads
 dynamicDestinations))
 .withSideInputs(writeTablesSideInputs));
 
-return WriteResult.in(input.getPipeline());
+PCollection empty =
+p.apply("CreateEmptyFailedInserts", 
Create.empty(TypeDescriptor.of(TableRow.class)));
+return WriteResult.in(input.getPipeline(), new 
TupleTag("failedInserts"), empty);
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/a57ff0ee/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index cf258ca..6a93279 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -674,6 +674,7 @@ public class BigQueryIO {
 abstract BigQueryServices getBigQueryServices();
 @Nullable abstract Integer getMaxFilesPerBundle();
 @Nullable abstract Long getMaxFileSize();
+@Nullable abstract InsertRetryPolicy getFailedInsertRetryPolicy();
 
 abstract Builder toBuilder();
 
@@ -693,6 +694,7 @@ public class BigQueryIO {
   abstract Builder setBigQueryServices(BigQueryServices 
bigQueryServices);
   abstract Builder setMaxFilesPerBundle(Integer maxFilesPerB

[GitHub] beam pull request #1609: BEAM-1151 Add failure handling to BigQueryIO.Write

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1609


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1609: BEAM-1151 Add failure handling to BigQueryIO.Write

[jkff]

This closes #1609: BEAM-1151 Add failure handling to BigQueryIO.Write


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/53c9bf4c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/53c9bf4c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/53c9bf4c

Branch: refs/heads/master
Commit: 53c9bf4cd325035fabde192c63652ef6d591b93c
Parents: eee6726 a57ff0e
Author: Eugene Kirpichov 
Authored: Fri Jun 2 13:54:01 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Jun 2 13:54:01 2017 -0700

--
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java|  8 +-
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 22 -
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  8 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 39 +
 .../sdk/io/gcp/bigquery/InsertRetryPolicy.java  | 86 +++
 .../sdk/io/gcp/bigquery/StreamingInserts.java   | 35 ++--
 .../sdk/io/gcp/bigquery/StreamingWriteFn.java   | 50 ---
 .../io/gcp/bigquery/StreamingWriteTables.java   | 26 --
 .../beam/sdk/io/gcp/bigquery/WriteResult.java   | 27 --
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 56 +
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 88 +---
 .../sdk/io/gcp/bigquery/BigQueryUtilTest.java   | 11 ++-
 .../sdk/io/gcp/bigquery/FakeDatasetService.java | 75 -
 .../io/gcp/bigquery/InsertRetryPolicyTest.java  | 79 ++
 14 files changed, 540 insertions(+), 70 deletions(-)
--

[jira] [Commented] (BEAM-1151) BigQueryIO.Write has no way of handling failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035411#comment-16035411
 ] 

ASF GitHub Bot commented on BEAM-1151:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1609


> BigQueryIO.Write has no way of handling failures
> 
>
> Key: BEAM-1151
> URL: https://issues.apache.org/jira/browse/BEAM-1151
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.4.0
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.3.0-incubating
>
>
> BigQueryIO.Write in streaming mode uses BigQuery's streaming insert API. 
> Inserts can fail for a variety of reasons, and the current connector provides 
> no way of managing those failures. This PR allows better failure handling.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3254: [BEAM-2375] upgrade beam-version to 2.0.0

[xumingming]

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3254


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2375) upgrade beam-version to 2.0.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035558#comment-16035558
 ] 

ASF GitHub Bot commented on BEAM-2375:
--

Github user xumingming closed the pull request at:

https://github.com/apache/beam/pull/3254


> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #3165: Allow the Distinct transform to deduplicate elements across panes

[kenn]

This closes #3165: Allow the Distinct transform to deduplicate elements across 
panes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/9cdae6ca
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/9cdae6ca
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/9cdae6ca

Branch: refs/heads/master
Commit: 9cdae6caf4c466dcc012a10380da219c18b56472
Parents: 53c9bf4 1bc84fb
Author: Kenneth Knowles 
Authored: Fri Jun 2 15:44:53 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 15:44:53 2017 -0700

--
 .../runners/spark/SparkRunnerDebuggerTest.java  |   2 +-
 .../apache/beam/sdk/transforms/Distinct.java|  80 +---
 .../beam/sdk/transforms/DistinctTest.java   | 130 ++-
 3 files changed, 188 insertions(+), 24 deletions(-)
--

[GitHub] beam pull request #3165: [BEAM-2305] Allow the Distinct transform to dedupli...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3165


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Allow the Distinct transform to deduplicate elements across panes

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 53c9bf4cd -> 9cdae6caf


Allow the Distinct transform to deduplicate elements across panes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1bc84fb5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1bc84fb5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1bc84fb5

Branch: refs/heads/master
Commit: 1bc84fb5ff4ca087c97da45247f1e445eadc48de
Parents: 53c9bf4
Author: Reuven Lax 
Authored: Tue May 16 12:12:01 2017 -0700
Committer: Kenneth Knowles 
Committed: Fri Jun 2 15:42:53 2017 -0700

--
 .../runners/spark/SparkRunnerDebuggerTest.java  |   2 +-
 .../apache/beam/sdk/transforms/Distinct.java|  80 +---
 .../beam/sdk/transforms/DistinctTest.java   | 130 ++-
 3 files changed, 188 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1bc84fb5/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
index 9009751..64ff98c 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java
@@ -142,7 +142,7 @@ public class SparkRunnerDebuggerTest {
 + "_.mapPartitions(new org.apache.beam.sdk.transforms.Distinct$2())\n"
 + "_.groupByKey()\n"
 + "_.map(new 
org.apache.beam.sdk.transforms.Combine$IterableCombineFn())\n"
-+ "_.mapPartitions(new org.apache.beam.sdk.transforms.Keys$1())\n"
++ "_.mapPartitions(new org.apache.beam.sdk.transforms.Distinct$3())\n"
 + "_.mapPartitions(new org.apache.beam.sdk.transforms.WithKeys$2())\n"
 + "_.";
 

http://git-wip-us.apache.org/repos/asf/beam/blob/1bc84fb5/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java
index 2d08cee..d751dbe 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Distinct.java
@@ -17,9 +17,15 @@
  */
 package org.apache.beam.sdk.transforms;
 
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * {@code Distinct} takes a {@code PCollection} and
@@ -59,6 +65,8 @@ import org.apache.beam.sdk.values.TypeDescriptor;
  */
 public class Distinct extends PTransform,
 PCollection> {
+  private static final Logger LOG = LoggerFactory.getLogger(Distinct.class);
+
   /**
* Returns a {@code Distinct} {@code PTransform}.
*
@@ -66,7 +74,7 @@ public class Distinct extends PTransform,
* {@code PCollection}s
*/
   public static  Distinct create() {
-return new Distinct();
+return new Distinct<>();
   }
 
   /**
@@ -78,26 +86,48 @@ public class Distinct extends PTransform,
*/
   public static  WithRepresentativeValues 
withRepresentativeValueFn(
   SerializableFunction fn) {
-return new WithRepresentativeValues(fn, null);
+return new WithRepresentativeValues<>(fn, null);
+  }
+
+  private static  void validateWindowStrategy(
+  WindowingStrategy strategy) {
+if (!strategy.getWindowFn().isNonMerging()
+&& (!strategy.getTrigger().getClass().equals(DefaultTrigger.class)
+|| strategy.getAllowedLateness().isLongerThan(Duration.ZERO))) {
+throw new UnsupportedOperationException(String.format(
+"%s does not support non-merging windowing strategies, except when 
using the default "
++ "trigger and zero allowed lateness.", 
Distinct.class.getSimpleName()));
+}
   }
 
   @Override
   public PCollection expand(PCollection in) {
-return in
-.apply("CreateIndex", MapElements.via(new SimpleFunction>() {
-  @Override
-  public KV apply(T element) {
-return KV.of(element, (Void) null);
-  }
-}))
-.apply(Combine.perKey(
-new SerializableFunction, Void>() {
+  

[jira] [Commented] (BEAM-2305) Dinstinct transform produces unexpected output when triggered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035559#comment-16035559
 ] 

ASF GitHub Bot commented on BEAM-2305:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3165


> Dinstinct transform produces unexpected output when triggered
> -
>
> Key: BEAM-2305
> URL: https://issues.apache.org/jira/browse/BEAM-2305
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>
> Distinct is expected to produce one unique element per window (if the 
> aggregation mode is accumulatingFiredPanes), however the current 
> implementation will produce one unique element per pane instead. This is 
> incorrect behavior.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-2405:
---

 Summary: Create a BigQuery sink for streaming using PTransform
 Key: BEAM-2405
 URL: https://issues.apache.org/jira/browse/BEAM-2405
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Sourabh Bajaj
Assignee: Sourabh Bajaj






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3014

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035608#comment-16035608
 ] 

ASF GitHub Bot commented on BEAM-2405:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3288

[BEAM-2405] Write to BQ using the streaming API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This should be ready for review so that we can test it with streaming 
pipelines

There are few followup items after this PR:
- Check if it is possible to get rid of the create DoFn
- Integration with the Dataflow runner to work in both Batch and Streaming
- Migrate the tornadoes example to use this and deprecate the sink interface

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-BQ-SINK

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3288


commit 4792096b1e3992d1edd296f2ccbe46d2a2536573
Author: Sourabh Bajaj 
Date:   2017-06-02T23:12:13Z

[BEAM-2405] Write to BQ using the streaming API




> Create a BigQuery sink for streaming using PTransform
> -
>
> Key: BEAM-2405
> URL: https://issues.apache.org/jira/browse/BEAM-2405
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3288: [BEAM-2405] Write to BQ using the streaming API

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3288

[BEAM-2405] Write to BQ using the streaming API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

This should be ready for review so that we can test it with streaming 
pipelines

There are few followup items after this PR:
- Check if it is possible to get rid of the create DoFn
- Integration with the Dataflow runner to work in both Batch and Streaming
- Migrate the tornadoes example to use this and deprecate the sink interface

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-BQ-SINK

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3288


commit 4792096b1e3992d1edd296f2ccbe46d2a2536573
Author: Sourabh Bajaj 
Date:   2017-06-02T23:12:13Z

[BEAM-2405] Write to BQ using the streaming API




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #3015

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #36

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Bumps dataflow container version to 0530

[kirpichov] Removes unused PropertyNames

[kirpichov] Dead-letter support for BigQuery. Allow users to specify a retry 
policy,

[klk] Allow the Distinct transform to deduplicate elements across panes

--
[...truncated 899.69 KB...]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(aeb1bca373e70742): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerE

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #36

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Bumps dataflow container version to 0530

[kirpichov] Removes unused PropertyNames

[kirpichov] Dead-letter support for BigQuery. Allow users to specify a retry 
policy,

[klk] Allow the Distinct transform to deduplicate elements across panes

--
[...truncated 878.01 KB...]
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-03\T\00:13:35.337 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-06-03\T\00:13:40.486 [INFO] Replacing original artifact with shaded 
artifact.
2017-06-03\T\00:13:40.486 [INFO] Replacing 

 with 

2017-06-03\T\00:13:40.486 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-06-03\T\00:13:40.486 [INFO] Replacing 

 with 

2017-06-03\T\00:13:40.486 [INFO] Dependency-reduced POM written at: 

2017-06-03\T\00:13:40.621 [INFO] 
2017-06-03\T\00:13:40.621 [INFO] --- 
maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
2017-06-03\T\00:13:40.739 [INFO] Failsafe report directory: 

2017-06-03\T\00:13:40.744 [INFO] parallel='all', perCoreThreadCount=true, 
threadCount=4, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-06-03\T\00:13:40.763 [INFO] 
2017-06-03\T\00:13:40.763 [INFO] 
---
2017-06-03\T\00:13:40.764 [INFO]  T E S T S
2017-06-03\T\00:13:40.764 [INFO] 
---
2017-06-03\T\00:19:39.502 [INFO] Running 
org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT
2017-06-03\T\00:19:39.515 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 10.473 s - in 
org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT
2017-06-03\T\00:19:39.516 [INFO] Running 
org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteIT
2017-06-03\T\00:19:39.518 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 277.076 s - in 
org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteIT
2017-06-03\T\00:19:39.521 [INFO] Running 
org.apache.beam.sdk.io.gcp.datastore.V1ReadIT
2017-06-03\T\00:19:39.538 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 331.623 s - in org.apache.beam.sdk.io.gcp.datastore.V1ReadIT
2017-06-03\T\00:19:39.538 [INFO] Running 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT
2017-06-03\T\00:19:39.581 [ERROR] Tests run: 1, Failures: 0, Errors: 1, 
Skipped: 0, Time elapsed: 288.782 s <<< FAILURE! - in 
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT
2017-06-03\T\00:19:39.582 [ERROR] 
testWrite(org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT)  Time elapsed: 
288.782 s  

[jira] [Created] (BEAM-2406) NullPointerException when writing an empty table to BigQuery

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-2406:
--

 Summary: NullPointerException when writing an empty table to 
BigQuery
 Key: BEAM-2406
 URL: https://issues.apache.org/jira/browse/BEAM-2406
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Affects Versions: 2.0.0
Reporter: Ben Chambers
Assignee: Reuven Lax
Priority: Minor


Originally reported on Stackoverflow:
https://stackoverflow.com/questions/44314030/handling-empty-pcollections-with-bigquery-in-apache-beam

It looks like if there is no data to write, then WritePartitions will return a 
null destination, as explicitly stated in the comments:

https://github.com/apache/beam/blob/v2.0.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WritePartition.java#L126

But, the ConstantTableDestination doesn't turn that into the constant 
destination as the comment promises, instead it returns that `null` destination:

https://github.com/apache/beam/blob/53c9bf4cd325035fabde192c63652ef6d591b93c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinationsHelpers.java#L74

This leads to a null pointer error here since the `tableDestination` is that 
null result from calling `getTable`:

https://github.com/apache/beam/blob/v2.0.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L97



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #86

[Apache Jenkins Server]

See 


Changes:

[kirpichov] Bumps dataflow container version to 0530

[kirpichov] Removes unused PropertyNames

[kirpichov] Dead-letter support for BigQuery. Allow users to specify a retry 
policy,

[klk] Allow the Distinct transform to deduplicate elements across panes

--
[...truncated 2.53 MB...]
2017-06-03T00:33:06.257 [INFO] 
2017-06-03T00:33:06.257 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-03T00:33:06.262 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-03T00:33:06.272 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 144.7 KB/sec)
2017-06-03T00:33:06.275 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-03T00:33:06.285 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 1125.6 KB/sec)
2017-06-03T00:33:06.289 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-03T00:33:06.297 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1604.2 KB/sec)
2017-06-03T00:33:06.302 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-03T00:33:06.310 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 271.6 KB/sec)
2017-06-03T00:33:06.314 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-03T00:33:06.321 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 224.7 KB/sec)
2017-06-03T00:33:06.325 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-03T00:33:06.333 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 656.2 KB/sec)
2017-06-03T00:33:06.337 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-03T00:33:06.344 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 356.4 KB/sec)
2017-06-03T00:33:06.348 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-03T00:33:06.358 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-03T00:33:06.363 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-03T00:33:06.370 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 143.3 KB/sec)
2017-06-03T00:33:06.374 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-03T00:33:06.382 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 212.0 KB/sec)
2017-06-03T00:33:06.387 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-03T00:33:06.395 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 220.5 KB/sec)
2017-06-03T00:33:06.400 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-03T00:33:06.408 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 331.8 KB/sec)
2017-06-03T00:33:06.413 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-03T00:33:06.421 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 2202.0 KB/sec)
2017-06-03T00:33:06.426 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-03T00:33:06.434 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 1178.5 KB/sec)
2017-06-03T00:33:06.437 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apac

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3998

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3016

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #37

[Apache Jenkins Server]

See 


--
[...truncated 872.75 KB...]
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 from the shaded 
jar.
2017-06-03\T\06:09:05.834 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-03\T\06:09:05.834 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-06-03\T\06:09:10.157 [INFO] Replacing original artifact with shaded 
artifact.
2017-06-03\T\06:09:10.157 [INFO] Replacing 

 with 

2017-06-03\T\06:09:10.157 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-06-03\T\06:09:10.157 [INFO] Replacing 

 with 

2017-06-03\T\06:09:10.158 [INFO] Dependency-reduced POM written at: 

2017-06-03\T\06:09:10.249 [INFO] 
2017-06-03\T\06:09:10.249 [INFO] --- 
maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
2017-06-03\T\06:09:10.323 [INFO] Failsafe report directory: 

2017-06-03\T\06:09:10.327 [INFO] parallel='all', perCoreThreadCount=true, 
threadCount=4, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-06-03\T\06:09:10.344 [INFO] 
2017-06-03\T\06:09:10.344 [INFO] 
---
2017-06-03\T\06:09:10.344 [INFO]  T E S T S
2017-06-03\T\06:09:10.344 [INFO] 
---
2017-06-03\T\06:14:16.738 [INFO] Running 
org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT
2017-06-03\T\06:14:16.747 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 1.009 s - in 
org.apache.beam.sdk.io.gcp.datastore.SplitQueryFnIT
2017-06-03\T\06:14:16.748 [INFO] Running 
org.apache.beam.sdk.io.gcp.datastore.V1ReadIT
2017-06-03\T\06:14:16.755 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 305.136 s - in org.apache.beam.sdk.io.gcp.datastore.V1ReadIT
2017-06-03\T\06:14:16.755 [INFO] Running 
org.apache.beam.sdk.io.gcp.b

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #37

[Apache Jenkins Server]

See 


--
[...truncated 873.90 KB...]
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.auth:google-auth-library-appengine:jar:0.6.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.appengine:appengine-api-1.0-sdk:jar:1.9.34 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Including 
org.apache.beam:beam-runners-core-construction-java:jar:2.1.0-SNAPSHOT in the 
shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.2.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding com.google.code.gson:gson:jar:2.7 
from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.oauth-client:google-oauth-client:jar:1.22.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.http-client:google-http-client:jar:1.22.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding commons-codec:commons-codec:jar:1.3 
from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.http-client:google-http-client-jackson2:jar:1.22.0 from the shaded 
jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev196-1.22.0 from the 
shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.1 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.1 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api-client:google-api-client-java6:jar:1.22.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api-client:google-api-client-jackson2:jar:1.22.0 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 from the shaded 
jar.
2017-06-03\T\06:11:28.241 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-03\T\06:11:28.241 [INFO] Ex

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #37

[Apache Jenkins Server]

See 


--
[...truncated 895.84 KB...]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:398)
... 13 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at io.grpc.Status.asRuntimeException(Status.java:540)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
at 
io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152)
... 3 more
(8439eb197940faa5): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
at 
com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
at 
com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
at 
com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
Caused by: com.google.cloud.spanner.SpannerException: NOT_FOUND: 
io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
resource_name: 
"projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
description: "Database does not exist."

at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cl

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #87

[Apache Jenkins Server]

See 


--
[...truncated 2.53 MB...]
2017-06-03T06:31:04.128 [INFO] 
2017-06-03T06:31:04.128 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-03T06:31:04.132 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-03T06:31:04.143 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 131.6 KB/sec)
2017-06-03T06:31:04.146 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-03T06:31:04.160 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 804.0 KB/sec)
2017-06-03T06:31:04.163 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-03T06:31:04.171 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1604.2 KB/sec)
2017-06-03T06:31:04.175 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-03T06:31:04.185 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 217.3 KB/sec)
2017-06-03T06:31:04.190 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-03T06:31:04.200 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 157.3 KB/sec)
2017-06-03T06:31:04.204 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-03T06:31:04.212 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 656.2 KB/sec)
2017-06-03T06:31:04.216 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-03T06:31:04.223 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 356.4 KB/sec)
2017-06-03T06:31:04.227 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-03T06:31:04.319 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 37.2 KB/sec)
2017-06-03T06:31:04.325 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-03T06:31:04.336 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 91.2 KB/sec)
2017-06-03T06:31:04.340 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-03T06:31:04.349 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 188.5 KB/sec)
2017-06-03T06:31:04.354 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-03T06:31:04.365 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 160.3 KB/sec)
2017-06-03T06:31:04.370 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-03T06:31:04.381 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 241.3 KB/sec)
2017-06-03T06:31:04.386 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-03T06:31:04.399 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1355.1 KB/sec)
2017-06-03T06:31:04.403 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-03T06:31:04.413 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 942.8 KB/sec)
2017-06-03T06:31:04.417 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
2017-06-03T06:31:04.425 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant-parent/1.8.4/ant-parent-1.8.4.pom
 (5 KB at 556.8 KB/sec)
2017-06-03T06:31:04.429 [INFO] Downloading: 
https://repo.maven.apache.org/maven