Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1179

[Apache Jenkins Server]

See 


--
[...truncated 775.24 KB...]
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-24_20_24_10-16182077139363237034]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-24_20_24_10-16182077139363237034?project=apache-beam-testing
root: INFO: Job 2018-03-24_20_24_10-16182077139363237034 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-25T03:24:10.359Z: JOB_MESSAGE_WARNING: Job 
2018-03-24_20_24_10-16182077139363237034 might autoscale up to 1000 workers.
root: INFO: 2018-03-25T03:24:10.391Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-24_20_24_10-16182077139363237034. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-25T03:24:10.414Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-24_20_24_10-16182077139363237034.
root: INFO: 2018-03-25T03:24:12.987Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-25T03:24:13.153Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-25T03:24:14.418Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-25T03:24:14.454Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-25T03:24:14.485Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-25T03:24:14.508Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-25T03:24:14.533Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-25T03:24:14.565Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-25T03:24:14.589Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-25T03:24:14.620Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-25T03:24:14.643Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-25T03:24:14.666Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-25T03:24:14.699Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-25T03:24:14.722Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-25T03:24:14.754Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11-u13 for input s12-reify-value0-c11
root: INFO: 2018-03-25T03:24:14.785Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Write, through flatten s11-u13, into 
producer assert_that/Group/GroupByKey/Reify
root: INFO: 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4499

[Apache Jenkins Server]

See 


--
[...truncated 290.12 KB...]
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-25 03:09:41.337551
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-25 03:09:42.293274
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-25 03:09:42.471370
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-25 03:09:42.488965
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5215

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1507

[Apache Jenkins Server]

See 


--
[...truncated 91.64 KB...]
'apache-beam-testing:bqjob_r7da483e4bd4ffd9c_01625a851a71_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-25 00:18:17,208 1506be9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-25 00:18:35,101 1506be9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-25 00:18:38,423 1506be9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r3dce9653197efffd_01625a856970_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r3dce9653197efffd_01625a856970_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3dce9653197efffd_01625a856970_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-25 00:18:38,423 1506be9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-25 00:19:04,118 1506be9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-25 00:19:06,566 1506be9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r6f64e7db442df5d_01625a85db5f_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_r6f64e7db442df5d_01625a85db5f_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r6f64e7db442df5d_01625a85db5f_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-25 00:19:06,566 1506be9a MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-25 00:19:34,538 1506be9a MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-25 00:19:37,099 1506be9a MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_reae1aa40bed052a_01625a8651f4_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_reae1aa40bed052a_01625a8651f4_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_reae1aa40bed052a_01625a8651f4_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #290

[Apache Jenkins Server]

See 


--
[...truncated 15.17 KB...]
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Installing collected packages: grpcio, hdfs, nose, apache-beam
  Found existing installation: grpcio 1.3.5
Not uninstalling grpcio at 
/home/jenkins/.local/lib/python2.7/site-packages, outside environment 

  Running setup.py develop for apache-beam
Successfully installed apache-beam grpcio-1.10.0 hdfs-2.1.0 nose-1.3.7
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5175440576051434168.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.tfrecordioit_pkb_results 
--temp_dir=
 --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/290/,--numberOfRecords=100]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-03-25 00:00:36,905 58bb22f6 MainThread INFO Verbose logging to: 

2018-03-25 00:00:36,905 58bb22f6 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-445-geb5f1b5
2018-03-25 00:00:36,906 58bb22f6 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/290/,--numberOfRecords=100]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.tfrecordioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-03-25 00:00:37,165 58bb22f6 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-03-25 00:00:37,165 58bb22f6 MainThread INFO Initializing the edw 
service decoder
2018-03-25 00:00:37,278 58bb22f6 MainThread beam_integration_benchmark(1/1) 
INFO Provisioning resources for benchmark beam_integration_benchmark
2018-03-25 00:00:37,280 58bb22f6 MainThread beam_integration_benchmark(1/1) 
INFO Preparing benchmark beam_integration_benchmark
2018-03-25 00:00:37,280 58bb22f6 MainThread beam_integration_benchmark(1/1) 
INFO Running: git clone https://github.com/apache/beam.git
2018-03-25 00:00:46,233 58bb22f6 MainThread beam_integration_benchmark(1/1) 
INFO Running benchmark beam_integration_benchmark
2018-03-25 00:00:46,243 58bb22f6 MainThread beam_integration_benchmark(1/1) 
INFO Running: /home/jenkins/tools/maven/latest/bin/mvn -e verify 
-Dit.test=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT -DskipITs=false -pl 
sdks/java/io/file-based-io-tests -Pio-it -Pdataflow-runner -Dfilesystem=gcs 
-DintegrationTestPipelineOptions=["--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-perf-tests","--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/290/","--numberOfRecords=100","--runner=TestDataflowRunner"]
2018-03-25 00:20:46,252 58bb22f6 Thread-2 ERRORIssueCommand timed out after 
1200 seconds. Killing command "/home/jenkins/tools/maven/latest/bin/mvn -e 
verify -Dit.test=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT -DskipITs=false 
-pl sdks/java/io/file-based-io-tests -Pio-it -Pdataflow-runner -Dfilesystem=gcs 
-DintegrationTestPipelineOptions=["--project=apache-beam-testing","--tempRoot=gs://temp-storage-for-perf-tests","--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/290/","--numberOfRecords=100","--runner=TestDataflowRunner"]".
2018-03-25 00:20:46,278 58bb22f6 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #59

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1063

[Apache Jenkins Server]

See 


--
[...truncated 61.52 KB...]
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.049s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.107s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  2.825 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  2.165 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.032 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  6.229 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  2.984 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  3.895 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.090 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 15.734 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED
[INFO] Apache Beam :: Runners :: Local Java Core .. SKIPPED
[INFO] Apache Beam :: Runners :: Direct Java .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Common  SKIPPED
[INFO] Apache Beam :: SDKs :: Java 

Build failed in Jenkins: beam_PerformanceTests_JDBC #370

[Apache Jenkins Server]

See 


--
[...truncated 48.75 KB...]
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ 
beam-sdks-java-io-jdbc ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.001 s 
<<< FAILURE! - in org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] 

[jira] [Commented] (BEAM-1678) Create MemcachedIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412809#comment-16412809
 ] 

Ismaël Mejía commented on BEAM-1678:


Hi, [~sesh...@gmail.com] just pinging to know if you would still be interested 
on working on this one or if I can assign this to someone else.

> Create MemcachedIO
> --
>
> Key: BEAM-1678
> URL: https://issues.apache.org/jira/browse/BEAM-1678
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Ismaël Mejía
>Assignee: Seshadri Raghunathan
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1178

[Apache Jenkins Server]

See 


--
[...truncated 779.18 KB...]
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-24_14_24_51-5163007450373142262]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-24_14_24_51-5163007450373142262?project=apache-beam-testing
root: INFO: Job 2018-03-24_14_24_51-5163007450373142262 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-24T21:24:52.006Z: JOB_MESSAGE_WARNING: Job 
2018-03-24_14_24_51-5163007450373142262 might autoscale up to 1000 workers.
root: INFO: 2018-03-24T21:24:52.025Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-24_14_24_51-5163007450373142262. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-24T21:24:52.047Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-24_14_24_51-5163007450373142262.
root: INFO: 2018-03-24T21:24:54.850Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-24T21:24:55.144Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-24T21:24:56.225Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T21:24:56.260Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-24T21:24:56.288Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T21:24:56.321Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-24T21:24:56.354Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-24T21:24:56.394Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-24T21:24:56.426Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-24T21:24:56.448Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-24T21:24:56.479Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-24T21:24:56.510Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-24T21:24:56.537Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-24T21:24:56.567Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-24T21:24:56.598Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11-u13 for input s12-reify-value0-c11
root: INFO: 2018-03-24T21:24:56.628Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Write, through flatten s11-u13, into 
producer assert_that/Group/GroupByKey/Reify
root: INFO: 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4498

[Apache Jenkins Server]

See 


--
[...truncated 290.12 KB...]
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-24 21:09:26.605181
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-24 21:09:27.178116
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 21:09:27.342745
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 21:09:27.360761
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5214

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1506

[Apache Jenkins Server]

See 


--
[...truncated 90.58 KB...]
'apache-beam-testing:bqjob_r2e629163fabcd562_0162593bef42_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 18:18:44,746 bcc5632b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 18:19:13,811 bcc5632b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 18:19:15,931 bcc5632b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r2d1337ea55b20d_0162593c691b_1 ... (0s) Current status: 
RUNNING 
   Waiting on bqjob_r2d1337ea55b20d_0162593c691b_1 ... (0s) Current 
status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2d1337ea55b20d_0162593c691b_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 18:19:15,932 bcc5632b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 18:19:45,870 bcc5632b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 18:19:48,046 bcc5632b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r7f7d08a7cfb36054_0162593ce649_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7f7d08a7cfb36054_0162593ce649_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r7f7d08a7cfb36054_0162593ce649_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 18:19:48,046 bcc5632b MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 18:20:10,734 bcc5632b MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 18:20:13,039 bcc5632b MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r506738a0a45d1b97_0162593d4792_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r506738a0a45d1b97_0162593d4792_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r506738a0a45d1b97_0162593d4792_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #61

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1062

[Apache Jenkins Server]

See 


--
[...truncated 61.52 KB...]
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.014s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.081s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  3.132 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.060 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.083 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  9.170 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.815 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  3.978 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.135 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 23.270 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED
[INFO] Apache Beam :: Runners :: Local Java Core .. SKIPPED
[INFO] Apache Beam :: Runners :: Direct Java .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Common  SKIPPED
[INFO] Apache Beam :: SDKs :: Java 

Build failed in Jenkins: beam_PerformanceTests_JDBC #369

[Apache Jenkins Server]

See 


--
[...truncated 38.54 KB...]
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ 
beam-sdks-java-io-jdbc ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0 s <<< 
FAILURE! - in org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #58

[Apache Jenkins Server]

See 


--
[...truncated 44.75 KB...]
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ 
beam-sdks-java-io-hadoop-input-format ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0 s <<< 
FAILURE! - in org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT
[ERROR] org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIOIT  Time 
elapsed: 0 s  <<< ERROR!
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at 
org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:215)
at org.postgresql.Driver.makeConnection(Driver.java:404)
at org.postgresql.Driver.connect(Driver.java:272)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at 

[jira] [Work logged] (BEAM-1763) TestPipeline should ensure that all assertions succeeded

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1763?focusedWorklogId=84063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84063
 ]

ASF GitHub Bot logged work on BEAM-1763:


Author: ASF GitHub Bot
Created on: 24/Mar/18 17:25
Start Date: 24/Mar/18 17:25
Worklog Time Spent: 10m 
  Work Description: aviemzur closed pull request #3807: [BEAM-1763] Ensure 
run waitUntilFinish in TestPipeline.
URL: https://github.com/apache/beam/pull/3807
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index be2f193ea93..8f3664534c2 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -354,6 +354,7 @@ public PipelineResult run(PipelineOptions options) {
   .as(TestValueProviderOptions.class)
   
.setProviderRuntimeValues(StaticValueProvider.of(providerRuntimeValues));
   pipelineResult = super.run(updatedOptions);
+  pipelineResult.waitUntilFinish();
   verifyPAssertsSucceeded(this, pipelineResult);
 } catch (RuntimeException exc) {
   Throwable cause = exc.getCause();


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84063)
Time Spent: 20m  (was: 10m)

> TestPipeline should ensure that all assertions succeeded
> 
>
> Key: BEAM-1763
> URL: https://issues.apache.org/jira/browse/BEAM-1763
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Aviem Zur
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This doesn't need to be part of each {{PipelineRunner}} implementation if it 
> goes through the {{PipelineResult}} APIs. The assertion can be of the form 
> that if the Pipeline is finished, then the number of successful assertions is 
> equal to the total number of assertions.
> Suggested solution:
> For runners which support metrics, use the counters for successful/failed 
> assertions and compare them to expected number of assertions.
> Runners which do not support metrics should either implement metrics or 
> override {{PAssert}} in a way that verifies its execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-981?focusedWorklogId=84061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84061
 ]

ASF GitHub Bot logged work on BEAM-981:
---

Author: ASF GitHub Bot
Created on: 24/Mar/18 17:25
Start Date: 24/Mar/18 17:25
Worklog Time Spent: 10m 
  Work Description: aviemzur closed pull request #2997: 
BEAM-981-Not-possible-to-directly-submit-on-spark-cluster
URL: https://github.com/apache/beam/pull/2997
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
index 26b549baa2e..48d0023cd6b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java
@@ -71,6 +71,10 @@
   String getCheckpointDir();
   void setCheckpointDir(String checkpointDir);
 
+  @Description("JARs to be submitted to the cluster, Usually a shaded JAR with 
all dependencies. ")
+  String getJars();
+  void setJars(String jars);
+
   /**
* Returns the default checkpoint directory of /tmp/${job.name}.
* For testing purposes only. Production applications should use a reliable
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
index cdeddadefab..2bb2169ec96 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.runners.spark.translation;
 
+import com.google.common.base.Strings;
 import org.apache.beam.runners.spark.SparkContextOptions;
 import org.apache.beam.runners.spark.SparkPipelineOptions;
 import org.apache.beam.runners.spark.coders.BeamSparkRunnerRegistrator;
@@ -93,6 +94,9 @@ private static JavaSparkContext 
createSparkContext(SparkContextOptions contextOp
 // set master if not set.
 conf.setMaster(contextOptions.getSparkMaster());
   }
+  if (!conf.contains("spark.jars") && 
!Strings.isNullOrEmpty(contextOptions.getJars())) {
+conf.setJars(contextOptions.getJars().split(","));
+  }
   conf.setAppName(contextOptions.getAppName());
   // register immutable collections serializers because the SDK uses them.
   conf.set("spark.kryo.registrator", 
BeamSparkRunnerRegistrator.class.getName());


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84061)
Time Spent: 20m  (was: 10m)

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> 

[jira] [Work logged] (BEAM-981) Not possible to directly submit a pipeline on spark cluster

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-981?focusedWorklogId=84060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84060
 ]

ASF GitHub Bot logged work on BEAM-981:
---

Author: ASF GitHub Bot
Created on: 24/Mar/18 17:25
Start Date: 24/Mar/18 17:25
Worklog Time Spent: 10m 
  Work Description: aviemzur commented on issue #2997: 
BEAM-981-Not-possible-to-directly-submit-on-spark-cluster
URL: https://github.com/apache/beam/pull/2997#issuecomment-375909972
 
 
   Closed for inactivity.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84060)
Time Spent: 10m
Remaining Estimate: 0h

> Not possible to directly submit a pipeline on spark cluster
> ---
>
> Key: BEAM-981
> URL: https://issues.apache.org/jira/browse/BEAM-981
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.6.0
>Reporter: Jean-Baptiste Onofré
>Assignee: Łukasz Gajowy
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It's not possible to directly run a pipeline on the spark runner (for 
> instance using {{mvn exec:java}}. It fails with:
> {code}
> [appclient-register-master-threadpool-0] INFO 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Connecting to 
> master spark://10.200.118.197:7077...
> [shuffle-client-0] ERROR org.apache.spark.network.client.TransportClient - 
> Failed to send RPC 6813731522650020739 to /10.200.118.197:7077: 
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:820)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:733)
> at 
> io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:284)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:748)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:740)
> at 
> io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1101)
> at 
> io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1148)
> at 
> io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1090)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401)
> at 
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
> at java.lang.Thread.run(Thread.java:745)
> [appclient-register-master-threadpool-0] WARN 
> org.apache.spark.deploy.client.AppClient$ClientEndpoint - Failed to connect 
> to master 10.200.118.197:7077
> java.io.IOException: Failed to send RPC 6813731522650020739 to 
> /10.200.118.197:7077: java.lang.AbstractMethodError: 
> org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
> at 
> 

[jira] [Work logged] (BEAM-1763) TestPipeline should ensure that all assertions succeeded

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1763?focusedWorklogId=84062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84062
 ]

ASF GitHub Bot logged work on BEAM-1763:


Author: ASF GitHub Bot
Created on: 24/Mar/18 17:25
Start Date: 24/Mar/18 17:25
Worklog Time Spent: 10m 
  Work Description: aviemzur commented on issue #3807: [BEAM-1763] Ensure 
run waitUntilFinish in TestPipeline.
URL: https://github.com/apache/beam/pull/3807#issuecomment-375910020
 
 
   Closed for inactivity.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 84062)
Time Spent: 10m
Remaining Estimate: 0h

> TestPipeline should ensure that all assertions succeeded
> 
>
> Key: BEAM-1763
> URL: https://issues.apache.org/jira/browse/BEAM-1763
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Aviem Zur
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This doesn't need to be part of each {{PipelineRunner}} implementation if it 
> goes through the {{PipelineResult}} APIs. The assertion can be of the form 
> that if the Pipeline is finished, then the number of successful assertions is 
> equal to the total number of assertions.
> Suggested solution:
> For runners which support metrics, use the counters for successful/failed 
> assertions and compare them to expected number of assertions.
> Runners which do not support metrics should either implement metrics or 
> override {{PAssert}} in a way that verifies its execution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1177

[Apache Jenkins Server]

See 


--
[...truncated 766.77 KB...]
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": "kind:bytes"
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "FlatMap()/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "FlatMap()/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-24_08_24_40-3584323830304294032]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-24_08_24_40-3584323830304294032?project=apache-beam-testing
root: INFO: Job 2018-03-24_08_24_40-3584323830304294032 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-24T15:24:40.472Z: JOB_MESSAGE_WARNING: Job 
2018-03-24_08_24_40-3584323830304294032 might autoscale up to 1000 workers.
root: INFO: 2018-03-24T15:24:40.497Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-24_08_24_40-3584323830304294032. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-24T15:24:40.530Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-24_08_24_40-3584323830304294032.
root: INFO: 2018-03-24T15:24:43.220Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-24T15:24:43.330Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-24T15:24:44.343Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T15:24:44.376Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-24T15:24:44.407Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T15:24:44.440Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-24T15:24:44.458Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-24T15:24:44.498Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-24T15:24:44.531Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s13 for input s11.out
root: INFO: 2018-03-24T15:24:44.564Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_0
root: INFO: 2018-03-24T15:24:44.598Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-24T15:24:44.631Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-24T15:24:44.659Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-24T15:24:44.684Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s13-u13 for input s14-reify-value0-c11
root: INFO: 2018-03-24T15:24:44.719Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Write, through flatten s13-u13, into 
producer assert_that/Group/GroupByKey/Reify
root: INFO: 2018-03-24T15:24:44.750Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-24T15:24:44.776Z: JOB_MESSAGE_DETAILED: Created new flatten 
s4-c17 to unzip producers of s18
root: INFO: 2018-03-24T15:24:44.800Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s4-c17 for input s2.out
root: INFO: 2018-03-24T15:24:44.815Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of FlatMap()/MapToVoidKey0, through 
flatten Flatten, into producer side1/Read

Build failed in Jenkins: beam_PostCommit_Python_Verify #4497

[Apache Jenkins Server]

See 


--
[...truncated 290.12 KB...]
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-24 15:09:25.817733
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-24 15:09:26.463524
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 15:09:26.626510
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 15:09:26.642178
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5213

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1505

[Apache Jenkins Server]

See 


--
[...truncated 94.51 KB...]
'apache-beam-testing:bqjob_r70548f23fe0351fb_016257f38e5b_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 12:20:04,183 736fa404 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 12:20:22,690 736fa404 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 12:20:25,207 736fa404 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r1a725d96ad767f97_016257f3e06c_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r1a725d96ad767f97_016257f3e06c_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r1a725d96ad767f97_016257f3e06c_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 12:20:25,207 736fa404 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 12:20:44,008 736fa404 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 12:20:46,469 736fa404 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r6a3b19df65951888_016257f43388_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r6a3b19df65951888_016257f43388_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r6a3b19df65951888_016257f43388_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 12:20:46,470 736fa404 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 12:21:06,699 736fa404 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 12:21:08,804 736fa404 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r4bda21f560e1bb2c_016257f48b6e_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4bda21f560e1bb2c_016257f48b6e_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4bda21f560e1bb2c_016257f48b6e_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #60

[Apache Jenkins Server]

See 


--
[...truncated 714.19 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] 

Build failed in Jenkins: beam_PerformanceTests_Python #1061

[Apache Jenkins Server]

See 


--
[...truncated 1.56 KB...]

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8891376667279288864.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2542610715143907719.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8778063250219388569.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3806017414532358519.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4356513893450273522.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2518256084645030848.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #3

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[tgroh] Enable Proto Round Trips in the DirectRunner Tests

[lcwik] [BEAM-3249, BEAM-3457] Make Apache Rat a dependency of the Java and

[sidhom] Add Java bounded read overrides

[herohde] [BEAM-3817] Switch BQ write to not use side input

[lcwik] [BEAM-3457] Exclude generated and data files part of the Go SDK.

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[coheigea] Replacing size() == 0 with isEmpty()

[markliu] [BEAM-3841] Fix TestDataflowRunner.run to run_pipeline

[coheigea] Put the String literal first when comparing it to an Object

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[ekirpichov] BigQueryIO.read().fromQuery() improvements

[szewinho] [BEAM-3060] Fixing mvn dependency issue when runnning filebasedIOIT

[lukasz.gajowy] [BEAM-3798] Remove error check on dataflow when getting batch 
job state

[tgroh] Add ExecutableStage -> ProcessBundleDescriptor

[tgroh] Clean up GrpcFnServer Constructor Signature

[tgroh] Add a RemoteExecutionTest

[tgroh] Clean up a couple of SdkHarnessClient Signatures

[tgroh] Separate RemoteInputDestinations from SdkHarnessClient

[XuMingmin] [SQL] Add support for DOT expression (#4863)

[herohde] Add TODO to revert Go IO to use side input

[jb] [BEAM-3500] "Attach" JDBC connection to the bundle (improve the pooling)

[jb] [BEAM-3500] Test if the user provides both withDataSourceConfiguration()

[jb] [BEAM-3500] Wrap the datasource as a poolable datasource and expose

[jb] [BEAM-3500] Add commons-pool2 dependency

[jb] [BEAM-3500] Only expose max number of connections in the pool to the

[jb] [BEAM-3500] Cleanup pool configuration parameters

[jb] [BEAM-3500] Remove dataSourceFactory

[jb] [BEAM-3500] Remove unecessary check on dataSourceConfiguration

[szewinho] Jax-api added to dependencyManagement.

[szewinho] Jaxb-api added to beam root pom.xml

[szewinho] Removed empty line

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

[axelmagn] Add a generic interface for the state service.

[herohde] Remove WindowedValue on PCollections for Go SDK

[cclauss] [BEAM-3872] Change Exception.message --> Exception.args[0]

[alan.myrvold] [BEAM-3249] Publish java artifacts using gradle

[herohde] CR: Fix comments to remove old windowing

[lcwik] Fix minor checkstyle/findbugs errors

[mariagh] Add support for streaming side inputs in the DirectRunner

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

[axelmagn] Fix StateRequestHandler interface to be idiomatic

[herohde] Add Go support for universal runners, incl Flink

[herohde] CR: Fixed comments for job service helper functions

[iemejia] Add missing ASF license to ExecutableStageTranslation file

[szewinho] Updated gradle build to use jaxb-api. Jaxb-api version set to 2.2.12

[mariand] Switched AvroIO default codec to snappyCodec().

[yifanzou] [BEAM-3840] Get python mobile-gaming automating on core runners

[coheigea] Remove "i == numSplits" condition, which can never be reached as i is

[sidhom] [BEAM-3565] Clean up ExecutableStage

[wcn] Fix incorrect read of atomic counter.

[tgroh] Add a suborder parameter to Networks#topologicalOrder

[herohde] [BEAM-3893] Add fallback to unauthenticated access for GCS IO

[robertwb] [BEAM-3865] Fix watermark hold handling bug.

[robertwb] [BEAM-2927] Python support for dataflow portable side inputs over Fn 
API

[Pablo] Fixing check for sideinput_io_metrics experiment flag.

[herohde] CR: fix typo

[herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns

[ccy] Replace side inputs when applying PTransformOverrides

[tgroh] Add QueryablePipeline#forTransforms

[herohde] Allow manual override of Go worker binary

[aaltay] [BEAM-3861] Improve test infra in Python SDK for streaming end-to-end

[iemejia] Remove testing package-info from main package for GCP IO

[iemejia] Update maven failsafe/surefire plugin to version 2.21.0

[iemejia] [BEAM-3873] Update commons-compress to version 1.16.1 (fix

[iemejia] Remove maven warnings

[tgroh] Add Side Inputs to ExecutableStage

[herohde] [BEAM-3866] Remove windowed value requirement for External

[lcwik] [BEAM-3903] Build the Go SDK examples using gradle

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #4

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[tgroh] Enable Proto Round Trips in the DirectRunner Tests

[lcwik] [BEAM-3249, BEAM-3457] Make Apache Rat a dependency of the Java and

[sidhom] Add Java bounded read overrides

[herohde] [BEAM-3817] Switch BQ write to not use side input

[lcwik] [BEAM-3457] Exclude generated and data files part of the Go SDK.

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[coheigea] Replacing size() == 0 with isEmpty()

[markliu] [BEAM-3841] Fix TestDataflowRunner.run to run_pipeline

[coheigea] Put the String literal first when comparing it to an Object

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[ekirpichov] BigQueryIO.read().fromQuery() improvements

[szewinho] [BEAM-3060] Fixing mvn dependency issue when runnning filebasedIOIT

[lukasz.gajowy] [BEAM-3798] Remove error check on dataflow when getting batch 
job state

[tgroh] Add ExecutableStage -> ProcessBundleDescriptor

[tgroh] Clean up GrpcFnServer Constructor Signature

[tgroh] Add a RemoteExecutionTest

[tgroh] Clean up a couple of SdkHarnessClient Signatures

[tgroh] Separate RemoteInputDestinations from SdkHarnessClient

[XuMingmin] [SQL] Add support for DOT expression (#4863)

[herohde] Add TODO to revert Go IO to use side input

[jb] [BEAM-3500] "Attach" JDBC connection to the bundle (improve the pooling)

[jb] [BEAM-3500] Test if the user provides both withDataSourceConfiguration()

[jb] [BEAM-3500] Wrap the datasource as a poolable datasource and expose

[jb] [BEAM-3500] Add commons-pool2 dependency

[jb] [BEAM-3500] Only expose max number of connections in the pool to the

[jb] [BEAM-3500] Cleanup pool configuration parameters

[jb] [BEAM-3500] Remove dataSourceFactory

[jb] [BEAM-3500] Remove unecessary check on dataSourceConfiguration

[szewinho] Jax-api added to dependencyManagement.

[szewinho] Jaxb-api added to beam root pom.xml

[szewinho] Removed empty line

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

[axelmagn] Add a generic interface for the state service.

[herohde] Remove WindowedValue on PCollections for Go SDK

[cclauss] [BEAM-3872] Change Exception.message --> Exception.args[0]

[alan.myrvold] [BEAM-3249] Publish java artifacts using gradle

[herohde] CR: Fix comments to remove old windowing

[lcwik] Fix minor checkstyle/findbugs errors

[mariagh] Add support for streaming side inputs in the DirectRunner

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

[axelmagn] Fix StateRequestHandler interface to be idiomatic

[herohde] Add Go support for universal runners, incl Flink

[herohde] CR: Fixed comments for job service helper functions

[iemejia] Add missing ASF license to ExecutableStageTranslation file

[szewinho] Updated gradle build to use jaxb-api. Jaxb-api version set to 2.2.12

[mariand] Switched AvroIO default codec to snappyCodec().

[yifanzou] [BEAM-3840] Get python mobile-gaming automating on core runners

[coheigea] Remove "i == numSplits" condition, which can never be reached as i is

[sidhom] [BEAM-3565] Clean up ExecutableStage

[wcn] Fix incorrect read of atomic counter.

[tgroh] Add a suborder parameter to Networks#topologicalOrder

[herohde] [BEAM-3893] Add fallback to unauthenticated access for GCS IO

[robertwb] [BEAM-3865] Fix watermark hold handling bug.

[robertwb] [BEAM-2927] Python support for dataflow portable side inputs over Fn 
API

[Pablo] Fixing check for sideinput_io_metrics experiment flag.

[herohde] CR: fix typo

[herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns

[ccy] Replace side inputs when applying PTransformOverrides

[tgroh] Add QueryablePipeline#forTransforms

[herohde] Allow manual override of Go worker binary

[aaltay] [BEAM-3861] Improve test infra in Python SDK for streaming end-to-end

[iemejia] Remove testing package-info from main package for GCP IO

[iemejia] Update maven failsafe/surefire plugin to version 2.21.0

[iemejia] [BEAM-3873] Update commons-compress to version 1.16.1 (fix

[iemejia] Remove maven warnings

[tgroh] Add Side Inputs to ExecutableStage

[herohde] [BEAM-3866] Remove windowed value requirement for External

[lcwik] [BEAM-3903] Build the Go SDK examples using 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #9

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[tgroh] Enable Proto Round Trips in the DirectRunner Tests

[lcwik] [BEAM-3249, BEAM-3457] Make Apache Rat a dependency of the Java and

[sidhom] Add Java bounded read overrides

[herohde] [BEAM-3817] Switch BQ write to not use side input

[lcwik] [BEAM-3457] Exclude generated and data files part of the Go SDK.

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[coheigea] Replacing size() == 0 with isEmpty()

[markliu] [BEAM-3841] Fix TestDataflowRunner.run to run_pipeline

[coheigea] Put the String literal first when comparing it to an Object

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[ekirpichov] BigQueryIO.read().fromQuery() improvements

[szewinho] [BEAM-3060] Fixing mvn dependency issue when runnning filebasedIOIT

[lukasz.gajowy] [BEAM-3798] Remove error check on dataflow when getting batch 
job state

[tgroh] Add ExecutableStage -> ProcessBundleDescriptor

[tgroh] Clean up GrpcFnServer Constructor Signature

[tgroh] Add a RemoteExecutionTest

[tgroh] Clean up a couple of SdkHarnessClient Signatures

[tgroh] Separate RemoteInputDestinations from SdkHarnessClient

[XuMingmin] [SQL] Add support for DOT expression (#4863)

[herohde] Add TODO to revert Go IO to use side input

[jb] [BEAM-3500] "Attach" JDBC connection to the bundle (improve the pooling)

[jb] [BEAM-3500] Test if the user provides both withDataSourceConfiguration()

[jb] [BEAM-3500] Wrap the datasource as a poolable datasource and expose

[jb] [BEAM-3500] Add commons-pool2 dependency

[jb] [BEAM-3500] Only expose max number of connections in the pool to the

[jb] [BEAM-3500] Cleanup pool configuration parameters

[jb] [BEAM-3500] Remove dataSourceFactory

[jb] [BEAM-3500] Remove unecessary check on dataSourceConfiguration

[szewinho] Jax-api added to dependencyManagement.

[szewinho] Jaxb-api added to beam root pom.xml

[szewinho] Removed empty line

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

[axelmagn] Add a generic interface for the state service.

[herohde] Remove WindowedValue on PCollections for Go SDK

[cclauss] [BEAM-3872] Change Exception.message --> Exception.args[0]

[alan.myrvold] [BEAM-3249] Publish java artifacts using gradle

[herohde] CR: Fix comments to remove old windowing

[lcwik] Fix minor checkstyle/findbugs errors

[mariagh] Add support for streaming side inputs in the DirectRunner

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

[axelmagn] Fix StateRequestHandler interface to be idiomatic

[herohde] Add Go support for universal runners, incl Flink

[herohde] CR: Fixed comments for job service helper functions

[iemejia] Add missing ASF license to ExecutableStageTranslation file

[szewinho] Updated gradle build to use jaxb-api. Jaxb-api version set to 2.2.12

[mariand] Switched AvroIO default codec to snappyCodec().

[yifanzou] [BEAM-3840] Get python mobile-gaming automating on core runners

[coheigea] Remove "i == numSplits" condition, which can never be reached as i is

[sidhom] [BEAM-3565] Clean up ExecutableStage

[wcn] Fix incorrect read of atomic counter.

[tgroh] Add a suborder parameter to Networks#topologicalOrder

[herohde] [BEAM-3893] Add fallback to unauthenticated access for GCS IO

[robertwb] [BEAM-3865] Fix watermark hold handling bug.

[robertwb] [BEAM-2927] Python support for dataflow portable side inputs over Fn 
API

[Pablo] Fixing check for sideinput_io_metrics experiment flag.

[herohde] CR: fix typo

[herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns

[ccy] Replace side inputs when applying PTransformOverrides

[tgroh] Add QueryablePipeline#forTransforms

[herohde] Allow manual override of Go worker binary

[aaltay] [BEAM-3861] Improve test infra in Python SDK for streaming end-to-end

[iemejia] Remove testing package-info from main package for GCP IO

[iemejia] Update maven failsafe/surefire plugin to version 2.21.0

[iemejia] [BEAM-3873] Update commons-compress to version 1.16.1 (fix

[iemejia] Remove maven warnings

[tgroh] Add Side Inputs to ExecutableStage

[herohde] [BEAM-3866] Remove windowed value requirement for External

[lcwik] [BEAM-3903] Build the Go SDK examples using gradle

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT_HDFS #4

[Apache Jenkins Server]

See 


Changes:

[alex] Correct BigQuery.write JavaDoc example

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[tgroh] Enable Proto Round Trips in the DirectRunner Tests

[lcwik] [BEAM-3249, BEAM-3457] Make Apache Rat a dependency of the Java and

[sidhom] Add Java bounded read overrides

[herohde] [BEAM-3817] Switch BQ write to not use side input

[lcwik] [BEAM-3457] Exclude generated and data files part of the Go SDK.

[sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction

[sidhom] Fix typo

[coheigea] Replacing size() == 0 with isEmpty()

[markliu] [BEAM-3841] Fix TestDataflowRunner.run to run_pipeline

[coheigea] Put the String literal first when comparing it to an Object

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[tgroh] Use InstructionRequestHandler in RemoteEnvironment

[ekirpichov] BigQueryIO.read().fromQuery() improvements

[szewinho] [BEAM-3060] Fixing mvn dependency issue when runnning filebasedIOIT

[lukasz.gajowy] [BEAM-3798] Remove error check on dataflow when getting batch 
job state

[tgroh] Add ExecutableStage -> ProcessBundleDescriptor

[tgroh] Clean up GrpcFnServer Constructor Signature

[tgroh] Add a RemoteExecutionTest

[tgroh] Clean up a couple of SdkHarnessClient Signatures

[tgroh] Separate RemoteInputDestinations from SdkHarnessClient

[XuMingmin] [SQL] Add support for DOT expression (#4863)

[herohde] Add TODO to revert Go IO to use side input

[jb] [BEAM-3500] "Attach" JDBC connection to the bundle (improve the pooling)

[jb] [BEAM-3500] Test if the user provides both withDataSourceConfiguration()

[jb] [BEAM-3500] Wrap the datasource as a poolable datasource and expose

[jb] [BEAM-3500] Add commons-pool2 dependency

[jb] [BEAM-3500] Only expose max number of connections in the pool to the

[jb] [BEAM-3500] Cleanup pool configuration parameters

[jb] [BEAM-3500] Remove dataSourceFactory

[jb] [BEAM-3500] Remove unecessary check on dataSourceConfiguration

[szewinho] Jax-api added to dependencyManagement.

[szewinho] Jaxb-api added to beam root pom.xml

[szewinho] Removed empty line

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

[axelmagn] Add a generic interface for the state service.

[herohde] Remove WindowedValue on PCollections for Go SDK

[cclauss] [BEAM-3872] Change Exception.message --> Exception.args[0]

[alan.myrvold] [BEAM-3249] Publish java artifacts using gradle

[herohde] CR: Fix comments to remove old windowing

[lcwik] Fix minor checkstyle/findbugs errors

[mariagh] Add support for streaming side inputs in the DirectRunner

[andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS 
calls

[ccy] Add support for PaneInfo in WindowedValues

[axelmagn] Write unit tests for GrpcStateService.

[XuMingmin] Bump calcite and avatica versions (#4887)

[axelmagn] Fix StateRequestHandler interface to be idiomatic

[herohde] Add Go support for universal runners, incl Flink

[herohde] CR: Fixed comments for job service helper functions

[iemejia] Add missing ASF license to ExecutableStageTranslation file

[szewinho] Updated gradle build to use jaxb-api. Jaxb-api version set to 2.2.12

[mariand] Switched AvroIO default codec to snappyCodec().

[yifanzou] [BEAM-3840] Get python mobile-gaming automating on core runners

[coheigea] Remove "i == numSplits" condition, which can never be reached as i is

[sidhom] [BEAM-3565] Clean up ExecutableStage

[wcn] Fix incorrect read of atomic counter.

[tgroh] Add a suborder parameter to Networks#topologicalOrder

[herohde] [BEAM-3893] Add fallback to unauthenticated access for GCS IO

[robertwb] [BEAM-3865] Fix watermark hold handling bug.

[robertwb] [BEAM-2927] Python support for dataflow portable side inputs over Fn 
API

[Pablo] Fixing check for sideinput_io_metrics experiment flag.

[herohde] CR: fix typo

[herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns

[ccy] Replace side inputs when applying PTransformOverrides

[tgroh] Add QueryablePipeline#forTransforms

[herohde] Allow manual override of Go worker binary

[aaltay] [BEAM-3861] Improve test infra in Python SDK for streaming end-to-end

[iemejia] Remove testing package-info from main package for GCP IO

[iemejia] Update maven failsafe/surefire plugin to version 2.21.0

[iemejia] [BEAM-3873] Update commons-compress to version 1.16.1 (fix

[iemejia] Remove maven warnings

[tgroh] Add Side Inputs to ExecutableStage

[herohde] [BEAM-3866] Remove windowed value requirement for External

[lcwik] [BEAM-3903] Build the Go SDK examples using gradle

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT_HDFS #3

[Apache Jenkins Server]

See 


--
[...truncated 78.84 KB...]
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Not a valid TFRecord. Fewer than 12 
bytes.
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at 
org.apache.beam.sdk.io.TFRecordIO$TFRecordCodec.read(TFRecordIO.java:631)
at 
org.apache.beam.sdk.io.TFRecordIO$TFRecordSource$TFRecordReader.readNextRecord(TFRecordIO.java:526)
at 
org.apache.beam.sdk.io.CompressedSource$CompressedReader.readNextRecord(CompressedSource.java:426)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:473)
at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.advance(OffsetBasedSource.java:267)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:602)
... 14 more
java.io.IOException: Failed to advance reader of source: 
hdfs://146.148.62.9:9000/TEXTIO_IT__1521892046445-1-of-3.tfrecord range 
[0, 1914)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:605)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.advance(ReadOperation.java:379)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:185)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:150)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:74)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:381)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:353)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:284)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Invalid data
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at 
org.apache.beam.sdk.io.TFRecordIO$TFRecordCodec.read(TFRecordIO.java:642)
at 
org.apache.beam.sdk.io.TFRecordIO$TFRecordSource$TFRecordReader.readNextRecord(TFRecordIO.java:526)
at 
org.apache.beam.sdk.io.CompressedSource$CompressedReader.readNextRecord(CompressedSource.java:426)
at 
org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:473)
at 
org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.advance(OffsetBasedSource.java:267)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:602)
... 14 more
java.io.IOException: Failed to advance reader of source: 
hdfs://146.148.62.9:9000/TEXTIO_IT__1521892046445-1-of-3.tfrecord range 
[0, 1914)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.advance(WorkerCustomSources.java:605)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.advance(ReadOperation.java:379)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:185)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:150)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:74)
at 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #2

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83990
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:44
Start Date: 24/Mar/18 11:44
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375877853
 
 
   Run Java TFRecordIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83990)
Time Spent: 6h 10m  (was: 6h)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83989=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83989
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:38
Start Date: 24/Mar/18 11:38
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375877105
 
 
   Run Java XmlIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83989)
Time Spent: 6h  (was: 5h 50m)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #3

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #3

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83988
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:31
Start Date: 24/Mar/18 11:31
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375876057
 
 
   Run Java AvroIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83988)
Time Spent: 5h 50m  (was: 5h 40m)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83987
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:26
Start Date: 24/Mar/18 11:26
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375875453
 
 
   Run Java CompressedTextIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83987)
Time Spent: 5h 40m  (was: 5.5h)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #8

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83985
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:18
Start Date: 24/Mar/18 11:18
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375874415
 
 
   Run Java TextIO Performance Test HDFS


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83985)
Time Spent: 5.5h  (was: 5h 20m)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3060) Add performance tests for commonly used file-based I/O PTransforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3060?focusedWorklogId=83983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-83983
 ]

ASF GitHub Bot logged work on BEAM-3060:


Author: ASF GitHub Bot
Created on: 24/Mar/18 11:13
Start Date: 24/Mar/18 11:13
Worklog Time Spent: 10m 
  Work Description: szewi commented on issue #4861: [BEAM-3060] Jenkins 
configuration allowing to run FilebasedIO tests on HDFS.
URL: https://github.com/apache/beam/pull/4861#issuecomment-375873823
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 83983)
Time Spent: 5h 20m  (was: 5h 10m)

> Add performance tests for commonly used file-based I/O PTransforms
> --
>
> Key: BEAM-3060
> URL: https://issues.apache.org/jira/browse/BEAM-3060
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Szymon Nieradka
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> We recently added a performance testing framework [1] that can be used to do 
> following.
> (1) Execute Beam tests using PerfkitBenchmarker
> (2) Manage Kubernetes-based deployments of data stores.
> (3) Easily publish benchmark results. 
> I think it will be useful to add performance tests for commonly used 
> file-based I/O PTransforms using this framework. I suggest looking into 
> following formats initially.
> (1) AvroIO
> (2) TextIO
> (3) Compressed text using TextIO
> (4) TFRecordIO
> It should be possibly to run these tests for various Beam runners (Direct, 
> Dataflow, Flink, Spark, etc.) and file-systems (GCS, local, HDFS, etc.) 
> easily.
> In the initial version, tests can be made manually triggerable for PRs 
> through Jenkins. Later, we could make some of these tests run periodically 
> and publish benchmark results (to BigQuery) through PerfkitBenchmarker.
> [1] https://beam.apache.org/documentation/io/testing/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1176

[Apache Jenkins Server]

See 


--
[...truncated 776.79 KB...]
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:pair", 
  "component_encodings": [
{
  "@type": "kind:bytes"
}, 
{
  "@type": 
"VarIntCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxhiUWeeSXOIA5XIYNmYyFjbSFTkh4A89cR+g==",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "compute/MapToVoidKey0.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s2"
}, 
"serialized_fn": "", 
"user_name": "compute/MapToVoidKey0"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
root: INFO: Create job: 
root: INFO: Created job with id: [2018-03-24_02_24_12-11264698875865366110]
root: INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-24_02_24_12-11264698875865366110?project=apache-beam-testing
root: INFO: Job 2018-03-24_02_24_12-11264698875865366110 is in state 
JOB_STATE_PENDING
root: INFO: 2018-03-24T09:24:12.568Z: JOB_MESSAGE_WARNING: Job 
2018-03-24_02_24_12-11264698875865366110 might autoscale up to 1000 workers.
root: INFO: 2018-03-24T09:24:12.593Z: JOB_MESSAGE_DETAILED: Autoscaling is 
enabled for job 2018-03-24_02_24_12-11264698875865366110. The number of workers 
will be between 1 and 1000.
root: INFO: 2018-03-24T09:24:12.615Z: JOB_MESSAGE_DETAILED: Autoscaling was 
automatically enabled for job 2018-03-24_02_24_12-11264698875865366110.
root: INFO: 2018-03-24T09:24:15.210Z: JOB_MESSAGE_DETAILED: Checking required 
Cloud APIs are enabled.
root: INFO: 2018-03-24T09:24:15.378Z: JOB_MESSAGE_DETAILED: Checking 
permissions granted to controller Service Account.
root: INFO: 2018-03-24T09:24:16.085Z: JOB_MESSAGE_DETAILED: Expanding 
CoGroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T09:24:16.117Z: JOB_MESSAGE_DEBUG: Combiner lifting 
skipped for step assert_that/Group/GroupByKey: GroupByKey not followed by a 
combiner.
root: INFO: 2018-03-24T09:24:16.149Z: JOB_MESSAGE_DETAILED: Expanding 
GroupByKey operations into optimizable parts.
root: INFO: 2018-03-24T09:24:16.180Z: JOB_MESSAGE_DETAILED: Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
root: INFO: 2018-03-24T09:24:16.216Z: JOB_MESSAGE_DEBUG: Annotating graph with 
Autotuner information.
root: INFO: 2018-03-24T09:24:16.255Z: JOB_MESSAGE_DETAILED: Fusing adjacent 
ParDo, Read, Write, and Flatten operations
root: INFO: 2018-03-24T09:24:16.278Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11 for input s10.out
root: INFO: 2018-03-24T09:24:16.310Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Reify, through flatten 
assert_that/Group/Flatten, into producer assert_that/Group/pair_with_1
root: INFO: 2018-03-24T09:24:16.341Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/GroupByKey/GroupByWindow into 
assert_that/Group/GroupByKey/Read
root: INFO: 2018-03-24T09:24:16.376Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Unkey into assert_that/Group/Map(_merge_tagged_vals_under_key)
root: INFO: 2018-03-24T09:24:16.400Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Match into assert_that/Unkey
root: INFO: 2018-03-24T09:24:16.428Z: JOB_MESSAGE_DETAILED: Fusing consumer 
assert_that/Group/Map(_merge_tagged_vals_under_key) into 
assert_that/Group/GroupByKey/GroupByWindow
root: INFO: 2018-03-24T09:24:16.475Z: JOB_MESSAGE_DETAILED: Unzipping flatten 
s11-u13 for input s12-reify-value0-c11
root: INFO: 2018-03-24T09:24:16.505Z: JOB_MESSAGE_DETAILED: Fusing unzipped 
copy of assert_that/Group/GroupByKey/Write, through flatten s11-u13, into 
producer 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4496

[Apache Jenkins Server]

See 


--
[...truncated 290.11 KB...]
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-03-23 17:50:29.204663
+++ 
:after
  2018-03-24 09:09:22.836612
@@ -29,15 +29,14 @@
 import unittest
 import uuid
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import streaming_wordcount
 from apache_beam.io.gcp.tests.pubsub_matcher import PubSubMessageMatcher
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 INPUT_TOPIC = 'wc_topic_input'
 OUTPUT_TOPIC = 'wc_topic_output'
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-24 09:09:23.397605
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 09:09:23.541815
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.internal import pickler
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.assert_that import assert_that as hc_assert_that
+from hamcrest.core.base_matcher import BaseMatcher
 
 
 # A simple matcher that is ued for testing extra options appending.
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-24 09:09:23.555981
@@ -25,12 +25,11 @@
 import logging
 import time
 
-from hamcrest.core.base_matcher import BaseMatcher
-
 from apache_beam.io.filesystems import FileSystems
 from apache_beam.runners.runner import PipelineState
 from apache_beam.testing import test_utils as utils
 from apache_beam.utils import retry
+from hamcrest.core.base_matcher import BaseMatcher
 
 __all__ = [
 'PipelineStateMatcher',
ERROR: 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5212

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1504

[Apache Jenkins Server]

See 


--
[...truncated 90.41 KB...]
'apache-beam-testing:bqjob_reec656584e56e5b_016256aabb92_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 06:20:54,641 d450e581 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 06:21:22,168 d450e581 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 06:21:29,333 d450e581 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r199fef8a42053c96_016256ab440e_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r199fef8a42053c96_016256ab440e_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r199fef8a42053c96_016256ab440e_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 06:21:29,333 d450e581 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 06:21:55,514 d450e581 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 06:21:57,681 d450e581 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_rb0748c781b78dda_016256abb2c6_1 ... (0s) Current status: 
RUNNING 
Waiting on bqjob_rb0748c781b78dda_016256abb2c6_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_rb0748c781b78dda_016256abb2c6_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-24 06:21:57,682 d450e581 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-24 06:22:25,830 d450e581 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-24 06:22:28,035 d450e581 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r78581c1d7712350d_016256ac2953_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r78581c1d7712350d_016256ac2953_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r78581c1d7712350d_016256ac2953_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #56

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1060

[Apache Jenkins Server]

See 


--
[...truncated 61.52 KB...]
[INFO] Copying 212 resources
[INFO] 
[INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-go-cmd-source) @ 
beam-sdks-go ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] 
[INFO] --- maven-assembly-plugin:3.1.0:single (export-go-pkg-sources) @ 
beam-sdks-go ---
[INFO] Reading assembly descriptor: descriptor.xml
[INFO] Building zip: 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) 
@ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:get (go-get-imports) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go get google.golang.org/grpc 
golang.org/x/oauth2/google google.golang.org/api/storage/v1 
github.com/spf13/cobra cloud.google.com/go/bigquery 
google.golang.org/api/googleapi google.golang.org/api/dataflow/v1b3
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build-linux-amd64) @ beam-sdks-go 
---
[INFO] Prepared command line : bin/go build -buildmode=default -o 

 github.com/apache/beam/sdks/go/cmd/beamctl
[INFO] The Result file has been successfuly created : 

[INFO] 
[INFO] --- maven-checkstyle-plugin:3.0.0:check (default) @ beam-sdks-go ---
[INFO] 
[INFO] --- mvn-golang-wrapper:2.1.6:test (go-test) @ beam-sdks-go ---
[INFO] Prepared command line : bin/go test ./...
[INFO] 
[INFO] -Exec.Out-
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/beamctl/cmd  [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/specialize   [no test files]
[INFO] ?github.com/apache/beam/sdks/go/cmd/symtab   [no test files]
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam 0.020s
[INFO] ok   github.com/apache/beam/sdks/go/pkg/beam/artifact0.124s
[INFO] 
[ERROR] 
[ERROR] -Exec.Err-
[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx
[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication
[ERROR] 
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Beam :: Parent .. SUCCESS [  7.882 s]
[INFO] Apache Beam :: SDKs :: Java :: Build Tools . SUCCESS [  3.867 s]
[INFO] Apache Beam :: Model ... SUCCESS [  0.106 s]
[INFO] Apache Beam :: Model :: Pipeline ... SUCCESS [  8.205 s]
[INFO] Apache Beam :: Model :: Job Management . SUCCESS [  3.367 s]
[INFO] Apache Beam :: Model :: Fn Execution ... SUCCESS [  4.161 s]
[INFO] Apache Beam :: SDKs  SUCCESS [  0.156 s]
[INFO] Apache Beam :: SDKs :: Go .. FAILURE [ 23.041 s]
[INFO] Apache Beam :: SDKs :: Go :: Container . SKIPPED
[INFO] Apache Beam :: SDKs :: Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Core  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Fn Execution  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Extensions :: Google Cloud Platform Core 
SKIPPED
[INFO] Apache Beam :: Runners . SKIPPED
[INFO] Apache Beam :: Runners :: Core Construction Java ... SKIPPED
[INFO] Apache Beam :: Runners :: Core Java  SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Harness . SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: Container ... SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Amazon Web Services SKIPPED
[INFO] Apache Beam :: Runners :: Local Java Core .. SKIPPED
[INFO] Apache Beam :: Runners :: Direct Java .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP .. SKIPPED
[INFO] Apache Beam :: SDKs :: Java :: IO :: Common  SKIPPED
[INFO] Apache Beam :: SDKs :: Java