See <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/9/display/redirect?page=changes>
Changes: [alex] Correct BigQuery.write JavaDoc example [rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not [tgroh] Enable Proto Round Trips in the DirectRunner Tests [lcwik] [BEAM-3249, BEAM-3457] Make Apache Rat a dependency of the Java and [sidhom] Add Java bounded read overrides [herohde] [BEAM-3817] Switch BQ write to not use side input [lcwik] [BEAM-3457] Exclude generated and data files part of the Go SDK. [sidhom] Add ExecutableStagePayload to make aid runner stage reconstruction [sidhom] Fix typo [coheigea] Replacing size() == 0 with isEmpty() [markliu] [BEAM-3841] Fix TestDataflowRunner.run to run_pipeline [coheigea] Put the String literal first when comparing it to an Object [ankurgoenka] Send error response for process bundle progress instead of throwing and [tgroh] Use InstructionRequestHandler in RemoteEnvironment [ekirpichov] BigQueryIO.read().fromQuery() improvements [szewinho] [BEAM-3060] Fixing mvn dependency issue when runnning filebasedIOIT [lukasz.gajowy] [BEAM-3798] Remove error check on dataflow when getting batch job state [tgroh] Add ExecutableStage -> ProcessBundleDescriptor [tgroh] Clean up GrpcFnServer Constructor Signature [tgroh] Add a RemoteExecutionTest [tgroh] Clean up a couple of SdkHarnessClient Signatures [tgroh] Separate RemoteInputDestinations from SdkHarnessClient [XuMingmin] [SQL] Add support for DOT expression (#4863) [herohde] Add TODO to revert Go IO to use side input [jb] [BEAM-3500] "Attach" JDBC connection to the bundle (improve the pooling) [jb] [BEAM-3500] Test if the user provides both withDataSourceConfiguration() [jb] [BEAM-3500] Wrap the datasource as a poolable datasource and expose [jb] [BEAM-3500] Add commons-pool2 dependency [jb] [BEAM-3500] Only expose max number of connections in the pool to the [jb] [BEAM-3500] Cleanup pool configuration parameters [jb] [BEAM-3500] Remove dataSourceFactory [jb] [BEAM-3500] Remove unecessary check on dataSourceConfiguration [szewinho] Jax-api added to dependencyManagement. [szewinho] Jaxb-api added to beam root pom.xml [szewinho] Removed empty line [Pablo] Fixing counter string name. [tgroh] Add Network Utility Methods [lcwik] Fix missing license header in test. [aaltay] Enable input subscription in Python streaming wordcount (#4864) [tgroh] fixup! Add Network Utility Methods [tgroh] Remove Future#set(null) in fn-execution [herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps [axelmagn] Add a generic interface for the state service. [herohde] Remove WindowedValue on PCollections for Go SDK [cclauss] [BEAM-3872] Change Exception.message --> Exception.args[0] [alan.myrvold] [BEAM-3249] Publish java artifacts using gradle [herohde] CR: Fix comments to remove old windowing [lcwik] Fix minor checkstyle/findbugs errors [mariagh] Add support for streaming side inputs in the DirectRunner [andreas.ehrencrona] [BEAM-2264] Credentials were not being reused between GCS calls [ccy] Add support for PaneInfo in WindowedValues [axelmagn] Write unit tests for GrpcStateService. [XuMingmin] Bump calcite and avatica versions (#4887) [axelmagn] Fix StateRequestHandler interface to be idiomatic [herohde] Add Go support for universal runners, incl Flink [herohde] CR: Fixed comments for job service helper functions [iemejia] Add missing ASF license to ExecutableStageTranslation file [szewinho] Updated gradle build to use jaxb-api. Jaxb-api version set to 2.2.12 [mariand] Switched AvroIO default codec to snappyCodec(). [yifanzou] [BEAM-3840] Get python mobile-gaming automating on core runners [coheigea] Remove "i == numSplits" condition, which can never be reached as i is [sidhom] [BEAM-3565] Clean up ExecutableStage [wcn] Fix incorrect read of atomic counter. [tgroh] Add a suborder parameter to Networks#topologicalOrder [herohde] [BEAM-3893] Add fallback to unauthenticated access for GCS IO [robertwb] [BEAM-3865] Fix watermark hold handling bug. [robertwb] [BEAM-2927] Python support for dataflow portable side inputs over Fn API [Pablo] Fixing check for sideinput_io_metrics experiment flag. [herohde] CR: fix typo [herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns [ccy] Replace side inputs when applying PTransformOverrides [tgroh] Add QueryablePipeline#forTransforms [herohde] Allow manual override of Go worker binary [aaltay] [BEAM-3861] Improve test infra in Python SDK for streaming end-to-end [iemejia] Remove testing package-info from main package for GCP IO [iemejia] Update maven failsafe/surefire plugin to version 2.21.0 [iemejia] [BEAM-3873] Update commons-compress to version 1.16.1 (fix [iemejia] Remove maven warnings [tgroh] Add Side Inputs to ExecutableStage [herohde] [BEAM-3866] Remove windowed value requirement for External [lcwik] [BEAM-3903] Build the Go SDK examples using gradle [markliu] Clean up terminal state check in TestDataflowRunner [robertwb] Add mapper microbenchmark. [szewinho] Scope removed from root pom.xml and jaxb-api dependency set to [Pablo] Updating dataflow API protocol buffers. [tgroh] Make RemoteEnvironment public [boyuanz] Add cython annotation to make DistributionAccumulator faster [tgroh] Move getId to the top-level pipeline node [tgroh] Add FusedPipeline#toPipeline [robertwb] Remove from docs generation. [wcn] BEAM-3894: Add Protocol Buffer support [altay] Convert big query writes to beam.io.WriteToBigQuery [aaltay] Improve test util for PubSub exist check (#4922) [XuMingmin] [BEAM-3851] Option to preserve element timestamp while publishing to [ankurgoenka] Send worker_id in all channels to runner Adding [wcn] Updating generated files. [ehudm] Reduce precommit test timeouts for Java and Go. [altay] More graceful fallback when grpc is not present. [markliu] [BEAM-3861] Complete streaming wordcount test in Python SDK [aaltay] [BEAM-3738] Enable py3 lint and cleanup tox.ini. (#4877) ------------------------------------------ [...truncated 2.05 KB...] INFO: Display format "default". DEBUG: SDK update checks are disabled. [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8042547548148976285.sh + cp /home/jenkins/.kube/config <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins8523930096014323346.sh + kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> create namespace filebasedioithdfs-1521890089887 namespace "filebasedioithdfs-1521890089887" created [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6156137348082625348.sh ++ kubectl config current-context + kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> config set-context gke_apache-beam-testing_us-central1-a_io-datastores --namespace=filebasedioithdfs-1521890089887 Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified. [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins3778855273628284256.sh + rm -rf PerfKitBenchmarker [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1468335882873496441.sh + rm -rf .env [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins2482970785995407264.sh + virtualenv .env --system-site-packages New python executable in .env/bin/python Installing setuptools, pip...done. [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins6024071469595385299.sh + .env/bin/pip install --upgrade setuptools pip Downloading/unpacking setuptools from https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc Downloading/unpacking pip from https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb Installing collected packages: setuptools, pip Found existing installation: setuptools 2.2 Uninstalling setuptools: Successfully uninstalled setuptools Found existing installation: pip 1.5.4 Uninstalling pip: Successfully uninstalled pip Successfully installed setuptools pip Cleaning up... [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins1578255523698464554.sh + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git Cloning into 'PerfKitBenchmarker'... [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins371988884710762486.sh + .env/bin/pip install -r PerfKitBenchmarker/requirements.txt Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14)) Requirement already satisfied: jinja2>=2.7 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 15)) Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 16)) Collecting colorlog[windows]==2.6.0 (from -r PerfKitBenchmarker/requirements.txt (line 17)) Using cached colorlog-2.6.0-py2.py3-none-any.whl Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18)) Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 19)) Using cached futures-3.2.0-py2-none-any.whl Requirement already satisfied: PyYAML==3.12 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 20)) Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21)) Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22)) Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: functools32 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 23)) Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt (line 24)) Using cached contextlib2-0.5.5-py2.py3-none-any.whl Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25)) Using cached pywinrm-0.3.0-py2.py3-none-any.whl Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages (from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r PerfKitBenchmarker/requirements.txt (line 15)) Requirement already satisfied: colorama; extra == "windows" in /usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r PerfKitBenchmarker/requirements.txt (line 17)) Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached xmltodict-0.11.0-py2.py3-none-any.whl Collecting requests-ntlm>=0.3.0 (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached requests_ntlm-1.1.0-py2.py3-none-any.whl Requirement already satisfied: requests>=2.9.1 in /usr/local/lib/python2.7/dist-packages (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Collecting ntlm-auth>=1.0.2 (from requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached ntlm_auth-1.1.0-py2.py3-none-any.whl Collecting cryptography>=1.3 (from requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached cryptography-2.2.1-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: enum34; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Collecting asn1crypto>=0.21.0 (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Using cached asn1crypto-0.24.0-py2.py3-none-any.whl Requirement already satisfied: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) Installing collected packages: absl-py, colorlog, blinker, futures, pint, numpy, contextlib2, xmltodict, ntlm-auth, pycparser, cffi, asn1crypto, cryptography, requests-ntlm, pywinrm Successfully installed absl-py-0.1.12 asn1crypto-0.24.0 blinker-1.4 cffi-1.11.5 colorlog-2.6.0 contextlib2-0.5.5 cryptography-2.2.1 futures-3.2.0 ntlm-auth-1.1.0 numpy-1.13.3 pint-0.8.1 pycparser-2.18 pywinrm-0.3.0 requests-ntlm-1.1.0 xmltodict-0.11.0 [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4495700473950752.sh + .env/bin/pip install -e 'src/sdks/python/[gcp,test]' Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/sdks/python> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.5.0.dev0) Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0) Collecting dill==0.2.6 (from apache-beam==2.5.0.dev0) Collecting grpcio<2,>=1.8 (from apache-beam==2.5.0.dev0) Using cached grpcio-1.10.0-cp27-cp27mu-manylinux1_x86_64.whl Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0) Collecting httplib2<0.10,>=0.8 (from apache-beam==2.5.0.dev0) Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.5.0.dev0) Using cached mock-2.0.0-py2.py3-none-any.whl Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.5.0.dev0) Using cached oauth2client-4.1.2-py2.py3-none-any.whl Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.5.0.dev0) Using cached protobuf-3.5.2.post1-cp27-cp27mu-manylinux1_x86_64.whl Requirement already satisfied: pyyaml<4.0.0,>=3.12 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0) Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.5.0.dev0) Requirement already satisfied: six<1.12,>=1.9 in /usr/local/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0) Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.5.0.dev0) Using cached typing-3.6.4-py2-none-any.whl Requirement already satisfied: futures<4.0.0,>=3.1.1 in ./.env/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0) Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.5.0.dev0) Using cached google_apitools-0.5.20-py2-none-any.whl Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from apache-beam==2.5.0.dev0) Collecting googledatastore==7.0.1 (from apache-beam==2.5.0.dev0) Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.5.0.dev0) Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl Collecting proto-google-cloud-pubsub-v1==0.15.4 (from apache-beam==2.5.0.dev0) Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.5.0.dev0) Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl Collecting nose>=1.3.7 (from apache-beam==2.5.0.dev0) Using cached nose-1.3.7-py2-none-any.whl Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.5.0.dev0) Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl Requirement already satisfied: enum34>=1.0.4 in /usr/local/lib/python2.7/dist-packages (from grpcio<2,>=1.8->apache-beam==2.5.0.dev0) Requirement already satisfied: requests>=2.7.0 in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages (from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Collecting funcsigs>=1; python_version < "3.3" (from mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0) Using cached funcsigs-1.0.2-py2.py3-none-any.whl Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0) Using cached pbr-3.1.1-py2.py3-none-any.whl Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0) Using cached rsa-3.4.2-py2.py3-none-any.whl Collecting pyasn1-modules>=0.0.5 (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0) Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0) Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages (from protobuf<4,>=3.5.0.post1->apache-beam==2.5.0.dev0) Collecting fasteners>=0.14 (from google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0) Using cached fasteners-0.14.1-py2.py3-none-any.whl Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.5.0.dev0) Collecting google-cloud-core<0.26dev,>=0.25.0 (from google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0) Collecting monotonic>=0.1 (from fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0) Using cached monotonic-1.4-py2.py3-none-any.whl Collecting google-auth<2.0.0dev,>=0.4.0 (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Using cached google_auth-1.4.1-py2.py3-none-any.whl Collecting google-auth-httplib2 (from google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl Collecting google-gax<0.16dev,>=0.15.7 (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Using cached google_gax-0.15.16-py2.py3-none-any.whl Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Collecting cachetools>=2.0.0 (from google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Using cached cachetools-2.0.1-py2.py3-none-any.whl Collecting future<0.17dev,>=0.16.0 (from google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Collecting ply==3.8 (from google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0) Installing collected packages: avro, dill, protobuf, grpcio, hdfs, httplib2, funcsigs, pbr, mock, rsa, pyasn1-modules, oauth2client, pyvcf, typing, monotonic, fasteners, google-apitools, googleapis-common-protos, proto-google-cloud-datastore-v1, googledatastore, cachetools, google-auth, google-auth-httplib2, google-cloud-core, future, ply, google-gax, grpc-google-iam-v1, proto-google-cloud-pubsub-v1, gapic-google-cloud-pubsub-v1, google-cloud-pubsub, google-cloud-bigquery, nose, pyhamcrest, apache-beam Running setup.py develop for apache-beam Successfully installed apache-beam avro-1.8.2 cachetools-2.0.1 dill-0.2.6 fasteners-0.14.1 funcsigs-1.0.2 future-0.16.0 gapic-google-cloud-pubsub-v1-0.15.4 google-apitools-0.5.20 google-auth-1.4.1 google-auth-httplib2-0.0.3 google-cloud-bigquery-0.25.0 google-cloud-core-0.25.0 google-cloud-pubsub-0.26.0 google-gax-0.15.16 googleapis-common-protos-1.5.3 googledatastore-7.0.1 grpc-google-iam-v1-0.11.4 grpcio-1.10.0 hdfs-2.1.0 httplib2-0.9.2 mock-2.0.0 monotonic-1.4 nose-1.3.7 oauth2client-4.1.2 pbr-3.1.1 ply-3.8 proto-google-cloud-datastore-v1-0.90.4 proto-google-cloud-pubsub-v1-0.15.4 protobuf-3.5.2.post1 pyasn1-modules-0.2.1 pyhamcrest-1.9.0 pyvcf-0.6.8 rsa-3.4.2 typing-3.6.4 [beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe /tmp/jenkins4258650102972799589.sh + .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing --dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn --bigquery_table=beam_performance.textioit_hdfs_pkb_results --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/> --official=true --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=1000000]' '--beam_extra_mvn_properties=[filesystem=hdfs]' --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml> --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster.yml,/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TextIOIT_HDFS/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster-for-local-dev.yml> 2018-03-24 12:00:50,879 f0cb1175 MainThread INFO Verbose logging to: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/runs/f0cb1175/pkb.log> 2018-03-24 12:00:50,880 f0cb1175 MainThread INFO PerfKitBenchmarker version: v1.12.0-445-geb5f1b5 2018-03-24 12:00:50,880 f0cb1175 MainThread INFO Flag values: --beam_extra_mvn_properties=[filesystem=hdfs] --beam_it_class=org.apache.beam.sdk.io.text.TextIOIT --beam_it_timeout=1200 --beam_sdk=java --temp_dir=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/> --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn --beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=1000000] --beam_prebuilt --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> --project=apache-beam-testing --beam_kubernetes_scripts=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster.yml,/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_TextIOIT_HDFS/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster-for-local-dev.yml> --bigquery_table=beam_performance.textioit_hdfs_pkb_results --official --beam_it_module=sdks/java/io/file-based-io-tests --dpb_log_level=INFO --beam_options_config_file=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml> --beam_it_profile=io-it --benchmarks=beam_integration_benchmark 2018-03-24 12:00:51,104 f0cb1175 MainThread WARNING The key "flags" was not in the default config, but was in user overrides. This may indicate a typo. 2018-03-24 12:00:51,104 f0cb1175 MainThread INFO Initializing the edw service decoder 2018-03-24 12:00:51,208 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Provisioning resources for benchmark beam_integration_benchmark 2018-03-24 12:00:51,210 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Preparing benchmark beam_integration_benchmark 2018-03-24 12:00:51,210 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: git clone https://github.com/apache/beam.git 2018-03-24 12:01:05,120 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> create -f <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster.yml> 2018-03-24 12:01:05,602 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> create -f <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster-for-local-dev.yml> 2018-03-24 12:01:06,060 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running benchmark beam_integration_benchmark 2018-03-24 12:01:06,067 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:01:16,405 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:01:26,615 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:01:36,748 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:01:46,891 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:01:57,015 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:07,155 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:17,312 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:27,507 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:37,627 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:47,760 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:02:57,883 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:08,007 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:18,129 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:28,255 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:38,375 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:48,504 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:03:58,627 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:04:08,761 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> get svc hadoop-external -ojsonpath={.status.loadBalancer.ingress[0].ip} 2018-03-24 12:04:08,876 f0cb1175 MainThread beam_integration_benchmark(1/1) ERROR Error during benchmark beam_integration_benchmark Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 624, in RunBenchmark DoRunPhase(spec, collector, detailed_timer) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 527, in DoRunPhase samples = spec.BenchmarkRun(spec) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,> line 148, in Run dynamic_pipeline_options) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 100, in GenerateAllPipelineOptions EvaluateDynamicPipelineOptions(dynamic_pipeline_options)) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 67, in EvaluateDynamicPipelineOptions argValue = RetrieveLoadBalancerIp(optionDescriptor) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 153, in RetrieveLoadBalancerIp raise "Could not retrieve LoadBalancer IP address" TypeError: exceptions must be old-style classes or derived from BaseException, not str 2018-03-24 12:04:08,877 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Cleaning up benchmark beam_integration_benchmark 2018-03-24 12:04:08,877 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> delete -f <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster.yml> 2018-03-24 12:04:09,322 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Running: kubectl --kubeconfig=<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/config-filebasedioithdfs-1521890089887> delete -f <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/src/.test-infra/kubernetes/hadoop/SmallITCluster/hdfs-single-datanode-cluster-for-local-dev.yml> 2018-03-24 12:04:09,484 f0cb1175 MainThread beam_integration_benchmark(1/1) ERROR Exception running benchmark Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 758, in RunBenchmarkTask RunBenchmark(spec, collector) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 624, in RunBenchmark DoRunPhase(spec, collector, detailed_timer) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 527, in DoRunPhase samples = spec.BenchmarkRun(spec) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py";,> line 148, in Run dynamic_pipeline_options) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 100, in GenerateAllPipelineOptions EvaluateDynamicPipelineOptions(dynamic_pipeline_options)) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 67, in EvaluateDynamicPipelineOptions argValue = RetrieveLoadBalancerIp(optionDescriptor) File "<https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/PerfKitBenchmarker/perfkitbenchmarker/beam_pipeline_options.py";,> line 153, in RetrieveLoadBalancerIp raise "Could not retrieve LoadBalancer IP address" TypeError: exceptions must be old-style classes or derived from BaseException, not str 2018-03-24 12:04:09,485 f0cb1175 MainThread beam_integration_benchmark(1/1) ERROR Benchmark 1/1 beam_integration_benchmark (UID: beam_integration_benchmark0) failed. Execution will continue. 2018-03-24 12:04:09,485 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Benchmark run statuses: --------------------------------------------------------------------------------- Name UID Status Failed Substatus --------------------------------------------------------------------------------- beam_integration_benchmark beam_integration_benchmark0 FAILED --------------------------------------------------------------------------------- Success rate: 0.00% (0/1) 2018-03-24 12:04:09,485 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Complete logs can be found at: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/runs/f0cb1175/pkb.log> 2018-03-24 12:04:09,485 f0cb1175 MainThread beam_integration_benchmark(1/1) INFO Completion statuses can be found at: <https://builds.apache.org/job/beam_PerformanceTests_TextIOIT_HDFS/ws/runs/f0cb1175/completion_statuses.json> Build step 'Execute shell' marked build as failure
