See <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/56/display/redirect?page=changes>
Changes: [chamikara] [BEAM-3060] Support for Perfkit execution of file-based-io-tests on HDFS ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam4 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision ad82ac140161cd5d88c972c88a1c8168cf26ff1c (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f ad82ac140161cd5d88c972c88a1c8168cf26ff1c Commit message: "[BEAM-3060] Support for Perfkit execution of file-based-io-tests on HDFS cluster. (#4401)" > git rev-list 3c6899c001b28ff98db461df8b712c691403b67d # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins7525012335634357582.sh + rm -rf PerfKitBenchmarker [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins537005116997630131.sh + rm -rf .env [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins8939756250110714810.sh + virtualenv .env --system-site-packages New python executable in .env/bin/python Installing setuptools, pip...done. [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins8189699785624850515.sh + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git Cloning into 'PerfKitBenchmarker'... [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins5167402494817098215.sh + .env/bin/pip install -r PerfKitBenchmarker/requirements.txt Requirement already satisfied (use --upgrade to upgrade): absl-py in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 14)) Requirement already satisfied (use --upgrade to upgrade): jinja2>=2.7 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 15)) Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 16)) Requirement already satisfied (use --upgrade to upgrade): colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 17)) Installing extra requirements: 'windows' Requirement already satisfied (use --upgrade to upgrade): blinker>=1.3 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 18)) Requirement already satisfied (use --upgrade to upgrade): futures>=3.0.3 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 19)) Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.12 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 20)) Requirement already satisfied (use --upgrade to upgrade): pint>=0.7 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 21)) Requirement already satisfied (use --upgrade to upgrade): numpy in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 22)) Requirement already satisfied (use --upgrade to upgrade): functools32 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 23)) Requirement already satisfied (use --upgrade to upgrade): contextlib2>=0.5.1 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 24)) Requirement already satisfied (use --upgrade to upgrade): pywinrm in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 25)) Cleaning up... [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins8405484500491001678.sh + .env/bin/pip install -e 'src/sdks/python/[gcp,test]' Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python> Running setup.py (path:<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python/setup.py)> egg_info for package from file://<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python> <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python/setup.py>:66: UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is recommended. _PIP_VERSION, REQUIRED_PIP_VERSION no previously-included directories found matching 'doc/.build' Installed <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python/nose-1.3.7-py2.7.egg> /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) warning: no files found matching 'README.md' warning: no files found matching 'NOTICE' warning: no files found matching 'LICENSE' Installing extra requirements: 'gcp,test' Requirement already satisfied (use --upgrade to upgrade): avro>=1.8.1,<2.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): crcmod>=1.7,<2.0 in /usr/lib/python2.7/dist-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): dill==0.2.6 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): grpcio>=1.0,<2 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): httplib2>=0.8,<0.10 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): mock>=1.0.1,<3.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): oauth2client>=2.0.1,<5 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): protobuf>=3.5.0.post1,<4 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.12,<4.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): pyvcf>=0.6.8,<0.7.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): six>=1.9,<1.12 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): typing>=3.6.0,<3.7.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): futures>=3.1.1,<4.0.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): hdfs3>=0.3.0,<0.4.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): pyhamcrest>=1.9,<2.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): google-apitools>=0.5.18,<=0.5.20 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): googledatastore==7.0.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): google-cloud-pubsub==0.26.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): google-cloud-bigquery==0.25.0 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): enum34>=1.0.4 in /home/jenkins/.local/lib/python2.7/site-packages (from grpcio>=1.0,<2->apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.7 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client>=2.0.1,<5->apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules>=0.0.5 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client>=2.0.1,<5->apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): rsa>=3.1.4 in /home/jenkins/.local/lib/python2.7/site-packages (from oauth2client>=2.0.1,<5->apache-beam==2.3.0.dev) Requirement already satisfied (use --upgrade to upgrade): googleapis-common-protos>=1.5.2,<2.0dev in /home/jenkins/.local/lib/python2.7/site-packages (from proto-google-cloud-datastore-v1>=0.90.0,<=0.90.4->apache-beam==2.3.0.dev) Installing collected packages: apache-beam Running setup.py develop for apache-beam <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python/setup.py>:66: UserWarning: You are using version 1.5.4 of pip. However, version 7.0.0 is recommended. _PIP_VERSION, REQUIRED_PIP_VERSION /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires' warnings.warn(msg) warning: no files found matching 'README.md' warning: no files found matching 'NOTICE' warning: no files found matching 'LICENSE' Creating <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/.env/lib/python2.7/site-packages/apache-beam.egg-link> (link to .) Adding apache-beam 2.3.0.dev to easy-install.pth file Installed <https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/src/sdks/python> Successfully installed apache-beam Cleaning up... [beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe /tmp/jenkins2153495598447736521.sh + .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing --dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn --bigquery_table=beam_performance.tfrecordioit_pkb_results --official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 --beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java --beam_it_module=sdks/java/io/file-based-io-tests --beam_it_class=org.apache.beam.sdk.io.tfrecord.TFRecordIOIT '--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=1000000,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TFRecordIOIT/56/]' '--beam_extra_mvn_properties=[filesystem=gcs]' Traceback (most recent call last): File "PerfKitBenchmarker/pkb.py", line 21, in <module> sys.exit(Main()) File "<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 869, in Main SetUpPKB() File "<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/pkb.py";,> line 740, in SetUpPKB vm_util.GenTempDir() File "<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/vm_util.py";,> line 121, in GenTempDir temp_dir.CreateTemporaryDirectories() File "<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/PerfKitBenchmarker/perfkitbenchmarker/temp_dir.py";,> line 62, in CreateTemporaryDirectories os.makedirs(path) File "<https://builds.apache.org/job/beam_PerformanceTests_TFRecordIOIT/ws/.env/lib/python2.7/os.py";,> line 157, in makedirs mkdir(name, mode) OSError: [Errno 13] Permission denied: '/tmp/perfkitbenchmarker/runs/7e997a83' Build step 'Execute shell' marked build as failure
