See <https://builds.apache.org/job/beam_PerformanceTests_Python/1185/display/redirect?page=changes>
Changes: [geet.kumar75] BEAM-4038: Support Kafka Headers in KafkaIO [geet.kumar75] BEAM-4038: Update license information [geet.kumar75] Update code formatting [geet.kumar75] Remove custom implementations of KafkaHeader, KafkaHeaders, etc. [geet.kumar75] Changes based on review comments [daniel.o.programmer] [BEAM-3513] Removing PrimitiveCombineGroupedValues override w/ FnAPI. [echauchot] Introduce MetricsPusher in runner core to regularly push aggregated [echauchot] Instanciate MetricsPusher in runner-specific code because we need [echauchot] Improve MetricsPusher: do not aggregate metrics when not needed, improve [echauchot] Create JsonMetricsSerializer [echauchot] Stop MetricsPusher thread by observing pipeline state and improve the [echauchot] Make metrics sink configurable through PipelineOptions, pass [echauchot] Add MetricsPusher tests specific to Spark (because Spark streaming tests [echauchot] Add a MetricksPusher test to runner-core (batch and streaming are done [echauchot] Push metrics at the end of a batch pipeline in flink runner [echauchot] improve MetricsPusher lifecycle and thread safety [echauchot] Make MetricsPusher merge a list a MetricsContainerStepMaps because there [echauchot] Fix thread synchronisation and replace usages of instance variable by [echauchot] Clear dummyMetricsSink before test [echauchot] Push metrics at the end of a batch pipeline in spark runner [echauchot] Improve MetricsPusher teardown to enable multiple pipelines in a single [echauchot] Manually generate json and remove jackson [echauchot] Replace use of http client by use of java.net.HttpUrlConnection and deal [echauchot] Remove DEFAULT_PERIOD constant in favor of already existing [echauchot] Remove unneeded null check, format [echauchot] convert MetricsSink to an interface with a single writeMetrics method [echauchot] Remove MetricsSerializer base class and inline serialization in [echauchot] Dynamically create the sinks by reflection [echauchot] Split DummyMetricsSink into NoOpMetricsSink (default sink) and [echauchot] Reduce overhead when no metricsSink is provided, do not start polling [echauchot] Make MetricsPusher a regular object instead of a singleton to allow [echauchot] Explicitely start MetricsPusher from the runners [echauchot] Separate MetricsHttpSink POC to a new runners-extensions artifact and [echauchot] Fix cycle bug between teardown() and pushmetrics() [echauchot] Update MetricsPusher and TestMetricsSink to new serializable [echauchot] Use regular jackson object mapper to serialize metrics now that they are [echauchot] Give MetricsPusher a bit of time to push before assert in test [echauchot] Make MetricsPusher thread a daemon [echauchot] Fix build and clean: dependencies, rat, checkstyle, findbugs, remove [geet.kumar75] Support kafka versions 0.10.1.0 and above [echauchot] Move build to gradle [echauchot] MetricsSink no more needs to be generic [echauchot] SparkRunnerDebugger does not need to export metrics as it does not run [tgroh] Use Existing Matchers in WatermarkManagerTest [echauchot] Move MetricsHttpSink and related classes to a new sub-module [kirpichov] Consistently handle EmptyMatchTreatment [rangadi] Add 10 millis sleep when there are no elements left in a partition. ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on beam3 (beam) in workspace <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 0f2ba71e1b6db88ed79744e363586a8ff16dcb08 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 0f2ba71e1b6db88ed79744e363586a8ff16dcb08 Commit message: "Merge pull request #5195: Use Existing Matchers in WatermarkManagerTest" > git rev-list --no-walk 247a62ff1d4368f1e7c2ade6bed5dec71d8d2bcc # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins5809426347260041593.sh + rm -rf PerfKitBenchmarker [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins3285690391011211747.sh + rm -rf .env [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins6101718920793672879.sh + virtualenv .env --system-site-packages New python executable in <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.env/bin/python> Installing setuptools, pip, wheel...done. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins5530998206000920874.sh + .env/bin/pip install --upgrade setuptools pip Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages (39.0.1) Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages (10.0.1) cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed. gapic-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. proto-google-cloud-datastore-v1 0.90.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. proto-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll have oauth2client 4.1.2 which is incompatible. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins845102992366333830.sh + git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git Cloning into 'PerfKitBenchmarker'... [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins4490238239090703186.sh + .env/bin/pip install -r PerfKitBenchmarker/requirements.txt Requirement already satisfied: absl-py in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 14)) (0.1.7) Requirement already satisfied: jinja2>=2.7 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 15)) (2.10) Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 16)) (39.0.1) Requirement already satisfied: colorlog[windows]==2.6.0 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0) Requirement already satisfied: blinker>=1.3 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 18)) (1.4) Requirement already satisfied: futures>=3.0.3 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0) Requirement already satisfied: PyYAML==3.12 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 20)) (3.12) Requirement already satisfied: pint>=0.7 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 21)) (0.8.1) Requirement already satisfied: numpy==1.13.3 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 22)) (1.13.3) Requirement already satisfied: functools32 in /usr/local/lib/python2.7/dist-packages (from -r PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2) Requirement already satisfied: contextlib2>=0.5.1 in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 24)) (0.5.5) Requirement already satisfied: pywinrm in /home/jenkins/.local/lib/python2.7/site-packages (from -r PerfKitBenchmarker/requirements.txt (line 25)) (0.3.0) Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages (from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14)) (1.11.0) Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r PerfKitBenchmarker/requirements.txt (line 15)) (1.0) Requirement already satisfied: colorama; extra == "windows" in /usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r PerfKitBenchmarker/requirements.txt (line 17)) (0.2.5) Requirement already satisfied: xmltodict in /home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (0.11.0) Requirement already satisfied: requests-ntlm>=0.3.0 in /home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.1.0) Requirement already satisfied: requests>=2.9.1 in /usr/local/lib/python2.7/dist-packages (from pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (2.18.4) Requirement already satisfied: ntlm-auth>=1.0.2 in /home/jenkins/.local/lib/python2.7/site-packages (from requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.0.6) Requirement already satisfied: cryptography>=1.3 in /home/jenkins/.local/lib/python2.7/site-packages (from requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (2.1.4) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.22) Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (2.6) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (2018.1.18) Requirement already satisfied: cffi>=1.7; platform_python_implementation != "PyPy" in /home/jenkins/.local/lib/python2.7/site-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.11.2) Requirement already satisfied: enum34; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.1.6) Requirement already satisfied: asn1crypto>=0.21.0 in /home/jenkins/.local/lib/python2.7/site-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (0.24.0) Requirement already satisfied: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (1.0.19) Requirement already satisfied: pycparser in /home/jenkins/.local/lib/python2.7/site-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.3->requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25)) (2.18) cheetah 2.4.4 requires Markdown>=2.0.1, which is not installed. gapic-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. proto-google-cloud-datastore-v1 0.90.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. proto-google-cloud-pubsub-v1 0.15.4 has requirement oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is incompatible. googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll have oauth2client 4.1.2 which is incompatible. [beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins3190979353997182633.sh + .env/bin/pip install -e 'src/sdks/python/[gcp,test]' Obtaining file://<https://builds.apache.org/job/beam_PerformanceTests_Python/ws/src/sdks/python> Requirement already satisfied: avro<2.0.0,>=1.8.1 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0) (1.8.2) Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0) (1.7) Requirement already satisfied: dill==0.2.6 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0) (0.2.6) Requirement already satisfied: grpcio<2,>=1.8 in /home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0) (1.8.3) Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0) <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py>:339: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings SNIMissingWarning <https://builds.apache.org/job/beam_PerformanceTests_Python/ws/.env/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py>:137: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecurePlatformWarning Could not find a version that satisfies the requirement hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0) (from versions: ) No matching distribution found for hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0) Build step 'Execute shell' marked build as failure Not sending mail to unregistered user hero...@google.com Not sending mail to unregistered user k...@google.com Not sending mail to unregistered user amyrv...@google.com Not sending mail to unregistered user valen...@google.com Not sending mail to unregistered user da...@getindata.com Not sending mail to unregistered user geet.kuma...@gmail.com Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com Not sending mail to unregistered user lukasz.gaj...@polidea.com Not sending mail to unregistered user bat...@batbat-linuxworkstation.sea.corp.google.com Not sending mail to unregistered user holdenka...@google.com Not sending mail to unregistered user samuel.waggo...@healthsparq.com Not sending mail to unregistered user z...@giggles.nyc.corp.google.com Not sending mail to unregistered user mark...@google.com Not sending mail to unregistered user grzegorz.kolakow...@getindata.com Not sending mail to unregistered user w...@google.com Not sending mail to unregistered user daniel.o.program...@gmail.com Not sending mail to unregistered user mari...@google.com Not sending mail to unregistered user sweg...@google.com Not sending mail to unregistered user j...@nanthrax.net Not sending mail to unregistered user xumingmi...@gmail.com Not sending mail to unregistered user arost...@google.com Not sending mail to unregistered user sdus...@google.com Not sending mail to unregistered user ankurgoe...@gmail.com Not sending mail to unregistered user rob...@frantil.com Not sending mail to unregistered user dariusz.aniszew...@polidea.com Not sending mail to unregistered user eh...@google.com Not sending mail to unregistered user xuming...@users.noreply.github.com Not sending mail to unregistered user aromanenko....@gmail.com Not sending mail to unregistered user ke...@google.com Not sending mail to unregistered user apill...@google.com Not sending mail to unregistered user arnaudfournier...@gmail.com Not sending mail to unregistered user bbassingthwa...@vendasta.com Not sending mail to unregistered user 1147435+chua...@users.noreply.github.com Not sending mail to unregistered user fjetum...@gmail.com Not sending mail to unregistered user jiang...@gmail.com Not sending mail to unregistered user rezaro...@google.com Not sending mail to unregistered user mair...@google.com Not sending mail to unregistered user ni...@google.com Not sending mail to unregistered user axelm...@gmail.com Not sending mail to unregistered user luke_...@brown.edu Not sending mail to unregistered user swegn...@gmail.com Not sending mail to unregistered user joey.bar...@gmail.com Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com Not sending mail to unregistered user zhouha...@meituan.com Not sending mail to unregistered user dcava...@google.com Not sending mail to unregistered user kirpic...@google.com Not sending mail to unregistered user wi...@bounceexchange.com Not sending mail to unregistered user samu...@google.com Not sending mail to unregistered user timrobertson...@gmail.com Not sending mail to unregistered user g...@telligent-data.com Not sending mail to unregistered user djiofack...@gmail.com Not sending mail to unregistered user alan.myrv...@comcast.net Not sending mail to unregistered user ro...@frantil.com Not sending mail to unregistered user c...@google.com Not sending mail to unregistered user re...@relax-macbookpro2.roam.corp.google.com Not sending mail to unregistered user git...@alasdairhodge.co.uk Not sending mail to unregistered user ekirpic...@gmail.com Not sending mail to unregistered user sho...@google.com Not sending mail to unregistered user 7542791+rezaro...@users.noreply.github.com Not sending mail to unregistered user aljoscha.kret...@gmail.com Not sending mail to unregistered user meliss...@google.com Not sending mail to unregistered user sid...@google.com Not sending mail to unregistered user andreas.ehrencr...@velik.it Not sending mail to unregistered user abhyan...@gmail.com Not sending mail to unregistered user aal...@gmail.com Not sending mail to unregistered user ccla...@bluewin.ch Not sending mail to unregistered user 1028332...@qq.com Not sending mail to unregistered user mott...@gmail.com Not sending mail to unregistered user arnaudfournier...@gmail.com Not sending mail to unregistered user szewi...@gmail.com Not sending mail to unregistered user yifan...@yifanzou-linuxworkstation.sea.corp.google.com Not sending mail to unregistered user wtan...@yahoo.com Not sending mail to unregistered user boyu...@google.com
