See <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/185/display/redirect?page=changes>
Changes: [echauchot] Introduce MetricsPusher in runner core to regularly push aggregated [echauchot] Instanciate MetricsPusher in runner-specific code because we need [echauchot] Improve MetricsPusher: do not aggregate metrics when not needed, improve [echauchot] Create JsonMetricsSerializer [echauchot] Stop MetricsPusher thread by observing pipeline state and improve the [echauchot] Make metrics sink configurable through PipelineOptions, pass [echauchot] Add MetricsPusher tests specific to Spark (because Spark streaming tests [echauchot] Add a MetricksPusher test to runner-core (batch and streaming are done [echauchot] Push metrics at the end of a batch pipeline in flink runner [echauchot] improve MetricsPusher lifecycle and thread safety [echauchot] Make MetricsPusher merge a list a MetricsContainerStepMaps because there [echauchot] Fix thread synchronisation and replace usages of instance variable by [echauchot] Clear dummyMetricsSink before test [echauchot] Push metrics at the end of a batch pipeline in spark runner [echauchot] Improve MetricsPusher teardown to enable multiple pipelines in a single [echauchot] Manually generate json and remove jackson [echauchot] Replace use of http client by use of java.net.HttpUrlConnection and deal [echauchot] Remove DEFAULT_PERIOD constant in favor of already existing [echauchot] Remove unneeded null check, format [echauchot] convert MetricsSink to an interface with a single writeMetrics method [echauchot] Remove MetricsSerializer base class and inline serialization in [echauchot] Dynamically create the sinks by reflection [echauchot] Split DummyMetricsSink into NoOpMetricsSink (default sink) and [echauchot] Reduce overhead when no metricsSink is provided, do not start polling [echauchot] Make MetricsPusher a regular object instead of a singleton to allow [echauchot] Explicitely start MetricsPusher from the runners [echauchot] Separate MetricsHttpSink POC to a new runners-extensions artifact and [echauchot] Fix cycle bug between teardown() and pushmetrics() [echauchot] Update MetricsPusher and TestMetricsSink to new serializable [echauchot] Use regular jackson object mapper to serialize metrics now that they are [echauchot] Give MetricsPusher a bit of time to push before assert in test [echauchot] Make MetricsPusher thread a daemon [echauchot] Fix build and clean: dependencies, rat, checkstyle, findbugs, remove [echauchot] Move build to gradle [echauchot] MetricsSink no more needs to be generic [echauchot] SparkRunnerDebugger does not need to export metrics as it does not run [echauchot] Move MetricsHttpSink and related classes to a new sub-module [kirpichov] Consistently handle EmptyMatchTreatment ------------------------------------------ Started by GitHub push by iemejia [EnvInject] - Loading node environment variables. Building remotely on beam8 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/> > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 14525484ac19e295ca2811323c04af4a10a2477e (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 14525484ac19e295ca2811323c04af4a10a2477e Commit message: "Merge pull request #4548: [BEAM-3310] Add metrics pusher" > git rev-list --no-walk e6681e8aac364b11304fdd4a85642a89307eeaee # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. [Gradle] - Launching build. [src] $ <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/gradlew> --info --continue --max-workers=2 :beam-runners-spark:validatesRunner Initialized native services in: /home/jenkins/.gradle/native Using 2 worker leases. Starting Build Settings evaluated using settings file '<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/settings.gradle'.> Using local directory build cache for the root build (location = /home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days). Projects loaded. Root project using build file '<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/build.gradle'.> Included projects: [root project 'beam', project ':beam-examples-java', project ':beam-local-artifact-service-java', project ':beam-model-fn-execution', project ':beam-model-job-management', project ':beam-model-pipeline', project ':beam-runners-apex', project ':beam-runners-core-construction-java', project ':beam-runners-core-java', project ':beam-runners-direct-java', project ':beam-runners-extensions-java-metrics', project ':beam-runners-flink_2.11', project ':beam-runners-gcp-gcemd', project ':beam-runners-gcp-gcsproxy', project ':beam-runners-gearpump', project ':beam-runners-google-cloud-dataflow-java', project ':beam-runners-java-fn-execution', project ':beam-runners-local-java-core', project ':beam-runners-reference-java', project ':beam-runners-reference-job-server', project ':beam-runners-spark', project ':beam-sdks-go', project ':beam-sdks-go-container', project ':beam-sdks-go-examples', project ':beam-sdks-java-build-tools', project ':beam-sdks-java-container', project ':beam-sdks-java-core', project ':beam-sdks-java-extensions-google-cloud-platform-core', project ':beam-sdks-java-extensions-join-library', project ':beam-sdks-java-extensions-json-jackson', project ':beam-sdks-java-extensions-protobuf', project ':beam-sdks-java-extensions-sketching', project ':beam-sdks-java-extensions-sorter', project ':beam-sdks-java-extensions-sql', project ':beam-sdks-java-fn-execution', project ':beam-sdks-java-harness', project ':beam-sdks-java-io-amazon-web-services', project ':beam-sdks-java-io-amqp', project ':beam-sdks-java-io-cassandra', project ':beam-sdks-java-io-common', project ':beam-sdks-java-io-elasticsearch', project ':beam-sdks-java-io-elasticsearch-tests-2', project ':beam-sdks-java-io-elasticsearch-tests-5', project ':beam-sdks-java-io-elasticsearch-tests-common', project ':beam-sdks-java-io-file-based-io-tests', project ':beam-sdks-java-io-google-cloud-platform', project ':beam-sdks-java-io-hadoop-common', project ':beam-sdks-java-io-hadoop-file-system', project ':beam-sdks-java-io-hadoop-input-format', project ':beam-sdks-java-io-hbase', project ':beam-sdks-java-io-hcatalog', project ':beam-sdks-java-io-jdbc', project ':beam-sdks-java-io-jms', project ':beam-sdks-java-io-kafka', project ':beam-sdks-java-io-kinesis', project ':beam-sdks-java-io-mongodb', project ':beam-sdks-java-io-mqtt', project ':beam-sdks-java-io-redis', project ':beam-sdks-java-io-solr', project ':beam-sdks-java-io-tika', project ':beam-sdks-java-io-xml', project ':beam-sdks-java-javadoc', project ':beam-sdks-java-maven-archetypes-examples', project ':beam-sdks-java-maven-archetypes-starter', project ':beam-sdks-java-nexmark', project ':beam-sdks-python', project ':beam-sdks-python-container', project ':release'] Parallel execution with configuration on demand is an incubating feature. Evaluating root project 'beam' using build file '<https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/ws/src/build.gradle'.> Invalidating in-memory cache of /home/jenkins/.gradle/caches/4.6/fileHashes/fileHashes.bin Invalidating in-memory cache of /home/jenkins/.gradle/caches/4.6/fileHashes/resourceHashesCache.bin FAILURE: Build failed with an exception. * What went wrong: A problem occurred configuring root project 'beam'. > Could not resolve all dependencies for configuration ':classpath'. > Unable to lock the information region for artifact cache (/home/jenkins/.gradle/caches/modules-2) * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 4m 8s Build cache (/home/jenkins/.gradle/caches/build-cache-1) has not been cleaned up in 4 days Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error? Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected]
