[ https://issues.apache.org/jira/browse/FLINK-18290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135526#comment-17135526 ]
Robert Metzger commented on FLINK-18290: ---------------------------------------- First occurrence of a related issue was build 20200612.19 (build ID 3416) First occurrence of the exact issue with exit code was build 20200612.26 (id 3434). Is a docs change. Potentially problematic commits: This commit (https://github.com/flink-ci/flink-mirror/commit/c004b119ef28dc7387935d8d3a4dbf296cc5f661) introduces a System.exit(-17) in the checkpoint coordinator. 256 - 17 = 239. Coincidence? Introducing a {{System.exit(-17);}} into any test will lead to exactly the failure reported here. This seems to be the reason why System.exit() gets called (from 20200612.19): {code} 14:56:33,906 [flink-akka.actor.default-dispatcher-2] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Stopped TaskExecutor akka://flink/user/rpc/taskmanager_28. 14:56:33,887 [jobmanager-future-thread-7] ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread 'jobmanager-future-thread-7' produced an uncaught exception. Stopping the process... java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1db3e21b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@198c75f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:838) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [?:1.8.0_242] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609) [?:1.8.0_242] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1db3e21b rejected from java.util.concurrent.ScheduledThreadPoolExecutor@198c75f2[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 20] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) ~[?:1.8.0_242] at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_242] at org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter.execute(ScheduledExecutorServiceAdapter.java:62) ~[flink-runtime_2.11-1.11-SNAPSHOT.jar:1.11-SNAPSHOT] at java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:826) ~[?:1.8.0_242] ... 10 more {code} Seems to be the same in the .26 run: {code} 20:51:18,122 [mini-cluster-io-thread-26] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - JobManager for job 63af744539889f9a6bf731aa05b02e97 with leader id 93f8812403b7e711da29465d96a74439 lost leadership. 20:51:18,122 [flink-akka.actor.default-dispatcher-5] INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Close JobManager connection for job 63af744539889f9a6bf731aa05b02e97. 20:51:18,121 [ Checkpoint Timer] ERROR org.apache.flink.runtime.util.FatalExitExceptionHandler [] - FATAL: Thread 'Checkpoint Timer' produced an uncaught exception. Stopping the process... java.util.concurrent.CompletionException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@59fa0f36 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@1cf89a6d[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 7] at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:838) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) [?:1.8.0_242] at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575) [?:1.8.0_242] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:594) [?:1.8.0_242] at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) [?:1.8.0_242] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_242] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242] Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@59fa0f36 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@1cf89a6d[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_242] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_242] at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622) ~[?:1.8.0_242] at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_242] at org.apache.flink.runtime.concurrent.ScheduledExecutorServiceAdapter.execute(ScheduledExecutorServiceAdapter.java:62) ~[flink-runtime_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT] at java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543) ~[?:1.8.0_242] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:826) ~[?:1.8.0_242] ... 12 more 20:51:18,123 [PermanentBlobCache shutdown hook] INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Shutting down BLOB cache {code} > Tests are crashing with exit code 239 > ------------------------------------- > > Key: FLINK-18290 > URL: https://issues.apache.org/jira/browse/FLINK-18290 > Project: Flink > Issue Type: Bug > Components: Build System / Azure Pipelines > Affects Versions: 1.11.0 > Reporter: Robert Metzger > Assignee: Robert Metzger > Priority: Blocker > Labels: test-stability > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3467&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=34f486e1-e1e4-5dd2-9c06-bfdd9b9c74a8] > Kafka011ProducerExactlyOnceITCase > > {code:java} > 2020-06-15T03:24:28.4677649Z [WARNING] The requested profile > "skip-webui-build" could not be activated because it does not exist. > 2020-06-15T03:24:28.4692049Z [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test > (integration-tests) on project flink-connector-kafka-0.11_2.11: There are > test failures. > 2020-06-15T03:24:28.4692585Z [ERROR] > 2020-06-15T03:24:28.4693170Z [ERROR] Please refer to > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire-reports > for the individual test results. > 2020-06-15T03:24:28.4693928Z [ERROR] Please refer to dump files (if any > exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. > 2020-06-15T03:24:28.4694423Z [ERROR] ExecutionException The forked VM > terminated without properly saying goodbye. VM crash or System.exit called? > 2020-06-15T03:24:28.4696762Z [ERROR] Command was /bin/sh -c cd > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target && > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m > -Dlog4j.configurationFile=log4j2-test.properties -Dmvn.forkNumber=2 > -XX:-UseGCOverheadLimit -jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire/surefirebooter617700788970993266.jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire > 2020-06-15T03-07-01_381-jvmRun2 surefire2676050245109796726tmp > surefire_602825791089523551074tmp > 2020-06-15T03:24:28.4698486Z [ERROR] Error occurred in starting fork, check > output in log > 2020-06-15T03:24:28.4699066Z [ERROR] Process Exit Code: 239 > 2020-06-15T03:24:28.4699458Z [ERROR] Crashed tests: > 2020-06-15T03:24:28.4699960Z [ERROR] > org.apache.flink.streaming.connectors.kafka.Kafka011ProducerExactlyOnceITCase > 2020-06-15T03:24:28.4700849Z [ERROR] > org.apache.maven.surefire.booter.SurefireBooterForkException: > ExecutionException The forked VM terminated without properly saying goodbye. > VM crash or System.exit called? > 2020-06-15T03:24:28.4703760Z [ERROR] Command was /bin/sh -c cd > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target && > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m > -Dlog4j.configurationFile=log4j2-test.properties -Dmvn.forkNumber=2 > -XX:-UseGCOverheadLimit -jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire/surefirebooter617700788970993266.jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire > 2020-06-15T03-07-01_381-jvmRun2 surefire2676050245109796726tmp > surefire_602825791089523551074tmp > 2020-06-15T03:24:28.4705501Z [ERROR] Error occurred in starting fork, check > output in log > 2020-06-15T03:24:28.4706297Z [ERROR] Process Exit Code: 239 > 2020-06-15T03:24:28.4706592Z [ERROR] Crashed tests: > 2020-06-15T03:24:28.4706895Z [ERROR] > org.apache.flink.streaming.connectors.kafka.Kafka011ProducerExactlyOnceITCase > 2020-06-15T03:24:28.4707386Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510) > 2020-06-15T03:24:28.4708053Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457) > 2020-06-15T03:24:28.4708908Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298) > 2020-06-15T03:24:28.4709720Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246) > 2020-06-15T03:24:28.4710497Z [ERROR] at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183) > 2020-06-15T03:24:28.4711448Z [ERROR] at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011) > 2020-06-15T03:24:28.4712395Z [ERROR] at > org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857) > 2020-06-15T03:24:28.4712997Z [ERROR] at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132) > 2020-06-15T03:24:28.4713524Z [ERROR] at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) > 2020-06-15T03:24:28.4714079Z [ERROR] at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > 2020-06-15T03:24:28.4714560Z [ERROR] at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > 2020-06-15T03:24:28.4715096Z [ERROR] at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) > 2020-06-15T03:24:28.4715672Z [ERROR] at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) > 2020-06-15T03:24:28.4716445Z [ERROR] at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > 2020-06-15T03:24:28.4717024Z [ERROR] at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120) > 2020-06-15T03:24:28.4717478Z [ERROR] at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355) > 2020-06-15T03:24:28.4717939Z [ERROR] at > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) > 2020-06-15T03:24:28.4718378Z [ERROR] at > org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) > 2020-06-15T03:24:28.4718852Z [ERROR] at > org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216) > 2020-06-15T03:24:28.4719230Z [ERROR] at > org.apache.maven.cli.MavenCli.main(MavenCli.java:160) > 2020-06-15T03:24:28.4719676Z [ERROR] at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > 2020-06-15T03:24:28.4720309Z [ERROR] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > 2020-06-15T03:24:28.4720882Z [ERROR] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > 2020-06-15T03:24:28.4721339Z [ERROR] at > java.lang.reflect.Method.invoke(Method.java:498) > 2020-06-15T03:24:28.4721888Z [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > 2020-06-15T03:24:28.4722658Z [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > 2020-06-15T03:24:28.4723430Z [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > 2020-06-15T03:24:28.4724062Z [ERROR] at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > 2020-06-15T03:24:28.4724657Z [ERROR] Caused by: > org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM > terminated without properly saying goodbye. VM crash or System.exit called? > 2020-06-15T03:24:28.4726770Z [ERROR] Command was /bin/sh -c cd > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target && > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m > -Dlog4j.configurationFile=log4j2-test.properties -Dmvn.forkNumber=2 > -XX:-UseGCOverheadLimit -jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire/surefirebooter617700788970993266.jar > /__w/2/s/flink-connectors/flink-connector-kafka-0.11/target/surefire > 2020-06-15T03-07-01_381-jvmRun2 surefire2676050245109796726tmp > surefire_602825791089523551074tmp > 2020-06-15T03:24:28.4728582Z [ERROR] Error occurred in starting fork, check > output in log > 2020-06-15T03:24:28.4729202Z [ERROR] Process Exit Code: 239 > 2020-06-15T03:24:28.4729612Z [ERROR] Crashed tests: > 2020-06-15T03:24:28.4730247Z [ERROR] > org.apache.flink.streaming.connectors.kafka.Kafka011ProducerExactlyOnceITCase > 2020-06-15T03:24:28.4730781Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669) > 2020-06-15T03:24:28.4731292Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115) > 2020-06-15T03:24:28.4731829Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444) > 2020-06-15T03:24:28.4732353Z [ERROR] at > org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420) > 2020-06-15T03:24:28.4732792Z [ERROR] at > java.util.concurrent.FutureTask.run(FutureTask.java:266) > 2020-06-15T03:24:28.4733235Z [ERROR] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 2020-06-15T03:24:28.4733718Z [ERROR] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 2020-06-15T03:24:28.4734170Z [ERROR] at java.lang.Thread.run(Thread.java:748) > 2020-06-15T03:24:28.4734682Z [ERROR] -> [Help 1] > 2020-06-15T03:24:28.4734859Z [ERROR] > 2020-06-15T03:24:28.4735312Z [ERROR] To see the full stack trace of the > errors, re-run Maven with the -e switch. > 2020-06-15T03:24:28.4735927Z [ERROR] Re-run Maven using the -X switch to > enable full debug logging. > 2020-06-15T03:24:28.4736439Z [ERROR] > 2020-06-15T03:24:28.4736952Z [ERROR] For more information about the errors > and possible solutions, please read the following articles: > 2020-06-15T03:24:28.4737706Z [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > 2020-06-15T03:24:28.4738167Z [ERROR] > 2020-06-15T03:24:28.4738553Z [ERROR] After correcting the problems, you can > resume the build with the command > 2020-06-15T03:24:28.4739663Z [ERROR] mvn <goals> -rf > :flink-connector-kafka-0.11_2.11 > 2020-06-15T03:24:29.0980029Z MVN exited with EXIT CODE: 1. > {code} > This could be a CI environment issue... > When did it start? -- This message was sent by Atlassian Jira (v8.3.4#803005)