Denis Chudov created IGNITE-27493:
-------------------------------------
Summary: Test timeout may not work due to FJP await in cluster
shutdown
Key: IGNITE-27493
URL: https://issues.apache.org/jira/browse/IGNITE-27493
Project: Ignite
Issue Type: Bug
Reporter: Denis Chudov
Sometimes we see test suite timeouts like this:
[https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3x_Test_IntegrationTests_SystemDisasterRecovery/9876671]
Full log is attached.
There is stack trace in thread dump:
{code:java}
"Test worker" #1 prio=5 os_prio=0 cpu=19887.94ms
elapsed=3540.59s tid=0x00007fab3002f000 nid=0x4e547 in Object.wait()
[0x00007fab363fa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at
java.util.concurrent.ForkJoinTask.externalAwaitDone([email protected]/ForkJoinTask.java:330)
- waiting to re-lock in wait() <0x000000072881e770> (a
java.util.stream.ForEachOps$ForEachTask)
at
java.util.concurrent.ForkJoinTask.doInvoke([email protected]/ForkJoinTask.java:412)
at
java.util.concurrent.ForkJoinTask.invoke([email protected]/ForkJoinTask.java:736)
at
java.util.stream.ForEachOps$ForEachOp.evaluateParallel([email protected]/ForEachOps.java:159)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel([email protected]/ForEachOps.java:173)
at
java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:233)
at
java.util.stream.ReferencePipeline.forEach([email protected]/ReferencePipeline.java:497)
at
org.apache.ignite.internal.Cluster.stopNodes(Cluster.java:690)
at
org.apache.ignite.internal.Cluster.shutdown(Cluster.java:672)
at
org.apache.ignite.internal.ClusterPerTestIntegrationTest.stopCluster(ClusterPerTestIntegrationTest.java:89)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:786)
at
org.junit.platform.commons.support.ReflectionSupport.invokeMethod(ReflectionSupport.java:514)
at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at
org.junit.jupiter.engine.extension.SameThreadTimeoutInvocation.proceed(SameThreadTimeoutInvocation.java:49)
at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:161)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:133)
at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:115)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$5077/0x0000000800d93440.apply(Unknown
Source)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:112)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall$$Lambda$277/0x00000008001da840.apply(Unknown
Source)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:94)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$$Lambda$505/0x000000080022dc40.apply(Unknown
Source)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:93)
at
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:87)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:547)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$21(ClassBasedTestDescriptor.java:537)
at
org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor$$Lambda$453/0x0000000800220840.invokeAfterEachMethod(Unknown
Source)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$6(TestMethodTestDescriptor.java:246)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor$$Lambda$5075/0x0000000800d86440.invoke(Unknown
Source)
at
org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$1(CallbackSupport.java:49)
at
org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda$5076/0x0000000800d93040.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.jupiter.engine.descriptor.CallbackSupport.lambda$invokeAfterCallbacks$2(CallbackSupport.java:49)
at
org.junit.jupiter.engine.descriptor.CallbackSupport$$Lambda$5074/0x0000000800d86040.accept(Unknown
Source)
at
org.junit.platform.commons.util.CollectionUtils.forEachInReverseOrder(CollectionUtils.java:262)
at
org.junit.jupiter.engine.descriptor.CallbackSupport.invokeAfterCallbacks(CallbackSupport.java:48)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:244)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:163)
at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:70)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:157)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$418/0x000000080020a040.accept(Unknown
Source)
at
java.util.ArrayList.forEach([email protected]/ArrayList.java:1541)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:161)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService$$Lambda$418/0x000000080020a040.accept(Unknown
Source)
at
java.util.ArrayList.forEach([email protected]/ArrayList.java:1541)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:161)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$414/0x000000080020b040.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:147)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$413/0x000000080020bc40.invoke(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:145)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask$$Lambda$412/0x000000080020b840.execute(Unknown
Source)
at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:144)
at
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:101)
at
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.executeEngine(EngineExecutionOrchestrator.java:230)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.failOrExecuteEngine(EngineExecutionOrchestrator.java:204)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:172)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:101)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:64)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator$$Lambda$349/0x00000008001ed040.accept(Unknown
Source)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:150)
at
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:63)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:109)
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:91)
at
org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
at
org.junit.platform.launcher.core.InterceptingLauncher.lambda$execute$1(InterceptingLauncher.java:39)
at
org.junit.platform.launcher.core.InterceptingLauncher$$Lambda$88/0x000000080016e040.proceed(Unknown
Source)
at
org.junit.platform.launcher.core.ClasspathAlignmentCheckingLauncherInterceptor.intercept(ClasspathAlignmentCheckingLauncherInterceptor.java:25)
at
org.junit.platform.launcher.core.InterceptingLauncher.execute(InterceptingLauncher.java:38)
at
org.junit.platform.launcher.core.DelegatingLauncher.execute(DelegatingLauncher.java:47)
at
org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:119)
at
org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:94)
at
org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:89)
at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at
org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at
org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
{code}
ClusterPerTestIntegrationTest#stopCluster is annotated with @Timeout which
should interrupt the thread after timeout, see
[https://docs.junit.org/5.12.0/user-guide/index.html]
But in stack trace we can see ForkJoinTask#externalAwaitDone which swallows
InterruptedException and waits forever even if cluster shutdown can't be
completed for some reason.
The actual reason of cluster shutdown failure is unclear in this case: thread
dump of test jvm has few threads, probably some of them were interrupted for
some reason, and no thread may fire an event that shutdown worker may wait for.
Proposal: make the test cluster shutdown limited by time using another way, or
avoid using FJP in the test cluster shutdown code.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
