[
https://issues.apache.org/jira/browse/IGNITE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663744#comment-16663744
]
Sergey Chugunov commented on IGNITE-9673:
-----------------------------------------
[~NSAmelchev],
I'm OK with moving *makeResponse* call of all handlers from striped pool but
let's use REST executor service instead of system one:
*GridKernalContext::getRestExecutorService*.
Could you please make this change and run TC again to make sure deadlock won't
pop up again? If this is the case then please proceed with merging.
> Timeout in Java Client suite.
> -----------------------------
>
> Key: IGNITE-9673
> URL: https://issues.apache.org/jira/browse/IGNITE-9673
> Project: Ignite
> Issue Type: Bug
> Reporter: Amelchev Nikita
> Assignee: Amelchev Nikita
> Priority: Major
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.8
>
> Attachments: ThreadDump.txt
>
>
> Example of timeout: [TC
> build|[https://ci.ignite.apache.org/viewLog.html?buildId=1919405&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_JavaClient].]
> The possible reason is non-interruptable future and starvation in stripped
> pool:
> {noformat}
> "test-runner-#2440%redis.RedisProtocolStringSelfTest%" #3843 prio=5 os_prio=0
> tid=0x00007f8f053fb000 nid=0x7b19 waiting on condition [0x00007f8d74f8f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2465)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2463)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4228)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2463)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2444)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2421)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1089)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
> at
> org.apache.ignite.internal.processors.rest.protocols.tcp.redis.RedisProtocolStringSelfTest.testStrlen(RedisProtocolStringSelfTest.java:310)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2177)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:143)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2092)
> at java.lang.Thread.run(Thread.java:748)
> [grid-timeout-worker-#2323%redis.RedisProtocolStringSelfTest0%][G] >>>
> Possible starvation in striped pool.
> Thread name: sys-stripe-3-#2304%redis.RedisProtocolStringSelfTest0%
> Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridCacheIdMessage [cacheId=1481046058]GridDistributedBaseMessage
> [ver=GridCacheVersion [topVer=148979816, order=1537499815759, nodeOrder=1],
> committedVers=ArrayList [], rolledbackVers=ArrayList [], cnt=0,
> super=]GridDistributedLockResponse
> [futId=e739f1af561-9bc10183-74c7-4b9a-a525-aef32c002efc, err=null,
> vals=ArrayList [null], super=]GridNearLockResponse [pending=ArrayList [],
> miniId=1, dhtVers=GridCacheVersion[] [GridCacheVersion [topVer=0, order=0,
> nodeOrder=0]], mappedVers=GridCacheVersion[] [GridCacheVersion
> [topVer=148979816, order=1537499815760, nodeOrder=2]], clientRemapVer=null,
> super=]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
> msg=GridDistributedTxFinishResponse [txId=GridCacheVersion [topVer=148979816,
> order=1537499815765, nodeOrder=1],
> futId=e849f1af561-9bc10183-74c7-4b9a-a525-aef32c002efc,
> part=-1]GridNearTxFinishResponse [err=null, miniId=1, nearThreadId=3843,
> super=]]]]
> Deadlock: false
> Completed: 5
> "sys-stripe-3-#2304%redis.RedisProtocolStringSelfTest0%" #3628 prio=5
> os_prio=0 tid=0x00007f8f054e1800 nid=0x7a41 waiting on condition
> [0x00007f8ed2ef0000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4715)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4689)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1413)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:332)
> at
> org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:562)
> at
> org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:557)
> at
> org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.retryTopologySafe(DataStructuresProcessor.java:1653)
> at
> org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:557)
> at
> org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:463)
> at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3569)
> at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3558)
> at
> org.apache.ignite.internal.processors.rest.handlers.redis.string.GridRedisGetCommandHandler.makeResponse(GridRedisGetCommandHandler.java:90)
> at
> org.apache.ignite.internal.processors.rest.handlers.redis.GridRedisRestCommandHandler$1.applyx(GridRedisRestCommandHandler.java:72)
> at
> org.apache.ignite.internal.processors.rest.handlers.redis.GridRedisRestCommandHandler$1.applyx(GridRedisRestCommandHandler.java:66)
> at
> org.apache.ignite.internal.util.lang.IgniteClosureX.apply(IgniteClosureX.java:38)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:179)
> at
> org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:176)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.processors.cache.GridCacheCompoundIdentityFuture.onDone(GridCacheCompoundIdentityFuture.java:56)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onDone(GridNearGetFuture.java:220)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onDone(GridNearGetFuture.java:69)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at
> org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onDone(GridNearGetFuture.java:914)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onDone(GridNearGetFuture.java:837)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.postProcessResultAndDone(GridNearGetFuture.java:1041)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onResult(GridNearGetFuture.java:1027)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onResult(GridNearGetFuture.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearCacheAdapter.processGetResponse(GridNearCacheAdapter.java:291)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearTransactionalCache$1.apply(GridNearTransactionalCache.java:92)
> at
> org.apache.ignite.internal.processors.cache.distributed.near.GridNearTransactionalCache$1.apply(GridNearTransactionalCache.java:90)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The main thread is waiting for runner thread interrupt:
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x00007f8f0400e000 nid=0x6c18 in Object.wait()
> [0x00007f8f0db58000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1252)
> - locked <0x000000078bfaf668> (a org.apache.ignite.thread.IgniteThread)
> at
> org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:4662)
> at
> org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:4647)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:2124)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at junit.framework.TestSuite.runTest(TestSuite.java:255)
> at junit.framework.TestSuite.run(TestSuite.java:250)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:206)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:160)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:83)
> at
> org.apache.maven.plugin.surefire.InPluginVMSurefireStarter.runSuitesInProcess(InPluginVMSurefireStarter.java:84)
> at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1107)
> at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
> at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> {noformat}
> It happens in the GridAbstractTest.runTest() method and prevents test
> interrupt by timeout:
> {noformat}
> U.interrupt(runner);
> U.join(runner, log);
> throw new TimeoutException("Test has been timed out [test=" +
> getName() + ", timeout=" +
> getTestTimeout() + ']' );
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
