[
https://issues.apache.org/jira/browse/IGNITE-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semen Boikov closed IGNITE-1843.
--------------------------------
Assignee: (was: Semen Boikov)
> Hang in GridJobProcessor on node stop
> -------------------------------------
>
> Key: IGNITE-1843
> URL: https://issues.apache.org/jira/browse/IGNITE-1843
> Project: Ignite
> Issue Type: Bug
> Components: compute
> Reporter: Semen Boikov
> Priority: Blocker
> Fix For: 1.5
>
>
> Observed hang in GridTaskFailoverAffinityRunTest.testNodeRestart:
> GridJobProcessor in onKernalStop tries to block operations and gets write
> lock:
> {noformat}
> [11:42:06] : [org.apache.ignite:ignite-core] Thread
> [name="restart-thread-2", id=27562, state=TIMED_WAITING, blockCnt=885,
> waitCnt=27131]
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.lang.Thread.sleep(Native Method)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.util.GridSpinReadWriteLock.writeLock(GridSpinReadWriteLock.java:210)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.job.GridJobProcessor.onKernalStop(GridJobProcessor.java:277)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.IgniteKernal.stop0(IgniteKernal.java:1824)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.IgniteKernal.stop(IgniteKernal.java:1770)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2133)
> [11:42:06] : [org.apache.ignite:ignite-core] - locked
> o.a.i.i.IgnitionEx$IgniteNamedInstance@6602227a
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2096)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.IgnitionEx.stop(IgnitionEx.java:314)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.Ignition.stop(Ignition.java:223)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:802)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.testframework.junits.GridAbstractTest.stopGrid(GridAbstractTest.java:1060)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.GridTaskFailoverAffinityRunTest.access$000(GridTaskFailoverAffinityRunTest.java:44)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.GridTaskFailoverAffinityRunTest$1.call(GridTaskFailoverAffinityRunTest.java:121)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)
> {noformat}
> Discovery listener thread is blocked trying to get read lock:
> {noformat}
> [11:42:06] : [org.apache.ignite:ignite-core] Thread
> [name="disco-event-worker-#26595%internal.GridTaskFailoverAffinityRunTest2%",
> id=32093, state=TIMED_WAITING, blockCnt=0, waitCnt=25843]
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.lang.Thread.sleep(Native Method)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.util.GridSpinReadWriteLock.readLock(GridSpinReadWriteLock.java:101)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.job.GridJobProcessor$JobDiscoveryListener.onEvent(GridJobProcessor.java:1854)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:770)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:755)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:295)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1949)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2156)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1989)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.lang.Thread.run(Thread.java:745)
> {noformat}
> There is hanging 'get' from marshaller cache inside GridJobProcessor read
> lock which probably depends on discovery event:
> {noformat}
> [11:42:06] : [org.apache.ignite:ignite-core] Thread
> [name="ignite-#26553%pub-internal.GridTaskFailoverAffinityRunTest2%",
> id=32037, state=WAITING, blockCnt=2, waitCnt=3]
> [11:42:06] : [org.apache.ignite:ignite-core] Lock
> [object=o.a.i.i.processors.cache.distributed.dht.GridPartitionedGetFuture@30ae29ee,
> ownerName=null, ownerId=-1]
> [11:42:06] : [org.apache.ignite:ignite-core] at
> sun.misc.Unsafe.park(Native Method)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:157)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.cache.GridCacheAdapter.getTopologySafe(GridCacheAdapter.java:1312)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.MarshallerContextImpl.className(MarshallerContextImpl.java:151)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:174)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:257)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:309)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.closure.GridClosureProcessor$C2.readExternal(GridClosureProcessor.java:1808)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:514)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:803)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:315)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:248)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.job.GridJobWorker.initialize(GridJobWorker.java:409)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1094)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1776)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:811)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.communication.GridIoManager.access$1500(GridIoManager.java:106)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> o.a.i.i.managers.communication.GridIoManager$5.run(GridIoManager.java:774)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [11:42:06] : [org.apache.ignite:ignite-core] at
> java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
