[jira] [Commented] (IGNITE-5125) Need to improve logging in case of hang
[ https://issues.apache.org/jira/browse/IGNITE-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715056#comment-16715056 ] ASF GitHub Bot commented on IGNITE-5125: Github user zstan closed the pull request at: https://github.com/apache/ignite/pull/1922 > Need to improve logging in case of hang > --- > > Key: IGNITE-5125 > URL: https://issues.apache.org/jira/browse/IGNITE-5125 > Project: Ignite > Issue Type: Improvement >Reporter: Yakov Zhdanov >Priority: Critical > Fix For: 2.1 > > > 1. When cache operation hangs on node it is not reported as hanged although > partition map exchange cannot finish. > {noformat} >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fdd260fd7c0> (a > org.apache.ignite.internal.util.future.GridEmbeddedFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2356) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4168) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2335) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2312) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1379) > {noformat} > {noformat} > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fa0698fee50> (a > org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4570) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4544) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1428) > at > org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:329) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:589) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.sequence(DataStructuresProcessor.java:397) > {noformat} > 2. Partition exchnage future dumps objects only limited number of times. I > would suggest to switch to mode when we double the delay between dumps each > time, but no more than 30min > 3. If exchange worker is stuck at > GridDhtPartitionsExchangeFuture.waitPartitionRelease then unreleased > partitions should be reported (same rules as of pt 2 apply) > {noformat} > "exchange-worker-#93%...%" #143 prio=5 os_prio=0 tid=0x7fd782df3000 > nid=0x1526 waiting on condition [0x7fc2dc9c5000] >java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fcab30006b8> (a > org.apache.ignite.internal.util.future.GridCompoundFuture) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronize
[jira] [Commented] (IGNITE-5125) Need to improve logging in case of hang
[ https://issues.apache.org/jira/browse/IGNITE-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992932#comment-15992932 ] Stanilovsky Evgeny commented on IGNITE-5125: Yakov, pt 1 it`s all due to : {code} // If exchange is in progress it will dump all hanging operations if any. if (lastFut != null && !lastFut.isDone()) return; {code} does ignite config property will be proper here ? pt 2\3 i hope GridLogThrottle is good candidate for this purpose (with some modifications) but found memory leak in this realization, force to create jira ticket. > Need to improve logging in case of hang > --- > > Key: IGNITE-5125 > URL: https://issues.apache.org/jira/browse/IGNITE-5125 > Project: Ignite > Issue Type: Improvement >Reporter: Yakov Zhdanov >Assignee: Stanilovsky Evgeny >Priority: Critical > Fix For: 2.1 > > > 1. When cache operation hangs on node it is not reported as hanged although > partition map exchange cannot finish. > {noformat} >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fdd260fd7c0> (a > org.apache.ignite.internal.util.future.GridEmbeddedFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2356) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4168) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2335) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2312) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1379) > {noformat} > {noformat} > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fa0698fee50> (a > org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4570) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4544) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1428) > at > org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:329) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:589) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.sequence(DataStructuresProcessor.java:397) > {noformat} > 2. Partition exchnage future dumps objects only limited number of times. I > would suggest to switch to mode when we double the delay between dumps each > time, but no more than 30min > 3. If exchange worker is stuck at > GridDhtPartitionsExchangeFuture.waitPartitionRelease then unreleased > partitions should be reported (same rules as of pt 2 apply) > {noformat} > "exchange-worker-#93%...%" #143 prio=5 os_prio=0 tid=0x7fd782df3000 > nid=0x1526 waiting on condition [0x7fc2dc9c5000] >
[jira] [Commented] (IGNITE-5125) Need to improve logging in case of hang
[ https://issues.apache.org/jira/browse/IGNITE-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004416#comment-16004416 ] ASF GitHub Bot commented on IGNITE-5125: GitHub user zstan opened a pull request: https://github.com/apache/ignite/pull/1922 IGNITE-5125, improve logging in case of hang while exchange cannot fi… …nish. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-5125 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/1922.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1922 commit 5587c33c99c3e40d4efca0068a0508c40f8abc5c Author: Evgeny Stanilovskiy Date: 2017-05-05T12:35:54Z IGNITE-5125, improve logging in case of hang while exchange cannot finish. > Need to improve logging in case of hang > --- > > Key: IGNITE-5125 > URL: https://issues.apache.org/jira/browse/IGNITE-5125 > Project: Ignite > Issue Type: Improvement >Reporter: Yakov Zhdanov >Assignee: Stanilovsky Evgeny >Priority: Critical > Fix For: 2.1 > > > 1. When cache operation hangs on node it is not reported as hanged although > partition map exchange cannot finish. > {noformat} >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fdd260fd7c0> (a > org.apache.ignite.internal.util.future.GridEmbeddedFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2356) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4168) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2354) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2335) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2312) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1379) > {noformat} > {noformat} > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x7fa0698fee50> (a > org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4570) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4544) > at > org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1428) > at > org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:329) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:589) > at > org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.sequence(DataStructuresProcessor.java:397) > {noformat} > 2. Partition exchnage future dumps objects only limited number of times. I > would suggest to switch to mode when we double the delay