Additionally, Client use the cache.putAllAsync () call. If you look at the Ignite log, you see a method call like updateAllAsyncInternal0. At the same time, does the client have a lock issue when it asynchronously calls after sending a cache entry? :(
2017-10-21 21:06 GMT+09:00 dark <[email protected]>: > Hello, I'm using Ignite 2.0.0, and I would like to ask if you have any > doubts > about the deadlock. > The first use pattern is to create a new cache time unit, and after a > certain period of time, it will perform Destroy. > > Example) > > We create a cache that keeps the data of the 3-minute cycle as shown below > > [00:00_Cache] [00:01_Cache] [00:02_Cache] > > After one minute, create a new cache [00: 03_Cache] and clear old cache > [00: > 00_Cache]. > > [00:00_Cache] is destroy! > [00:03_Cache] is new! > > below current cache list > [00:01_Cache] [00:02_Cache] [00:03_Cache] > > The reason for using this is to remove the data of a certain time period > quickly rather than the expiry of Cache. As a result of eye observation, it > was possible to quickly remove data in the time zone without using a lot of > CPU. > In this state, I kept it for about 5 hours, and then I took down 5 Client > nodes that existed in Topology for a while and then uploaded them again. > Then, about ten minutes later, a deadlock occurred with the following > message. > > [19:48:51,290][WARN ][grid-timeout-worker-#15%null%][G] >>> Possible > starvation in striped pool. > Thread name: sys-stripe-3-#4%null% > Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, > topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, > msg=GridDhtAtomicUpdateRequest [keys=[KeyCacheObjectImpl [part=179, val > Deadlock: true > Completed: 1054320 > Thread [name="sys-stripe-3-#4%null%", id=21, state=BLOCKED, blockCnt=5364, > waitCnt=1261740] > Lock > [object=o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCacheEntry@6c7a9d31, > ownerName=sys-stripe-6-#7%null%, ownerId=24] > at > o.a.i.i.processors.cache.GridCacheMapEntry.markObsoleteIfEmpty( > GridCacheMapEntry.java:2095) > at > o.a.i.i.processors.cache.CacheOffheapEvictionManager.touch( > CacheOffheapEvictionManager.java:44) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > unlockEntries(GridDhtAtomicCache.java:2896) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal0(GridDhtAtomicCache.java:1853) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal(GridDhtAtomicCache.java:1630) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3016) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:127) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:282) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:277) > at > o.a.i.i.processors.cache.GridCacheIoManager.processMessage( > GridCacheIoManager.java:863) > at > o.a.i.i.processors.cache.GridCacheIoManager.onMessage0( > GridCacheIoManager.java:386) > at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage( > GridCacheIoManager.java:308) > at > o.a.i.i.processors.cache.GridCacheIoManager.access$000( > GridCacheIoManager.java:100) > at > o.a.i.i.processors.cache.GridCacheIoManager$1. > onMessage(GridCacheIoManager.java:253) > at > o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager. > java:1257) > at > o.a.i.i.managers.communication.GridIoManager.processRegularMessage0( > GridIoManager.java:885) > at > o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager. > java:114) > at > o.a.i.i.managers.communication.GridIoManager$7.run(GridIoManager.java:802) > at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor. > java:483) > at java.lang.Thread.run(Thread.java:745) > > [19:48:51,423][WARN ][grid-timeout-worker-#15%null%][G] >>> Possible > starvation in striped pool. > Thread name: sys-stripe-5-#6%null% > Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, > topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, > msg=GridDhtAtomicUpdateRequest [keys=[KeyCacheObjectImpl [part=541, val > Deadlock: true > Completed: 932925 > Thread [name="sys-stripe-5-#6%null%", id=23, state=BLOCKED, blockCnt=5629, > waitCnt=1137576] > Lock > [object=o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCacheEntry@449f1914, > ownerName=sys-stripe-6-#7%null%, ownerId=24] > at sun.misc.Unsafe.monitorEnter(Native Method) > at o.a.i.i.util.GridUnsafe.monitorEnter(GridUnsafe.java:1193) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > lockEntries(GridDhtAtomicCache.java:2815) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal0(GridDhtAtomicCache.java:1741) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > updateAllAsyncInternal(GridDhtAtomicCache.java:1630) > at > o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache. > processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3016) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:127) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:282) > at > o.a.i.i.processors.cache.distributed.dht.atomic. > GridDhtAtomicCache$6.apply(GridDhtAtomicCache.java:277) > at > o.a.i.i.processors.cache.GridCacheIoManager.processMessage( > GridCacheIoManager.java:863) > at > o.a.i.i.processors.cache.GridCacheIoManager.onMessage0( > GridCacheIoManager.java:386) > at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage( > GridCacheIoManager.java:308) > at > o.a.i.i.processors.cache.GridCacheIoManager.access$000( > GridCacheIoManager.java:100) > at > o.a.i.i.processors.cache.GridCacheIoManager$1. > onMessage(GridCacheIoManager.java:253) > at > o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager. > java:1257) > at > o.a.i.i.managers.communication.GridIoManager.processRegularMessage0( > GridIoManager.java:885) > at > o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager. > java:114) > at > o.a.i.i.managers.communication.GridIoManager$7.run(GridIoManager.java:802) > at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor. > java:483) > at java.lang.Thread.run(Thread.java:745) > > Deadlock jmc picture > <http://apache-ignite-users.70518.x6.nabble.com/file/ > t1415/ignite-deadlock-1.ignite-deadlock-1> > <http://apache-ignite-users.70518.x6.nabble.com/file/ > t1415/ignite-deadlock-2.ignite-deadlock-2> > <http://apache-ignite-users.70518.x6.nabble.com/file/ > t1415/ignite-deadlock-3.png> > > As you can see in the picture above, we can see that sys-stripe-5 and > sys-stripe-6 are the owner of the thread. Besides Ignite Cache > Configuration > is shown below. > > return ignite.getOrCreateCache(new CacheConfiguration<String, > RollupMetric>() > .setName(cacheName) > .setCacheMode(CacheMode.PARTITIONED) > .setAtomicityMode(CacheAtomicityMode.ATOMIC) > .setRebalanceMode(CacheRebalanceMode.ASYNC) > .setMemoryPolicyName(MEMORY_POLICY_NAME) > .setBackups(1) > .setStatisticsEnabled(true) > .setManagementEnabled(true) > .setCopyOnRead(false) > .setQueryParallelism(20) > .setLongQueryWarningTimeout(10000) // 10s > .setEagerTtl(false) > .setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new > Duration(TimeUnit.DAYS, 365))) > > .setMaxConcurrentAsyncOperations(CacheConfiguration.DFLT_MAX_ > CONCURRENT_ASYNC_OPS > * 10) > .setAffinity(new CoupangAffinityFunction()) > .setIndexedTypes(String.class, RollupMetric.class)); > > The reason for setting the CacheExpiryPolicy to 1 year above is because the > entry is evicted by clearing the cache as described previously. > > Ignite Memory Configuration > <property name="memoryConfiguration"> > <bean class="org.apache.ignite.configuration.MemoryConfiguration"> > > <property name="memoryPolicies"> > <list> > <bean > class="org.apache.ignite.configuration.MemoryPolicyConfiguration"> > <property name="name" value="RollupMemory"/> > > <property name="pageEvictionMode" value="RANDOM_LRU"/> > <property name="metricsEnabled" value="true"/> > > <property name="initialSize" value="21474836480"/> > > <property name="maxSize" value="21474836480"/> > </bean> > </list> > </property> > <property name="pageSize" value="4096"/> > <property name="concurrencyLevel" value="8"/> > </bean> > </property> > > For what reason did Deadlock occur? Is there an option or usage pattern to > solve this? > > I think it is due to the client's topology changes. If so, how would you > handle it? > > Please let me know if you have any additional questions. > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
