Hi all. We have 3 servers and cache configuration like:
<bean class="org.apache.ignite.configuration.CacheConfiguration" name="DEFAULT"> <property name="atomicityMode" value="ATOMIC" /> <property name="cacheMode" value="PARTITIONED" /> <property name="memoryMode" value="OFFHEAP_TIERED" /> <property name="backups" value="1" /> <property name="affinity"> <bean class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction"> <constructor-arg index="0" type="int" value="#{evamProperties['topology.cache.partition.size']}"/> </bean> </property> <!-- max memory is set to 0 means unlimited --> <property name="offHeapMaxMemory" value="0" /> <property name="writeThrough" value="false" /> <property name="writeBehindEnabled" value="false" /> <property name="eagerTtl" value="false"/> </bean> For our worker threads we check heartbeat and if a thread did not sent heart beat for 10 minutes we consider it as stucked and interrrupt and recreate it. As I can see all our worker threads are stucked in cache.put() state and interrupted and recreated regularly. What can be the reason we are stucked at put? Following is stacktrace for interruption error. javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Failed to wait for asynchronous operation permit (thread got interrupted). at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1502) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2021) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1221) at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(ConfigManagerHelperEE.java:90) at com.intellica.project.helper.ee.StateMachineConfigManagerEEImpl.store(StateMachineConfigManagerEEImpl.java:53) at com.evelopers.unimod.runtime.AbstractEventProcessor.storeConfig(AbstractEventProcessor.java:175) at com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:130) at com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:80) at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine.java:199) at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHandler.java:46) at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenarioLogic(AbstractScenarioWorker.java:172) at com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenario(AbstractScenarioWorker.java:130) at com.intellica.evam.engine.server.worker.AsyncWorker.processEvent(AsyncWorker.java:156) at com.intellica.evam.engine.server.worker.AsyncWorker.run(AsyncWorker.java:88) Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to wait for asynchronous operation permit (thread got interrupted). at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:747) at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:745) ... 14 more Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1214) ... 11 more -- Alper Tekinalp Software Developer Evam Streaming Analytics Atatürk Mah. Turgut Özal Bulv. Gardenya 5 Plaza K:6 Ataşehir 34758 İSTANBUL Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 www.evam.com.tr <http://www.evam.com>