I'm assuming you're seeing a lot of threads that are BLOCKED waiting on that locked GridLocalCacheEntry (<70d32489> in that example you pasted above). Looking at the code, it looks like it does block on individual cache entries (so two reads of the same key within the same JVM will block). In your particular example above, the thread in question is publishing an EVT_CACHE_OBJECT_EXPIRED event. If you don't need that, turning it off (along with EVT_CACHE_OBJECT_READ) will speed up the time that the cache entry spends blocking other reads (and speed things up, generally). It's locking to make sure it's deserialized from swap once and expired once (if necessary; looks like it was in this particular case).
matt On Fri, Apr 22, 2016 at 8:07 AM, Vladimir Ozerov <voze...@gridgain.com> wrote: > Hi, > > Could you please explain why do you think that the thread is blocked? I > see it is in a RUNNABLE state. > > Vladimir. > > On Fri, Apr 22, 2016 at 2:41 AM, ccanning <ccann...@stubhub.com> wrote: > >> We seem to be having some serious performance issues after adding Apache >> Ignite Local cache to our APIs'. Looking at a heap dump, we seem to have a >> bunch of threads blocked by this lock: >> >> "ajp-0.0.0.0-8009-70" - Thread t@641 >> java.lang.Thread.State: RUNNABLE >> at >> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.<init>(BinaryReaderExImpl.java:166) >> at >> >> org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1486) >> at >> >> org.apache.ignite.internal.binary.BinaryUtils.deserializeOrUnmarshal(BinaryUtils.java:1830) >> at >> >> org.apache.ignite.internal.binary.BinaryUtils.doReadMap(BinaryUtils.java:1813) >> at >> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1597) >> at >> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(BinaryReaderExImpl.java:1646) >> at >> >> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read(BinaryFieldAccessor.java:643) >> at >> >> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:714) >> at >> >> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450) >> at >> >> org.apache.ignite.internal.binary.BinaryObjectImpl.deserializeValue(BinaryObjectImpl.java:537) >> at >> >> org.apache.ignite.internal.binary.BinaryObjectImpl.value(BinaryObjectImpl.java:117) >> at >> >> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinary(CacheObjectContext.java:280) >> at >> >> org.apache.ignite.internal.processors.cache.CacheObjectContext.unwrapBinaryIfNeeded(CacheObjectContext.java:145) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:276) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:159) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:92) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet0(GridCacheMapEntry.java:862) >> - locked <70d32489> (a >> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerGet(GridCacheMapEntry.java:669) >> at >> >> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:587) >> at >> >> org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.get(GridLocalAtomicCache.java:483) >> at >> >> org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1378) >> at >> >> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:864) >> at >> org.apache.ignite.cache.spring.SpringCache.get(SpringCache.java:52) >> >> **** - locked <70d32489> (a >> org.apache.ignite.internal.processors.cache.local.GridLocalCacheEntry) >> >> Should this be causing blocking in a high-throughput API? Do you have any >> pointers in how we could solve this issue? >> >> Thanks. >> >> >> >> -- >> View this message in context: >> http://apache-ignite-users.70518.x6.nabble.com/Performance-Issue-Threads-blocking-tp4433.html >> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >> > >
