[
https://issues.apache.org/jira/browse/IGNITE-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850277#comment-15850277
]
Nikolay Tikhonov commented on IGNITE-4589:
------------------------------------------
Cause of the problem in processing ordered messages. During balancing between
two nodes happens the following scenario:
* Ordered {{GridDhtPartitionSupplyMessageV2}} message was sent for
{{ignite-marshaller-sys-cache}} cache with {{GridIoPolicy#MARSH_CACHE_POOL}};
* Ordered {{GridDhtPartitionSupplyMessageV2}} message was sent for
{{ignite-sys-cache}} cache with {{GridIoPolicy#UTILITY_CACHE_POOL}};
* On receiver node, the first message creates {{GridCommunicationMessageSet}}
and start polling messages from queue in marshaller thread pool.
* Second supply message added in the same {{GridCommunicationMessageSet}} and
that thread (from marshaller thread pool) is proccessed this message, although
the message have different io policy.
> Possible starvation during rebalancing for marshaller cache
> -----------------------------------------------------------
>
> Key: IGNITE-4589
> URL: https://issues.apache.org/jira/browse/IGNITE-4589
> Project: Ignite
> Issue Type: Bug
> Components: cache
> Affects Versions: 1.8
> Reporter: Nikolay Tikhonov
> Assignee: Nikolay Tikhonov
>
> When Ignite processing supply messages (rebalancing) in marshaller threads
> and calling marshaller cache from them, but responses also should be
> proccessed in marshaller threadpool. It leads to starvation.
> {noformat}
> marshaller-cache-#26%test-node%" #117 prio=5 os_prio=0 tid=0x00007f89e40c6000
> nid=0x1dc waiting on condition [0x00007f8a3fffc000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000c206d318> (a
> org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedGetFuture)
> at java.util.concurrent.locks.LockSupport.park(Unknown Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown
> Source)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown
> Source)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:159)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getTopologySafe(GridCacheAdapter.java:1340)
> at
> org.apache.ignite.internal.MarshallerContextImpl.className(MarshallerContextImpl.java:195)
> at
> org.apache.ignite.internal.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:174)
> at
> org.apache.ignite.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:266)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:318)
> at java.io.ObjectInputStream.readObject(Unknown Source)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:491)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:579)
> at
> org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:324)
> at java.io.ObjectInputStream.readObject(Unknown Source)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readFields(OptimizedObjectInputStream.java:491)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readSerializable(OptimizedObjectInputStream.java:579)
> at
> org.apache.ignite.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:927)
> at
> org.apache.ignite.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:324)
> at java.io.ObjectInputStream.readObject(Unknown Source)
> at
> org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:218)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1614)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1680)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1450)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:298)
> at
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:100)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9722)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9751)
> at
> org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.unmarshal(IgniteCacheObjectProcessorImpl.java:111)
> at
> org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.unmarshal(CacheObjectBinaryProcessorImpl.java:811)
> at
> org.apache.ignite.internal.processors.cache.CacheObjectImpl.finishUnmarshal(CacheObjectImpl.java:109)
> at
> org.apache.ignite.internal.processors.cache.GridCacheEntryInfo.unmarshal(GridCacheEntryInfo.java:416)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionSupplyMessageV2.finishUnmarshal(GridDhtPartitionSupplyMessageV2.java:245)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1229)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:358)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$400(GridCacheIoManager.java:89)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1276)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1082)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:102)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2332)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1042)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$1900(GridIoManager.java:102)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$6.run(GridIoManager.java:1011)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
