Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello!

At this point I recommend debugging which statements are ran on Oracle and
why they take long.

Also I have noticed: appDataSource - is it behind some kind of connection
pool? I am afraid it is possible that this data source is single-threaded
in the absense of connection pool, hence you might see contention.

Regards,

-- 
Ilya Kasnacheev

2018-07-18 20:48 GMT+03:00 Shailendrasinh Gohil <
shailendrasinh.go...@salientcrgt.com>:

> Here you go...
>
> 
> 
> 
> 
> 
>
> 
>  class="org.apache.ignite.cache.store.jdbc.
> CacheJdbcPojoStoreFactory">
>  value="appDataSource" />
> 
> 
> 
> 
>
> 
> 
> 
>  value="GeneratedProdIdCache" />
>  value="java.lang.String" />
> 
> value="ignite.model.GeneratedProdId" />
>  name="databaseSchema" value="APPSCHEMA" />
>  name="databaseTable" value="GENERATED_PROD_ID" />
>
> 
> 
>  class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> 
>
>   
>
> 
>
> 
>
> 
>
> 
> 
> 
> 
>
>  name="valueFields">
> 
>  class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> 
>
>   
>
> 
>
> 
>
> 
>
> 
> 
>
>   class="org.apache.ignite.cache.store.jdbc.JdbcTypeField">
>
> 
>
>   
>
> 
>
> 
>
> 
>
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
>
> 
> 
> 
>  value="java.lang.String" />
>  value="ignite.model.GeneratedProdId"
> />
>  value="GENERATED_PROD_ID" />
>  value="prodId" />
>
> 
> 
> prodId
> 
> 
>
> 
> 
>  value="java.lang.String" />
>  value="java.lang.String" />
> 
> 
>
> 
> 
>  value="PROD_ID" />
>  value="PROD_ID_KEY" />
> 
> 
>
> 
> 
>  class="org.apache.ignite.cache.QueryIndex">
>  name="name" value="INDX_GNRTD_PROD_ID" />
>  name="indexType" value="FULLTEXT" />
>
>  name="fields">
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Possible starvation in striped pool

[Shailendrasinh Gohil]

Here you go...




























































 












 


























prodId






































--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello again!

I have just noticed the following stack trace:

"flusher-0-#588%AppCluster%" #633 prio=5 os_prio=0
tid=0x7f18d424f800 nid=0xe1bb runnable [0x7f197c1cd000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at oracle.net.ns.Packet.receive(Packet.java:311)
at oracle.net.ns.DataPacket.receive(DataPacket.java:105)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:249)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:171)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:89)
at 
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at 
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at 
oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
at 
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:225)
at 
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:53)
at 
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:943)
at 
oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:12029)
at 
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12140)
- locked <0x7f2aaa591778> (a oracle.jdbc.driver.T4CConnection)
at 
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
at 
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.executeBatch(CacheAbstractJdbcStore.java:1226)
at 
org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore.writeAll(CacheAbstractJdbcStore.java:)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.updateStore(GridCacheWriteBehindStore.java:809)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.applyBatch(GridCacheWriteBehindStore.java:725)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.access$2400(GridCacheWriteBehindStore.java:75)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.flushCacheCoalescing(GridCacheWriteBehindStore.java:1113)
at 
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore$Flusher.body(GridCacheWriteBehindStore.java:1011)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)

It looks like you're waiting for Oracle to complete batch execution, but it
won't.

Regards,

-- 
Ilya Kasnacheev

2018-07-18 17:47 GMT+03:00 Ilya Kasnacheev :

> Hello!
>
> Can you please share the configuration of your Apache Ignite nodes,
> especially the cache store's of caches. I have just noticed that you're
> actually waiting on cache store lock.
>
> Regards,
>
> --
> Ilya Kasnacheev
>
> 2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil  salientcrgt.com>:
>
>> We are using the TreeMap for all the putAll operations. We also tried
>> streamer API to create the automatic batches. Still the issue is same.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>

Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello!

Can you please share the configuration of your Apache Ignite nodes,
especially the cache store's of caches. I have just noticed that you're
actually waiting on cache store lock.

Regards,

-- 
Ilya Kasnacheev

2018-07-17 19:11 GMT+03:00 Shailendrasinh Gohil <
shailendrasinh.go...@salientcrgt.com>:

> We are using the TreeMap for all the putAll operations. We also tried
> streamer API to create the automatic batches. Still the issue is same.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Possible starvation in striped pool

[Shailendrasinh Gohil]

We are using the TreeMap for all the putAll operations. We also tried
streamer API to create the automatic batches. Still the issue is same.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Sambhaji Sawant]

Hello same issue occurred when trying to put object in cache using
cache.put method.after changing put to putAsync issue was solved.

I have read about when you using putAll methode pass sorted collection to
it so it avoid deadlock. So is it true?

On Tue, Jul 17, 2018, 8:22 PM ilya.kasnacheev 
wrote:

> Hello!
>
> I have noticed that you are using putAll in your code.
>
> Apache Ignite is susceptible to deadlocks in the same fashion as regular
> multi-threaded code: i.e., if you take multiple locks (as putAll does, on
> partitions for its keys), you can get deadlock unless you maintain sequence
> of locks, i.e., always lock B after A and not the other way around.
>
> You can attain that by passing TreeMap's to putAll as opposed to HashMap's.
>
> Can you try to pass TreeMap's to putAll, see if it fixed your issue?
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Possible starvation in striped pool

[ilya.kasnacheev]

Hello!

I have noticed that you are using putAll in your code.

Apache Ignite is susceptible to deadlocks in the same fashion as regular
multi-threaded code: i.e., if you take multiple locks (as putAll does, on
partitions for its keys), you can get deadlock unless you maintain sequence
of locks, i.e., always lock B after A and not the other way around.

You can attain that by passing TreeMap's to putAll as opposed to HashMap's.

Can you try to pass TreeMap's to putAll, see if it fixed your issue?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Shailendrasinh Gohil]

Please find attached thread dump as requested.

ServerThreadDump0716.txt

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello!

Can you please provide the thread dump of problematic cluster after removal
of close statements on caches?

Regards,

-- 
Ilya Kasnacheev

2018-07-16 17:21 GMT+03:00 Shailendrasinh Gohil <
shailendrasinh.go...@salientcrgt.com>:

> Thanks again for the response.
>
> We have tried removing the close statements but the result was same. And
> yes, other threads accessing cache from the same Dao.
>
> We also tried both the atomicityMode to see if any improvement. We also
> have
> write behind enabled for the large tables with frequent get and put
> operations.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Possible starvation in striped pool

[Shailendrasinh Gohil]

Thanks again for the response.

We have tried removing the close statements but the result was same. And
yes, other threads accessing cache from the same Dao.

We also tried both the atomicityMode to see if any improvement. We also have
write behind enabled for the large tables with frequent get and put
operations.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello!

I can see here that you are trying to destroy a cache:

at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
at
org.apache.ignite.internal.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.close(GatewayProtectedCacheProxy.java:1514)
at
dao.impl.IgniteLookupServiceImpl.getProdCountForEst(IgniteLookupServiceImpl.java:2645)
at
service.impl.RegistrationServiceImpl.getProdCountForEst(RegistrationServiceImpl.java:2588)

Is it supposed to be destroyed in this method? I can see other threads
accessing cache from the same Dao.

Regards,

-- 
Ilya Kasnacheev

2018-07-12 19:35 GMT+03:00 Shailendrasinh Gohil <
shailendrasinh.go...@salientcrgt.com>:

> Thank you for your response. Please find attached thread dumps for client
> and
> server nodes. ClientThreadDump.txt
>  t1917/ClientThreadDump.txt>
> ThreadDumpServer1.txt
>  t1917/ThreadDumpServer1.txt>
> ThreadDumpServer2.txt
>  t1917/ThreadDumpServer2.txt>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Possible starvation in striped pool

[Shailendrasinh Gohil]

Thank you for your response. Please find attached thread dumps for client and
server nodes. ClientThreadDump.txt

  
ThreadDumpServer1.txt

  
ThreadDumpServer2.txt

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool

[Ilya Kasnacheev]

Hello!

Unfortunately it's hard to say without looking at full thread dumps from
all nodes in cluster. Can you post them somewhere?

Regards,

-- 
Ilya Kasnacheev

2018-07-12 18:04 GMT+03:00 Gohil, Shailendrasinh (INTL) <
shailendrasinh.go...@salientcrgt.com>:

> We are working on integrating the JEE web application with Apache Ignite
> and caching around 900 millions records from Oracle database to off heap
> cache. The client was started from WebLogic using the servlet listener.
> When users are trying to access the application, the application queries
> their data from cache. We see the below issue when there are more than 2
> users performing the similar operation on their own data. This was not the
> performance we expected from the documentation.
>
>
>
> WARN [org.apache.ignite.internal.util.typedef.G] - >>> Possible
> starvation in striped pool.
>
> Thread name: sys-stripe-14-#15%AppCluster%
>
> Queue: []
>
> Deadlock: false
>
> Completed: 4
>
> Thread [name="sys-stripe-14-#15% AppCluster%", id=107, state=WAITING,
> blockCnt=0, waitCnt=6]
>
> Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$
> NonfairSync@5a7be62, ownerName=exchange-worker-#57% AppCluster%,
> ownerId=171]
>
> at sun.misc.Unsafe.park(Native Method)
>
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireShared(AbstractQueuedSynchronizer.java:967)
>
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireShared(AbstractQueuedSynchronizer.java:1283)
>
> at java.util.concurrent.locks.ReentrantReadWriteLock$
> ReadLock.lock(ReentrantReadWriteLock.java:727)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(
> GridCacheIoManager.java:317)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(
> GridCacheIoManager.java:304)
>
> at o.a.i.i.processors.cache.GridCacheIoManager.access$100(
> GridCacheIoManager.java:99)
>
> at o.a.i.i.processors.cache.GridCacheIoManager$1.
> onMessage(GridCacheIoManager.java:293)
>
> at o.a.i.i.managers.communication.GridIoManager.
> invokeListener(GridIoManager.java:1555)
>
> at o.a.i.i.managers.communication.GridIoManager.
> processRegularMessage0(GridIoManager.java:1183)
>
> at o.a.i.i.managers.communication.GridIoManager.
> access$4200(GridIoManager.java:126)
>
> at o.a.i.i.managers.communication.GridIoManager$9.
> run(GridIoManager.java:1090)
>
> at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.
> java:505)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
>

Re: Possible starvation in striped pool. message

[ezhuravlev]

peerClassLoading used only for compute, for example for sharing job classes
between nodes, it's not working for objects that put into cache. If you want
to work without this classes on nodes, take a look to BinaryObjects:
https://apacheignite.readme.io/v2.0/docs/binary-marshaller

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible starvation in striped pool. message

[kestas]

I did try your suggestion. My code:
 some = ignite.compute().affinityCall(CACHE_NAME,1, () ->
cacheBinary.get(1).field("some"));
And i get ClassNotFoundException for the object class which is put into
cache. peerClassLoadingEnabled is set to true.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-tp15993p16482.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Possible starvation in striped pool. message

[Yakov Zhdanov]

Well, we need to take a closer look then. This may be affected by
transaction protocol. Viacheslav Koptilin, can you please create a test and
see what time goes to?

kestas, you can switch to Ignite.compute().affinityCall("key", () ->
{return cacheBinary.get("key").field("f"));}); This should work fine for
both transactional and atomic caches.

Thanks!
--
Yakov Zhdanov, Director R&D
*GridGain Systems*
www.gridgain.com

2017-08-09 16:30 GMT+03:00 kestas :

> I did some testing on #invoke vs #get performance. It works as expected on
> ATOMIC cache, however on TRANSACTIONAL cache #invoke has even lower
> performance than pure #get. Is this to be expected?
>
> simplified code:
> some = cacheBinary.invoke(1,(mutableEntry, objects) ->
> mutableEntry.getValue().field("some"));
> and
> some = cache.get(1).getSome();
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Possible-starvation-in-striped-pool-
> message-tp15993p16081.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Possible starvation in striped pool. message

[kestas]

I did some testing on #invoke vs #get performance. It works as expected on
ATOMIC cache, however on TRANSACTIONAL cache #invoke has even lower
performance than pure #get. Is this to be expected?

simplified code:
some = cacheBinary.invoke(1,(mutableEntry, objects) ->
mutableEntry.getValue().field("some"));
and
some = cache.get(1).getSome();



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-tp15993p16081.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Possible starvation in striped pool. message

[Yakov Zhdanov]

Hi!

Ignite prints stacktrace of an internal thread if it detects that thread
runs the same task (single get response in your case) for a long time,
which is 30 seconds by default.

Of course, getting large objects from remote nodes is expensive. Can you
please estimate the value size?

Possible ways of speeding up the application:
1. Refactor object model to reduce values size
2. use org.apache.ignite.IgniteCache#withKeepBinary - this will return
IgniteCache that will not deserialize binary objects, therefore there will
be no deserialization. Take a look at this page for info -
https://apacheignite.readme.io/docs/binary-marshaller
3. use org.apache.ignite.IgniteCache#invoke(K,
org.apache.ignite.cache.CacheEntryProcessor, java.lang.Object...)
instead of pure get() providing the processor that will not be altering the
value, but return only some part of the object (e.g. tuple of last name and
city instead of returning the entire User object).
4. confiure NearCache - https://apacheignite.readme.io/docs/near-caches and
set copy on read
(org.apache.ignite.configuration.CacheConfiguration#isCopyOnRead) to false
- however, bringing large objects to client may overload the client and
introduce new problems.

I would first try 1-3. Please let us know the results.

--Yakov

Re: Possible starvation in striped pool. message

[kestas]

Yes, this seems to appear when we start working with large objects. Is there
a way to solve this? Does it affect cache put/get operations performance
directly ?



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-tp15993p16025.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Possible starvation in striped pool. message

[slava.koptilin]

Hi Kestas,

There are several possible reasons for that.
In your case, I think you are trying to put or get huge objects, that
contain internal collections and so on.

>at
> o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016) 
>at
> o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904) 
...
>at
> o.a.i.i.binary.BinaryUtils.doReadCollection(BinaryUtils.java:2016) 
>at
> o.a.i.i.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1904) 

Another possible reason is intensive use of getAll()/putAll() methods from
different threads.
In that case, you need to sort collection of keys before, because batch
operation on the same entries in different order could lead to deadlock.


Thanks.



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Possible-starvation-in-striped-pool-message-tp15993p16002.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Possible starvation in striped pool

[Andrey Mashenkov]

Hi Alper,

Seems, sometime stripe-pool threads stucks for awhile due to short network
issues.
Possibly, it happens when nodes doesn't left the grid gracefully.

Try to set java.net.preferIPv4Stack=true on all nodes (incl. clients),
it can help in some cases when client is droped from grid due to IPv6
connectivity issues.

On Mon, Jul 17, 2017 at 11:04 AM, Alper Tekinalp  wrote:

> Hi Andrey.
>
> Logs for two server nodes are attached.
>
> On Fri, Jul 14, 2017 at 8:41 PM, Andrey Mashenkov <
> andrey.mashen...@gmail.com> wrote:
>
>> Hi Alper,
>>
>> Stripe pool threads do message processing.
>> In stacktrace, I see node can send message to other node due to network
>> issue or node failure.
>> Thread try to reconnect to the node during failure detection timeout.
>>
>> What  Ignite version do you use?
>> Would you please share full logs?
>>
>> On Fri, Jul 14, 2017 at 1:24 PM, Alper Tekinalp  wrote:
>>
>>> Hi.
>>>
>>> What does following log means:
>>>
>>> [WARN ] 2017-07-12 23:00:50.786 [grid-timeout-worker-#71%cache-server%]
>>> G - >>> Possible starvation in striped pool: sys-stripe-10-#11%cache-server
>>> %
>>> [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE,
>>> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
>>> msg=GridNearSingleGetRequest
>>> 
>>> 43f6-946a-6593f8dcde7d, taskNameHash=0, createTtl=0, accessTtl=-1
>>> deadlock: false
>>> completed: 699284
>>> Thread [name="sys-stripe-10-#11%cache-server%", id=44, state=WAITING,
>>> blockCnt=18197, waitCnt=642544]
>>> Lock [object=o.a.i.spi.communication.tcp.TcpCommunicationSpi$Conn
>>> ectFuture@604f22f, ownerName=null, ownerId=-1]
>>> at sun.misc.Unsafe.park(Native Method)
>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java
>>> :186)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn
>>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu
>>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
>>> at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter
>>> .java:161)
>>> at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.
>>> java:119)
>>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClien
>>> t(TcpCommunicationSpi.java:2515)
>>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0
>>> (TcpCommunicationSpi.java:2340)
>>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(
>>> TcpCommunicationSpi.java:2304)
>>> at o.a.i.i.managers.communication.GridIoManager.send(GridIoMana
>>> ger.java:1281)
>>> at o.a.i.i.managers.communication.GridIoManager.sendOrderedMess
>>> age(GridIoManager.java:1517)
>>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1335)
>>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1306)
>>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi
>>> thRetries(GridContinuousProcessor.java:1288)
>>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendNo
>>> tification(GridContinuousProcessor.java:949)
>>> at o.a.i.i.processors.continuous.GridContinuousProcessor.addNot
>>> ification(GridContinuousProcessor.java:892)
>>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:814)
>>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler.access$800(CacheContinuousQueryHandler.java:91)
>>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:426)
>>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue
>>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:421)
>>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:450)
>>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:56)
>>> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapt
>>> er.java:321)
>>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.registerResponse(GridDhtAtomicAbstract
>>> UpdateFuture.java:343)
>>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cAbstractUpdateFuture.onResult(GridDhtAtomicAbstractUpdateFu
>>> ture.java:399)
>>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi
>>> cCache.processDhtAtomicDeferredUpdateResponse(GridDhtAtomicC
>>> ache.java:3388)
>>> at 

Re: Possible starvation in striped pool

[Andrey Mashenkov]

Hi Alper,

Stripe pool threads do message processing.
In stacktrace, I see node can send message to other node due to network
issue or node failure.
Thread try to reconnect to the node during failure detection timeout.

What  Ignite version do you use?
Would you please share full logs?

On Fri, Jul 14, 2017 at 1:24 PM, Alper Tekinalp  wrote:

> Hi.
>
> What does following log means:
>
> [WARN ] 2017-07-12 23:00:50.786 [grid-timeout-worker-#71%cache-server%] G
> - >>> Possible starvation in striped pool: sys-stripe-10-#11%cache-server%
> [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
> ordered=false, timeout=0, skipOnTimeout=false, msg=GridNearSingleGetRequest
> 
> 43f6-946a-6593f8dcde7d, taskNameHash=0, createTtl=0, accessTtl=-1
> deadlock: false
> completed: 699284
> Thread [name="sys-stripe-10-#11%cache-server%", id=44, state=WAITING,
> blockCnt=18197, waitCnt=642544]
> Lock [object=o.a.i.spi.communication.tcp.TcpCommunicationSpi$
> ConnectFuture@604f22f, ownerName=null, ownerId=-1]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:186)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.
> acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at o.a.i.i.util.future.GridFutureAdapter.get0(
> GridFutureAdapter.java:161)
> at o.a.i.i.util.future.GridFutureAdapter.get(
> GridFutureAdapter.java:119)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(
> TcpCommunicationSpi.java:2515)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(
> TcpCommunicationSpi.java:2340)
> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage(
> TcpCommunicationSpi.java:2304)
> at o.a.i.i.managers.communication.GridIoManager.
> send(GridIoManager.java:1281)
> at o.a.i.i.managers.communication.GridIoManager.
> sendOrderedMessage(GridIoManager.java:1517)
> at o.a.i.i.processors.continuous.GridContinuousProcessor.
> sendWithRetries(GridContinuousProcessor.java:1335)
> at o.a.i.i.processors.continuous.GridContinuousProcessor.
> sendWithRetries(GridContinuousProcessor.java:1306)
> at o.a.i.i.processors.continuous.GridContinuousProcessor.
> sendWithRetries(GridContinuousProcessor.java:1288)
> at o.a.i.i.processors.continuous.GridContinuousProcessor.
> sendNotification(GridContinuousProcessor.java:949)
> at o.a.i.i.processors.continuous.GridContinuousProcessor.
> addNotification(GridContinuousProcessor.java:892)
> at o.a.i.i.processors.cache.query.continuous.
> CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.
> java:814)
> at o.a.i.i.processors.cache.query.continuous.
> CacheContinuousQueryHandler.access$800(CacheContinuousQueryHandler.
> java:91)
> at o.a.i.i.processors.cache.query.continuous.
> CacheContinuousQueryHandler$1$1.apply(CacheContinuousQueryHandler.
> java:426)
> at o.a.i.i.processors.cache.query.continuous.
> CacheContinuousQueryHandler$1$1.apply(CacheContinuousQueryHandler.
> java:421)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFut
> ure.java:450)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFut
> ure.java:56)
> at o.a.i.i.util.future.GridFutureAdapter.onDone(
> GridFutureAdapter.java:321)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicAbstractUpdateFuture.registerResponse(
> GridDhtAtomicAbstractUpdateFuture.java:343)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicAbstractUpdateFuture.onResult(GridDhtAtomicAbstractUpdateFut
> ure.java:399)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicCache.processDhtAtomicDeferredUpdate
> Response(GridDhtAtomicCache.java:3388)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicCache.access$2000(GridDhtAtomicCache.java:126)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicCache$10.apply(GridDhtAtomicCache.java:412)
> at o.a.i.i.processors.cache.distributed.dht.atomic.
> GridDhtAtomicCache$10.apply(GridDhtAtomicCache.java:407)
> at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(
> GridCacheIoManager.java:827)
> at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(
> GridCacheIoManager.java:369)
> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(
> GridCacheIoManager.java:293)
> at o.a.i.i.processors.cache.GridC