Re: What does "First 10 long running cache futures" ?

[Zhenya Stanilovsky]

Hi, this is just a warning shows that something suspicious observed.
There is no simple reply for your question, in common case all these messages 
are due to cluster (resources or settings)  limitation.
Check documentation for tuning performance [1]
 
[1] 
https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
 
>Hi, using 2.8.1 I understand the message as in my async TRX is taking longer 
>but is there a way to prevent it?
> 
>When this happened I was pushing about 50, 000 get/puts per second from my 
>API. 
 
 
 
 

Re: Help with ContinuousQuery issue

[Latha Masters]

Thanks for that idea – we can see that the listener works as expected, but our 
issue is at the beginning, when we run the initial query. At that time, it does 
not retrieve the data that is in the cache and we can see that because when we 
retry, then we get the results.  This also does not happen every time, but 
seems to be a sporadic issue and we don’t understand why this happens.


Latha Masters
Lead Software Engineer
Office: +1 (434) 270-7713
Pronouns:   She, Her, Hers
[Merkle Inc]
[LinkedIn][Twitter] 
  [Facebook]   
  [Youtube]   [Instagram] 




[cid:image761666.jpg@1343CD97.76E3E4E9]



From: Surinder Mehra 
Date: Monday, September 27, 2021 at 2:44 PM
To: user@ignite.apache.org 
Subject: Re: Help with ContinuousQuery issue
Hi,
Did you try to print data inside local listener. As I understand it, initial 
query is only when cintinuous query starts, after that, local listener and 
remote filter (if provided) play role.
On Tue, Sep 28, 2021, 00:02 Latha Masters 
mailto:smast...@merkleinc.com>> wrote:
Hi all,

We’ve run into an issue when using ContinuousQuery and hoping someone can help 
provide some insight.

1   ContinuousQuery query = new ContinuousQuery<>();
2query.setLocalListener(listener);
3query.setInitialQuery(new ScanQuery<>());
4query.setPageSize(updatePageSize);
5query.setTimeInterval(updateTimeInterval);
6cursor = igniteCache.query(query);
7
8for (Entry entry : cursor) {
9cache.put(entry.getKey(), entry.getValue());
10}


With the code snippet above, we have verified that the igniteCache has data but 
occasionally the query on line 6 results in an empty cursor.

Our questions:

Is this expected behavior?

Is there some additional configuration step we are missing which could lead to 
inconsistent results from the initialQuery?

Is there a best practice for retrieving results from the initialQuery such as 
retry? We have noticed that if we retry after it fails to get results the first 
time, it will then succeed on a subsequent try with a one-second wait time 
between retries. We didn’t think this should be necessary so wondered if there 
was something we were doing incorrectly.

Please let me know if any additional information/logs are needed and thank you 
for your help!

Latha
Latha Masters
Lead Software Engineer
Office:
+1 (434) 270-7713
Pronouns:
She, Her, Hers
[Merkle Inc]
[LinkedIn]
[Twitter]
[Facebook]
[Youtube]
[Instagram]



[cid:image507282.jpg@21231AA9.74FA36DC]




Merkle is named a Leader among Customer Database and Engagement Agencies. Get 
the report.

Access Merkle’s award-winning Digital Marketing Report for the latest trends 
and benchmarks in digital 
marketing

This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 
communication may be confidential and may be subject to the attorney-client 
privilege.

Merkle is named a Leader among Customer Database and Engagement Agencies. Get 
the report.

Access Merkle’s award-winning Digital Marketing Report for the latest trends 
and benchmarks in digital 
marketing

This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 
communication may be confidential

Re: What does "First 10 long running cache futures" ?

[John Smith]

Yeah ever since I got this erro for example the REST APi wont return and
the request are slower. But when I connect with visor I can get stats I can
scan the cache etc...

Is it possible that these async futures/threads are not released?

On Tue, 5 Oct 2021 at 04:11, Zhenya Stanilovsky  wrote:

> Hi, this is just a warning shows that something suspicious observed.
> There is no simple reply for your question, in common case all these
> messages are due to cluster (resources or settings)  limitation.
> Check documentation for tuning performance [1]
>
> [1]
> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
>
>
> Hi, using 2.8.1 I understand the message as in my async TRX is taking
> longer but is there a way to prevent it?
>
> When this happened I was pushing about 50, 000 get/puts per second from my
> API.
>
>
>
>
>
>

Re: What does "First 10 long running cache futures" ?

[John Smith]

Actually to be more clear...

http://xx-0001:8080/ignite?cmd=version responds immediately.

http://xx-0001:8080/ignite?cmd=size&cacheName=my-cache doesn't respond
at all.

On Tue, 5 Oct 2021 at 09:59, John Smith  wrote:

> Yeah ever since I got this erro for example the REST APi wont return and
> the request are slower. But when I connect with visor I can get stats I can
> scan the cache etc...
>
> Is it possible that these async futures/threads are not released?
>
> On Tue, 5 Oct 2021 at 04:11, Zhenya Stanilovsky 
> wrote:
>
>> Hi, this is just a warning shows that something suspicious observed.
>> There is no simple reply for your question, in common case all these
>> messages are due to cluster (resources or settings)  limitation.
>> Check documentation for tuning performance [1]
>>
>> [1]
>> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
>>
>>
>> Hi, using 2.8.1 I understand the message as in my async TRX is taking
>> longer but is there a way to prevent it?
>>
>> When this happened I was pushing about 50, 000 get/puts per second from
>> my API.
>>
>>
>>
>>
>>
>>
>

Re: What does "First 10 long running cache futures" ?

[John Smith]

And I see this...

[14:04:15,150][WARNING][exchange-worker-#43%raange%][GridDhtPartitionsExchangeFuture]
Unable to await partitions release latch within timeout. For more details
please check coordinator node logs [crdNode=TcpDiscoveryNode
[id=36ad785d-e344-43bb-b685-e79557572b54,
consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList
[127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, /
127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1633370987399, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=false]] [latch=ClientLatch
[coordinator=TcpDiscoveryNode [id=36ad785d-e344-43bb-b685-e79557572b54,
consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList
[127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, /
127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1633370987399, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=false], ackSent=true,
super=CompletableLatch [id=CompletableLatchUid [id=exchange,
topVer=AffinityTopologyVersion [topVer=103, minorTopVer=0]

On Tue, 5 Oct 2021 at 10:02, John Smith  wrote:

> Actually to be more clear...
>
> http://xx-0001:8080/ignite?cmd=version responds immediately.
>
> http://xx-0001:8080/ignite?cmd=size&cacheName=my-cache doesn't
> respond at all.
>
> On Tue, 5 Oct 2021 at 09:59, John Smith  wrote:
>
>> Yeah ever since I got this erro for example the REST APi wont return and
>> the request are slower. But when I connect with visor I can get stats I can
>> scan the cache etc...
>>
>> Is it possible that these async futures/threads are not released?
>>
>> On Tue, 5 Oct 2021 at 04:11, Zhenya Stanilovsky 
>> wrote:
>>
>>> Hi, this is just a warning shows that something suspicious observed.
>>> There is no simple reply for your question, in common case all these
>>> messages are due to cluster (resources or settings)  limitation.
>>> Check documentation for tuning performance [1]
>>>
>>> [1]
>>> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
>>>
>>>
>>> Hi, using 2.8.1 I understand the message as in my async TRX is taking
>>> longer but is there a way to prevent it?
>>>
>>> When this happened I was pushing about 50, 000 get/puts per second from
>>> my API.
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: What does "First 10 long running cache futures" ?

[John Smith]

And finally this on the coordinator node

[14:07:41,282][WARNING][exchange-worker-#42%xx%][GridDhtPartitionsExchangeFuture]
Unable to await partitions release latch within timeout. Some nodes have
not sent acknowledgement for latch completion. It's possible due to
unfinishined atomic updates, transactions or not released explicit locks on
that nodes. Please check logs for errors on nodes with ids reported in
latch `pendingAcks` collection [latch=ServerLatch [permits=1,
pendingAcks=HashSet [36edbfd5-4feb-417e-b965-bdc34a0a6f4f],
super=CompletableLatch [id=CompletableLatchUid [id=exchange,
topVer=AffinityTopologyVersion [topVer=103, minorTopVer=0]

On Tue, 5 Oct 2021 at 10:07, John Smith  wrote:

> And I see this...
>
> [14:04:15,150][WARNING][exchange-worker-#43%raange%][GridDhtPartitionsExchangeFuture]
> Unable to await partitions release latch within timeout. For more details
> please check coordinator node logs [crdNode=TcpDiscoveryNode
> [id=36ad785d-e344-43bb-b685-e79557572b54,
> consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList
> [127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, /
> 127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1633370987399, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=false]] [latch=ClientLatch
> [coordinator=TcpDiscoveryNode [id=36ad785d-e344-43bb-b685-e79557572b54,
> consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList
> [127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, /
> 127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1633370987399, loc=false,
> ver=2.8.1#20200521-sha1:86422096, isClient=false], ackSent=true,
> super=CompletableLatch [id=CompletableLatchUid [id=exchange,
> topVer=AffinityTopologyVersion [topVer=103, minorTopVer=0]
>
> On Tue, 5 Oct 2021 at 10:02, John Smith  wrote:
>
>> Actually to be more clear...
>>
>> http://xx-0001:8080/ignite?cmd=version responds immediately.
>>
>> http://xx-0001:8080/ignite?cmd=size&cacheName=my-cache doesn't
>> respond at all.
>>
>> On Tue, 5 Oct 2021 at 09:59, John Smith  wrote:
>>
>>> Yeah ever since I got this erro for example the REST APi wont return and
>>> the request are slower. But when I connect with visor I can get stats I can
>>> scan the cache etc...
>>>
>>> Is it possible that these async futures/threads are not released?
>>>
>>> On Tue, 5 Oct 2021 at 04:11, Zhenya Stanilovsky 
>>> wrote:
>>>
 Hi, this is just a warning shows that something suspicious observed.
 There is no simple reply for your question, in common case all these
 messages are due to cluster (resources or settings)  limitation.
 Check documentation for tuning performance [1]

 [1]
 https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips


 Hi, using 2.8.1 I understand the message as in my async TRX is taking
 longer but is there a way to prevent it?

 When this happened I was pushing about 50, 000 get/puts per second from
 my API.






>>>

Re[2]: What does "First 10 long running cache futures" ?

[Zhenya Stanilovsky]

Ok, seems something goes wrong on node with 
id=36edbfd5-4feb-417e-b965-bdc34a0a6f4f If you still have a problem, can u send 
here or directly by me these logs ?


 
>And finally this on the coordinator node
>
>[14:07:41,282][WARNING][exchange-worker-#42%xx%][GridDhtPartitionsExchangeFuture]
> Unable to await partitions release latch within timeout. Some nodes have not 
>sent acknowledgement for latch completion. It's possible due to unfinishined 
>atomic updates, transactions or not released explicit locks on that nodes. 
>Please check logs for errors on nodes with ids reported in latch `pendingAcks` 
>collection [latch=ServerLatch [permits=1, pendingAcks=HashSet 
>[36edbfd5-4feb-417e-b965-bdc34a0a6f4f], super=CompletableLatch 
>[id=CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion 
>[topVer=103, minorTopVer=0]  
>On Tue, 5 Oct 2021 at 10:07, John Smith < java.dev@gmail.com > wrote:
>>And I see this...
>>
>>[14:04:15,150][WARNING][exchange-worker-#43%raange%][GridDhtPartitionsExchangeFuture]
>> Unable to await partitions release latch within timeout. For more details 
>>please check coordinator node logs [crdNode=TcpDiscoveryNode 
>>[id=36ad785d-e344-43bb-b685-e79557572b54, 
>>consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList 
>>[127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, / 
>>127.0.0.1:47500 ], discPort=47500, order=1, intOrder=1, 
>>lastExchangeTime=1633370987399, loc=false, ver=2.8.1#20200521-sha1:86422096, 
>>isClient=false]] [latch=ClientLatch [coordinator=TcpDiscoveryNode 
>>[id=36ad785d-e344-43bb-b685-e79557572b54, 
>>consistentId=8172e45d-3ff8-4fe4-aeda-e7d30c1e11e2, addrs=ArrayList 
>>[127.0.0.1, xx.65], sockAddrs=HashSet [xx-0002/xx.65:47500, / 
>>127.0.0.1:47500 ], discPort=47500, order=1, intOrder=1, 
>>lastExchangeTime=1633370987399, loc=false, ver=2.8.1#20200521-sha1:86422096, 
>>isClient=false], ackSent=true, super=CompletableLatch [id=CompletableLatchUid 
>>[id=exchange, topVer=AffinityTopologyVersion [topVer=103, minorTopVer=0]  
>>On Tue, 5 Oct 2021 at 10:02, John Smith < java.dev@gmail.com > wrote:
>>>Actually to be more clear...
>>>
>>>http://xx-0001:8080/ignite?cmd=version responds immediately.
>>>
>>>http://xx-0001:8080/ignite?cmd=size&cacheName=my-cache doesn't respond 
>>>at all.  
>>>On Tue, 5 Oct 2021 at 09:59, John Smith < java.dev@gmail.com > wrote:
Yeah ever since I got this erro for example the REST APi wont return and 
the request are slower. But when I connect with visor I can get stats I can 
scan the cache etc...

Is it possible that these async futures/threads are not released?  
On Tue, 5 Oct 2021 at 04:11, Zhenya Stanilovsky < arzamas...@mail.ru > 
wrote:
>Hi, this is just a warning shows that something suspicious observed.
>There is no simple reply for your question, in common case all these 
>messages are due to cluster (resources or settings)  limitation.
>Check documentation for tuning performance [1]
> 
>[1]  
>https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
>   
>>Hi, using 2.8.1 I understand the message as in my async TRX is taking 
>>longer but is there a way to prevent it?
>> 
>>When this happened I was pushing about 50, 000 get/puts per second from 
>>my API. 
> 
> 
> 
>