RE: Multiple ignite nodes crashed at the same time due to "Maximum number of retries 100000 reached for Put operation" error

[Lo, Marcus]

Hi Ilya,

Unfortunately I have to rebuild the database and did not keep the persistence 
files. But you are right that the failed nodes fails every time on restart.

I will see if I can reproduce the issue – in the meantime do you have any 
suggestions on what I should check?

Regards,
Marcus

From: [gmail.com] Ilya Kasnacheev 
Sent: Thursday, May 20, 2021 11:11 PM
To: user@ignite.apache.org
Subject: Re: Multiple ignite nodes crashed at the same time due to "Maximum 
number of retries 10 reached for Put operation" error

Hello!

This looks like a PDS corruption to me. Can you by chance share persistence 
files from problematic node? I am assuming that it fails every time on restart?

Regards,
--
Ilya Kasnacheev


чт, 20 мая 2021 г. в 12:52, Lo, Marcus 
mailto:marcus...@citi.com>>:
Hi,

We have a 4 node ignite cluster setup. After running the cluster for 1 day, we 
encounter the following error almost at the same time at node #2, #3, and #4:

Critical system error detected. Will be handled accordingly to configured 
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class 
o.a.i.IgniteCheckedException: Maximum number of retries 1000 reached for Put 
operation (the tree may be corrupted). Increase IGNITE_BPLUS_TREE_LOCK_RETRIES 
system property if you regularly see this message (current value is 1000).]] 
org.apache.ignite.IgniteCheckedException: Maximum number of retries 1000 
reached for Put operation (the tree may be corrupted). Increase 
IGNITE_BPLUS_TREE_LOCK_RETRIES system property if you regularly see this 
message (current value is 1000). at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Get.checkLockRetry(BPlusTree.java:3109)
 [ignite-core-2.10.0.jar:2.10.0] at 
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.checkLockRetry(BPlusTree.java:3906)
 [ignite-core-2.10.0.jar:2.10.0]

Tried increasing IGNITE_BPLUS_TREE_LOCK_RETRIES to 100,000 and restarted the 
nodes, but it didn’t help and the node went into the same error straight away.

Can you please shed some lights on how to resolve the issue? Thanks.

I also attach the logs for your reference:
ignite-node-[1,2,3,4].log: the full log files for all nodes
ignite-restart.log: the log for node 2 when it crashed

Regards,
Marcus

Re: BinaryInvalidTypeException: Requesting mapping from grid failed for error reading cache

[Joan Pujol]

A lot of thanks for your response Alex.

And if we are not using persistence, any idea of what can be the
origin of the problem or where to look?
About the upgrade, it's something we have pending for some time, but
sure, I will try also.

Cheers,

On Thu, 20 May 2021 at 14:23, akorensh  wrote:
>
> Hi,
>   The error is coming from here:
> https://github.com/apache/ignite/blob/9c2ba403fae5079201e9df6bc7b7eea5bacdf4c0/modules/core/src/main/java/org/apache/ignite/internal/MarshallerContextImpl.java#L386
>
>   This means that the model class metadata is missing/corrupt.
>It is located in /binary_meta/
>and in /marshaller
>
> To fix:
>If you have persistence enabled, destroy the cache in question (as well
> as all caches using this type),
>remove the given types from the above dirs and retry.
>Make sure the model class is in the class path.
>
>
>To determine the types in question:  grep "fully qualified type name"
> 
>  remove -1293425677 file in the above mentioned dirs. Take a look at
>  marshaller/-1293425677.classname0 to see what type this typeid
> represents. Make sure to remove it
>  and all caches using this type.
>
>
>   I encourage you to use the latest ignite version:
> https://ignite.apache.org/download.cgi#binaries
>
> Thanks, Alex
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/



-- 
Joan Jesús Pujol Espinar
http://www.joanpujol.cat
https://play.google.com/store/apps/developer?id=lujop&hl=ca

Re: System.InvalidOperationException: 'No coercion operator is defined between types 'Apache.Ignite.Core.Impl.Binary.BinaryObjectBuilder' and 'System.DateTime'

[Pavel Tupitsyn]

Josh, can you please provide the code?
Ideally, a simple reproducer I could run?

On Fri, May 21, 2021 at 1:47 AM Josh Katz <
josh.katz.contrac...@dodgeandcox.com> wrote:

> Using .NET UnitTest to connect to the cluster and persistence enabled.
>
> Put works without errors. When calling Get we get the following exception:
>
> System.InvalidOperationException: 'No coercion operator is defined between
> types 'Apache.Ignite.Core.Impl.Binary.BinaryObjectBuilder' and
> 'System.DateTime'
>
>
>
> We are using System.Runtime.Serialization for the DateTime property with
> DataMemberAttribute.
>
>
>
> Thanks,
>
>
>
> *Josh Katz*
>
> *Dodge & Cox*
>
> 555 California Street | 40th floor | San Francisco, CA 94104
>
> 415-262-7520
>
>
>
> josh.katz.contrac...@dodgeandcox.com
>
> www.dodgeandcox.com
>
>
>
>
> --
> Please follow the hyperlink to important disclosures.
> https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html
>
>

Re: Unsubscribe

[pinak sawhney]

Unsubscribe

On Thu, 20 May 2021, 7:27 pm Manan Joshi,  wrote:

> Unsubscribe
>
> --
>
> *Manan Joshi | Engineering Manager*
> *Izenda, Inc. *| manan,jo...@izenda.com  |
> 404-287-1941
>
>

Unsubscribe

[Manan Joshi]

 Unsubscribe

-- 

*Manan Joshi | Engineering Manager*
*Izenda, Inc. *| manan,jo...@izenda.com  |
404-287-1941

System.InvalidOperationException: 'No coercion operator is defined between types 'Apache.Ignite.Core.Impl.Binary.BinaryObjectBuilder' and 'System.DateTime'

[Josh Katz]

Using .NET UnitTest to connect to the cluster and persistence enabled.
Put works without errors. When calling Get we get the following exception:
System.InvalidOperationException: 'No coercion operator is defined between 
types 'Apache.Ignite.Core.Impl.Binary.BinaryObjectBuilder' and 'System.DateTime'

We are using System.Runtime.Serialization for the DateTime property with 
DataMemberAttribute.

Thanks,

Josh Katz
Dodge & Cox
555 California Street | 40th floor | San Francisco, CA 94104
415-262-7520

josh.katz.contrac...@dodgeandcox.com
www.dodgeandcox.com


--
Please follow the hyperlink to important 
disclosures.https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html

Unsubscribe

[Nathan Neely]

Thanks,

Nathan Neely
Engineer IV, Software 5x Core
Benefitfocus for life™
Mobile: 864.395.1532
Office: 864.335.6109



Benefitfocus® (NASDAQ: BNFT) unifies the entire benefits industry on a single 
technology platform to protect consumers for life. Our powerful cloud-based 
software, data-driven insights and thoughtfully-designed services enable 
employers, insurance brokers, health plans and suppliers to simplify the 
complexity of benefits administration and deliver health, wealth, property and 
lifestyle products through a world-class benefits experience. Our mission is 
simple: to improve lives with benefits. Learn more at 
www.benefitfocus.com, 
LinkedIn and 
Twitter.

Re: Execute Ignite Callable Jobs with set priorities

[Ilya Kasnacheev]

Hello!

It just sounds that you want to prioritize some other resource rather than
Ignite compute capacity, if you want cluster-wide priorities.

This means that Ignite prioritizing is a poor fit for you and you may need
to roll out your own, perhaps based on IgniteQueue.

Regards,
-- 
Ilya Kasnacheev


вс, 16 мая 2021 г. в 13:20, Krish :

> Thanks Stephen and Ilya,
>
> As mentioned in job scheduling documentation, collisionAPI will take care
> of
> job scheduling when jobs arrive at destination node. Lets say I use
> PriorityQueueCollisionSpi and send three jobs with priorities 5, 7 and 10
> to
> one node, then that node will execute job with priority 10 first then job
> with priority 7 and then 5.
>
> However, My use case is different than this. Going with above example, in
> my
> use case, ignite client will send three jobs to three different nodes and I
> would still want these jobs to be executed based on their priorities.
> Basically, no matter how these jobs are distributed across the cluster they
> should be executed based on priority. Can this be achieved using
> CollisionAPI?
>
> Many Thanks,
> K
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Data replication from kafka-topic to ignite cluster

[Ilya Kasnacheev]

Hello!

Maybe you should ask on Kafka list about the specifics of its class
loading. "org.apache.kafka.connect.runtime.isolation" suggests that there
may be some limitations.

Regards,
-- 
Ilya Kasnacheev


пн, 17 мая 2021 г. в 13:32, shubhamshirur :

> Thank you sir for replying. I have added ignite-core.jar and I have
> mentioned
> its path in plugin.path. This particular jar is loaded in while connector
> instantiation I have checked it in the log. Can you please enlighten me on
> this ?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Migration from Apache ignite 2.7.0 to 2.10.0

[Ilya Kasnacheev]

Hello!

2.10.0 is definitely more stable than 2.7.0.

Regards,
-- 
Ilya Kasnacheev


пн, 17 мая 2021 г. в 08:00, BEELA GAYATRI :

> Dear Team,
>
>We are planning to migrate Apache 2.7.0 to 2.10.0. We have seen few
> warnings in 2.10.0 release notes. Can we move to 2.10.0 in production?
> Please suggest.
>
> Thanks and Regards,
>
> Beela Gayatri.
>
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>

Re: Running sql query on partitioned cache

[Ilya Kasnacheev]

Hello!

1. Hard to say what happens here. Do you have a runnable reproducer for
this behavior? Can you share?

2.
https://ignite.apache.org/releases/latest/cppdoc/classignite_1_1cache_1_1Cache.html#a03574797da901a76180aad88476ef8ce
- Cache.Size()?

Regards,
-- 
Ilya Kasnacheev


чт, 20 мая 2021 г. в 09:39, rakshita04 :

> Hi Team,
>
> We are running sql query on our cache(which is configured in "Partitioned"
> mode with backup =1)
> We have two nodes connected over network to each other.
> Our application is C++ based and running on ARM environment(linux)
> We are facing 2 issues now-
> 1. when we are running below query on our code while in background we are
> adding entries in DB-
> const SqlFieldsQuery countQuery(
> "select count(reqType) "
> "from DBStorage "
> "where reqType=0"); DataBaseConfig.xml
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2857/DataBaseConfig.xml>
>
> The count is returned is increasing with proper entry till 10,000 entries
> but after that it is getting reset again and starting again with 0.
> Is there anything else, we need to do for running sql query in partitioned
> mode for correct results?
>
> 2. What C++ API we need to use to get exact size of cache on partitioned
> mode? So cache.localSize() is returning half the size of actual size (may
> be
> due to partitioned mode)?
>
> I am attaching the xml for your reference.
>
> regards,
> Rakshita Chaudhary
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Partitioned cache behaviour when saving the same key on different nodes

[Ilya Kasnacheev]

Hello!

For every key there is a primary node. Both puts will happen on that node
in some order.

Please see
https://ignite.apache.org/docs/latest/data-modeling/data-partitioning

Regards,
-- 
Ilya Kasnacheev


чт, 20 мая 2021 г. в 18:10, r_s :

> Hi all,
>
> I am trying to understand what happens in the following situation:
> I have a partitioned cache that is accessed via the IgniteRepository from
> org.apache.ignite.springdata22.repository. Now I realized, that it might
> happen that two nodes will call the save method on the repository with the
> same key but different value objects. We can expect that the key is not
> present in the cache yet. What is the expected behaviour of the cache in
> this situation? Will the nodes hold the conflicting data on different
> partitions until they are somehow synchronized?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Multiple ignite nodes crashed at the same time due to "Maximum number of retries 100000 reached for Put operation" error

[Ilya Kasnacheev]

Hello!

This looks like a PDS corruption to me. Can you by chance share persistence
files from problematic node? I am assuming that it fails every time on
restart?

Regards,
-- 
Ilya Kasnacheev


чт, 20 мая 2021 г. в 12:52, Lo, Marcus :

> Hi,
>
>
>
> We have a 4 node ignite cluster setup. After running the cluster for 1
> day, we encounter the following error almost at the same time at node #2,
> #3, and #4:
>
>
>
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [
> SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.IgniteCheckedException: Maximum number of retries 1000 reached for
> Put operation (the tree may be corrupted). Increase
> IGNITE_BPLUS_TREE_LOCK_RETRIES system property if you regularly see this
> message (current value is 1000).]]
> org.apache.ignite.IgniteCheckedException: Maximum number of retries 1000
> reached for Put operation (the tree may be corrupted). Increase
> IGNITE_BPLUS_TREE_LOCK_RETRIES system property if you regularly see this
> message (current value is 1000). at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Get.checkLockRetry
> (BPlusTree.java:3109) [ignite-core-2.10.0.jar:2.10.0] at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Put.checkLockRetry
> (BPlusTree.java:3906) [ignite-core-2.10.0.jar:2.10.0]
>
>
>
> Tried increasing IGNITE_BPLUS_TREE_LOCK_RETRIES to 100,000 and restarted
> the nodes, but it didn’t help and the node went into the same error
> straight away.
>
>
>
> Can you please shed some lights on how to resolve the issue? Thanks.
>
>
>
> I also attach the logs for your reference:
>
> ignite-node-[1,2,3,4].log: the full log files for all nodes
>
> ignite-restart.log: the log for node 2 when it crashed
>
>
>
> Regards,
>
> Marcus
>
>
>

Partitioned cache behaviour when saving the same key on different nodes

[r_s]

Hi all,

I am trying to understand what happens in the following situation:
I have a partitioned cache that is accessed via the IgniteRepository from
org.apache.ignite.springdata22.repository. Now I realized, that it might
happen that two nodes will call the save method on the repository with the
same key but different value objects. We can expect that the key is not
present in the cache yet. What is the expected behaviour of the cache in
this situation? Will the nodes hold the conflicting data on different
partitions until they are somehow synchronized?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Exception on CacheEntryProcessor invoke (2.10.0)

[Ilya Kasnacheev]

Hello!

Can you please share a runnable reproducer project which works on previous
version but fails on 2.10?

Regards,
-- 
Ilya Kasnacheev


чт, 20 мая 2021 г. в 17:08, ihalilaltun :

> Hi igniters,
>
> recenlty we have upgraded from 2.7.6 to 2.10.0 and some of
> cacheentryprocessors started to throw following errors on cache.invoke(...)
> calls.
>
> Caused by: java.lang.ClassNotFoundException:
> com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_261]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_261]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> ~[?:1.8.0_261]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_261]
> at java.lang.Class.forName0(Native Method) ~[?:1.8.0_261]
> at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_261]
>
>
> on 2.7.6 version we also get these error from time to time, but when
> application that uses these cacaheentryprocessors is restarted errors does
> not occur. but on 2.10.0 version this solution did not solve our problem.
>
> currently we have 23 different cacheentryprocessors runs on the system.
> after many different test scenarios and checks we found a pattern on above
> error case. only 4 out of 23 cacheentryprocessor keeps getting this error,
> *3 of these are invoked by ExecutorServices*;
>
> sample usage is somithing like the following;
>
> private ExecutorService executorService = Executors.newCachedThreadPool();
> 
> executorService.submit(() -> {
> ...
> igniteCache.withKeepBinary()
> .invoke(record.getKey(), new
> RockScoreUpdateProcessor(),
> "arg1", "arg2", "arg3");
> });
>
> *one cacheentryprocessor is invoked by XSync
> (https://github.com/antkorwin/xsync) *
>
>
> so what we see here, somehow when a cacheentryprocessor is invoked from a
> runnable context classnotfoundexception is thrown.
>
> *peerclassloading* property is set to true, *deploymentmode* is set to
> SHARED and *persistenceEnabled* is set to true.
>
>
> can this be a bug either known or unknown?
>
>
> currently this is a blocker issue for us to upgrade on production
> environment. any help is appriciated.
>
> Thanks.
>
>
>
>
> -
> İbrahim Halil Altun
> Senior Software Engineer @ Segmentify
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Exception on CacheEntryProcessor invoke (2.10.0)

[ihalilaltun]

Hi igniters,

recenlty we have upgraded from 2.7.6 to 2.10.0 and some of
cacheentryprocessors started to throw following errors on cache.invoke(...)
calls.

Caused by: java.lang.ClassNotFoundException:
com.segmentify.lotr.frodo.cacheentryprocessor.RockScoreUpdateProcessor
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_261]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_261]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
~[?:1.8.0_261]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_261]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_261]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_261]


on 2.7.6 version we also get these error from time to time, but when
application that uses these cacaheentryprocessors is restarted errors does
not occur. but on 2.10.0 version this solution did not solve our problem.

currently we have 23 different cacheentryprocessors runs on the system.
after many different test scenarios and checks we found a pattern on above
error case. only 4 out of 23 cacheentryprocessor keeps getting this error,
*3 of these are invoked by ExecutorServices*;

sample usage is somithing like the following;

private ExecutorService executorService = Executors.newCachedThreadPool();

executorService.submit(() -> { 
...
igniteCache.withKeepBinary()
.invoke(record.getKey(), new
RockScoreUpdateProcessor(),
"arg1", "arg2", "arg3");
});

*one cacheentryprocessor is invoked by XSync
(https://github.com/antkorwin/xsync) *


so what we see here, somehow when a cacheentryprocessor is invoked from a
runnable context classnotfoundexception is thrown.

*peerclassloading* property is set to true, *deploymentmode* is set to
SHARED and *persistenceEnabled* is set to true.


can this be a bug either known or unknown?


currently this is a blocker issue for us to upgrade on production
environment. any help is appriciated.

Thanks.




-
İbrahim Halil Altun
Senior Software Engineer @ Segmentify
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: BinaryInvalidTypeException: Requesting mapping from grid failed for error reading cache

[akorensh]

Hi,
  The error is coming from here:
https://github.com/apache/ignite/blob/9c2ba403fae5079201e9df6bc7b7eea5bacdf4c0/modules/core/src/main/java/org/apache/ignite/internal/MarshallerContextImpl.java#L386
   
  This means that the model class metadata is missing/corrupt.
   It is located in /binary_meta/
   and in /marshaller

To fix:
   If you have persistence enabled, destroy the cache in question (as well
as all caches using this type),
   remove the given types from the above dirs and retry.
   Make sure the model class is in the class path.

 
   To determine the types in question:  grep "fully qualified type name"

 remove -1293425677 file in the above mentioned dirs. Take a look at
 marshaller/-1293425677.classname0 to see what type this typeid
represents. Make sure to remove it
 and all caches using this type.


  I encourage you to use the latest ignite version:
https://ignite.apache.org/download.cgi#binaries

Thanks, Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Features in ignite 2.11

[akurbanov]

Hi,

That's a better discussion for a dev list, it has an active thread on 2.11
release:
http://apache-ignite-developers.2346864.n4.nabble.com/Apache-Ignite-2-11-td52605.html

However, it is always a good thing to search up JIRA for resolved tickets
with fix version 2.11, the list is pretty big, so I would suggest to take a
look into it and see if there is anything that might have your attention:
https://issues.apache.org/jira/browse/IGNITE-14152?jql=project%20%3D%20IGNITE%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/