Ibrahim,
I've checked logs and found the following issue:
[2019-09-27T15:00:06,164][ERROR][sys-stripe-32-#33][atomic] Received
message without registered handler (will ignore)
[msg=GridDhtAtomicDeferredUpdateResponse [futIds=GridLongList [idx=1,
arr=[6389728]]], node=e39bd72e-acee-48a7-ad45-2019dfff9df4,
locTopVer=AffinityTopologyVersion [topVer=92, minorTopVer=1], ...
This response was needed to complete (finish) AtomicUpdateFuture:
[2019-09-27T15:00:36,287][WARN ][exchange-worker-#219][diagnostic] >>>
GridDhtAtomicSingleUpdateFuture [allUpdated=true,
super=GridDhtAtomicAbstractUpdateFuture [futId=6389728, resCnt=0,
addedReader=false, dhtRes={e39bd72e-acee-48a7-ad45-2019dfff9df4=[res=false,
size=1, nearSize=0]}]]
During exchange, all nodes wait for atomic updates and transaction
completion and then send an acknowledgment to the coordinator to continue
processing exchange.
Because atomic update on that node was not finished, the node didn't send
the acknowledgement to the coordinator and that's why you have seen
messages like:
[2019-09-27T15:00:17,727][WARN ][exchange-worker-#219][
GridDhtPartitionsExchangeFuture] Unable to await partitions release latch
within timeout: ServerLatch [permits=1, pendingAcks=[
*3561ac09-6752-4e2e-8279-d975c268d045*], super=CompletableLatch
[id=exchange, topVer=AffinityTopologyVersion [topVer=92, minorTopVer=2]]]
The handler to complete AtomicUpdateFuture was not found due to the
concurrency issue in 2.7.6 codebase. There is a map that contains handlers
for cache messages:
org/apache/ignite/internal/processors/cache/GridCacheIoManager.java:1575
In 2.7.6 it's just HashMap with volatile read/write publishing. However,
because of improper synchronization with adding and getting a handler in
rare cases, it can lead to false-positive missing a handler for a message
that you may see in logs.
This issue was fixed at https://issues.apache.org/jira/browse/IGNITE-8006 which
will be in 2.8 release.
However, if it's critical, you can make a hotfix by yourself:
Checkout ignite-2.7.6 branch from https://github.com/apache/ignite
Change HashMap declaration to ConcurrentHashMap here:
org/apache/ignite/internal/processors/cache/GridCacheIoManager.java:1575
Rebuild ignite-core module and deploy new ignite-core-jar on your server
nodes.
This hotfix will work for your case.
Another option is you can use the last version of GridGain Community
Edition instead of Apache Ignite which is fully compatible with Ignite.
Regarding message:
[sys-#337823][GridDhtPartitionsExchangeFuture] Partition states validation
has failed for group: acc_1306acd07be78000_userPriceDrop. Partitions
cachesizes are inconsistent for Part 129
I see that you create caches with ExpiryPolicy. If you use expiry policies
you can have different partition sizes on primary-backup nodes, because
expiring is not synchronized and performed independently on different nodes.
So it's OK to see such warnings. They are false-positive. Such warning
messages will not be printed if a cache has an expiry policy set. That was
fixed in https://issues.apache.org/jira/browse/IGNITE-12206
пт, 11 окт. 2019 г. в 14:40, ihalilaltun <ibrahim.al...@segmentify.com>:
> Hi Pavel,
>
> Here is the logs from node with
> localId:3561ac09-6752-4e2e-8279-d975c268d045
> ignite-2019-10-06.gz
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t2515/ignite-2019-10-06.gz>
>
>
> cache creation is done with java code on our side, we use getOrCreateCache
> method, here is the piece of code how we configure and create caches;
>
> ...
> ignite.getOrCreateCache(getCommonCacheConfigurationForAccount(accountId,
> initCacheType));
>
> private <K, V> CacheConfiguration<K, V>
> getCommonCacheConfigurationForAccount(String accountId, IgniteCacheType
> cacheType) {
> CacheConfiguration<K, V> cacheConfiguration = new
> CacheConfiguration<>();
>
>
> cacheConfiguration.setName(accountId.concat(cacheType.getCacheNameSuffix()));
> if (cacheType.isSqlTable()) {
> cacheConfiguration.setIndexedTypes(cacheType.getKeyClass(),
> cacheType.getValueClass());
> cacheConfiguration.setSqlSchema(accountId);
> cacheConfiguration.setSqlEscapeAll(true);
> }
> cacheConfiguration.setEventsDisabled(true);
> cacheConfiguration.setStoreKeepBinary(true);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfiguration.setBackups(1);
> if (!cacheType.getCacheGroupName().isEmpty()) {
> cacheConfiguration.setGroupName(cacheType.getCacheGroupName());
> }
> if (cacheType.getExpiryDurationInDays().getDurationAmount() > 0) {
>
>
> cacheConfiguration.setExpiryPolicyFactory(TouchedExpiryPolicy.factoryOf(cacheType.getExpiryDurationInDays()));
> }
> return cacheConfiguration;
> }
>
>
>
> -----
> İbrahim Halil Altun
> Senior Software Engineer @ Segmentify
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
