Hi, In one of our Ignite cluster's node, we have got B+Tree exception due to which the node got stopped/disconnected from the cluster. Ignite Version : 2.11.0.
Is this a product bug? If Yes then what is the workaround. Also has this been fixed in any version? Logs: [2022-04-14T14:50:17,989][INFO ][wal-file-archiver%EDIFMEDIA_PROD-#235%EDIFMEDIA_PROD%][FileWriteAheadLogManager] Copied file [src=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000000000.wal, dst=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000211270.wal] [2022-04-14T14:50:21,441][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550, val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123], val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14 14:50:21.437, partyId=null, subscriptionIdList=22414145021425514379098800228113357] ][ ]]]] org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550, val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123], val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14 14:50:21.437, partyId=null, subscriptionIdList=22414145021425514379098800228113357] ][ ]] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.corruptedTreeException(InlineIndexTree.java:581) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2464) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2411) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.putx(InlineIndexImpl.java:269) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:251) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndex(IndexProcessor.java:419) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndexes(IndexProcessor.java:291) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.IndexProcessor.store(IndexProcessor.java:142) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2512) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2697) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1773) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1748) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2794) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2342) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3338) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569) [ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) [ignite-core-2.11.0.jar:2.11.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311] Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException: java.lang.IllegalStateException: Item not found: 20 at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:345) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.createIndexRow(InlineIndexTree.java:377) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:127) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:37) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:420) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:70) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:5423) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:326) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:70) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5410) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5330) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1100(BPlusTree.java:100) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:310) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5921) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:290) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5907) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:174) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:397) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:6108) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2796) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808) ~[ignite-core-2.11.0.jar:2.11.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2431) [ignite-core-2.11.0.jar:2.11.0] 663 [2022-04-14T14:50:23,192][WARN ][sys-stripe-17-#18%EDIFMEDIA_PROD%][DiagnosticProcessor] CorruptedTreeException has occurred. To diagnose it, make a backup of the following directories: [/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142, /datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142]. Then, run the following command: java -cp <classpath> org.apache.ignite.development.utils.IgniteWalConverter walDir=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142 walArchiveDir=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142 pages=/datastore1/metadataWorkDir/diagnostic/corruptedPages_2022-04-14_14-50-23_184.txt [2022-04-14T14:50:23,192][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550, val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache, indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key: com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213, hash=2147463230, serviceType=Z0114, partyId=p123], val: com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655, serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14 14:50:21.437, partyId=null, subscriptionIdList=22414145021425514379098800228113357] ][ ]]]] Thanks and Regards, "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."