Hi,
In one of our Ignite cluster's node, we have got B+Tree exception due to which
the node got stopped/disconnected from the cluster.
Ignite Version : 2.11.0.
Is this a product bug? If Yes then what is the workaround. Also has this been
fixed in any version?
Logs:
[2022-04-14T14:50:17,989][INFO
][wal-file-archiver%EDIFMEDIA_PROD-#235%EDIFMEDIA_PROD%][FileWriteAheadLogManager]
Copied file
[src=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000000000.wal,
dst=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142/0000000000211270.wal]
[2022-04-14T14:50:21,441][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][] Critical
system error detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550,
val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache,
indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key:
com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213,
hash=2147463230, serviceType=Z0114, partyId=p123], val:
com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655,
serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14
14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]]]
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550,
val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache,
indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key:
com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213,
hash=2147463230, serviceType=Z0114, partyId=p123], val:
com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655,
serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14
14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.corruptedTreeException(InlineIndexTree.java:581)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2464)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2411)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.putx(InlineIndexImpl.java:269)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexImpl.onUpdate(InlineIndexImpl.java:251)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndex(IndexProcessor.java:419)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.IndexProcessor.updateIndexes(IndexProcessor.java:291)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.IndexProcessor.store(IndexProcessor.java:142)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:2512)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:2697)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1773)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1748)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2794)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2342)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3338)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomicCache.java:143)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:322)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$7.apply(GridDhtAtomicCache.java:317)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1151)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:393)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:319)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:110)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:309)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1908)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1529)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:242)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1422)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:569)
[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.11.0.jar:2.11.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_311]
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
java.lang.IllegalStateException: Item not found: 20
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:345)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.createIndexRow(InlineIndexTree.java:377)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:127)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.io.AbstractInlineLeafIO.getLookupRow(AbstractInlineLeafIO.java:37)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:420)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.getRow(InlineIndexTree.java:70)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:5423)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:326)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.cache.query.index.sorted.inline.InlineIndexTree.compare(InlineIndexTree.java:70)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5410)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5330)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1100(BPlusTree.java:100)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:310)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5921)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:290)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5907)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:174)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:397)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:6108)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2796)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2808)
~[ignite-core-2.11.0.jar:2.11.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2431)
[ignite-core-2.11.0.jar:2.11.0]
663
[2022-04-14T14:50:23,192][WARN
][sys-stripe-17-#18%EDIFMEDIA_PROD%][DiagnosticProcessor]
CorruptedTreeException has occurred. To diagnose it, make a backup of the
following directories:
[/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142,
/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142]. Then, run the
following command: java -cp <classpath>
org.apache.ignite.development.utils.IgniteWalConverter
walDir=/datastore2/wal/node00-060224e3-53ac-4555-876e-166a7480c142
walArchiveDir=/datastore2/archive/node00-060224e3-53ac-4555-876e-166a7480c142
pages=/datastore1/metadataWorkDir/diagnostic/corruptedPages_2022-04-14_14-50-23_184.txt
[2022-04-14T14:50:23,192][ERROR][sys-stripe-17-#18%EDIFMEDIA_PROD%][] JVM will
be halted immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class
o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is
corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-291739550,
val2=844420635435161]], cacheId=-291739550, cacheName=MapCustOTTCache,
indexName=_key_PK, msg=Runtime failure on row: Row@5eaffcf8[ key:
com.media.digitalapi.edif.model.MapCustOTTKey [idHash=1519412213,
hash=2147463230, serviceType=Z0114, partyId=p123], val:
com.media.digitalapi.edif.model.MapCustOTT [idHash=529725123, hash=-1059328655,
serviceType=null, updatedBy=TB00001KGQEC, updateddatetime=2022-04-14
14:50:21.437, partyId=null,
subscriptionIdList=22414145021425514379098800228113357] ][ ]]]]
Thanks and Regards,
"Confidentiality Warning: This message and any attachments are intended only
for the use of the intended recipient(s).
are confidential and may be privileged. If you are not the intended recipient.
you are hereby notified that any
review. re-transmission. conversion to hard copy. copying. circulation or other
use of this message and any attachments is
strictly prohibited. If you are not the intended recipient. please notify the
sender immediately by return email.
and delete this message and any attachments from your system.
Virus Warning: Although the company has taken reasonable precautions to ensure
no viruses are present in this email.
The company cannot accept responsibility for any loss or damage arising from
the use of this email or attachment."
