[
https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Voronkin updated IGNITE-10644:
------------------------------------
Description:
Partition eviction process on the other hand:
2018-12-10 20:59:24.426
[ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager]
Partition eviction failed, this can cause grid hang.
org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513,
hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]" [50000-195]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:295)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513,
hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]" [50000-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
... 22 common frames omitted
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513,
hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*][*][kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290)
... 19 common frames omitted
Caused by: java.lang.IllegalStateException: Item not found: 9
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782)
... 21 common frames omitted
And transaction on the other:
2018-12-10 20:59:34.881
[ERROR]sys-stripe-33-#34%DPL_GRID%DplGridNodeName%[org.apache.ignite.Ignite]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
transaction has produced runtime exception]]
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Committing a transaction has produced runtime exception
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]" [50000-195]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:295)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617)
... 19 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]" [50000-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
... 34 common frames omitted
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248)
... 31 common frames omitted
Caused by: java.lang.IllegalStateException: Item not found: 9
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100)
... 33 common frames omitted
Are failing to remove and put the same index entry on findIndirectItemIndex().
Because of node force (kill -9) stop index can be corrupted.
was:
Partition eviction process on the other hand:
2018-12-10 20:59:24.426
[ERROR][sys-#204%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.t.PartitionsEvictManager]
Partition eviction failed, this can cause grid hang.
org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]" [50000-195]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:295)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915)
at
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]" [50000-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
... 22 common frames omitted
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
lastChangeDate=1544464745135, errorMessage=No api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, partition_DPL_id=5,
messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
[topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No
api
[ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*].
List of services violations:
NODE MODULE FILTER VIOLATIONS
No services or violations were found for routing, 4071535538120363041, FALSE, 5
]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290)
... 19 common frames omitted
Caused by: java.lang.IllegalStateException: Item not found: 9
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782)
... 21 common frames omitted
And transaction on the other:
2018-12-10 20:59:34.881
[ERROR][sys-stripe-33-#34%DPL_GRID%DplGridNodeName%][org.apache.ignite.Ignite]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
transaction has produced runtime exception]]
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Committing a transaction has produced runtime exception
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.h2.message.DbException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]" [50000-195]
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:295)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251)
at
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617)
... 19 common frames omitted
Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]" [50000-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
... 34 common frames omitted
Caused by:
org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
[idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
[topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
null, 4071535538120364041, FALSE, 5 ]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080)
at
org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248)
... 31 common frames omitted
Caused by: java.lang.IllegalStateException: Item not found: 9
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
at
org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
at
org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
at
org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100)
... 33 common frames omitted
Are failing to remove and put the same index entry on findIndirectItemIndex().
Because of node force (kill -9) stop index can be corrupted.
> CorruptedTreeException might occur after force node kill during transaction
> ---------------------------------------------------------------------------
>
> Key: IGNITE-10644
> URL: https://issues.apache.org/jira/browse/IGNITE-10644
> Project: Ignite
> Issue Type: Bug
> Reporter: Pavel Voronkin
> Priority: Major
>
> Partition eviction process on the other hand:
>
> 2018-12-10 20:59:24.426
> [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager]
> Partition eviction failed, this can cause grid hang.
> org.h2.message.DbException: General error: "class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
> X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
> lastChangeDate=1544464745135, errorMessage=No api
> [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
> available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, partition_DPL_id=5,
> messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
> [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
> 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115,
> No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
> services available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, 4071535538120363041, FALSE,
> 5 ]" [50000-195]
> at org.h2.message.DbException.get(DbException.java:168)
> at org.h2.message.DbException.convert(DbException.java:295)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293)
> at
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487)
> at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423)
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782)
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
> X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
> lastChangeDate=1544464745135, errorMessage=No api
> [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
> available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, partition_DPL_id=5,
> messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
> [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
> 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115,
> No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
> services available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, 4071535538120363041, FALSE,
> 5 ]" [50000-195]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
> ... 22 common frames omitted
> Caused by:
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
> X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
> lastChangeDate=1544464745135, errorMessage=No api
> [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
> available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, partition_DPL_id=5,
> messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
> [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
> 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115,
> No api
> [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
> services available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, 4071535538120363041, FALSE,
> 5 ]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290)
> ... 19 common frames omitted
> Caused by: java.lang.IllegalStateException: Item not found: 9
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
> at
> org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
> at
> org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782)
> ... 21 common frames omitted
>
> And transaction on the other:
> 2018-12-10 20:59:34.881
> [ERROR]sys-stripe-33-#34%DPL_GRID%DplGridNodeName%[org.apache.ignite.Ignite]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
> transaction has produced runtime exception]]
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Committing a transaction has produced runtime exception
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.h2.message.DbException: General error: "class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
> com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
> lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
> messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
> [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
> 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
> null, 4071535538120364041, FALSE, 5 ]" [50000-195]
> at org.h2.message.DbException.get(DbException.java:168)
> at org.h2.message.DbException.convert(DbException.java:295)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251)
> at
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
> at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617)
> ... 19 common frames omitted
> Caused by: org.h2.jdbc.JdbcSQLException: General error: "class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
> com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
> lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
> messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
> [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
> 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
> null, 4071535538120364041, FALSE, 5 ]" [50000-195]
> at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
> ... 34 common frames omitted
> Caused by:
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val:
> com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I,
> lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5,
> messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion
> [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null,
> 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856,
> null, 4071535538120364041, FALSE, 5 ]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248)
> ... 31 common frames omitted
> Caused by: java.lang.IllegalStateException: Item not found: 9
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
> at
> org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
> at
> org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100)
> ... 33 common frames omitted
>
> Are failing to remove and put the same index entry on
> findIndirectItemIndex(). Because of node force (kill -9) stop index can be
> corrupted.
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
