[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Voronkin updated IGNITE-10644: ------------------------------------ Description: Partition eviction process on the other hand: 2018-12-10 20:59:24.426 [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang. org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ... 22 common frames omitted Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290) ... 19 common frames omitted Caused by: java.lang.IllegalStateException: Item not found: 9 at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782) ... 21 common frames omitted And transaction on the other: 2018-12-10 20:59:34.881 [ERROR]sys-stripe-33-#34%X_GRID%XGridNodeName%[org.apache.ignite.Ignite] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception]] org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617) ... 19 common frames omitted Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ... 34 common frames omitted Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248) ... 31 common frames omitted Caused by: java.lang.IllegalStateException: Item not found: 9 at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100) ... 33 common frames omitted Are failing to remove and put the same index entry on findIndirectItemIndex(). Because of node force (kill -9) stop index can be corrupted. was: Partition eviction process on the other hand: 2018-12-10 20:59:24.426 [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang. org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ... 22 common frames omitted Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290) ... 19 common frames omitted Caused by: java.lang.IllegalStateException: Item not found: 9 at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782) ... 21 common frames omitted And transaction on the other: 2018-12-10 20:59:34.881 [ERROR]sys-stripe-33-#34%DPL_GRID%DplGridNodeName%[org.apache.ignite.Ignite] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception]] org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709) at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043) at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617) ... 19 common frames omitted Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) ... 34 common frames omitted Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_DPL_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248) ... 31 common frames omitted Caused by: java.lang.IllegalStateException: Item not found: 9 at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100) ... 33 common frames omitted Are failing to remove and put the same index entry on findIndirectItemIndex(). Because of node force (kill -9) stop index can be corrupted. > CorruptedTreeException might occur after force node kill during transaction > --------------------------------------------------------------------------- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug > Reporter: Pavel Voronkin > Priority: Major > > Partition eviction process on the other hand: > > 2018-12-10 20:59:24.426 > [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] > Partition eviction failed, this can cause grid hang. > org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY > [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [50000-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, > hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [50000-195] > at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) > ... 22 common frames omitted > Caused by: > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, > hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api > [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290) > ... 19 common frames omitted > Caused by: java.lang.IllegalStateException: Item not found: 9 > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) > at > org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) > at > org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) > at > org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) > at > org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782) > ... 21 common frames omitted > > And transaction on the other: > 2018-12-10 20:59:34.881 > [ERROR]sys-stripe-33-#34%X_GRID%XGridNodeName%[org.apache.ignite.Ignite] > Critical system error detected. Will be handled accordingly to configured > handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, > failureCtx=FailureContext [type=CRITICAL_ERROR, err=class > o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a > transaction has produced runtime exception]] > org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: > Committing a transaction has produced runtime exception > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720) > at > org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769) > at > org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217) > at > org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: > com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY > [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, > lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, > messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion > [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, > 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, > null, 4071535538120364041, FALSE, 5 ]" [50000-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043) > at > org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617) > ... 19 common frames omitted > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: > com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY > [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, > lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, > messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion > [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, > 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, > null, 4071535538120364041, FALSE, 5 ]" [50000-195] > at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) > ... 34 common frames omitted > Caused by: > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: > com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY > [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, > lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, > messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion > [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, > 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, > null, 4071535538120364041, FALSE, 5 ] > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248) > ... 31 common frames omitted > Caused by: java.lang.IllegalStateException: Item not found: 9 > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341) > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450) > at > org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) > at > org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152) > at > org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67) > at > org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212) > at > org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843) > at > org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161) > at > org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100) > ... 33 common frames omitted > > Are failing to remove and put the same index entry on > findIndirectItemIndex(). Because of node force (kill -9) stop index can be > corrupted. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)