[jira] [Updated] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Voronkin updated IGNITE-10644: Attachment: (was: IndexingTest.java) > CorruptedTreeException might occur after force node kill during transaction > --- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug >Reporter: Pavel Voronkin >Priority: Major > > Partition eviction process on the other hand: > > 2018-12-10 20:59:24.426 > [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] > Partition eviction failed, this can cause grid hang. > org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY > [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [5-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, > hash=529139710, colocationKey
[jira] [Updated] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Voronkin updated IGNITE-10644: Attachment: IndexingTest.java > CorruptedTreeException might occur after force node kill during transaction > --- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug >Reporter: Pavel Voronkin >Priority: Major > Attachments: IndexingTest.java > > > Partition eviction process on the other hand: > > 2018-12-10 20:59:24.426 > [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] > Partition eviction failed, this can cause grid hang. > org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY > [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [5-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513,
[jira] [Updated] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Voronkin updated IGNITE-10644: Description: Partition eviction process on the other hand: 2018-12-10 20:59:24.426 [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang. org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [5-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [top
[jira] [Updated] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Voronkin updated IGNITE-10644: Description: Partition eviction process on the other hand: 2018-12-10 20:59:24.426 [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang. org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [5-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersi
[jira] [Updated] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Voronkin updated IGNITE-10644: Description: Partition eviction process on the other hand: 2018-12-10 20:59:24.426 [ERROR][sys-#204%DPL_GRID%DplGridNodeName%][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang. org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [5-195] at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convert(DbException.java:295) at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: com.sbt.bm.ucp.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*]-[*]-[kbt] (zone-node-module).IP: [*]. List of services violations: NODE MODULE FILTER VIOLATIONS No services or violations were found for routing, partition_DPL_id=5, messageId=1211871172446406939, entityId=1211871174131851
