[jira] [Commented] (IGNITE-8435) Inconsistent data after StopNodeFailureHandler handles process termination
[
https://issues.apache.org/jira/browse/IGNITE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467454#comment-16467454
]
Andrey Gura commented on IGNITE-8435:
-
LGTM. Merged to master and ignite-2.5 branches. Thanks for contribution!
> Inconsistent data after StopNodeFailureHandler handles process termination
> --
>
> Key: IGNITE-8435
> URL: https://issues.apache.org/jira/browse/IGNITE-8435
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
> Fix For: 2.5
>
>
> Steps to reproduce:
> 1. Cluster started with StopNodeFailureHandler in config.
> 2. Upload data to the caches and run accounts task (transfer the amount from
> one account to another for some cache).
> 3. Terminate one thread (ttl-cleanup-worker for example for one node).
> 4. Wait until accounts task finished the work.
> 5. Check the sum for caches.
> Expected result:
> The sum is the same as before thread rermination.
> Actual result:
> The sum is not the same for some caches.
> Also, I see these exception in the logs for node with terminated thread:
> {code:java}
> [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
> class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Commit produced a runtime exception (all transaction entries will be
> invalidated):
> GridDhtTxRemote[xid=8e4906dd261--0815-47fa--0004,
> xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
> eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ,
> state=COMMITTING, invalidate=false, rollbackOnly=false,
> nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> search row:
> org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3604)
> at
> org.apache.ignite.internal.processors.ca
[jira] [Commented] (IGNITE-8435) Inconsistent data after StopNodeFailureHandler handles process termination
[
https://issues.apache.org/jira/browse/IGNITE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465682#comment-16465682
]
Anton Kalashnikov commented on IGNITE-8435:
---
Thanks for review. I fixed your comments.
> Inconsistent data after StopNodeFailureHandler handles process termination
> --
>
> Key: IGNITE-8435
> URL: https://issues.apache.org/jira/browse/IGNITE-8435
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
>
> Steps to reproduce:
> 1. Cluster started with StopNodeFailureHandler in config.
> 2. Upload data to the caches and run accounts task (transfer the amount from
> one account to another for some cache).
> 3. Terminate one thread (ttl-cleanup-worker for example for one node).
> 4. Wait until accounts task finished the work.
> 5. Check the sum for caches.
> Expected result:
> The sum is the same as before thread rermination.
> Actual result:
> The sum is not the same for some caches.
> Also, I see these exception in the logs for node with terminated thread:
> {code:java}
> [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
> class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Commit produced a runtime exception (all transaction entries will be
> invalidated):
> GridDhtTxRemote[xid=8e4906dd261--0815-47fa--0004,
> xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
> eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ,
> state=COMMITTING, invalidate=false, rollbackOnly=false,
> nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> search row:
> org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3604)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEnt
[jira] [Commented] (IGNITE-8435) Inconsistent data after StopNodeFailureHandler handles process termination
[
https://issues.apache.org/jira/browse/IGNITE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465565#comment-16465565
]
Alexey Goncharuk commented on IGNITE-8435:
--
[~akalashnikov], I left some review comments in the upsource (mostly,
renaming). Good to merge after they are fixed.
> Inconsistent data after StopNodeFailureHandler handles process termination
> --
>
> Key: IGNITE-8435
> URL: https://issues.apache.org/jira/browse/IGNITE-8435
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
>
> Steps to reproduce:
> 1. Cluster started with StopNodeFailureHandler in config.
> 2. Upload data to the caches and run accounts task (transfer the amount from
> one account to another for some cache).
> 3. Terminate one thread (ttl-cleanup-worker for example for one node).
> 4. Wait until accounts task finished the work.
> 5. Check the sum for caches.
> Expected result:
> The sum is the same as before thread rermination.
> Actual result:
> The sum is not the same for some caches.
> Also, I see these exception in the logs for node with terminated thread:
> {code:java}
> [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
> class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Commit produced a runtime exception (all transaction entries will be
> invalidated):
> GridDhtTxRemote[xid=8e4906dd261--0815-47fa--0004,
> xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
> eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ,
> state=COMMITTING, invalidate=false, rollbackOnly=false,
> nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> search row:
> org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3604)
> at
> org.apache.ign
[jira] [Commented] (IGNITE-8435) Inconsistent data after StopNodeFailureHandler handles process termination
[
https://issues.apache.org/jira/browse/IGNITE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463602#comment-16463602
]
Anton Kalashnikov commented on IGNITE-8435:
---
UP - [https://reviews.ignite.apache.org/ignite/review/IGNT-CR-596]
TC -
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&tab=projectOverview&branch_IgniteTests24Java8=pull%2F3944%2Fhead
> Inconsistent data after StopNodeFailureHandler handles process termination
> --
>
> Key: IGNITE-8435
> URL: https://issues.apache.org/jira/browse/IGNITE-8435
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
>
> Steps to reproduce:
> 1. Cluster started with StopNodeFailureHandler in config.
> 2. Upload data to the caches and run accounts task (transfer the amount from
> one account to another for some cache).
> 3. Terminate one thread (ttl-cleanup-worker for example for one node).
> 4. Wait until accounts task finished the work.
> 5. Check the sum for caches.
> Expected result:
> The sum is the same as before thread rermination.
> Actual result:
> The sum is not the same for some caches.
> Also, I see these exception in the logs for node with terminated thread:
> {code:java}
> [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
> class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Commit produced a runtime exception (all transaction entries will be
> invalidated):
> GridDhtTxRemote[xid=8e4906dd261--0815-47fa--0004,
> xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
> eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ,
> state=COMMITTING, invalidate=false, rollbackOnly=false,
> nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> search row:
> org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
> at
> org.apache.ignite.internal.proc
[jira] [Commented] (IGNITE-8435) Inconsistent data after StopNodeFailureHandler handles process termination
[
https://issues.apache.org/jira/browse/IGNITE-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463592#comment-16463592
]
ASF GitHub Bot commented on IGNITE-8435:
GitHub user akalash opened a pull request:
https://github.com/apache/ignite/pull/3944
IGNITE-8435 StorageException is handled like NodeStoppingException du…
…ring failing transaction commit
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gridgain/apache-ignite ignite-8435
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/ignite/pull/3944.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3944
commit 86d4b7d5526dc2195db376c87e1265b042190162
Author: Anton Kalashnikov
Date: 2018-04-23T10:36:08Z
IGNITE-8435 StorageException is handled like NodeStoppingException during
failing transaction commit
> Inconsistent data after StopNodeFailureHandler handles process termination
> --
>
> Key: IGNITE-8435
> URL: https://issues.apache.org/jira/browse/IGNITE-8435
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kalashnikov
>Assignee: Anton Kalashnikov
>Priority: Major
>
> Steps to reproduce:
> 1. Cluster started with StopNodeFailureHandler in config.
> 2. Upload data to the caches and run accounts task (transfer the amount from
> one account to another for some cache).
> 3. Terminate one thread (ttl-cleanup-worker for example for one node).
> 4. Wait until accounts task finished the work.
> 5. Check the sum for caches.
> Expected result:
> The sum is the same as before thread rermination.
> Actual result:
> The sum is not the same for some caches.
> Also, I see these exception in the logs for node with terminated thread:
> {code:java}
> [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
> class
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> Commit produced a runtime exception (all transaction entries will be
> invalidated):
> GridDhtTxRemote[xid=8e4906dd261--0815-47fa--0004,
> xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
> eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ,
> state=COMMITTING, invalidate=false, rollbackOnly=false,
> nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> search row:
> org.apache.ignite.in
