[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785797#comment-16785797
]
Roman Kondakov commented on IGNITE-10561:
-
[~amashenkov], patch looks good for me.
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:933)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.app
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785446#comment-16785446
]
Andrew Mashenkov commented on IGNITE-10561:
---
[~rkondakov],TC look ok.
Would you please review PR once again?
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:933)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTr
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785442#comment-16785442
]
Ignite TC Bot commented on IGNITE-10561:
{panel:title=--> Run :: All: No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *--> Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=3243337&buildTypeId=IgniteTests24Java8_RunAll]
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
> at
> org.apache.ignite.internal.processors.cache.d
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784287#comment-16784287
]
Andrew Mashenkov commented on IGNITE-10561:
---
[~rkondakov], thanks for pointing.
I've fixed typos, redundant row state actualization.
What about effective row history check: no, without closure Ignite will fetch
all rows from page, but we want just check there is smth or not.
I'll fix this using proper method with ClosureCursor instead of ForwardCursor.
TC tests restarted.
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAs
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783613#comment-16783613
]
Roman Kondakov commented on IGNITE-10561:
-
[~amashenkov], patch looks very good for me. You've fixed a lot of stuff. But
there are several unclear points for me:
# {{GridCacheMvccEntryInfo}} - looks like the estimate of the entry size is
wrong:
{{SIZE_OVERHEAD = 4 * 8 /* long */ + 2 /* int */;}}
Shouldn't it be
{{SIZE_OVERHEAD = 4 * 8 /* long */ + 2 * 4* /* int */;}} ?
# I've noticed that we set missed hints to {{opCntr}} in the three places:
## {{GridDhtPartitionSupplier#extractEntryInfo}}
## {{MvccDataRow#readHeader}}
## {{GridDhtTxAbstractEnlistFuture#fetchHistoryInfo}}
What if instead of duplicating such a messy and low-level code in several
places we would left it only within {{MvccDataRow#readHeader}}?
# A bit weird naming of the {{added}} variable in
{{GridCacheMapEntry.java:6735}}. Looks like the real meaning of this variable
is opposite. Would it be better to remove negation and then do {{if (!added)
break;}}?
# Why do we need additional WAL entry for the latest entry ({{entries[0]}}) in
the case of delete operation at {{GridCacheMapEntry.java:6765}}?
# Is it possible to use {{BPlusTree#find(Max, Min, RowData.LINK_ONLY)}} and
then check for {{GridCursor#next}} to find out if there is any history of row
exists in the tree, Instead of using {{CheckHistoryExistsClosure}} in
{{IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#mvccApplyHistoryIfAbsent}}?
# Why do we need 400 GB region size in mvcc case? See
{{GridCTextacheRebalancingSyncSelfTest.java:111}}
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOff
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781757#comment-16781757
]
Ignite TC Bot commented on IGNITE-10561:
{panel:title=--> Run :: All: No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *--> Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=3211244&buildTypeId=IgniteTests24Java8_RunAll]
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Critical
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
> at
> org.apache.ignite.internal.processors.cache.d
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780380#comment-16780380
]
Andrew Mashenkov commented on IGNITE-10561:
---
The main issue is Rebalance iterator can cache rows and doesn't coordinates
with Vacuum, so cached rows can easily become stale.
E.g. Vacuum has completed after iterator read rows from page and before cursor
requests next row from iterator. In this case cursor can get row without hints,
and TxLog has no states for the row versions anymore.
Usually, an mvcc cache iterator is used with snapshot in queries that prevents
Vacuum from cleaning up TxLog, but rebalance iterator takes no snapshot.
Seems, we have to actualize row states on read instantly, to make sure all
committed rows will be rebalanced.
Note, vacuum should see real row state as it stored in data page.
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Major
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.int
[jira] [Commented] (IGNITE-10561) MVCC: Fix MVCC cache rebalance.
[
https://issues.apache.org/jira/browse/IGNITE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773812#comment-16773812
]
Andrew Mashenkov commented on IGNITE-10561:
---
Just for upd.
I've found a number of issues that are fixed in the PR, but the test still
fails, however other TC tests looks good.
For now, the issue with unexpected tx state looks resolved, but sometimes
latest entry updates can be missed after rebalance.
What was done:
* entry full history is sent within single message and apply either entire
history or nothing under same entry lock.
* fix version hints usage in "update with history" operation.
* minor code cleanup.
> MVCC: Fix MVCC cache rebalance.
> ---
>
> Key: IGNITE-10561
> URL: https://issues.apache.org/jira/browse/IGNITE-10561
> Project: Ignite
> Issue Type: Bug
> Components: mvcc
>Reporter: Roman Kondakov
>Assignee: Andrew Mashenkov
>Priority: Major
> Labels: failover, mvcc_stabilization_stage_1
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Unexpected transaction state may be observed on some nodes after rebalance.
> Reproducer: {{GridCacheRebalancingSyncSelfTest#testComplexRebalancing}}
> Stacktrace:
> {noformat}
> javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
> Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null,
> snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224,
> cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]],
> upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
> at
> org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on
> bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse
> [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823,
> txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.in
