[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult

2016-10-19 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590134#comment-15590134
 ] 

ASF subversion and git services commented on GEODE-538:
---

Commit 11ef3ebbe30a8340f57776bf4063684b91ccd0a3 in incubator-geode's branch 
refs/heads/develop from [~khowe]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=11ef3eb ]

GEODE-538: Add check for persistent data recovery

PartitionedRegion.getNodeForBucketReadOrLoad can return an invalid node
if persistent data recovery is in process and a get() targets a bucket
that
hasn't been recoverd yet. This can result in returning an incorrect
value (null) or throwing ConflictingPersistentDataException from a get()
or put() on the region.

This change adds a check for persistent recovery to be completed
before creating the new bucket. If recovery isn't complete then the
operation on the region will fail with a PartitionOfflineException.

Queries on a region while persistent recovery is in progress can also
result in incorrect results so a similar check is added to
DefaultQuery.checkQueryOnPR.

This closes #264


> PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion
>  failed if do not wait on getResult
> 
>
> Key: GEODE-538
> URL: https://issues.apache.org/jira/browse/GEODE-538
> Project: Geode
>  Issue Type: Bug
>  Components: persistence
>Reporter: Jinmei Liao
>Assignee: Kenneth Howe
> Attachments: logs.zip
>
>
> While attempting to work on GEODE-506, I commented out line s1538-1540 like 
> the following
> async1 = vm1.invokeAsync(createChildPR);
> async0 = vm0.invokeAsync(createChildPR);
> AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
> //async0.getResult(MAX_WAIT);  -- line 1538
> //async1.getResult(MAX_WAIT);  -- line 1539
> //async2.getResult(MAX_WAIT);  -- line 1540
> This will reproduce the error similar to the one reported in GEODE506, but it 
> failed on checking the data on parent region instead of the child region. Log 
> files attached.
> dunit.RMIException: While invoking 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run
>  in VM 0 running on Host localhost with 4 VMs
>   at dunit.VM.invoke(VM.java:368)
>   at dunit.VM.invoke(VM.java:312)
>   at dunit.VM.invoke(VM.java:266)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:252)
>   at junit.framework.TestSuite.run(TestSuite.java:247)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: junit.framework.AssertionFailedError: For key 1 expected: but 
> was:
>   at

[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult

2016-10-17 Thread Kenneth Howe (JIRA) 

[ 
https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582559#comment-15582559
 ] 

Kenneth Howe commented on GEODE-538:


Queries on a persistent colocated PR can also cause this problem.

> PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion
>  failed if do not wait on getResult
> 
>
> Key: GEODE-538
> URL: https://issues.apache.org/jira/browse/GEODE-538
> Project: Geode
>  Issue Type: Bug
>  Components: persistence
>Reporter: Jinmei Liao
>Assignee: Kenneth Howe
> Attachments: logs.zip
>
>
> While attempting to work on GEODE-506, I commented out line s1538-1540 like 
> the following
> async1 = vm1.invokeAsync(createChildPR);
> async0 = vm0.invokeAsync(createChildPR);
> AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
> //async0.getResult(MAX_WAIT);  -- line 1538
> //async1.getResult(MAX_WAIT);  -- line 1539
> //async2.getResult(MAX_WAIT);  -- line 1540
> This will reproduce the error similar to the one reported in GEODE506, but it 
> failed on checking the data on parent region instead of the child region. Log 
> files attached.
> dunit.RMIException: While invoking 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run
>  in VM 0 running on Host localhost with 4 VMs
>   at dunit.VM.invoke(VM.java:368)
>   at dunit.VM.invoke(VM.java:312)
>   at dunit.VM.invoke(VM.java:266)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:252)
>   at junit.framework.TestSuite.run(TestSuite.java:247)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: junit.framework.AssertionFailedError: For key 1 expected: but 
> was:
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.TestCase.assertEquals(TestCase.java:244)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run(PersistentPartitionedRegionTestBase.java:177)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at hydra.MethExecutor.executeObject(MethExecutor.java:267)
>   at 
> dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:77)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult

2016-09-27 Thread Kenneth Howe (JIRA) 

[ 
https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527484#comment-15527484
 ] 

Kenneth Howe commented on GEODE-538:


With the async*.getResult(MAX_WAIT) commented out, then checkData() is called 
while persistent data recovery is still in progress following the child region 
creation on all the members. The test now (usually) fails because a 
PartitionedRegion.get() call targets a bucket that hasn't yet been recovered. 
PR.getNodeForBucketReadOrLoad() ends up creating a new bucket, and the get() 
ends up returning null.

Examining full logs there will also be one or more members that throw 
ConflictingPersistentDataException when the recovery thread(s) tries to recover 
the bucket that was erroneously created.

put() operations on PRs with incomplete persistent recovery can also cause this 
failure.

> PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion
>  failed if do not wait on getResult
> 
>
> Key: GEODE-538
> URL: https://issues.apache.org/jira/browse/GEODE-538
> Project: Geode
>  Issue Type: Bug
>  Components: persistence
>Reporter: Jinmei Liao
>Assignee: Kenneth Howe
> Attachments: logs.zip
>
>
> While attempting to work on GEODE-506, I commented out line s1538-1540 like 
> the following
> async1 = vm1.invokeAsync(createChildPR);
> async0 = vm0.invokeAsync(createChildPR);
> AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
> //async0.getResult(MAX_WAIT);  -- line 1538
> //async1.getResult(MAX_WAIT);  -- line 1539
> //async2.getResult(MAX_WAIT);  -- line 1540
> This will reproduce the error similar to the one reported in GEODE506, but it 
> failed on checking the data on parent region instead of the child region. Log 
> files attached.
> dunit.RMIException: While invoking 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run
>  in VM 0 running on Host localhost with 4 VMs
>   at dunit.VM.invoke(VM.java:368)
>   at dunit.VM.invoke(VM.java:312)
>   at dunit.VM.invoke(VM.java:266)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:252)
>   at junit.framework.TestSuite.run(TestSuite.java:247)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: junit.framework.AssertionFailedError: For key 1 expected: but 
> was:
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.TestCase.assertEquals(TestCase.java:244)
>   at 
>

[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult

2016-08-16 Thread Darrel Schneider (JIRA) 

[ 
https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423587#comment-15423587
 ] 

Darrel Schneider commented on GEODE-538:


In the current source code I think this is lines: 1562-1564. I made commented 
out these lines like so:
async1 = vm1.invokeAsync(createChildPR);
async0 = vm0.invokeAsync(createChildPR);
AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
//async0.getResult(MAX_WAIT);
//async1.getResult(MAX_WAIT);
//async2.getResult(MAX_WAIT);

but the unit test still passes.

Ken has reported seeing operations on parent regions that are created but still 
off-line waiting for a colocated child region hang.

> PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion
>  failed if do not wait on getResult
> 
>
> Key: GEODE-538
> URL: https://issues.apache.org/jira/browse/GEODE-538
> Project: Geode
>  Issue Type: Bug
>  Components: persistence
>Reporter: Jinmei Liao
>Assignee: Dan Smith
> Attachments: logs.zip
>
>
> While attempting to work on GEODE-506, I commented out line s1538-1540 like 
> the following
> async1 = vm1.invokeAsync(createChildPR);
> async0 = vm0.invokeAsync(createChildPR);
> AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
> //async0.getResult(MAX_WAIT);  -- line 1538
> //async1.getResult(MAX_WAIT);  -- line 1539
> //async2.getResult(MAX_WAIT);  -- line 1540
> This will reproduce the error similar to the one reported in GEODE506, but it 
> failed on checking the data on parent region instead of the child region. Log 
> files attached.
> dunit.RMIException: While invoking 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run
>  in VM 0 running on Host localhost with 4 VMs
>   at dunit.VM.invoke(VM.java:368)
>   at dunit.VM.invoke(VM.java:312)
>   at dunit.VM.invoke(VM.java:266)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:252)
>   at junit.framework.TestSuite.run(TestSuite.java:247)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: junit.framework.AssertionFailedError: For key 1 expected: but 
> was:
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.TestCase.assertEquals(TestCase.java:244)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run(PersistentPartitionedRegionTestBase.java:177)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at

[jira] [Commented] (GEODE-538) PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion failed if do not wait on getResult

2015-11-16 Thread Dan Smith (JIRA) 

[ 
https://issues.apache.org/jira/browse/GEODE-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007632#comment-15007632
 ] 

Dan Smith commented on GEODE-538:
-

I was working with Jinmei when we noticed this.

I think the basic problem that we are allowing operations to proceed on a 
region that is not really initialized.

What's going on is that a member is restarting a member with two persistent 
colocated regions. After it creates the parent region, we return a region 
object from the RegionFactory.create method. However, geode hasn't actually 
recovered the parent region from disk because it's waiting waiting for the 
child region to also be created.

Unfortunately if someone calls region.get on the parent at this point it looks 
like geode returns null, rather than throwing some sort of exception. That's 
bad because it looks like a key is missing when if fact geode hasn't actually 
recovered the region yet.

> PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion
>  failed if do not wait on getResult
> 
>
> Key: GEODE-538
> URL: https://issues.apache.org/jira/browse/GEODE-538
> Project: Geode
>  Issue Type: Bug
>  Components: persistence
>Reporter: Jinmei Liao
>Assignee: Dan Smith
> Attachments: logs.zip
>
>
> While attempting to work on GEODE-506, I commented out line s1538-1540 like 
> the following
> async1 = vm1.invokeAsync(createChildPR);
> async0 = vm0.invokeAsync(createChildPR);
> AsyncInvocation async2 = vm2.invokeAsync(createChildPR);
> //async0.getResult(MAX_WAIT);  -- line 1538
> //async1.getResult(MAX_WAIT);  -- line 1539
> //async2.getResult(MAX_WAIT);  -- line 1540
> This will reproduce the error similar to the one reported in GEODE506, but it 
> failed on checking the data on parent region instead of the child region. Log 
> files attached.
> dunit.RMIException: While invoking 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase$5.run
>  in VM 0 running on Host localhost with 4 VMs
>   at dunit.VM.invoke(VM.java:368)
>   at dunit.VM.invoke(VM.java:312)
>   at dunit.VM.invoke(VM.java:266)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:182)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentPartitionedRegionTestBase.checkData(PersistentPartitionedRegionTestBase.java:165)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.rebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1543)
>   at 
> com.gemstone.gemfire.internal.cache.partitioned.PersistentColocatedPartitionedRegionDUnitTest.testRebalanceWithOfflineChildRegion(PersistentColocatedPartitionedRegionDUnitTest.java:1223)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at junit.framework.TestCase.runTest(TestCase.java:176)
>   at junit.framework.TestCase.runBare(TestCase.java:141)
>   at junit.framework.TestResult$1.protect(TestResult.java:122)
>   at junit.framework.TestResult.runProtected(TestResult.java:142)
>   at junit.framework.TestResult.run(TestResult.java:125)
>   at junit.framework.TestCase.run(TestCase.java:129)
>   at junit.framework.TestSuite.runTest(TestSuite.java:252)
>   at junit.framework.TestSuite.run(TestSuite.java:247)
>   at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> Caused by: junit.framework.AssertionFailedError: For key 1 expected: but 
> was:
>   at junit.framework.Assert.fail(Assert.java:57)
>   at junit.framework.Assert.failNotEquals(Assert.java:329)
>   at junit.framework.Assert.assertEquals(Assert.java:78)
>   at junit.framework.TestCase.assertEquals(TestCase.java:244)
>