[
https://issues.apache.org/jira/browse/GEODE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dan Smith reopened GEODE-244:
-----------------------------
This issue occurred again. I was able to get the logs, which showed that the
time the rebalance happened, some the buckets had radically different sizes.
I tracked this down to the fact the the queue is configured for async overflow.
The overflow size appears to be much larger than reported in memory size, which
is what causes this issue.
{noformat}
[vm_0][info 2015/10/25 15:59:32.719 PDT <ResourceManagerRecoveryThread 0>
tid=0x4c16] For Region: Partitioned Region @69974c36
[path='/AsyncEventQueue_parallelQueue_PARALLEL_GATEWAY_SENDER_QUEUE';
dataPolicy=PARTITION; prId=236; isDestroyed=false; isClosed=false;
retryTimeout=3600000; serialNumber=14495; hdfsStoreName=null;
hdfsWriteOnly=false; partition
attributes=PartitionAttributes@2085037449[redundantCopies=1;localMaxMemory=1;totalMaxMemory=2147483647;totalNumBuckets=113;partitionResolver=null;colocatedWith=/region1;recoveryDelay=-1;startupRecoveryDelay=-1;FixedPartitionAttributes=null;partitionListeners=null];
on VM cc1-co(10919)<v1767>:50556], Member:
cc1-co(10919)<v1767>:50556LOAD=PRLoad@63fcba02, weight: 1.0, numBuckets: 113,
bucketReadLoads: [16.0, 524415.0, 524415.0, 524415.0, 524415.0, 524415.0,
524415.0, 524415.0, 524415.0, 16.0, 16.0, 16.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
bucketWriteLoads: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0], equivalentMembers=[cc1-co(10914:locator)<v0>:56869,
cc1-co(10933)<v1769>:46926, cc1-co(10919)<v1767>:50556,
cc1-co(10928)<v1768>:19882]
{noformat}
> CI failure: RebalanceOperationDUnitTest
> testRecoverRedundancyParallelAsyncEventQueueSimulation failed an assertion
> ------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-244
> URL: https://issues.apache.org/jira/browse/GEODE-244
> Project: Geode
> Issue Type: Bug
> Affects Versions: 1.0.0-incubating
> Reporter: Kirk Lund
> Assignee: Dan Smith
> Labels: CI
>
> RebalanceOperationDUnitTest
> testRecoverRedundancyParallelAsyncEventQueueSimulation failed an assertion in
> nightly build #190:
> {color:red}
> {noformat}
> com.gemstone.gemfire.internal.cache.control.RebalanceOperationDUnitTest >
> testRecoverRedundancyParallelAsyncEventQueueSimulation FAILED
> dunit.RMIException: While invoking
> com.gemstone.gemfire.internal.cache.control.RebalanceOperationDUnitTest$43.run
> in VM 0 running on Host jenkins-ubuntu-1404-4gb-f59 with 4 VMs
> at dunit.VM.invoke(VM.java:359)
> at dunit.VM.invoke(VM.java:303)
> at dunit.VM.invoke(VM.java:257)
> at
> com.gemstone.gemfire.internal.cache.control.RebalanceOperationDUnitTest.recoverRedundancyParallelAsyncEventQueue(RebalanceOperationDUnitTest.java:1148)
> at
> com.gemstone.gemfire.internal.cache.control.RebalanceOperationDUnitTest.testRecoverRedundancyParallelAsyncEventQueueSimulation(RebalanceOperationDUnitTest.java:1090)
> Caused by:
> junit.framework.AssertionFailedError: expected:<6> but was:<4>
> {noformat}
> {color}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
