Eric Shu created GEODE-8206:
-------------------------------
Summary: CI Failure:
ReconnectWithClusterConfigurationDUnitTest.testReconnectAfterMeltdown hang
Key: GEODE-8206
URL: https://issues.apache.org/jira/browse/GEODE-8206
Project: Geode
Issue Type: Bug
Components: ci, membership
Reporter: Eric Shu
This test hangs in:
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/DistributedTestOpenJDK8/builds/221#A
{noformat}
"RMI TCP Connection(1)-172.17.0.23" #32 daemon prio=5 os_prio=0
tid=0x00007fea58001800 nid=0x27d waiting on condition [0x00007feb23bf7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000e0e6cb10> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
at
org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4693)
at
org.apache.geode.internal.cache.LocalRegion.waitOnInitialization(LocalRegion.java:4671)
at
org.apache.geode.internal.cache.LocalRegion.getSubregion(LocalRegion.java:4558)
at
org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3322)
at
org.apache.geode.internal.cache.GemFireCacheImpl.getRegion(GemFireCacheImpl.java:3153)
at
org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.getConfigurationRegion(InternalConfigurationPersistenceService.java:792)
at
org.apache.geode.distributed.internal.InternalConfigurationPersistenceService.destroySharedConfiguration(InternalConfigurationPersistenceService.java:639)
at
org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest.lambda$teardown$bb17a952$1(ReconnectWithClusterConfigurationDUnitTest.java:112)
at
org.apache.geode.cache30.ReconnectWithClusterConfigurationDUnitTest$$Lambda$339/1176989958.run(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
at
org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$15/2049270388.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000000e0a75928> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
{noformat}
It is waiting for initializationLatchAfterGetInitialImage to countdown. Not
sure why gii was not able to finish.
There is another reconnect thread is waiting in another process. I do not have
enough context to tell if it is the gii provider or not.
{noformat}
"ReconnectThread" #77 prio=5 os_prio=0 tid=0x00007f42281f7000 nid=0x353 in
Object.wait() [0x00007f43082db000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2568)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2423)
- locked <0x00000000e0867ca8> (a java.lang.Object)
- locked <0x00000000e08b1778> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
- locked <0x00000000e0867ce8> (a java.lang.Class for
org.apache.geode.internal.cache.InternalCacheBuilder)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1275)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:2315)
at
org.apache.geode.distributed.internal.membership.gms.GMSMembership.uncleanShutdown(GMSMembership.java:1287)
at
org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.lambda$forceDisconnect$0(GMSMembership.java:2030)
at
org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl$$Lambda$261/2070151959.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
