Shelley Lynn Hughes-Godfrey created GEODE-4675: --------------------------------------------------
Summary: CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during shutdown Key: GEODE-4675 URL: https://issues.apache.org/jira/browse/GEODE-4675 Project: Geode Issue Type: Bug Components: membership Reporter: Shelley Lynn Hughes-Godfrey This failure occurred during CI on geode: https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 {noformat} org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > testPartitionedParallelPropagationHA FAILED java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. ----------------------------------------------------------------------- Found suspect string in log4j at line 9339 [fatal 2018/02/13 21:12:48.099 UTC <DM-MemberEventInvoker> tid=891] Unexpected exception: org.apache.geode.distributed.DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected. at org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) at org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) at org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) at org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) at org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) at org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) at org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) at org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) at org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) at org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) at org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) at org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) at org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) at java.lang.Thread.run(Thread.java:748) ----------------------------------------------------------------------- {noformat} According to the logs, this looks like it occurs during shutdown ... {noformat} [vm1] [info 2018/02/13 21:12:48.075 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] Stopping membership services [vm0] [info 2018/02/13 21:12:48.077 UTC <Geode Failure Detection Server thread 0> tid=398] GMSHealthMonitor server thread exiting [vm0] [info 2018/02/13 21:12:48.078 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] GMSHealthMonitor serverSocketExecutor is terminated [vm3] [info 2018/02/13 21:12:48.079 UTC <unicast receiver,b85562d33221-49229> tid=896] received leave request from 172.17.0.2<ec><v0>:32771 for 172.17.0.2(180)<ec><v0>:32771 [vm0] [info 2018/02/13 21:12:48.084 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] DistributionManager stopped in 121ms. [vm0] [info 2018/02/13 21:12:48.086 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] Marking DistributionManager 172.17.0.2(176)<ec><v0>:32770 as closed. [vm1] [info 2018/02/13 21:12:48.087 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] GMSHealthMonitor server socket is closed in stopServices(). [vm0] [info 2018/02/13 21:12:48.089 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] Got result: null [vm0] from org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1443408061.run with 0 args on object: org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1443408061@13fa96af (took 139 ms) [vm1] [info 2018/02/13 21:12:48.090 UTC <Geode Failure Detection Server thread 0> tid=402] GMSHealthMonitor server thread exiting [vm1] [info 2018/02/13 21:12:48.090 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] GMSHealthMonitor serverSocketExecutor is terminated [vm1] [info 2018/02/13 21:12:48.101 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] DistributionManager stopped in 135ms. [vm1] [info 2018/02/13 21:12:48.102 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] Marking DistributionManager 172.17.0.2(180)<ec><v0>:32771 as closed. [vm3] [info 2018/02/13 21:12:48.122 UTC <RMI TCP Connection(2)-172.17.0.2> tid=30] Shutting down DistributionManager 172.17.0.2(189)<v2>:32773. [vm1] [info 2018/02/13 21:12:48.115 UTC <RMI TCP Connection(5)-172.17.0.2> tid=30] Got result: null [vm1] from org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1846294803.run with 0 args on object: org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1846294803@76a45c45 (took 161 ms) [vm3] [fatal 2018/02/13 21:12:48.099 UTC <DM-MemberEventInvoker> tid=891] Unexpected exception: [vm3] org.apache.geode.distributed.DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected. [vm3] at org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) [vm3] at org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) [vm3] at org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) [vm3] at org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) [vm3] at org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) [vm3] at org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) [vm3] at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) [vm3] at java.lang.Thread.run(Thread.java:748) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)