[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379003#comment-16379003
]
ASF subversion and git services commented on GEODE-4675:
--------------------------------------------------------
Commit 889da898b5adf1864dd61a05ab01549ae6edcaf5 in geode's branch
refs/heads/feature/GEODE-4685 from [~dschneider]
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=889da89 ]
GEODE-4675: remove checkConnected calls while notifying listeners (#1495)
A call of checkConnected ended up being made from notifyListenersMemberRemoved
which could cause it to throw DistributedSystemDisconnectedException if
the distributed system was being shutdown.
So now the distribution manager is made without this check being done.
Also change to log level from fatal to warn and improved the log message.
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
> Affects Versions: 1.5.0
> Reporter: Shelley Lynn Hughes-Godfrey
> Assignee: Darrel Schneider
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC <DM-MemberEventInvoker> tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
> at
> org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
> at
> org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
> at
> org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
> at java.lang.Thread.run(Thread.java:748)
> -----------------------------------------------------------------------
> {noformat}
> According to the logs, this looks like it occurs during shutdown ...
> {noformat}
> [vm1] [info 2018/02/13 21:12:48.075 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] Stopping membership services
> [vm0] [info 2018/02/13 21:12:48.077 UTC <Geode Failure Detection Server
> thread 0> tid=398] GMSHealthMonitor server thread exiting
> [vm0] [info 2018/02/13 21:12:48.078 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] GMSHealthMonitor serverSocketExecutor is terminated
> [vm3] [info 2018/02/13 21:12:48.079 UTC <unicast receiver,b85562d33221-49229>
> tid=896] received leave request from 172.17.0.2<ec><v0>:32771 for
> 172.17.0.2(180)<ec><v0>:32771
> [vm0] [info 2018/02/13 21:12:48.084 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] DistributionManager stopped in 121ms.
> [vm0] [info 2018/02/13 21:12:48.086 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] Marking DistributionManager 172.17.0.2(176)<ec><v0>:32770 as closed.
> [vm1] [info 2018/02/13 21:12:48.087 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] GMSHealthMonitor server socket is closed in stopServices().
> [vm0] [info 2018/02/13 21:12:48.089 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] Got result: null
> [vm0] from
> org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1443408061.run
> with 0 args on object:
> org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1443408061@13fa96af
> (took 139 ms)
> [vm1] [info 2018/02/13 21:12:48.090 UTC <Geode Failure Detection Server
> thread 0> tid=402] GMSHealthMonitor server thread exiting
> [vm1] [info 2018/02/13 21:12:48.090 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] GMSHealthMonitor serverSocketExecutor is terminated
> [vm1] [info 2018/02/13 21:12:48.101 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] DistributionManager stopped in 135ms.
> [vm1] [info 2018/02/13 21:12:48.102 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] Marking DistributionManager 172.17.0.2(180)<ec><v0>:32771 as closed.
> [vm3] [info 2018/02/13 21:12:48.122 UTC <RMI TCP Connection(2)-172.17.0.2>
> tid=30] Shutting down DistributionManager 172.17.0.2(189)<v2>:32773.
> [vm1] [info 2018/02/13 21:12:48.115 UTC <RMI TCP Connection(5)-172.17.0.2>
> tid=30] Got result: null
> [vm1] from
> org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1846294803.run
> with 0 args on object:
> org.apache.geode.internal.cache.wan.WANTestBase$$Lambda$45/1846294803@76a45c45
> (took 161 ms)
> [vm3] [fatal 2018/02/13 21:12:48.099 UTC <DM-MemberEventInvoker> tid=891]
> Unexpected exception:
> [vm3] org.apache.geode.distributed.DistributedSystemDisconnectedException:
> This connection to a distributed system has been disconnected.
> [vm3] at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> [vm3] at
> org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
> [vm3] at
> org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
> [vm3] at
> org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
> [vm3] at
> org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
> [vm3] at
> org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
> [vm3] at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
> [vm3] at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
