[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[ https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379003#comment-16379003 ] ASF subversion and git services commented on GEODE-4675: Commit 889da898b5adf1864dd61a05ab01549ae6edcaf5 in geode's branch refs/heads/feature/GEODE-4685 from [~dschneider] [ https://gitbox.apache.org/repos/asf?p=geode.git;h=889da89 ] GEODE-4675: remove checkConnected calls while notifying listeners (#1495) A call of checkConnected ended up being made from notifyListenersMemberRemoved which could cause it to throw DistributedSystemDisconnectedException if the distributed system was being shutdown. So now the distribution manager is made without this check being done. Also change to log level from fatal to warn and improved the log message. > CI failure (suspect strings): DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected reported as fatal > log message during shutdown > - > > Key: GEODE-4675 > URL: https://issues.apache.org/jira/browse/GEODE-4675 > Project: Geode > Issue Type: Bug > Components: regions >Affects Versions: 1.5.0 >Reporter: Shelley Lynn Hughes-Godfrey >Assignee: Darrel Schneider >Priority: Major > Labels: pull-request-available > Fix For: 1.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This failure occurred during CI on geode: > https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 > {noformat} > org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > > testPartitionedParallelPropagationHA FAILED > java.lang.AssertionError: Suspicious strings were written to the log > during this run. > Fix the strings or use IgnoredException.addIgnoredException to ignore. > --- > Found suspect string in log4j at line 9339 > [fatal 2018/02/13 21:12:48.099 UTC tid=891] > Unexpected exception: > org.apache.geode.distributed.DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected. > at > org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) > at > org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) > at > org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) > at > org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) > at > org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) > at > org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) > at > org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) > at > org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) > at > org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) > at > org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) > at > org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) > at java.lang.Thread.run(Thread.java:748) > --- > {noformat} > According to the logs, this looks like it occurs during shutdown ... > {noformat} > [vm1] [info 2018/02/13 21:12:48.075 UTC > tid=30] Stopping membership service
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[ https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377589#comment-16377589 ] ASF subversion and git services commented on GEODE-4675: Commit 889da898b5adf1864dd61a05ab01549ae6edcaf5 in geode's branch refs/heads/develop from [~dschneider] [ https://gitbox.apache.org/repos/asf?p=geode.git;h=889da89 ] GEODE-4675: remove checkConnected calls while notifying listeners (#1495) A call of checkConnected ended up being made from notifyListenersMemberRemoved which could cause it to throw DistributedSystemDisconnectedException if the distributed system was being shutdown. So now the distribution manager is made without this check being done. Also change to log level from fatal to warn and improved the log message. > CI failure (suspect strings): DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected reported as fatal > log message during shutdown > - > > Key: GEODE-4675 > URL: https://issues.apache.org/jira/browse/GEODE-4675 > Project: Geode > Issue Type: Bug > Components: regions >Affects Versions: 1.5.0 >Reporter: Shelley Lynn Hughes-Godfrey >Assignee: Darrel Schneider >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This failure occurred during CI on geode: > https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 > {noformat} > org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > > testPartitionedParallelPropagationHA FAILED > java.lang.AssertionError: Suspicious strings were written to the log > during this run. > Fix the strings or use IgnoredException.addIgnoredException to ignore. > --- > Found suspect string in log4j at line 9339 > [fatal 2018/02/13 21:12:48.099 UTC tid=891] > Unexpected exception: > org.apache.geode.distributed.DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected. > at > org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) > at > org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) > at > org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) > at > org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) > at > org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) > at > org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) > at > org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) > at > org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) > at > org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) > at > org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) > at > org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) > at java.lang.Thread.run(Thread.java:748) > --- > {noformat} > According to the logs, this looks like it occurs during shutdown ... > {noformat} > [vm1] [info 2018/02/13 21:12:48.075 UTC > tid=30] Stopping membership services > [vm0] [info 2018/02/13 21:12:48.077 UT
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[ https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377475#comment-16377475 ] xiaojian zhou commented on GEODE-4675: -- {noformat} Found in CI develop DistributedTest #162 org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > testParallelPropagationWithUnEqualBucketDivision FAILED java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore . --- Found suspect string in log4j at line 9083 [fatal 2018/02/24 06:27:04.231 UTC tid=1086] Unexpected exception: org.apache.geode.distributed.DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected. at org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) at org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1493) at org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) at org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) at org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) at org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) at org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) at org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) at org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) at org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) at org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) at org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) at org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) at java.lang.Thread.run(Thread.java:748{noformat} [ |https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/162#L5a6ba88b:628] > CI failure (suspect strings): DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected reported as fatal > log message during shutdown > - > > Key: GEODE-4675 > URL: https://issues.apache.org/jira/browse/GEODE-4675 > Project: Geode > Issue Type: Bug > Components: regions >Affects Versions: 1.5.0 >Reporter: Shelley Lynn Hughes-Godfrey >Assignee: Darrel Schneider >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This failure occurred during CI on geode: > https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 > {noformat} > org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > > testPartitionedParallelPropagationHA FAILED > java.lang.AssertionError: Suspicious strings were written to the log > during this run. > Fix the strings or use IgnoredException.addIgnoredException to ignore. > --- > Found suspect string in log4j at line 9339 > [fatal 2018/02/13 21:12:48.099 UTC tid=891] > Unexpected exception: > org.apache.geode.distributed.DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected. > at > org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) > at > org.apache.geode.distributed.internal.InternalDistributedSystem.getDis
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[ https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365966#comment-16365966 ] Shelley Lynn Hughes-Godfrey commented on GEODE-4675: Seen again in another test in CI: https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/144 ``` org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest > testPersistentPRWithGatewaySenderPersistenceEnabled_Restart2 FAILED java.lang.AssertionError: Suspicious strings were written to the log during this run. Fix the strings or use IgnoredException.addIgnoredException to ignore. --- Found suspect string in log4j at line 16228 [fatal 2018/02/15 03:31:53.578 UTC tid=447] Unexpected exception: org.apache.geode.distributed.DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected. at org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) at org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) at org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) at org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) at org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) at org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) at org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) at org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) at org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) at org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) at org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) at org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) at org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) at org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) at java.lang.Thread.run(Thread.java:748) --- ``` > CI failure (suspect strings): DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected reported as fatal > log message during shutdown > - > > Key: GEODE-4675 > URL: https://issues.apache.org/jira/browse/GEODE-4675 > Project: Geode > Issue Type: Bug > Components: regions >Affects Versions: 1.5.0 >Reporter: Shelley Lynn Hughes-Godfrey >Assignee: Darrel Schneider >Priority: Major > > This failure occurred during CI on geode: > https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 > {noformat} > org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > > testPartitionedParallelPropagationHA FAILED > java.lang.AssertionError: Suspicious strings were written to the log > during this run. > Fix the strings or use IgnoredException.addIgnoredException to ignore. > --- > Found suspect string in log4j at line 9339 > [fatal 2018/02/13 21:12:48.099 UTC tid=891] > Unexpected exception: > org.apache.geode.distributed.DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected. > at > org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) >
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[ https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364668#comment-16364668 ] Bruce Schuchardt commented on GEODE-4675: - This was caused by Darrel's recent commit, fc3e14440afbb66f19b3e6dc79ded168adb8dca3, that added a getDistributionManager() call to DistributionAdvisor.notifyListenersMemberRemoved(). The method used to get the distribution manager in this case should avoid checking to see if the DistributedSystem is closed. > CI failure (suspect strings): DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected reported as fatal > log message during shutdown > - > > Key: GEODE-4675 > URL: https://issues.apache.org/jira/browse/GEODE-4675 > Project: Geode > Issue Type: Bug > Components: regions >Affects Versions: 1.5.0 >Reporter: Shelley Lynn Hughes-Godfrey >Assignee: Darrel Schneider >Priority: Major > > This failure occurred during CI on geode: > https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140 > {noformat} > org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest > > testPartitionedParallelPropagationHA FAILED > java.lang.AssertionError: Suspicious strings were written to the log > during this run. > Fix the strings or use IgnoredException.addIgnoredException to ignore. > --- > Found suspect string in log4j at line 9339 > [fatal 2018/02/13 21:12:48.099 UTC tid=891] > Unexpected exception: > org.apache.geode.distributed.DistributedSystemDisconnectedException: This > connection to a distributed system has been disconnected. > at > org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911) > at > org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499) > at > org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757) > at > org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380) > at > org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225) > at > org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897) > at > org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964) > at > org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926) > at > org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183) > at > org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391) > at > org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218) > at > org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109) > at > org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250) > at java.lang.Thread.run(Thread.java:748) > --- > {noformat} > According to the logs, this looks like it occurs during shutdown ... > {noformat} > [vm1] [info 2018/02/13 21:12:48.075 UTC > tid=30] Stopping membership services > [vm0] [info 2018/02/13 21:12:48.077 UTC thread 0> tid=398] GMSHealthMonitor server thread exiting > [vm0] [info 2018/02/13 21:12:48.078 UTC > tid=30] GMSHealthMonitor serverSocketExecutor is terminated > [vm3] [info 2018/02/13 21:12:48.079 UTC > tid=896] received leave request from 172.17.0.2:32771 for > 172.17.0.2(180):32771 > [vm0] [info 2018/02/13 21:12:48.084 UTC > tid=30] DistributionManager stopped in 121ms. > [vm0] [