[
https://issues.apache.org/jira/browse/GEODE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17404051#comment-17404051
]
Eric Shu commented on GEODE-9531:
---------------------------------
This is a resource issue as multiple vms (in various tests at the time) all
encountered same suspect process.
For the failing test, the vm failed was just starting up joined the ds:
{noformat}
[vm2] [info 2021/05/25 16:57:54.603 GMT <RMI TCP Connection(23)-172.17.0.39>
tid=0x87] DistributionManager 172.17.0.39(278)<v185>:41003 started on
localhost[43925]. There were 3 other DMs. others:
[172.17.0.39(255)<v182>:41002, 172.17.0.39(246)<v179>:41001,
1797ac7f43c4(107:locator)<ec><v0>:41000] (took 1800 ms)
[vm2] [info 2021/05/25 16:57:54.850 GMT <StatSampler> tid=0x556] Disabling
statistic archival.
[vm2] [info 2021/05/25 16:57:54.896 GMT <RMI TCP Connection(23)-172.17.0.39>
tid=0x87] No locator(s) found with cluster configuration service
[vm2] [info 2021/05/25 16:57:55.161 GMT <RMI TCP Connection(23)-172.17.0.39>
tid=0x87] Initialized cache service
org.apache.geode.cache.query.internal.QueryConfigurationServiceImpl
{noformat}
And at the time all vms got suspected as Geode Failure Detection kicked in.
{noformat}
[vm1] [info 2021/05/25 16:58:08.119 GMT <Geode Failure Detection thread 3>
tid=0x76b] received suspect message from myself for
1797ac7f43c4(107:locator)<ec><v0>:41000: Member isn't responding to heartbeat
requests
[vm1] [info 2021/05/25 16:58:08.118 GMT <Geode Failure Detection thread 2>
tid=0x76a] received suspect message from myself for
172.17.0.39(278)<v185>:41003: Member isn't responding to heartbeat requests
[vm1] [info 2021/05/25 16:58:08.143 GMT <Geode Failure Detection thread 4>
tid=0x76c] received suspect message from myself for
172.17.0.39(246)<v179>:41001: Member isn't responding to heartbeat requests
[vm1] [info 2021/05/25 16:58:08.264 GMT <Geode Failure Detection thread 2>
tid=0x76a] Performing availability check for suspect member
172.17.0.39(278)<v185>:41003 reason=Member isn't responding to heartbeat
requests
[vm1] [info 2021/05/25 16:58:08.266 GMT <Geode Failure Detection thread 2>
tid=0x76a] All other members are suspect at this point
{noformat}
locator did not get enough cpu cycles as well, but managed to respond the
suspect process just in time.
{noformat}
[locator] [warn 2021/05/25 16:58:11.415 GMT <Geode Heartbeat Sender> tid=0x37]
Failure detection heartbeat-generation thread overslept by more than a full
period. Asleep time: 15,705,239,045 nanoseconds. Period: 2,500,000,000
nanoseconds.
[locator] [info 2021/05/25 16:58:11.541 GMT <unicast
receiver,1797ac7f43c4-30841> tid=0x32] received suspect message from
172.17.0.39(255)<v182>:41002 for 172.17.0.39(278)<v185>:41003: Member isn't
responding to heartbeat requests
{noformat}
vm2 did not and so it was kicked out of the ds.
{noformat}
[vm2] [warn 2021/05/25 16:58:13.131 GMT <StatSampler> tid=0x556] Statistics
sampling thread detected a wakeup delay of 16556 ms, indicating a possible
resource issue. Check the GC, memory, and CPU statistics.
[vm2] [warn 2021/05/25 16:58:13.147 GMT <Geode Heartbeat Sender> tid=0x54a]
Failure detection heartbeat-generation thread overslept by more than a full
period. Asleep time: 19,938,476,329 nanoseconds. Period: 2,500,000,000
nanoseconds.
[vm1] [info 2021/05/25 16:58:13.351 GMT <Geode Failure Detection thread 2>
tid=0x76a] Availability check failed for member 172.17.0.39(278)<v185>:41003
[vm1] [info 2021/05/25 16:58:13.351 GMT <Geode Failure Detection thread 2>
tid=0x76a] Requesting removal of suspect member 172.17.0.39(278)<v185>:41003
{noformat}
I also tried to see if other tests run experiencing the same issue or not. At
the time, following tests are run concurrently.
org.apache.geode.cache.lucene.RollingUpgradeQueryReturnsCorrectResultAfterTwoLocatorsWithTwoServersAreRolled
luceneQueryReturnsCorrectResultAfterTwoLocatorsWithTwoServersAreRolled[from_v1.3.0,
with reindex=true, singleHopEnabled=true]
org.apache.geode.cache.lucene.RollingUpgradeQueryReturnsCorrectResultsAfterClientAndServersAreRolledOver
luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOver[from_v1.3.0,
with reindex=true, singleHopEnabled=true]
org.apache.geode.cache.lucene.RollingUpgradeQueryReturnsCorrectResultsAfterClientAndServersAreRolledOverAllBucketsCreated
test[from_v1.4.0, with reindex=true, singleHopEnabled=true]
org.apache.geode.cache.lucene.RollingUpgradeQueryReturnsCorrectResultsAfterServersRollOverOnPartitionRegion
luceneQueryReturnsCorrectResultsAfterServersRollOverOnPartitionRegion[from_v1.2.0,
with reindex=false, singleHopEnabled=true]
org.apache.geode.cache.lucene.RollingUpgradeQueryReturnsCorrectResultsAfterServersRollOverOnPersistentPartitionRegion
luceneQueryReturnsCorrectResultsAfterServersRollOverOnPersistentPartitionRegion[from_v1.2.0,
with reindex=false, singleHopEnabled=true]
org.apache.geode.cache.lucene.RollingUpgradeReindexShouldBeSuccessfulWhenAllServersRollToCurrentVersion
luceneReindexShouldBeSuccessfulWhenAllServersRollToCurrentVersion[from_v1.3.0,
with reindex=false, singleHopEnabled=true]
org.apache.geode.cache.wan.WANRollingUpgradeCreateGatewaySenderMixedSiteOneCurrentSiteTwo
CreateGatewaySenderMixedSiteOneCurrentSiteTwo[from_v1.8.0]
org.apache.geode.cache.wan.WANRollingUpgradeEventProcessingMixedSiteOneCurrentSiteTwo
EventProcessingMixedSiteOneCurrentSiteTwo[from_v1.7.0]
org.apache.geode.cache.wan.WANRollingUpgradeEventProcessingMixedSiteOneOldSiteTwo
EventProcessingMixedSiteOneOldSiteTwo[from_v1.6.0]
org.apache.geode.cache.wan.WANRollingUpgradeEventProcessingOldSiteOneCurrentSiteTwo
testEventProcessingOldSiteOneCurrentSiteTwo[from_v1.8.0]
org.apache.geode.cache.wan.WANRollingUpgradeMultipleReceiversDefinedInClusterConfiguration
testMultipleReceiversRemovedDuringRoll[from_v1.3.0; attributes=[default=];
expectedReceiverCount=1]
org.apache.geode.cache.wan.WANRollingUpgradeNewSenderProcessOldEvent
bothOldAndNewEventsShouldBeProcessedByOldSender[from_v1.3.0]
org.apache.geode.cache.wan.WANRollingUpgradeSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailover
testSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailover[from_v1.9.1]
org.apache.geode.cache.wan.WANRollingUpgradeSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient
testSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient[from_v1.7.0]
org.apache.geode.cache.wan.WANRollingUpgradeSecondaryEventsNotReprocessedAfterOldSiteMemberFailover
testSecondaryEventsNotReprocessedAfterOldSiteMemberFailover[from_v1.7.0]
org.apache.geode.cache.wan.WANRollingUpgradeVerifyGatewayReceiverDoesNotSendRemoveCacheServerProfileToMembersOlderThan1dot5
VerifyGatewayReceiverDoesNotSendRemoveCacheServerProfileToMembersOlderThan1dot5[from_v1.12.1]
org.apache.geode.cache.wan.WANRollingUpgradeVerifyGatewaySenderProfile
testVerifyGatewaySenderProfile[from_v1.13.1]
org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxBothTest
test[14]
org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxPartitionTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxReplicateTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxBothTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxPartitionTest
test[14]
org.apache.geode.internal.cache.TxCommitMessageBCOldClientToServerTxReplicateTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerTxBothTest
test[14]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerTxPartitionTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerTxReplicateTest
test[14]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerViaPoolTxBothTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerViaPoolTxPartitionTest
test[11]
org.apache.geode.internal.cache.TxCommitMessageBCServerToServerViaPoolTxReplicateTest
test[11]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeClients
testClients[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeConcurrentPutsReplicated
testConcurrentPutsReplicated[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeCreateIndexesMixedServersOnPartitionedRegions
testCreateIndexesMixedServersOnPartitionedRegions[from_v1.9.1]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeHARegionNameOnDifferentServerVersions
testHARegionNameOnDifferentServerVersions[from_v1.11.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeNonHAFunction
functionExceptionsThrownFromDifferentVersionServerShouldCorrectlyWrapFunctionExceptionCauses[from_v1.11.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradePutAndGetMixedServerPartitionedRegion
testPutAndGetMixedServerPartitionedRegion[from_v1.11.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradePutAndGetMixedServersReplicateRegion
testPutAndGetMixedServersReplicateRegion[from_v1.10.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeQueryMixedServersOnPartitionedRegions
testQueryMixedServersOnPartitionedRegions[from_v1.9.1]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeQueryMixedServersOnReplicatedRegions
testQueryMixedServersOnReplicatedRegions[from_v1.10.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollLocatorWithTwoServers
testRollLocatorWithTwoServers[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollLocatorsWithOldServer
testRollLocatorsWithOldServer[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollServersOnPartitionedRegion_dataserializable
testRollServersOnPartitionedRegion_dataserializable[from_v1.5.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollServersOnPersistentRegion_dataserializable
testRollServersOnPersistentRegion_dataserializable[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollServersOnReplicatedRegion_dataserializable
testRollServersOnReplicatedRegion_dataserializable[from_v1.5.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeRollSingleLocatorWithMultipleServersReplicatedRegion
testRollSingleLocatorWithMultipleServersReplicatedRegion[from_v1.6.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeTracePRQuery
testTracePRQuery[from_v1.11.0]
org.apache.geode.internal.cache.rollingupgrade.RollingUpgradeVerifyXmlEntity
testVerifyXmlEntity[from_v1.12.1]
org.apache.geode.internal.cache.tier.sockets.ClientServerMiscBCDUnitTest
testOnSeverMethodsWithCachingProxyClientHA[0]
org.apache.geode.management.DeploymentManagementUpgradeTest
newLocatorCanReadOldConfigurationData[1.12.0]
org.apache.geode.management.GfshCompatibilityTest
currentGfshConnectToOlderVersionsOfLocator[1.10.0]
org.apache.geode.management.GfshRebalanceCommandCompatibilityTest
whenCurrentVersionLocatorsExecuteRebalanceOnOldServersThenItMustSucceed[1.12.1]
org.apache.geode.management.OperationManagementUpgradeTest
newLocatorCanReadOldConfigurationData[1.13.1]
org.apache.geode.management.RollingUpgradeWithGfshDUnitTest
testRollingUpgradeWithDeployment[1.10.0]
org.apache.geode.security.ClientAuthorizationCQDUnitTest
testAllOpsWithFailover2[from_v1.8.0]
org.apache.geode.session.tests.Tomcat8ClientServerRollingUpgradeTest
canDoARollingUpgradeOfGeodeServersWithSessionModules[1.8.0]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat7079WithOldModuleCanDoPutsTest
test[4]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat7079WithOldModulesMixedWithCurrentCanDoPutFromCurrentModuleTest
test[3]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat7079WithOldModulesMixedWithCurrentCanDoPutFromOldModuleTest
test[4]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat8WithOldModuleCanDoPutsTest
test[4]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat8WithOldModulesMixedWithCurrentCanDoPutFromCurrentModuleTest
test[4]
org.apache.geode.session.tests.TomcatSessionBackwardsCompatibilityTomcat8WithOldModulesMixedWithCurrentCanDoPutFromOldModuleTest
test[3]
I randomly selected some tests to do some checking. And every test checked
showing some vms experienced resource issue at about the same time. I think we
can just treat this failure as a resource issue.
In
org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxReplicateTest
[vm2] [warn 2021/05/25 16:58:11.675 GMT <Geode Heartbeat Sender> tid=0x557]
Failure detection heartbeat-generation thread overslept by more than a full
period. Asleep time: 5,478,533,251 nanoseconds. Period: 2,500,000,000
nanoseconds.
[vm2] [info 2021/05/25 16:58:12.929 GMT <unicast receiver,70dc021acda9-709>
tid=0x552] received suspect message from 172.17.0.32(252)<v182>:41002 for
172.17.0.32(264)<v185>:41003: Member isn't responding to heartbeat requests
[vm2] [info 2021/05/25 16:58:13.340 GMT <unicast receiver,70dc021acda9-709>
tid=0x552] received suspect message from 172.17.0.32(245)<v179>:41001 for
172.17.0.32(264)<v185>:41003: Member isn't responding to heartbeat requests
[vm2] [info 2021/05/25 16:58:13.347 GMT <unicast receiver,70dc021acda9-709>
tid=0x552] received suspect message from
70dc021acda9(106:locator)<ec><v0>:41000 for 172.17.0.32(264)<v185>:41003:
Member isn't responding to heartbeat requests
{noformat}
In org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxBothTest
[vm0] [warn 2021/05/25 16:58:13.492 GMT <Geode Heartbeat Sender> tid=0x864]
Failure detection heartbeat-generation thread overslept by more than a full
period. Asleep time: 8,042,469,216 nanoseconds. Period: 2,500,000,000
nanoseconds.
In org.apache.geode.cache.wan.WANRollingUpgradeVerifyGatewaySenderProfile
[vm0_v1.13.1] [warn 2021/05/25 16:58:11.628 GMT <StatSampler> tid=0x3c]
Statistics sampling thread detected a wakeup delay of 14482 ms, indicating a
possible resource issue. Check the GC, memory, and CPU statistics.
In org.apache.geode.cache.wan.WANRollingUpgradeNewSenderProcessOldEvent
[vm0] [warn 2021/05/25 16:58:13.015 GMT <StatSampler> tid=0x4c] Statistics
sampling thread detected a wakeup delay of 15418 ms, indicating a possible
resource issue. Check the GC, memory, and CPU statistics.
> CI Failure: TxCommitMessageBCClientToServerTxPartitionTest fails with
> ForcedDisconnectException
> -----------------------------------------------------------------------------------------------
>
> Key: GEODE-9531
> URL: https://issues.apache.org/jira/browse/GEODE-9531
> Project: Geode
> Issue Type: Bug
> Affects Versions: 1.14.0
> Reporter: Donal Evans
> Assignee: Eric Shu
> Priority: Major
> Labels: GeodeOperationAPI, blocks-1.14.0
>
> {noformat}
> org.apache.geode.internal.cache.TxCommitMessageBCClientToServerTxPartitionTest
> > test[11] FAILED
> org.apache.geode.test.dunit.RMIException: While invoking
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase$$Lambda$55/2050040059.run
> in VM 2 running on Host 1797ac7f43c4 with 5 VMs
> Caused by:
> org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> Caused by:
> org.apache.geode.ForcedDisconnectException: Member isn't
> responding to heartbeat requests
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 993
> [fatal 2021/05/25 16:58:13.700 GMT <unicast receiver,1797ac7f43c4-36391>
> tid=1349] Membership service failure: Member isn't responding to heartbeat
> requests
>
> org.apache.geode.distributed.internal.membership.api.MemberDisconnectedException:
> Member isn't responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.forceDisconnect(GMSMembership.java:1783)
> at
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1122)
> at
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveMemberMessage(GMSJoinLeave.java:725)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1366)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1302)
> at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
> at org.jgroups.JChannel.up(JChannel.java:741)
> at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
> at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
> at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
> at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
> at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
> at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:72)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:70)
> at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
> at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
> at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
> at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
> at org.jgroups.protocols.TP.receive(TP.java:1714)
> at
> org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:159)
> at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
> at java.lang.Thread.run(Thread.java:748)
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 1041
> [error 2021/05/25 16:58:14.206 GMT <RMI TCP Connection(23)-172.17.0.39>
> tid=135] Cache initialization for GemFireCache[id = 664332017; isClosing =
> false; isShutDownAll = false; created = Tue May 25 16:57:54 GMT 2021; server
> = false; copyOnRead = false; lockLease = 120; lockTimeout = 60] failed
> because:
> org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.DistributionImpl.checkCancelled(DistributionImpl.java:313)
> at
> org.apache.geode.distributed.internal.DistributionImpl.send(DistributionImpl.java:243)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2053)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:1981)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2018)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1083)
> at
> org.apache.geode.internal.cache.CreateRegionProcessor.initializeRegion(CreateRegionProcessor.java:115)
> at
> org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1161)
> at
> org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1092)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3104)
> at
> org.apache.geode.internal.cache.InternalRegionFactory.create(InternalRegionFactory.java:78)
> at
> org.apache.geode.pdx.internal.PeerTypeRegistration.initialize(PeerTypeRegistration.java:202)
> at
> org.apache.geode.pdx.internal.TypeRegistry.initialize(TypeRegistry.java:116)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializePdxRegistry(GemFireCacheImpl.java:1671)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1605)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1448)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> at
> org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
> at
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase.createServerCacheWithPool(TxCommitMessageBCTestBase.java:187)
> at
> org.apache.geode.internal.cache.TxCommitMessageBCTestBase.lambda$postSetUp$384cd611$1(TxCommitMessageBCTestBase.java:117)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
> at
> org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
> at sun.reflect.GeneratedMethodAccessor333.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.ForcedDisconnectException: Member isn't
> responding to heartbeat requests
> at
> org.apache.geode.distributed.internal.DistributionImpl.checkCancelled(DistributionImpl.java:312)
> ... 42 more
> -----------------------------------------------------------------------
> Found suspect string in 'dunit_suspect-vm2.log' at line 1130
> [error 2021/05/25 16:58:15.274 GMT <RMI TCP Connection(23)-172.17.0.39>
> tid=135] org.apache.geode.distributed.DistributedSystemDisconnectedException:
> membership shutdown, caused by org.apache.geode.ForcedDisconnectException:
> Member isn't responding to heartbeat requests
> 576 tests completed, 1 failed, 36 skipped
> {noformat}
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-= Test Results URI
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> http://files.apachegeode-ci.info/builds/apache-support-1-14-main/1.14.0-build.0787/test-results/upgradeTest/1621966586/
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Test report artifacts from this job are available at:
> http://files.apachegeode-ci.info/builds/apache-support-1-14-main/1.14.0-build.0787/test-artifacts/1621966586/upgradetestfiles-OpenJDK8-1.14.0-build.0787.tgz
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
