[ https://issues.apache.org/jira/browse/CASSANDRA-16238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334402#comment-17334402 ]
David Capwell commented on CASSANDRA-16238: ------------------------------------------- If this is a race reading un-committed data, then the patch below might work around it (double check locking, but with queues) Patch: {code} $ git diff diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java index 699f235bd3..e82b107bac 100644 --- a/src/java/org/apache/cassandra/gms/Gossiper.java +++ b/src/java/org/apache/cassandra/gms/Gossiper.java @@ -928,6 +928,11 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean { logger.info("FatClient {} has been silent for {}ms, removing from gossip", endpoint, fatClientTimeout); runInGossipStageBlocking(() -> { + if (!isGossipOnlyMember(endpoint)) + { + // updating gossip and token metadata are not atomic, but rely on the single threaded gossip stage + // since status checks are done outside the gossip stage, need to confirm the state of the endpoint + // to make sure that the previous read data was correct + logger.info("Race condition marking {} as a FatClient; ignoring", endpoint); + return; + } removeEndpoint(endpoint); // will put it in justRemovedEndpoints to respect quarantine delay evictFromMembership(endpoint); // can get rid of the state immediately }); {code} > Fix flaky jvm-dtests that fail with Unable to contact any seeds > --------------------------------------------------------------- > > Key: CASSANDRA-16238 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16238 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest/java > Reporter: David Capwell > Assignee: Brandon Williams > Priority: Normal > Fix For: 4.0-rc > > Attachments: 16238-archived-failures.txt > > > https://app.circleci.com/pipelines/github/dcapwell/cassandra/745/workflows/1c7e589e-b5af-4a56-b40a-43da424602c7/jobs/4231 > {code} > test teardown failure > Unexpected error found in node logs (see stdout for full details). Errors: > [ERROR [main] 2020-10-29 17:38:13,808 CassandraDaemon.java:817 - Exception > encountered during startup > java.lang.IllegalStateException: Unable to contact any seeds! > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1601) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:699) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:635) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795), > ERROR [main] 2020-10-29 17:38:13,808 CassandraDaemon.java:817 - Exception > encountered during startup > java.lang.IllegalStateException: Unable to contact any seeds! > at > org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1601) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931) > at > org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:699) > at > org.apache.cassandra.service.StorageService.initServer(StorageService.java:635) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org