[
https://issues.apache.org/jira/browse/CASSANDRA-12653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818621#comment-15818621
]
Stefan Podkowinski commented on CASSANDRA-12653:
------------------------------------------------
The firstSynSendAt value is only set while sending the first syn during regular
gossip. All ACKs will be ignored that have been received and queued before that
point. Please keep in mind that the first SYN is send by GossipTasks by using
delayed execution, so it's not enough to simply check if the Gossiper has
already been started.
You are correct in pointing out that the timestamp of the incoming ACK will be
a local nanoTime() value. In-flight ACKs triggered during the shadow round will
not be filtered by the timestamp check unless already deserialized. But I'd
prefer comparing local timestamps anyways, as they will not require some kind
of clock skew tolerance, which would be hard to come up with given the kind of
races we like to address. Also this shouldn't be a problem as, like you
mentioned, the current handling of empty shadow round ACK replies should be
safe. But it's not a very clean solution and I'd rather be able to either have
a causal relationship to the corresponding SYN/Gossip round or even a custom
ShadowGossipMessage type. But this is probably better taken care of in e.g.
CASSANDRA-12345 instead of this bug fix.
> In-flight shadow round requests
> -------------------------------
>
> Key: CASSANDRA-12653
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12653
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: Stefan Podkowinski
> Assignee: Stefan Podkowinski
> Priority: Minor
> Attachments: 12653-2.2.patch, 12653-3.0.patch, 12653-trunk.patch
>
>
> Bootstrapping or replacing a node in the cluster requires to gather and check
> some host IDs or tokens by doing a gossip "shadow round" once before joining
> the cluster. This is done by sending a gossip SYN to all seeds until we
> receive a response with the cluster state, from where we can move on in the
> bootstrap process. Receiving a response will call the shadow round done and
> calls {{Gossiper.resetEndpointStateMap}} for cleaning up the received state
> again.
> The issue here is that at this point there might be other in-flight requests
> and it's very likely that shadow round responses from other seeds will be
> received afterwards, while the current state of the bootstrap process doesn't
> expect this to happen (e.g. gossiper may or may not be enabled).
> One side effect will be that MigrationTasks are spawned for each shadow round
> reply except the first. Tasks might or might not execute based on whether at
> execution time {{Gossiper.resetEndpointStateMap}} had been called, which
> effects the outcome of {{FailureDetector.instance.isAlive(endpoint))}} at
> start of the task. You'll see error log messages such as follows when this
> happend:
> {noformat}
> INFO [SharedPool-Worker-1] 2016-09-08 08:36:39,255 Gossiper.java:993 -
> InetAddress /xx.xx.xx.xx is now UP
> ERROR [MigrationStage:1] 2016-09-08 08:36:39,255 FailureDetector.java:223
> - unknown endpoint /xx.xx.xx.xx
> {noformat}
> Although is isn't pretty, I currently don't see any serious harm from this,
> but it would be good to get a second opinion (feel free to close as "wont
> fix").
> /cc [~Stefania] [~thobbs]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
