Re: Cassandra Token ownership split-brain (3.0.14)

[C. Scott Andreas]

Hi Jaydeep, Thanks for reaching out and for bumping this thread. This is probably not the answer you’re after, but mentioning as it may address the issue. C* 3.0.14 was released over five years ago, with many hundreds of important bug fixes landing since July 2017. These include fixes for issues that have affected gossip in the past which may be related to this issue. Note that 3.0.14 also is susceptible to several critical data loss bugs including C-14513 and C-14515. I’d strongly recommend upgrading to Cassandra 3.0.27 as a starting point. If this doesn’t resolve your issue, members of the community may be in a better position to help triage a bug report against a current release of the database. - Scott On Sep 6, 2022, at 5:13 PM, Jaydeep Chovatia <chovatia.jayd...@gmail.com> wrote:  If anyone has seen this issue and knows a fix, it would be a great help! Thanks in advance. Jaydeep On Fri, Sep 2, 2022 at 1:56 PM Jaydeep Chovatia <chovatia.jayd...@gmail.com> wrote: Hi, We are running a production Cassandra version (3.0.14) with 256 tokens v-node configuration. Occasionally, we see that different nodes show different ownership for the same key. Only a node restart corrects; otherwise, it continues to behave in a split-brain. Say, for example, NodeA nodetool getendpoints ks1 table1 10 - n1 - n2 - n3 NodeB nodetool getendpoints ks1 table1 10 - n1 - n2 - n5 If I restart NodeB, then it shows the correct ownership {n1,n2,n3}. The majority of the nodes in the ring show correct ownership {n1,n2,n3}, only a few show this issue, and restarting them solves the problem. To me, it seems I think Cassandra's Gossip cache and StorageService cache (TokenMetadata) are having some sort of cache coherence. Anyone has observed this behavior? Any help would be highly appreciated. Jaydeep