[ https://issues.apache.org/jira/browse/CASSANDRA-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675737#comment-17675737 ]
Paulo Motta commented on CASSANDRA-16418: ----------------------------------------- In order to check the tests were reliably reproducing the issue on Lindsey's [branch|https://github.com/apache/cassandra/pull/2061] I commented out the following excerpt: {noformat} InetAddressAndPort localAddress = FBUtilities.getBroadcastAddressAndPort(); Integer pendingRangesCount = tokenMetadata.getPendingRanges(keyspaceName, localAddress).size(); if (pendingRangesCount > 0) { throw new RuntimeException("Node is involved in cluster membership changes. Not safe to run cleanup."); } {noformat} And expected both [testCleanupFailsDuringOngoingDecommission|https://github.com/apache/cassandra/pull/2061/files#diff-68d2cd75caa0e4091c7206717116594bdcb0aab38f72f6d6afa44eac60466e13R41] and [testCleanupFailsDuringOngoingBootstrap|https://github.com/apache/cassandra/pull/2061/files#diff-68d2cd75caa0e4091c7206717116594bdcb0aab38f72f6d6afa44eac60466e13R85] to fail. Even though the tests failed most of the time, sometimes the tests passed so data was not being wrongly cleaned up as expected. The reason for this is that these tests require that the cleanup is executed between the sstables are transferred by streaming and the ring membership operation is finished. There is a small chance cleanup is not executed within this window so the issue will not reproduce, especially if we run this test on faster hardware. I took a slightly different testing approach on [this commit|https://github.com/pauloricardomg/cassandra/blob/702f77d247893a51461823268ad6a20cd6c1a021/test/distributed/org/apache/cassandra/distributed/test/ring/CleanupFailureTest.java#L40] that inserts data while a node is bootstrapping or decommissioning and checks the data is present after a cleanup is run. This was able to reliably reproduce the issue when the excerpt above is commented out. The updated test is more deterministic because we don't depend on streaming nor timing. Furthermore this makes the test faster since we don't need so many rows to reproduce the issue, which is needed with the streaming approach. A nice benefit of this approach is that since we only run cleanup a single time while the node is bootstrapping/decommissioning, we're able to [verify that the cleanup fails with the expected error message|https://github.com/pauloricardomg/cassandra/blob/702f77d247893a51461823268ad6a20cd6c1a021/test/distributed/org/apache/cassandra/distributed/test/ring/CleanupFailureTest.java#L105]. > Unsafe to run nodetool cleanup during bootstrap or decommission > --------------------------------------------------------------- > > Key: CASSANDRA-16418 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16418 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Bootstrap and Decommission > Reporter: James Baker > Assignee: Lindsey Zurovchak > Priority: Normal > Fix For: 4.0.x > > Time Spent: 50m > Remaining Estimate: 0h > > What we expected: Running a cleanup is a safe operation; the result of > running a query after a cleanup should be the same as the result of running a > query before a cleanup. > What actually happened: We ran a cleanup during a decommission. All the > streamed data was silently deleted, the bootstrap did not fail, the cluster's > data after the decommission was very different to the state before. > Why: Cleanups do not take into account pending ranges and so the cleanup > thought that all the data that had just been streamed was redundant and so > deleted it. We think that this is symmetric with bootstraps, though have not > verified. > Not sure if this is technically a bug but it was very surprising (and > seemingly undocumented) behaviour. > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org