[ https://issues.apache.org/jira/browse/CASSANDRA-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leon Zaruvinsky updated CASSANDRA-15327: ---------------------------------------- Description: Hey, We've come across a scenario in production (noticed on Cassandra 2.2.14) where data that is deleted from Cassandra at consistency {{ALL}} can be resurrected. I've added a reproduction in a comment. If a {{delete}} is issued during a range movement (i.e. bootstrap, decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is finished, then the tombstones from the {{delete}} can be purged from the recipient node before the data is streamed. Once the move is complete, the data now exists on the recipient node without a tombstone. We noticed this because our bootstrapping time occasionally exceeds our configured gc_grace_seconds, so we lose the consistency guarantee. As an operator, it would be great to not have to worry about this edge case. I've attached a patch that we have tested and successfully used in production, and haven't noticed any ill effects. was: Hey, We've come across a scenario in production (noticed on Cassandra 2.2.14) where data that is deleted from Cassandra at consistency {{ALL}} can be resurrected. I've added a reproduction in a comment. If a {{delete}} is issued during a range movement (i.e. bootstrap, decommission, move), and {{gc_grace_seconds}} is surpassed before the stream is finished, then the tombstones from the {{delete}} can be purged from the recipient node before the data is streamed. Once the move is complete, the data now exists on the recipient node without a tombstone. We noticed this because our bootstrapping time occasionally exceeds our configured gc_grace_seconds, so we lose the consistency guarantee. As an operator, it would be great to not have to worry about this edge case. I've attached a patch that we have tested and successfully used in production, and haven't noticed any ill effects. > Deleted data can re-appear if range movement streaming time exceeds > gc_grace_seconds > ------------------------------------------------------------------------------------ > > Key: CASSANDRA-15327 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15327 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Bootstrap and Decommission > Reporter: Leon Zaruvinsky > Priority: Normal > Attachments: CASSANDRA-15327-2.1.txt > > > Hey, > We've come across a scenario in production (noticed on Cassandra 2.2.14) > where data that is deleted from Cassandra at consistency {{ALL}} can be > resurrected. I've added a reproduction in a comment. > If a {{delete}} is issued during a range movement (i.e. bootstrap, > decommission, move), and {{gc_grace_seconds}} is surpassed before the stream > is finished, then the tombstones from the {{delete}} can be purged from the > recipient node before the data is streamed. Once the move is complete, the > data now exists on the recipient node without a tombstone. > We noticed this because our bootstrapping time occasionally exceeds our > configured gc_grace_seconds, so we lose the consistency guarantee. As an > operator, it would be great to not have to worry about this edge case. > I've attached a patch that we have tested and successfully used in > production, and haven't noticed any ill effects. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org