[
https://issues.apache.org/jira/browse/CASSANDRA-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124962#comment-18124962
]
Jerome BAROTIN edited comment on CASSANDRA-21729 at 10/8/26 10:03 AM:
----------------------------------------------------------------------
Thanks very much, [~paulo] for this very clear analysis.
I'm trying to find a workaround. If we are assuming that negative timestamps
are not normal, do you think a patch that includes again the lines reverted in
CASSANDRA-18735 will help to bypass the IOException ?
I'm thinking about :
{code:java}
public boolean validate()
{
return markedForDeleteAt >= 0 || isLive();
}{code}
Then enabling {{corrupted_tombstone_strategy}} to {{warn}} will log the
negative tombstones "hidden" in a legacy cluster, this will help better
understand where they are coming from.
Do you think it's also necessary to apply your proposed fix (with the side
effect you explain) ?
In a more general way, maybe introducing a new configuration parameter
*disable_negative_timestamp_in_tombstone* might help solve the dilemma with the
CQL specification ?
was (Author: jbarotin):
Thanks very much, Paulo Motta for this very clear analysis.
I'm trying to find a workaround. If we are assuming that negative timestamps
are not normal, do you think a patch that includes again the lines reverted in
CASSANDRA-18735 will help to bypass the IOException ?
I'm thinking about :
{code:java}
public boolean validate()
{
return markedForDeleteAt >= 0 || isLive();
}{code}
Then enabling {{corrupted_tombstone_strategy}} to {{warn}} will log the
negative tombstones "hidden" in a legacy cluster, this will help better
understand where they are coming from.
Do you think it's also necessary to apply your proposed fix (with the side
effect you explain) ?
In a more general way, maybe introducing a new configuration parameter
*disable_negative_timestamp_in_tombstone* might help solve the dilemma with the
CQL specification ?
> Negative timestamps present in historical clusters create unsolvable
> corruptions
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21729
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jerome BAROTIN
> Priority: Normal
>
> After a successful migration from 4 to 5, some exceptions happened executing
> repair or verify :
> {code:java}
> Corrupted sstable. Invalid flags found deserializing DeletionTime:
> 10100000{code}
>
>
> Launching scrub is not working as well :
> {code:java}
> INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64
> - Scrub of
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
> partitions dropped
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74
> - Unrecoverable error while scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
> cannot continue. The sstable will be marked for deletion. You can attempt
> manual recovery from the pre-scrub snapshot. You can also run nodetool repair
> to transfer the data from a healthy replica, if any.
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79
> - Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
> (stacktrace follows):
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
> at
> org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
> deserializing DeletionTime: 11001100
> at
> org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
> ... 13 common frames omitted
> INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64
> - Scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> (2.277TiB){code}
> This seems linked to this change:
> https://issues.apache.org/jira/browse/CASSANDRA-18648
>
> This exception below is a bit hard for the node : With some tombstones with a
> negative timestamp somewhere in an historical sstable that doesn't not fit
> these news rules introduced in 5.0, all the sstable is lost.
> In theory, it could be restored through repair, but in reality, these
> corruptions could appear in several nodes and on very big SSTables.A solution
> (enabled with a special flag maybe) to skip the tombstones on the new sstable
> during the scrub will help Cassandra administrators arbitrate the choice of
> avoiding these corrupted tombstones and keeping other data in the new
> scrubbed SSTable or rebuilding the SSTable from the cluster ?
> Also, to avoid this in the future, handling negative timestamps during
> migration from Cassandra 4 to 5 might be a good idea.
> I have checked in Cassandra 5.0.9, and it is still possible to set some
> negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
> -4548 WHERE key = "value1";. I don't understand what is the interest in
> pushing a negative timestamp. It shouldn't be possible, cause it creates
> corruptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
