[
https://issues.apache.org/jira/browse/CASSANDRA-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124220#comment-18124220
]
Paulo Motta commented on CASSANDRA-21729:
-----------------------------------------
{quote}[~paulo], what could be an explanation for having negative timestamps if
one never does DELETE .. USING with negative timestamps (this is what I think
original report stated)?
{quote}
It's not clear what causes it but I believe this issue will most likely happen
when using client-side timestamps (or USING TIMESTAMP clause) and providing a
negative timestamp either due to bug like buffer overflow or custom application
logic. When not using client-side timestamps it could be a a time source
anomaly in the coordinator emitting negative timestamps. Regarding deletions,
use of collections can also cause tombstones. It would be worth checking with
affected users if they are using collections and/or client-side timestamps. And
the earliest version the cluster was on to rule out legacy corruption.
{quote}{quote} Could anything else lead to the same situation, e.g. reader
thinks that it sees DeletionTime record, but it is some other type of record (I
know nothing about how things are encoded, just trying to imagine all possible
scenarios)?
{quote}{quote}
It may be possible but I think it would most likely be related to DeletionTime
encoding since it happens in both BTI and BIG formats.
{quote}99.22% success rate with the proposed fix is not super impressive. Would
it be possible to improve by trying to read ahead and maybe figuring out
whether the next bytes belong to DeletionTime or something else?
{quote}
I think that the 0.78% negative timestamp range are already corrupted and may
cause CorruptedSSTableException or fail silently. I think they are unlikely to
happen because the affected range is close to the minimum possible negative
date which would be unusual for a timestamp. Alternatively this could be
bitflip indicating a prior corruption which was previously tolerated by 4.1 but
not by 5.0. It would be good to check if these are rare occurrences or happened
on many sstables.
> Negative timestamps present in historical clusters create unsolvable
> corruptions
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21729
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jerome BAROTIN
> Priority: Normal
>
> After a successful migration from 4 to 5, some exceptions happened executing
> repair or verify :
> {code:java}
> Corrupted sstable. Invalid flags found deserializing DeletionTime:
> 10100000{code}
>
>
> Launching scrub is not working as well :
> {code:java}
> INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64
> - Scrub of
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
> partitions dropped
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74
> - Unrecoverable error while scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
> cannot continue. The sstable will be marked for deletion. You can attempt
> manual recovery from the pre-scrub snapshot. You can also run nodetool repair
> to transfer the data from a healthy replica, if any.
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79
> - Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
> (stacktrace follows):
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
> at
> org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
> deserializing DeletionTime: 11001100
> at
> org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
> ... 13 common frames omitted
> INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64
> - Scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> (2.277TiB){code}
> This seems linked to this change:
> https://issues.apache.org/jira/browse/CASSANDRA-18648
>
> This exception below is a bit hard for the node : With some tombstones with a
> negative timestamp somewhere in an historical sstable that doesn't not fit
> these news rules introduced in 5.0, all the sstable is lost.
> In theory, it could be restored through repair, but in reality, these
> corruptions could appear in several nodes and on very big SSTables.A solution
> (enabled with a special flag maybe) to skip the tombstones on the new sstable
> during the scrub will help Cassandra administrators arbitrate the choice of
> avoiding these corrupted tombstones and keeping other data in the new
> scrubbed SSTable or rebuilding the SSTable from the cluster ?
> Also, to avoid this in the future, handling negative timestamps during
> migration from Cassandra 4 to 5 might be a good idea.
> I have checked in Cassandra 5.0.9, and it is still possible to set some
> negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
> -4548 WHERE key = "value1";. I don't understand what is the interest in
> pushing a negative timestamp. It shouldn't be possible, cause it creates
> corruptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
