[
https://issues.apache.org/jira/browse/CASSANDRA-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerome BAROTIN updated CASSANDRA-21729:
---------------------------------------
Description:
After a successful migration from 4 to 5, some exceptions happened executing
repair or verify :
{code:java}
Corrupted sstable. Invalid flags found deserializing DeletionTime:
10100000{code}
Launching scrub is not working as well :
{code:java}
INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64 -
Scrub of
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
partitions dropped
WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74 -
Unrecoverable error while scrubbing
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
cannot continue. The sstable will be marked for deletion. You can attempt
manual recovery from the pre-scrub snapshot. You can also run nodetool repair
to transfer the data from a healthy replica, if any.
WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79 -
Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
(stacktrace follows):
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
at
org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
at
org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
deserializing DeletionTime: 11001100
at
org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
... 13 common frames omitted
INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64 -
Scrubbing
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
(2.277TiB){code}
This seems linked to this change:
https://issues.apache.org/jira/browse/CASSANDRA-18648
This exception below is a bit hard for the node : With some tombstones with a
negative timestamp somewhere in an historical sstable that doesn't not fit
these news rules introduced in 5.0, all the sstable is lost.
In theory, it could be restored through repair, but in reality, these
corruptions could appear in several nodes and on very big SSTables.A solution
(enabled with a special flag maybe) to skip the tombstones on the new sstable
during the scrub will help Cassandra administrators arbitrate the choice of
avoiding these corrupted tombstones and keeping other data in the new scrubbed
SSTable or rebuilding the SSTable from the cluster ?
Also, to avoid this in the future, handling negative timestamps during
migration from Cassandra 4 to 5 might be a good idea.
I have checked in Cassandra 5.0.9, and it is still possible to set some
negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
-4548 WHERE key = "value1";. I don't understand what is the interest in
pushing a negative timestamp. It shouldn't be possible, cause it creates
corruptions.
was:
After a successful migration from 4 to 5, some exceptions happened executing
repair or verify :
{code:java}
Corrupted sstable. Invalid flags found deserializing DeletionTime:
10100000{code}
Launching scrub is not working as well :
{code:java}
INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64 -
Scrub of
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
partitions dropped
WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74 -
Unrecoverable error while scrubbing
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
cannot continue. The sstable will be marked for deletion. You can attempt
manual recovery from the pre-scrub snapshot. You can also run nodetool repair
to transfer the data from a healthy replica, if any.
WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79 -
Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
(stacktrace follows):
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
at
org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
at
org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
at
org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
at
org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
at
org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
deserializing DeletionTime: 11001100
at
org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
... 13 common frames omitted
INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64 -
Scrubbing
BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
(2.277TiB){code}
This seems linked to this change:
https://issues.apache.org/jira/browse/CASSANDRA-18648
This exception below is a bit hard for the node : With some tombstones with a
negative timestamp somewhere in an historical sstable that doesn't not fit
these news rules introduced in 5.0, all the sstable is lost.
In theory, it could be restored through repair, but in reality, these
corruptions could appear in several nodes and on very big SSTables.A solution
(enabled with a special flag maybe) to skip the tombstones on the new sstable
during the scrub will help Cassandra administrators arbitrate the choice of
avoiding these corrupted tombstones and keeping other data in the new scrubbed
SSTable or rebuilding the SSTable from the cluster ?
Also, to avoid this in the future, handling negative timestamps during
migration from Cassandra 4 to 5 might be a good idea.
To finished, I check in Cassandra 5.0.9, and is it still possible to set some
negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
-4548 WHERE key = "value1";. I don't understand what is the interest in
pushing a negative timestamp. It shouldn't be possible, cause it creates
corruptions.
> Negative timestamps present in historical clusters create unsolvable
> corruptions
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21729
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jerome BAROTIN
> Priority: Normal
>
> After a successful migration from 4 to 5, some exceptions happened executing
> repair or verify :
> {code:java}
> Corrupted sstable. Invalid flags found deserializing DeletionTime:
> 10100000{code}
>
>
> Launching scrub is not working as well :
> {code:java}
> INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64
> - Scrub of
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
> partitions dropped
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74
> - Unrecoverable error while scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
> cannot continue. The sstable will be marked for deletion. You can attempt
> manual recovery from the pre-scrub snapshot. You can also run nodetool repair
> to transfer the data from a healthy replica, if any.
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79
> - Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
> (stacktrace follows):
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
> at
> org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
> deserializing DeletionTime: 11001100
> at
> org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
> ... 13 common frames omitted
> INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64
> - Scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> (2.277TiB){code}
> This seems linked to this change:
> https://issues.apache.org/jira/browse/CASSANDRA-18648
>
> This exception below is a bit hard for the node : With some tombstones with a
> negative timestamp somewhere in an historical sstable that doesn't not fit
> these news rules introduced in 5.0, all the sstable is lost.
> In theory, it could be restored through repair, but in reality, these
> corruptions could appear in several nodes and on very big SSTables.A solution
> (enabled with a special flag maybe) to skip the tombstones on the new sstable
> during the scrub will help Cassandra administrators arbitrate the choice of
> avoiding these corrupted tombstones and keeping other data in the new
> scrubbed SSTable or rebuilding the SSTable from the cluster ?
> Also, to avoid this in the future, handling negative timestamps during
> migration from Cassandra 4 to 5 might be a good idea.
> I have checked in Cassandra 5.0.9, and it is still possible to set some
> negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
> -4548 WHERE key = "value1";. I don't understand what is the interest in
> pushing a negative timestamp. It shouldn't be possible, cause it creates
> corruptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
