[
https://issues.apache.org/jira/browse/CASSANDRA-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124156#comment-18124156
]
Paulo Motta commented on CASSANDRA-21729:
-----------------------------------------
The affected sstables are not corrupt. They contain a negative
markedForDeleteAt, which the 5.0 DeletionTime encoding cannot represent.
This is not an upgrade problem. It reproduces on a fresh 5.0 cluster with a
single DELETE. Upgraded clusters are more likely to hit this because this issue
did not affect 4.0 when negative mfda are present, only after sstables are
upgraded to 5.0 format (big or bti).
h3. Cause
CASSANDRA-18648 changed the DeletionTime serialization. It put
markedForDeleteAt first and reused the top bit of the first byte to flag LIVE.
LIVE is one byte, 0x80. Any other deletion is 12 bytes. The patch states its
assumption in a comment: "mfda is a positive timestamp".
The reader does not test for the marker. It tests for the range:
{code:java}
int flags = in.readByte();
if ((flags & IS_LIVE_DELETION) != 0) // 0x80 or more
{
if ((flags & 0xFF) != IS_LIVE_DELETION) // not exactly 0x80
throw new IOException("Corrupted sstable. Invalid flags found
deserializing "
+ "DeletionTime: " + Integer.toBinaryString(flags
& 0xFF));
return LIVE;
}
{code}
A negative timestamp always sets that bit, so it writes without complaint and
throws on every read afterwards.
CASSANDRA-18676 added a check that would have caught this. CASSANDRA-18735
reverted it:
{code:java}
- return markedForDeleteAt >= 0 || isLive();
+ return true;
{code}
with the message "Contrary to expectations, we do permit negative timestamps".
The revert is correct on its own terms. Negative timestamps are legal CQL and
Attributes only validates that the bind value is a long.
The flag values reported in the exception are the first byte of the timestamp.
11111111 is 255, a small negative value such as -5. 10100000, 10100111 and
11001100 are 160, 167 and 204, large negative values.
h3. Reproduction
Single 5.0 node, storage_compatibility_mode NONE. No upgrade and no
pre-existing sstable required.
{code:sql}
CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
CREATE TABLE ks.t (k int PRIMARY KEY, v int);
INSERT INTO ks.t (k, v) VALUES (1, 1) USING TIMESTAMP 100;
DELETE FROM ks.t USING TIMESTAMP -5 WHERE k = 1;
SELECT * FROM ks.t WHERE k = 1; -- returns the row, nothing serialized yet
{code}
{noformat}
nodetool flush ks t
{noformat}
{code:sql}
SELECT * FROM ks.t WHERE k = 1; -- fails from here on
{code}
{noformat}
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
.../oa-1-big-Data.db
at
org.apache.cassandra.io.sstable.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:177)
at
org.apache.cassandra.io.sstable.format.big.SSTableIterator.<init>(SSTableIterator.java:53)
at
org.apache.cassandra.io.sstable.format.big.BigTableReader.rowIterator(BigTableReader.java:143)
...
Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
deserializing DeletionTime: 11111111
at
org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
at
org.apache.cassandra.io.sstable.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:119)
{noformat}
-5 does not supersede timestamp 100, so the deletion shadows nothing. The
partition becomes unreadable anyway.
The same sequence with sstable format bti fails identically, via
BtiTableReader.rowIterator. This is not BTI specific. The partition header is
written by SortedTablePartitionWriter.start() and read by
AbstractSSTableIterator, both shared. For identical input the Data.db payload
is byte for byte identical between oa and da.
h3. The data is recoverable
The writer emits all 8 bytes of markedForDeleteAt, sign included, and the
failed read does not modify the file. Decoding an affected partition header by
hand returns markedForDeleteAt=-5 and the correct localDeletionTime.
readBytesToMFDA already
reconstructs negative values correctly.
h3. Proposed fix, and what it does not fix
This is a partial fix so not sure it would be appropriate. It restores access
to the data behind the failures reported in this ticket. It does not make the
encoding sound, and a gap remains that can produce wrong query results with no
error.
Change the reader's test from "0x80 or more" to "exactly 0x80", in
deserialize(DataInputPlus), deserialize(ByteBuffer, int) and skip. All three
together, since skip steps over the record during index deserialization.
{code:java}
int flags = in.readByte();
if ((flags & 0xFF) == IS_LIVE_DELETION)
return LIVE;
// otherwise read the remaining 11 bytes
{code}
This is a strict widening. Bytes below 0x80 and exactly 0x80 behave as they do
today, only the case that currently throws changes. No sstable that reads today
would read differently, no new bytes are written, so mixed version clusters and
downgrades are unaffected. It covers leading bytes 0x81 to 0xFF, which is
99.22% of the negative timestamp space and every value reported in this ticket.
h3. The gap: leading byte exactly 0x80
A timestamp in [Long.MIN_VALUE, Long.MIN_VALUE + 2^56) has a leading byte of
exactly 0x80 and is indistinguishable from the LIVE marker. The reader consumes
1 byte of the 12 written and the rest of the partition is parsed 11 bytes out
of step. The reader change above does not alter this, because both readings are
legal at that point.
This case does not reliably fail, which is what makes it worse than the one
reported here. The outcome depends on the remaining bytes of the timestamp,
which the operator does not control:
{noformat}
ts=-9223372036854775807 throws, EOFException inside
Columns$Serializer.deserializeSubset
ts=-9200000000000000000 returns 0 rows, no exception
ts=-9151314442816847873 returns 0 rows, no exception
{noformat}
In the silent cases the rows are still in the file. Writing three rows with
distinctive values, deleting the partition at -9200000000000000000 and flushing:
{noformat}
before flush: 3 rows
after flush: 0 rows
CANARY1 present in Data.db at offset 37
CANARY2 present in Data.db at offset 63
CANARY3 present in Data.db at offset 89
Data.db size=97 statsMetadata rows=3 minTimestamp=-9200000000000000000
{noformat}
The correct answer is 3 rows, since -9200000000000000000 does not supersede the
rows' timestamp of 100 and the tombstone shadows nothing. The query returns 0.
Nothing detects this. There is no exception, no log line and no metric. Since
the value arrives in a client write replicated to every replica, all replicas
write the same bytes, lose their place identically and return empty. Digests
match, read repair has nothing to
reconcile and repair sees no difference between replicas. Only the sstable
metadata disagrees with the query.
> Negative timestamps present in historical clusters create unsolvable
> corruptions
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-21729
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21729
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jerome BAROTIN
> Priority: Normal
>
> After a successful migration from 4 to 5, some exceptions happened executing
> repair or verify :
> {code:java}
> Corrupted sstable. Invalid flags found deserializing DeletionTime:
> 10100000{code}
>
>
> Launching scrub is not working as well :
> {code:java}
> INFO [CompactionExecutor:4107] 2026-09-17T10:32:28,177 OutputHandler.java:64
> - Scrub of
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> complete: 59158928 partitions in new sstable and 0 empty (tombstoned)
> partitions dropped
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:74
> - Unrecoverable error while scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db').Scrubbing
> cannot continue. The sstable will be marked for deletion. You can attempt
> manual recovery from the pre-scrub snapshot. You can also run nodetool repair
> to transfer the data from a healthy replica, if any.
> WARN [CompactionExecutor:4107] 2026-09-17T10:32:27,675 OutputHandler.java:79
> - Error reading partition 20210525-247c9972-928d-4bd2-bf83-1b012b4a44ec
> (stacktrace follows):
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:77)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.getIterator(SortedTableScrubber.java:321)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.tryAppend(SortedTableScrubber.java:291)
> at
> org.apache.cassandra.io.sstable.format.bti.BtiTableScrubber.scrubInternal(BtiTableScrubber.java:187)
> at
> org.apache.cassandra.io.sstable.format.SortedTableScrubber.scrub(SortedTableScrubber.java:193)
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:1319)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.execute(CompactionManager.java:529)
> at
> org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:457)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: java.io.IOException: Corrupted sstable. Invalid flags found
> deserializing DeletionTime: 11001100
> at
> org.apache.cassandra.db.DeletionTime$Serializer.deserialize(DeletionTime.java:228)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.create(SSTableIdentityIterator.java:67)
> ... 13 common frames omitted
> INFO [CompactionExecutor:4107] 2026-09-16T20:36:05,313 OutputHandler.java:64
> - Scrubbing
> BtiTableReader:bti(path='/var/lib/cassandra/data/keyspace1/table1-49c3302007d111eb95f19517b13fd7f9/da-3h2f_1en8_438xc2sy59frg5f9i3-bti-Data.db')
> (2.277TiB){code}
> This seems linked to this change:
> https://issues.apache.org/jira/browse/CASSANDRA-18648
>
> This exception below is a bit hard for the node : With some tombstones with a
> negative timestamp somewhere in an historical sstable that doesn't not fit
> these news rules introduced in 5.0, all the sstable is lost.
> In theory, it could be restored through repair, but in reality, these
> corruptions could appear in several nodes and on very big SSTables.A solution
> (enabled with a special flag maybe) to skip the tombstones on the new sstable
> during the scrub will help Cassandra administrators arbitrate the choice of
> avoiding these corrupted tombstones and keeping other data in the new
> scrubbed SSTable or rebuilding the SSTable from the cluster ?
> Also, to avoid this in the future, handling negative timestamps during
> migration from Cassandra 4 to 5 might be a good idea.
> I have checked in Cassandra 5.0.9, and it is still possible to set some
> negative timestamps with such CQL queries: DELETE FROM table USING TIMESTAMP
> -4548 WHERE key = "value1";. I don't understand what is the interest in
> pushing a negative timestamp. It shouldn't be possible, cause it creates
> corruptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
