Nadav Har'El created CASSANDRA-21733:
----------------------------------------
Summary: Reads fail after ALTER TABLE DROP ... USING TIMESTAMP
older than the column's data
Key: CASSANDRA-21733
URL: https://issues.apache.org/jira/browse/CASSANDRA-21733
Project: Apache Cassandra
Issue Type: Bug
Components: CQL/Interpreter, Legacy/Local Write-Read Paths
Reporter: Nadav Har'El
*Affects versions:*{color:#616161} 4.1.12, 5.0.9, 6.0-alpha2 (tested on the
official Docker images); probably all versions since 4.0 or earlier{color}
When a regular column is dropped with {{{}ALTER TABLE ... DROP col USING
TIMESTAMP t{}}}, and the column still holds cells with timestamps newer than
{{{}t{}}}, reads of that table fail. The coordinator returns a ReadFailure to
the client, and the replica logs an IllegalStateException while serializing the
read response.
Reproducer (single node, cqlsh):
{code:sql}
CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
CREATE TABLE ks.t (a int PRIMARY KEY, c int, f int);
INSERT INTO ks.t (a, c, f) VALUES (1, 2, 3);
ALTER TABLE ks.t DROP f USING TIMESTAMP 20000;
SELECT * FROM ks.t;
{code}
The {{SELECT}} fails:
{noformat}
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read]
message="Operation failed - received 0 responses and 1 failures: UNKNOWN from
/127.0.0.1:7000"
{noformat}
and the node logs:
{noformat}
java.lang.RuntimeException: java.lang.IllegalStateException: [c, f] is not a
subset of [c]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3159)
...
Caused by: java.lang.IllegalStateException: [c, f] is not a subset of [c]
at org.apache.cassandra.db.Columns$Serializer.encodeBitmap(Columns.java:618)
at org.apache.cassandra.db.Columns$Serializer.serializeSubset(Columns.java:548)
at
org.apache.cassandra.db.rows.UnfilteredSerializer.serializeRowBody(UnfilteredSerializer.java:242)
...
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:262)
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:453)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2756)
{noformat}
(Stack trace from 6.0-alpha2; 4.1 and 5.0 fail the same way.)
Observations:
* The same sequence without {{USING TIMESTAMP}} (so the drop time is newer
than the data) works correctly.
* The INSERT doesn't need an explicit timestamp. Any data newer than the drop
timestamp triggers the bug. In practice that can happen with clock skew,
client-provided timestamps, or a schema restored with explicit {{DROP ... USING
TIMESTAMP}} statements, like the ones DESCRIBE and snapshots' schema.cql emit
for dropped columns.
* The table doesn't need another regular column. With {{{}CREATE TABLE ks.t (a
int PRIMARY KEY, f int){}}}, the error becomes {{{}[f] is not a subset of
[]{}}}.
* Range scans ({{{}SELECT * FROM ks.t{}}}) fail for every table we tried.
Single-partition reads ({{{}WHERE a = 1{}}}) also fail when the table has a
clustering key, e.g. {{{}PRIMARY KEY (a, b){}}}, but succeed in the example
above without one.
* The error is in serializing the read response. The row still contains the
dropped column {{{}f{}}}, apparently because its cells are newer than the drop
time, but the serialization header's column set doesn't include it.
*Why the existing unit tests don't catch this:*
The unit tests that drop a column {{USING TIMESTAMP}} older than its data don't
detect this bug:
* {{{}AlterTest.testDropWithTimestamp{}}}, {{testDropStaticWithTimestamp}} and
{{testDropMultipleWithTimestamp}} re-ADD the dropped column before reading the
table, so they never read a table whose dropped column still has newer data.
* {{CreateLikeTest.testCopyAfterAlterTable}} does exactly that: it runs
{{ALTER TABLE ... DROP f USING TIMESTAMP 20000}} after writing {{f}} with
current timestamps, and then reads the table. It passes because CQLTester's
{{execute()}} runs queries through {{{}QueryProcessor.executeInternal(){}}},
which reads locally without building a serialized {{{}ReadResponse{}}}, and
that serialization is where the bug is. The same test fails when its statements
are sent through the native protocol (e.g. with {{{}executeNet(){}}}). We found
this while translating CreateLikeTest to ScyllaDB's Python-based CQL tests,
which run against a real Cassandra node through the Python driver.
Related: CASSANDRA-21607 reports a similar scenario, a dropped non-frozen
collection that still holds cells newer than the drop time, which causes a
NullPointerException in reads and compaction. This issue is the same situation
for a simple (non-collection) column. It fails differently, in read-response
serialization, but may have the same root cause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
