[
https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17134575#comment-17134575
]
Zhuqi Jin edited comment on CASSANDRA-15794 at 6/25/20, 2:53 PM:
-----------------------------------------------------------------
Hi, [~ifesdjeen].
I created patches for 3.0 and 3.11, according to the third method we discussed
earlier.
[^CASSANDRA-15794-branch-3.0.patch]
I'd like to move on to the second point and make sure we don't write any commit
log messages in such case. Could you help give me some pointers of where commit
logs are generated in this scenario?
was (Author: zhuqi1108):
Hi, [~ifesdjeen].
I created patches for 3.0 and 3.11, according to the third method we discussed
earlier.
[^CASSANDRA-15794-branch-3.0.patch]
I've attached the patches. Would you mind reviewing them?
And I'd like to move on to the second method. Could you please do me a favor?
We don‘t want to generate new commit logs before we hit the error in 4.x, so I
need to know when and where the commit logs were written.
> Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact
> tables in downgraded C* (3.11.4) introduces non-existent columns
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15794
> Project: Cassandra
> Issue Type: Bug
> Reporter: Zhuqi Jin
> Priority: Normal
> Attachments: CASSANDRA-15794-branch-3.0.patch,
> CASSANDRA-15794-branch-3.11.patch
>
>
> We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the
> following problems.
> * We started a single 3.11.4 C* node.
> * We ran cassandra-stress like this
> {code:java}
> ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code}
> * We stopped this node, and started a C* node running C* compiled from trunk
> (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb)
> * New C* failed to start with the following error message
> {code:java}
> ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while
> loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245
> - Error while loading schema: java.lang.IllegalArgumentException: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception
> (java.lang.IllegalArgumentException) encountered during startup: Compact
> Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ...
> DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to
> migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520
> CassandraDaemon.java:792 - Exception encountered during
> startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in
> Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE`
> command supplied in 3.x/3.11 Cassandra in order to migrate off Compact
> Storage. at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874)
> at
> org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862)
> at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at
> org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code}
> * We stopped the trunk version C* and started the 3.11.4 version C*.
> * 3.11.4 C* failed to start with the following error messages:
> {code:java}
> Exception (java.lang.IllegalStateException) encountered during startup:
> Unknown commitlog version 7Exception (java.lang.IllegalStateException)
> encountered during startup: Unknown commitlog version 7ERROR [main]
> 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered
> during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at
> org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167)
> ~[main/:na] at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148)
> ~[main/:na] at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
> [main/:na] at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
> [main/:na]
> {code}
> We manually deleted all new version commit logs and it worked. However, it's
> not obvious whether manually deleting these commit logs are safe.
> * And when we ran the "ALTER ... DROP COMPACT STORAGE" command on old
> version node as the error message suggested, non-existent columns appeared
> with null value.
> original table:
> {code:java}
> key | C0
> | C1
> | C2
> | C3
> | C4
> ------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------
> 0x36384b30374f33314c30 |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986
> 0x30374b37384e364c3531 |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb{code}
> After dropping compact storage:
> {code:java}
> key | column1 | C0
> | C1
> | C2
> | C3
> | C4
> | value
> ------------------------+---------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+------------------------------------------------------------------------+-------
> 0x36384b30374f33314c30 | null |
> 0x3ab9b990a5dbbe3dbf0354d35bc32122ab09f963a9a29b04434fac1ab16af8992ed0 |
> 0xdd9f9d6c207f3720d01cc2f9ab4e83a5c59226c779de8cf79cfe8f1a6c5a092fbee4 |
> 0x9116a0bb80af7f73d803ceeb78f8aada9555e78936b26678202f6fb1f4993df92089 |
> 0xbde6fe3df138b0428bcec8ad4ab0dcf256b5bd57c2b6ac42523913522d78f710ee98 |
> 0x63f1cac68e33a7375a8370fbf300dcc70f76052c3ce84588f3b7d69ef6bc470db986 | null
> 0x30374b37384e364c3531 | null |
> 0xce7990de95e1516101cbbd6ca3bdc2819e799c8f9b1bfd1b08aa1d1edf09dd409b7d |
> 0xc99b2076286ee4d4be742508653ed1178fb04192ae192d31745235e57dead6bf7f45 |
> 0xb492df82f1f2055af30694f135d3c99b0eac4e8d7d4d8e8b2d8ce49a9a3e50e3c63c |
> 0xc42bcb9b1a215a8d9629887bee918437fd580f0d15c48e1402fe11f6caab069e95aa |
> 0x329f193b16024ea72ace70571848e56b36496a05896454d13e1696c5c21053b5bcbb | null
> {code}
> * When we ran stress tool to read, IOExceptions were thrown, while it
> worked well on old version nodes
> * The error message is shown below:
> {code:java}
> java.io.IOException: Operation x0 on key(s) [4d32304c33314e374c30]: Data
> returned was not validatedjava.io.IOException: Operation x0 on key(s)
> [4d32304c33314e374c30]: Data returned was not validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)java.io.IOException:
> Operation x0 on key(s) [4d304f334b3432353231]: Data returned was not
> validated
> at org.apache.cassandra.stress.Operation.error(Operation.java:127) at
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:105) at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:91)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:99)
> at
> org.apache.cassandra.stress.operations.predefined.CqlOperation.run(CqlOperation.java:248)
> at
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:466)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
