[jira] [Created] (CASSANDRA-16257) When upgrading from 2.1.0 to 2.2.0, UnknownColumnFamilyException encountered on upgraded nodes
Zhuqi Jin created CASSANDRA-16257: - Summary: When upgrading from 2.1.0 to 2.2.0, UnknownColumnFamilyException encountered on upgraded nodes Key: CASSANDRA-16257 URL: https://issues.apache.org/jira/browse/CASSANDRA-16257 Project: Cassandra Issue Type: Bug Reporter: Zhuqi Jin Both using 2.1.0 node as seed and using 2.2.0 node as seed will fail with this similar error: {code:java} org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=49a44701-d6bc-11ea-848f-4766f428c026 (when using 2.1.0 node as seed) org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=727d9450-d6c7-11ea-9370-4766f428c026 (when using 2.2.0 node as seed)at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:165) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:147) ~[apache-cassandra-2.1.0.jar:2.1.0]at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.0.jar:2.1.0]{code} This seems to originate from the difference in the settings in the stress tests, and is not a data format bug. The field `cfId` is a UUID string generated from another two fields: `ksName` and `cfName`. These two names are not the interior settings in the cluster, instead they come from user specification. In 2.1.0, the stress test configures these two names to be “Keyspace1” and “Standard1”. See [https://github.com/apache/cassandra/blob/cassandra-2.1.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L84] . Here is my log: {code:java} WARN [MigrationStage:1] 2020-08-05 03:17:20,514 Schema.java:330 - Schema.load: Adding org.apache.cassandra.config.CFMetaData@6299796f…..., key = (Keyspace1,Standard1), val = 49a44701-d6bc-11ea-848f-4766f428c026 {code} In 2.2.0, the stress test configures these two names to be “keyspace1” and “standard1” (note that the names are case-sensitive). See https://github.com/apache/cassandra/blob/cassandra-2.2.0/tools/stress/src/org/apache/cassandra/stress/settings/SettingsSchema.java#L225 . Here is my log: {code:java} WARN [MigrationStage:1] 2020-08-05 02:57:49,680 Schema.java:350 - Schema.load: Adding org.apache.cassandra.config.CFMetaData@719822c2……..., key = (keyspace1,standard1), val = 727d9450-d6c7-11ea-9370-4766f428c026 {code} When a 2.2.0 message reaches a 2.1.0 node, or vice versa, the receiver will look up the `cfId` field in a map. Since the key is not present in its own map, the look up will fail with the above message. Unluckily this problem is quite hard to fix, since many other places also depend on the name. So I can not overcome this problem and find actual data format bugs between these two versions currently. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134575#comment-17134575 ] Zhuqi Jin edited comment on CASSANDRA-15794 at 6/25/20, 2:53 PM: - Hi, [~ifesdjeen]. I created patches for 3.0 and 3.11, according to the third method we discussed earlier. [^CASSANDRA-15794-branch-3.0.patch] I'd like to move on to the second point and make sure we don't write any commit log messages in such case. Could you help give me some pointers of where commit logs are generated in this scenario? was (Author: zhuqi1108): Hi, [~ifesdjeen]. I created patches for 3.0 and 3.11, according to the third method we discussed earlier. [^CASSANDRA-15794-branch-3.0.patch] I've attached the patches. Would you mind reviewing them? And I'd like to move on to the second method. Could you please do me a favor? We don‘t want to generate new commit logs before we hit the error in 4.x, so I need to know when and where the commit logs were written. > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > - > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug >Reporter: Zhuqi Jin >Priority: Normal > Attachments: CASSANDRA-15794-branch-3.0.patch, > CASSANDRA-15794-branch-3.11.patch > > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} > * We stopped the
[jira] [Comment Edited] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134575#comment-17134575 ] Zhuqi Jin edited comment on CASSANDRA-15794 at 6/16/20, 7:57 PM: - Hi, [~ifesdjeen]. I created patches for 3.0 and 3.11, according to the third method we discussed earlier. [^CASSANDRA-15794-branch-3.0.patch] I've attached the patches. Would you mind reviewing them? And I'd like to move on to the second method. Could you please do me a favor? We don‘t want to generate new commit logs before we hit the error in 4.x, so I need to know when and where the commit logs were written. was (Author: zhuqi1108): Hi, [~ifesdjeen]. I created patches for 3.0 and 3.11, according to the third method we discussed earlier. [^CASSANDRA-15794-branch-3.0.patch] I've attached the patches. Would you mind reviewing them? > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > - > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug >Reporter: Zhuqi Jin >Priority: Normal > Attachments: CASSANDRA-15794-branch-3.0.patch, > CASSANDRA-15794-branch-3.11.patch > > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} > * We stopped the trunk version C* and started the 3.11.4 version C*. > * 3.11.4 C* failed to start with the following error messages: > {code:java} >
[jira] [Commented] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134575#comment-17134575 ] Zhuqi Jin commented on CASSANDRA-15794: --- Hi, [~ifesdjeen]. I created patches for 3.0 and 3.11, according to the third method we discussed earlier. [^CASSANDRA-15794-branch-3.0.patch] I've attached the patches. Would you mind reviewing them? > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > - > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug >Reporter: Zhuqi Jin >Priority: Normal > Attachments: CASSANDRA-15794-branch-3.0.patch, > CASSANDRA-15794-branch-3.11.patch > > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} > * We stopped the trunk version C* and started the 3.11.4 version C*. > * 3.11.4 C* failed to start with the following error messages: > {code:java} > Exception (java.lang.IllegalStateException) encountered during startup: > Unknown commitlog version 7Exception (java.lang.IllegalStateException) > encountered during startup: Unknown commitlog version 7ERROR [main] > 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered > during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at > org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) > ~[main/:na] at >
[jira] [Updated] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuqi Jin updated CASSANDRA-15794: -- Attachment: CASSANDRA-15794-branch-3.11.patch CASSANDRA-15794-branch-3.0.patch > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > - > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug >Reporter: Zhuqi Jin >Priority: Normal > Attachments: CASSANDRA-15794-branch-3.0.patch, > CASSANDRA-15794-branch-3.11.patch > > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} > * We stopped the trunk version C* and started the 3.11.4 version C*. > * 3.11.4 C* failed to start with the following error messages: > {code:java} > Exception (java.lang.IllegalStateException) encountered during startup: > Unknown commitlog version 7Exception (java.lang.IllegalStateException) > encountered during startup: Unknown commitlog version 7ERROR [main] > 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered > during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at > org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) > ~[main/:na] at > org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) > ~[main/:na] at >
[jira] [Commented] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110435#comment-17110435 ] Zhuqi Jin commented on CASSANDRA-15794: --- Thanks for you prompt reply. I still think that users who have already used compact storage should be provided with a solution (even assuming CASSANDRA-15811 is fixed). Otherwise, users may be confused, especially when they roll it back but cannot start 3.11. For example, 1) Instead of directly block cassandra startup, is it possible to correctly drop existing compact storage in 4.x (with the fix from CASSANDRA-15811)? 2) Or, if we stick to ask users to downgrade to 3.x and drop compact storage, can we not generate new commit log before we hit the error in 4.x, so that we avoid blocking 3.x from starting? 3) Or, block or warn users when they try to introduce compact storage in 3.x? Throw a warning telling them that compact storage is deprecated in 3.x and will cause problem upgrading to 4.x, or simply block them from doing that, or at least modify the manual of 3.X, indicating that you should not use compact storage. If you think any of these is reasonable, I can provide help. > Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact > tables in downgraded C* (3.11.4) introduces non-existent columns > - > > Key: CASSANDRA-15794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 > Project: Cassandra > Issue Type: Bug >Reporter: Zhuqi Jin >Priority: Normal > > We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the > following problems. > * We started a single 3.11.4 C* node. > * We ran cassandra-stress like this > {code:java} > ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} > * We stopped this node, and started a C* node running C* compiled from trunk > (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) > * New C* failed to start with the following error message > {code:java} > ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while > loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 > - Error while loading schema: java.lang.IllegalArgumentException: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception > (java.lang.IllegalArgumentException) encountered during startup: Compact > Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... > DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to > migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 > CassandraDaemon.java:792 - Exception encountered during > startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in > Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` > command supplied in 3.x/3.11 Cassandra in order to migrate off Compact > Storage. at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) > at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at > org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) > at >
[jira] [Updated] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuqi Jin updated CASSANDRA-15794: -- Description: We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the following problems. * We started a single 3.11.4 C* node. * We ran cassandra-stress like this {code:java} ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} * We stopped this node, and started a C* node running C* compiled from trunk (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) * New C* failed to start with the following error message {code:java} ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: java.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception (java.lang.IllegalArgumentException) encountered during startup: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 CassandraDaemon.java:792 - Exception encountered during startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} * We stopped the trunk version C* and started the 3.11.4 version C*. * 3.11.4 C* failed to start with the following error messages: {code:java} Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7ERROR [main] 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [main/:na] {code} We manually deleted all new version commit logs and it worked. However, it's
[jira] [Updated] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & dropping compact tables in downgraded C* (3.11.4) introduces non-existent columns
[ https://issues.apache.org/jira/browse/CASSANDRA-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhuqi Jin updated CASSANDRA-15794: -- Discovered By: User Report Severity: Normal Since Version: 3.11.4 Description: We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the following problems. * We started a single 3.11.4 C* node. * We ran cassandra-stress like this {code:java} ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} * We stopped this node, and started a C* node running C* compiled from trunk (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) * New C* failed to start with the following error message {code:java} ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: java.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception (java.lang.IllegalArgumentException) encountered during startup: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 CassandraDaemon.java:792 - Exception encountered during startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} * We stopped the trunk version C* and started the 3.11.4 version C*. * 3.11.4 C* failed to start with the following error messages: {code:java} Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7ERROR [main] 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [main/:na]
[jira] [Created] (CASSANDRA-15794) Upgraded C* (4.x) fail to start because of Compact Tables & downgraded C* (3.11.4) fail to start because of commit log
Zhuqi Jin created CASSANDRA-15794: - Summary: Upgraded C* (4.x) fail to start because of Compact Tables & downgraded C* (3.11.4) fail to start because of commit log Key: CASSANDRA-15794 URL: https://issues.apache.org/jira/browse/CASSANDRA-15794 Project: Cassandra Issue Type: Bug Reporter: Zhuqi Jin We tried to test upgrading a 3.11.4 C* cluster to 4.x and run into the following problems. * We started a single 3.11.4 C* node. * We ran cassandra-stress like this {code:java} ./cassandra-stress write n = 30 -rate threads = 10 -node 172.17.0.2 {code} * We stopped this node, and started a C* node running C* compiled from trunk (git commit: e394dc0bb32f612a476269010930c617dd1ed3cb) * New C* failed to start with the following error message {code:java} ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: ERROR [main] 2020-05-07 00:58:18,503 CassandraDaemon.java:245 - Error while loading schema: java.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770)Exception (java.lang.IllegalArgumentException) encountered during startup: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage.ERROR [main] 2020-05-07 00:58:18,520 CassandraDaemon.java:792 - Exception encountered during startupjava.lang.IllegalArgumentException: Compact Tables are not allowed in Cassandra starting with 4.0 version. Use `ALTER ... DROP COMPACT STORAGE` command supplied in 3.x/3.11 Cassandra in order to migrate off Compact Storage. at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:965) at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:924) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:883) at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:874) at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:862) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:102) at org.apache.cassandra.schema.Schema.loadFromDisk(Schema.java:91) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:241) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:653) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:770){code} * We stopped the trunk version C* and started the 3.11.4 version C*. * 3.11.4 C* failed to start with the following error messages: {code:java} Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7Exception (java.lang.IllegalStateException) encountered during startup: Unknown commitlog version 7ERROR [main] 2020-05-07 01:13:26,798 CassandraDaemon.java:749 - Exception encountered during startupjava.lang.IllegalStateException: Unknown commitlog version 7 at org.apache.cassandra.db.commitlog.CommitLogDescriptor.getMessagingVersion(CommitLogDescriptor.java:227) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.shouldSkipSegmentId(CommitLogReader.java:276) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:220) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:132) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:137) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:167) ~[main/:na] at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:148) ~[main/:na] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:324) [main/:na] at