[
https://issues.apache.org/jira/browse/CASSANDRA-18934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782603#comment-17782603
]
Maxwell Guo edited comment on CASSANDRA-18934 at 11/3/23 3:28 PM:
------------------------------------------------------------------
I think any change that modifies the schema of the system table will cause the
risk of failure during downgrade. Not only the above case, the reason for this
problem is that between 4.1 and 5.0, I was the only one who changed the system
table:D.
The reason for this problem is that the higher version of the commitlog will
contain data from the new schema column (compaction_properties). However,
because the system table follows the schema structure of the lower version that
does not contain compaction_properties column such as 4.1, the data cannot be
played back and an error is reported. see this code [here
|https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L488]
I think we can make a simple fix for downgrade with commitlog's replay failure
. That may be add a flag to permit skip the replay of column data that is not
useful for older versions just at downgrade stage(By the way , do we have some
flag to show we are doing downgrade?[~dcapwell][~claude], I think we can use
the current process's version and the release version from system.local table
to verify this ). Because the data for the useless column (like
compaction_properties in 4.1) exists but is not used, so I think we shouldn't
need to play back the data either.
wdyt ? [~dcapwell][~e.dimitrova][~claude][~mmuzaf]，Looking forward to your
reply. I think I can fix this pretty quickly if everyone comes to a unified
conclusion.
was (Author: maxwellguo):
I think any change that modifies the schema of the system table will cause the
risk of failure during downgrade. Not only the above case, the reason for this
problem is that between 4.1 and 5.0, I was the only one who changed the system
table:D.
The reason for this problem is that the higher version of the commitlog will
contain data from the new schema column (compaction_properties). However,
because the system table follows the schema structure of the lower version that
does not contain compaction_properties column such as 4.1, the data cannot be
played back and an error is reported. see this code [here
|https://github.com/apache/cassandra/blob/cassandra-4.1/src/java/org/apache/cassandra/db/Columns.java#L488]
I think we can make a simple fix for downgrade with commitlog's replay failure
. That may be add a flag to permit skip the replay of column data that is not
useful for older versions just at downgrade stage(By the way , do we have some
flag to show we are doing downgrade?[~dcapwell][~claude] ). Because the data
for the useless column (like compaction_properties in 4.1) exists but is not
used, so I think we shouldn't need to play back the data either.
wdyt ? [~dcapwell][~e.dimitrova][~claude][~mmuzaf]，Looking forward to your
reply. I think I can fix this pretty quickly if everyone comes to a unified
conclusion.
> Downgrade to 4.1 fails due to schema changes
> --------------------------------------------
>
> Key: CASSANDRA-18934
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18934
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
> Reporter: David Capwell
> Assignee: Maxwell Guo
> Priority: Normal
> Fix For: 5.0-beta, 5.0.x, 5.x
>
>
> We are required to support 5.0 downgrading to 4.1 as a migration step, but we
> don’t have tests to show this is working… I wrote a quick test to make sure a
> change we needed in Accord wouldn’t block the downgrade and see that we fail
> right now.
> {code}
> ERROR 20:56:39 Exiting due to error while processing commit log during
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException:
> Unexpected error deserializing mutation; saved to
> /var/folders/h1/s_3p1x3s3hl0hltbpck67m0h0000gn/T/mutation4184214444767150092dat.
> This may be caused by replaying a mutation against a table with the same
> name but incompatible schema. Exception follows: java.lang.RuntimeException:
> Unknown column compaction_properties during deserialization
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readMutation(CommitLogReader.java:464)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readSection(CommitLogReader.java:397)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:244)
> at
> org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:147)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:191)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:223)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:204)
> {code}
> This was caused by a schema change in CASSANDRA-18061
> {code}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one
> * or more contributor license agreements. See the NOTICE file
> * distributed with this work for additional information
> * regarding copyright ownership. The ASF licenses this file
> * to you under the Apache License, Version 2.0 (the
> * "License"); you may not use this file except in compliance
> * with the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.cassandra.distributed.upgrade;
> import java.io.IOException;
> import java.io.File;
> import java.util.concurrent.atomic.AtomicBoolean;
> import org.junit.Test;
> import org.apache.cassandra.distributed.api.IUpgradeableInstance;
> public class DowngradeTest extends UpgradeTestBase
> {
> @Test
> public void test() throws Throwable
> {
> AtomicBoolean first = new AtomicBoolean(true);
> new TestCase()
> .nodes(1)
> .withConfig(c -> {
> if (first.compareAndSet(true, false))
> c.set("storage_compatibility_mode", "CASSANDRA_4");
> })
> .downgradeTo(v41)
> .setup(cluster -> {})
> // Uncomment if you want to test what happens after reading the commit log,
> which fails right now
> // .runBeforeNodeRestart((cluster, nodeId) -> {
> // IUpgradeableInstance inst = cluster.get(nodeId);
> // File f = new File((String)
> inst.config().get("commitlog_directory"));
> // deleteRecursive(f);
> // })
> .runAfterClusterUpgrade(cluster -> {})
> .run();
> }
> private void deleteRecursive(File f)
> {
> if (f.isDirectory())
> {
> File[] children = f.listFiles();
> if (children != null)
> {
> for (File c : children)
> deleteRecursive(c);
> }
> }
> f.delete();
> }
> }
> {code}
> {code}
> diff --git
> a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
>
> b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
> index 5ee8780204..b4111e3b44 100644
> ---
> a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
> +++
> b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTestBase.java
> @@ -226,6 +226,12 @@ public class UpgradeTestBase extends DistributedTestBase
> return this;
> }
> + public TestCase downgradeTo(Semver to)
> + {
> + upgrade.add(new TestVersions(versions.getLatest(CURRENT),
> Collections.singletonList(versions.getLatest(to))));
> + return this;
> + }
> +
> /**
> * performs all supported upgrade paths that exist in between from
> and to that include the current version.
> * This call is equivalent to calling {@code upgradesTo(from,
> CURRENT).upgradesFrom(CURRENT, to)}.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
