[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723930#comment-17723930
]
Claude Warren edited comment on CASSANDRA-8928 at 5/18/23 2:59 PM:
---
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
Also, I think this code should be rebased to 4.1 branch. I wonder if this
change is enough to require bumping to 4.2.
There are changes to:
* The column family store - exposing the sstable ID generator
* CompressedMetadata - adding a flag to indicate the presence of the max
compressed length variable.
* CompressedSequentialWriter - using the compressed length variable for writes.
* TableMetadata.Builder – to replace an existing column definition with
another one.
was (Author: claudenw):
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Tools
>Reporter: Jeremy Hanna
>Assignee: Claude Warren
>Priority: Low
> Labels: remove-reopen
> Fix For: 5.x
>
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17723930#comment-17723930
]
Claude Warren edited comment on CASSANDRA-8928 at 5/18/23 2:52 PM:
---
[~nasnousssi]
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the {*}-Filter.db, *-Index.db, *-TOC.txt, *-Digest.{*}, and
*-Summary.db for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
was (Author: claudenw):
I have updated the pull request: [https://github.com/apache/cassandra/pull/2045]
The problem has been that between v3 and v4 there was a breaking change in the
system local table where the columns "broadcast_port", "listen_port", and
"rpc_port" were added. The code (in the current pull request) provides
functionality to remove those columns when the older table is written. The
code also allows for other transformations of the columns, though none are
implemented.
In order for the downgrade to work the following steps are taken (not all are
in this code, some are in a script I have for testing the process)
# Execute the standalone downgrade on the desired table(s).
# Delete the system_schema tables.
# Delete the *-Filter.db, *-Index.db, *-TOC.txt, *-Digest.*, and *-Summary.db
for the modified table(s)
# Delete the original files (e.g. nb-*)
# Start the earlier version of the software.
I tested the current code by starting 4.1 to create the tables. Downgraded all
the tables in the database to "ma" version, followed the above steps and
started 3.11 According to the logs 3.11.14 started.
The current pull request code is not as clean as I would like it, but it does
work correctly.
I would like some comments on the general approach for removing columns where
they are filtered out of the row definition during writing.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Tools
>Reporter: Jeremy Hanna
>Assignee: Claude Warren
>Priority: Low
> Labels: remove-reopen
> Fix For: 5.x
>
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648684#comment-17648684 ] T Jake Luciani edited comment on CASSANDRA-8928 at 12/16/22 3:33 PM: - [~claude] has written an initial [PR|https://github.com/apache/cassandra/pull/2045] that supports downgrading via an offline tool (like sstableupgrader) Following a similar approach described in [this comment|https://issues.apache.org/jira/browse/CASSANDRA-11877?focusedCommentId=15310504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15310504]. It requires extending the same contract we have in place for major upgrades. We support reading from the last major sstable version and we support reading/writing to the previous MessagingService version. With this PR we add the ability to Write SSTables in the prior major format (though with a new format key which includes the major version it belongs to). This will allow someone who wants to revert major versions a simpler path. Just stop each node (w/ flush), run offline downgrader, then restart with previous version. This will require some new docs and also a sstable downgrade dtest as well. Would it be ok to re-assign this ticket to Claude? was (Author: tjake): [~claude] has written an initial [PR|https://github.com/apache/cassandra/pull/2045] that supports downgrading via an offline tool (like sstableupgrader) Following a similar approach described in this comment. It requires extending the same contract we have in place for major upgrades. We support reading from the last major sstable version and we support reading/writing to the previous MessagingService version. With this PR we add the ability to Write SSTables in the prior major format (though with a new format key which includes the major version it belongs to). This will allow someone who wants to revert major versions a simpler path. Just stop each node (w/ flush), run offline downgrader, then restart with previous version. This will require some new docs and also a sstable downgrade dtest as well. Would it be ok to re-assign this ticket to Claude? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329849#comment-15329849 ] Jeremy Hanna edited comment on CASSANDRA-8928 at 6/14/16 4:50 PM: -- [~pauloricardomg] I didn't realize that a later version sstable loader could write to an earlier version cluster. It makes sense with backwards compatible streaming, but I didn't know if there were other obstacles to get that to work. That should be good then I think. Thanks. was (Author: jeromatron): [~pauloricardomg] I didn't realize that a later version sstable loader could write to an earlier version cluster. That would work. Thanks. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328361#comment-15328361 ] Jeremy Hanna edited comment on CASSANDRA-8928 at 6/13/16 9:50 PM: -- The primary use case for downgrading sstables is to abort an upgrade. So for example, if you're on 2.1 and want to go to 2.2 or 3.x, the upgrade plan often includes a contingency plan to abort and revert back to the older version, including preserving data written while in the upgraded version. While backwards compatible streaming (CASSANDRA-8110) is helpful in other scenarios, it wouldn't help for that use case. Would it be possible to limit the implementation of this to the core sstable data file? We could then have an offline downgrade sstable process that would convert to the older sstable data file and those files could be bulk loaded through that version's sstable loader into the reverted cluster. was (Author: jeromatron): The primary use case for downgrading sstables is to abort an upgrade. So for example, if you're on 2.1 and want to go to 2.2 or 3.x, the upgrade plan often includes a contingency plan to abort and revert back to the older version, including preserving data written while in the upgraded version. While backwards compatible streaming (CASSANDRA-8110) is helpful in other scenarios, it wouldn't help for that use case. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208166#comment-15208166 ] Kaide Mu edited comment on CASSANDRA-8928 at 3/23/16 10:21 AM: --- Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? was (Author: kdmu): Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, thus I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174611#comment-15174611 ] Kaide Mu edited comment on CASSANDRA-8928 at 3/6/16 5:54 PM: - Hi, [~yukim] Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for [SSTable|https://github.com/apache/cassandra/blob/cassandra-2.2/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FSSTable.java#L57] but I don't know if it's the right one. Best regards, Kaide Mu was (Author: kdmu): Hi, dear Yuki. Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for
