[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723930#comment-17723930 ] Claude Warren commented on CASSANDRA-8928: -- I have updated the pull request: [https://github.com/apache/cassandra/pull/2045] The problem has been that between v3 and v4 there was a breaking change in the system local table where the columns "broadcast_port", "listen_port", and "rpc_port" were added. The code (in the current pull request) provides functionality to remove those columns when the older table is written. The code also allows for other transformations of the columns, though none are implemented. In order for the downgrade to work the following steps are taken (not all are in this code, some are in a script I have for testing the process) # Execute the standalone downgrade on the desired table(s). # Delete the system_schema tables. # Delete the *-Filter.db, *-Index.db, *-TOC.txt, *-Digest.*, and *-Summary.db for the modified table(s) # Delete the original files (e.g. nb-*) # Start the earlier version of the software. I tested the current code by starting 4.1 to create the tables. Downgraded all the tables in the database to "ma" version, followed the above steps and started 3.11 According to the logs 3.11.14 started. The current pull request code is not as clean as I would like it, but it does work correctly. I would like some comments on the general approach for removing columns where they are filtered out of the row definition during writing. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718819#comment-17718819 ] Claude Warren commented on CASSANDRA-8928: -- I also encountered the bigV3 problem and fixed that in a later push. The problem I am trying to solve is to remove columns that were added to system tables between 3 and 4. I also think there were some format changes that need to be undone. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715810#comment-17715810 ] anis ben brahim commented on CASSANDRA-8928: >From our tests we have downgraded from the 4.0.9 to the 3.11.14. In our first >quick tests we have noticed that the keyword bigV3 is problematic when >cassandra 3 start. By renaming this keyword by "big" in sstable filenames, >Cassandra successfully start. For this quick test as mentioned in the tickets, >system keysapces are not downgradable for now as it contains new columns and >new tables. As DDL are not allowed during the migration, restoring system >keyspaces from backups make this downgrade tool working. We are currently try >to figure out all the points from system keyspaces that make the downgrade >process failed and try to find some workarounds. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713920#comment-17713920 ] Claude Warren commented on CASSANDRA-8928: -- [~nasnousssi] , Yes. it is one of the 3 tickets I am working on. There is currently an issue with downgrading from 4.x to 3.x You said you were able to get the downgrade to work on a single node. Can you tell me what versions you downgraded from and to? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713762#comment-17713762 ] anis ben brahim commented on CASSANDRA-8928: Hi [~claude] do you have any updates on this feature ? After doing some test on a single node, it seems that recovering only system keyspaces from backups and downgrading other keyspaces with this tool works. Is there any plan to continue working on this subject ? Best regards, Anis > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704164#comment-17704164 ] Claude Warren commented on CASSANDRA-8928: -- Downgrading from V4 to V3 is not possible until CASSANDRA-18358 is resolved > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: remove-reopen > Fix For: 5.x > > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648690#comment-17648690 ] Paulo Motta commented on CASSANDRA-8928: reassigned to [~claude] as I think previous assignee is inactive. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Claude Warren >Priority: Low > Labels: gsoc2016, mentor, remove-reopen > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648684#comment-17648684 ] T Jake Luciani commented on CASSANDRA-8928: --- [~claude] has written an initial [PR|https://github.com/apache/cassandra/pull/2045] that supports downgrading via an offline tool (like sstableupgrader) Following a similar approach described in this comment. It requires extending the same contract we have in place for major upgrades. We support reading from the last major sstable version and we support reading/writing to the previous MessagingService version. With this PR we add the ability to Write SSTables in the prior major format (though with a new format key which includes the major version it belongs to). This will allow someone who wants to revert major versions a simpler path. Just stop each node (w/ flush), run offline downgrader, then restart with previous version. This will require some new docs and also a sstable downgrade dtest as well. Would it be ok to re-assign this ticket to Claude? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Low > Labels: gsoc2016, mentor, remove-reopen > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351164#comment-15351164 ] Aleksey Yeschenko commented on CASSANDRA-8928: -- bq. The same end result can be achieved, although with maybe a bit more of work, by backing up sstables of the upgraded node and wiping/recreating the node in a previous version, and then use sstableloader of the newer version to reload the data in the recreated node, which would be easily enabled by CASSANDRA-8110. sstableloader itself however is not backward or future compatible at the moment (schema loading code isn't, for one). Not a big deal, but something we should make work and add tests for. Otherwise I agree that CASSANDRA-8110 will do, at least for now. We can close this JIRA now and reopen later, if necessary. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329849#comment-15329849 ] Jeremy Hanna commented on CASSANDRA-8928: - [~pauloricardomg] I didn't realize that a later version sstable loader could write to an earlier version cluster. That would work. Thanks. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328411#comment-15328411 ] Paulo Motta commented on CASSANDRA-8928: bq. The primary use case for downgrading sstables is to abort an upgrade. So for example, if you're on 2.1 and want to go to 2.2 or 3.x, the upgrade plan often includes a contingency plan to abort and revert back to the older version, including preserving data written while in the upgraded version. The same end result can be achieved, although with maybe a bit more of work, by backing up sstables of the upgraded node and wiping/recreating the node in a previous version, and then use sstableloader of the newer version to reload the data in the recreated node, which would be easily enabled by CASSANDRA-8110. Although standalone downgrading could be nice, I'm not sure it's really worth the effort and risk of carrying sstable writing code across multiple versions. bq. Would it be possible to limit the implementation of this to the core sstable data file? We could then have an offline downgrade sstable process that would convert to the older sstable data file and those files could be bulk loaded through that version's sstable loader into the reverted cluster. This is more or less the approach I'm proposing, but instead of doing an offline downgrade only of the data file + sstableloading afterwards, you would only need to use sstableloader of the newer version to load newer version sstables in the reverted cluster automatically. Wouldn't this be sufficient? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328361#comment-15328361 ] Jeremy Hanna commented on CASSANDRA-8928: - The primary use case for downgrading sstables is to abort an upgrade. So for example, if you're on 2.1 and want to go to 2.2 or 3.x, the upgrade plan often includes a contingency plan to abort and revert back to the older version, including preserving data written while in the upgraded version. While backwards compatible streaming (CASSANDRA-8110) is helpful in other scenarios, it wouldn't help for that use case. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328321#comment-15328321 ] Paulo Motta commented on CASSANDRA-8928: After digging into this for a while we realized it may be too much effort to support writing full-fledged sstables in previous formats in a dependable way, specially after the large scale changes introduced by CASSANDRA-8099. This is due to the fact that we not only have to translate the data component, which is already a large effort per se, but also make sure other components such as indexes, samples, stats, etc, are downgraded correctly, which means a large bunch of legacy code that needs to be kept around until it becomes unsupported. We initially thought this could easily enable CASSANDRA-8110 but it can actually be seen as the opposite: once we make streaming backward compatible, downgrading is just a matter of sstableloading the new-format sstables in a previous-version node, which will already make the sstable be rewritten in the old format correctly, since all the components are rewritten during streaming. Of course there would be some manual juggling needed to restore the node with the correct tokens and schema, but we can probably add some kind of recovery/downgrade mode that would allow an operator to reload the data with sstableloader before the node becomes available. Maybe this could be made easier after CASSANDRA-6038 and/or CASSANDRA-9587. So my take is that we should pursue CASSANDRA-8110 first, since that seems much more attainable, and rethink this later, maybe via recovery mode + sstableloading combo as suggested previously. Since streaming basically transfer raw sstable partitions over the wire, downgraded stream support is just a matter of translating partitions on-the-fly to the previous data format, without the need to support downgrading other components extensively as required here. With this said, I will attach a design document to CASSANDRA-8110 shortly. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210268#comment-15210268 ] Paulo Motta commented on CASSANDRA-8928: bq. Regarding OldFormatIterator and OldFormatDeserializer, I am trying to find them on Git repository but I didn't find anything [OldFormatIterator|https://github.com/apache/cassandra/search?q=OldFormatIterator] and [OldFormatDeserializer|https://github.com/apache/cassandra/search?q=OldFormatDeserializer] ;-) bq. Also, can I use openJDK (7) instead of Oracle JDK? Because I'm trying to setting up Cassandra (trunk) with IDEA according to RunningCassandraInIDEA but it gives me a build failed error when I execute ant generate-idea-files. Generally you can, but you can probably save yourself few headaches by using Oracle JDK. It would help if you posted the compiler error output to find out what's the problem. Also, for more practical questions like this, #cassandra and/or #cassandra-dev on irc.freenode.net is usually a better (and faster) forum. bq. I'll try it again, once I have it setted up, I'll take a look along with other materials CASSANDRA- which seems more suitable to start with. Sounds good. Make sure to post any comments/questions related to that ticket there. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210247#comment-15210247
]
Kaide Mu commented on CASSANDRA-8928:
-
Hi, [~pauloricardomg].
Thank you so much, just fixed format error and added ccm and StandaloneUpgrader
to the proposal, I didn't realize that dtest runs on ccm. I'll check them out.
Regarding {{OldFormatIterator}} and {{OldFormatDeserializer}}, I am trying to
find them on Git repository but I didn't find anything, maybe after setting up
the development environment I can decompile them. Also, can I use openJDK (7)
instead of Oracle JDK? Because I'm trying to setting up Cassandra (trunk) with
IDEA according to
[RunningCassandraInIDEA|http://wiki.apache.org/cassandra/RunningCassandraInIDEA|http://wiki.apache.org/cassandra/RunningCassandraInIDEA]
but it gives me a build failed error when I execute {{ant
generate-idea-files}}.
{code}
build-project:
[echo] apache-cassandra: /home/km/cassandra/build.xml
[javac] Compiling 45 source files to /home/km/cassandra/build/classes/thrift
[javac] javac: invalid target release: 1.8
[javac] Usage: javac
[javac] use -help for a list of possible options
BUILD FAILED
/home/km/cassandra/build.xml:769: Compile failed; see the compiler error output
for details.
{code}
I'll try it again, once I have it setted up, I'll take a look along with other
materials [CASSANDRA-|https://issues.apache.org/jira/browse/CASSANDRA-]
which seems more suitable to start with.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: gsoc2016, mentor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209507#comment-15209507
]
Paulo Motta commented on CASSANDRA-8928:
LGTM, a few nits in your updated proposal:
* Update {{We will start working with ma (3.0.x) and jb (2.2.x) formats}} to ma
and la.
* Maybe add [ccm|https://github.com/pauloricardomg/ccm] tool to {{Familiarize
with Cassandra dtest suit.}}
bq. In addition, do you mean with double cycle support that our
sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"?
Would be something like ma -> la -> ma -> la -> ma and check the data is
correct. please note that upgrade from la -> ma is already supported via the
upgradesstables tool and you would add support to downgrade from ma -> la via
the sstabledowngrader tool. Actually {{StandaloneUpgrader}} is probably more
relevant then {{SStableScrubber}} in this context so you should mention this in
your proposal instead(although sstable scrubber is also important for learning
purposes because it validates the sstable when reading). You should also
probably have a look on {{OldFormatIterator}} and {{OldFormatDeserializer}}
classes to learn how sstables are read in the previous format during your
familiarization period.
Another thing that could possibly help in your Apache GSoC application is to
work on a [low hanging fruit
ticket|https://issues.apache.org/jira/browse/CASSANDRA-11352?jql=project%20%3D%20CASSANDRA%20AND%20status%20%3D%20Open%20AND%20labels%20%3D%20lhf]
to prove your ability to dive into a complex codebase (CASSANDRA- and
CASSANDRA-8777 are probably good for beginners). Please note that this is not
required, but would only be a plus in your application to the Apache selection
committee and does not need to be done before the application deadline since
the decision will probably be taken only a few weeks after that.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: gsoc2016, mentor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208166#comment-15208166 ] Kaide Mu commented on CASSANDRA-8928: - Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, thus I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204623#comment-15204623 ] Yuki Morishita commented on CASSANDRA-8928: --- Thanks Paulo, +1 for these. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204582#comment-15204582 ] Paulo Motta commented on CASSANDRA-8928: Overall looks good. A few comments to improve your proposal: * We should probably limit the scope from "ma" to "la" and "ka", and leave "jb" as an extra if there's time. Given that "ka" is pretty much the same as "la", except for the file name format, then the bulk of the work would be on downgrading from "ma" to "la". * I think we can divide this proposal into two major milestones: PoC and Productization. In the PoC milestone (midterm) you could deliver a basic sstabledowngrader tool supporting downgrade from "ma" to "la", but without much attention to the framework aspect, basically hacking the SStableScrubber tool and reusing existing code from previous versions to have a basic functional version along with tests. In the second phase after you're more familiarized with the problem you would refactor your initial PoC to deal with more complex scenarios (if there are any), and extract version-independent structures and interfaces to facilitate adding support downgrading to newer formats in the future, along with adding downgrade support to "ka" which should be easy enough after you have downgrade to "la" in place. With that said, you would have the following deliverables: ** sstabledowngrader tool with hard-coded downgrade support from "ma" to "la" (midterm) ** sstabledowngrader tool with extension points/flexible support to other sstable formats + documentation (final) ** comprehensive dtest suite for "ma" and "la" downgrade support with double cycle of upgrade/downgrade based on CASSANDRA-10563 (final) * You can probably move the code reading/familiarization from the coding period to the community bonding period, to focus on your first deliverable in the coding period. WDYT [~yukim]? Any other suggestion? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198548#comment-15198548 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear [~pauloricardomg]. Thank you for the reply and all these references! I'll look through all of them, especially StandaloneScrubbler which seems it's the key for our design. If I have any questions I'll leave a message in this ticket or via irc. Also, since the student application deadline is next Friday, I had drawn up a [proposal|https://docs.google.com/document/d/10Xor1LggGxEihZ39ItOJ8QZ-EKTOVqcAXaE_B2TY3BM/edit?usp=sharing] for this issue, if you have any time, would you mind take a look and tell me if I missed anything or anything that I should detailed more. Thank you so much. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196064#comment-15196064
]
Paulo Motta commented on CASSANDRA-8928:
bq. I tried to find something similar of changes on version 2.2 but I didn't
find anything, do you have any similar references or similar things, I'll try
to see if I can breakdown it, but it seems quite difficult in the sense that
I'm not familiar with Cassandra code base.
This [blog
post|http://distributeddatastore.blogspot.com.br/2013/08/cassandra-sstable-storage-format.html]
might have some useful info on previous sstable format to start with. I
understand this is not a trivial task so you might need to do a lot of
background reading before start doing some real work. You might want to watch
these
[tutorials|http://www.datastax.com/dev/blog/deep-into-cassandra-internals] on
C* internals, in particular the read write path and compactions to understand
in which context sstables fit in. The [ccm|https://github.com/pcmanus/ccm] tool
is very handy to create throw-away cassandra clusters.
A simple exercise you can make to get acquainted with the sstable format is to
create a simple table on C* 2.2, insert some data on it, and inspect it with
[sstable2json
tool|https://docs.datastax.com/en/cassandra/1.2/cassandra/tools/toolsSStable2json_t.html].
Then create the same table on C* 3.X and inspect it with
[sstabledump|http://www.datastax.com/dev/blog/debugging-sstables-in-3-0-with-sstabledump]
and compare the results.
After that you might inspect the {{StandaloneScrubber}} class on {{trunk}}
(which implements the
[sstablescrub|https://docs.datastax.com/en/cassandra/1.2/cassandra/tools/toolsSSTableScrub_t.html]
tool) to understand the flow of reading an sstable and rewriting it in the
current format + fix corruptions. Then you might want to hack this tool to read
in the current format and write in the {{ka}} format by replacing the
{{SSTableRewriter}} with a {{LegacySSTableWriter}} that could initially be a
copy of the 2.2 {{BigTableWriter}} (you'll probably have fun for a few weeks
with this already).
bq. Also before jumping to draw up the framework, I'd like to know is our
purpose downgrade an existing SSTable to certain older version or add the
possibility to write older version of SSTable?
SSTable are immutable structures, so in order to downgrade and existing sstable
you need to read it in the current format and then write it in an older format.
So the flow is: la-sstable -> -> ka-sstable
Please note that if you have questions along the way you can also reach out on
#cassandra-dev on irc.freenode.net.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: gsoc2016, mentor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15191399#comment-15191399 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear Paulo. I'd read such blog, I think now I'm clear with changes of Data.db in 3.x, I tried to find something similar of changes on version 2.2 but I didn't find anything, do you have any similar references or similar things, I'll try to see if I can breakdown it, but it seems quite difficult in the sense that I'm not familiar with Cassandra code base. Also before jumping to draw up the framework, I'd like to know is our purpose downgrade an existing SSTable to certain older version or add the possibility to write older version of SSTable? Regards. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183144#comment-15183144 ] Paulo Motta commented on CASSANDRA-8928: [~kdmu] Nice start! You're in the right direction but I suggest focusing on downgrading from "ma" (3.x) to "la" (2.2) format first, since the new native storage format introduced on 3.0 is what will probably take the most of your time to understand and get right. In order to familiarize yourself with the 3.0 storage format (which is brand new) I suggest you read [this blog post|http://thelastpickle.com/blog/2016/03/04/introductiont-to-the-apache-cassandra-3-storage-engine.html]. For a more practical task you can you can do a similar breakdown of the 2.2 format (if you can't find any available), what will make visualize better the main changes needed between these formats. Also, another important goal of this ticket is to provide a generic framework that will allow to downgrade sstables in future versions, so we should design it in a way that it can be reused to downgrade between different formats, rather than hard-coding to specific conversions. So before diving into hardcore implementation I suggest you to write a brief design document describing an overview of the downgrade process and its main interfaces and/or modules, so we can make sure we're on the same page before you start. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174611#comment-15174611 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear Yuki. Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for [SSTable|https://github.com/apache/cassandra/blob/cassandra-2.2/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FSSTable.java#L57] but I don't know if it's the right one. Best regards, Kaide Mu > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173951#comment-15173951
]
Yuki Morishita commented on CASSANDRA-8928:
---
Hi Kaide, thanks for your interest.
As you can see in
[BigVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116],
cassandra-3.x supports *reading* SSTable as old as version 2.0.1('jb' in
SSTable file name). But right now, cassandra only writes SSTable in the latest
(la) format. So, we'd like to add function to write in those older version
defined in BigVersion (jb, ka, la). Cassandra already can read from those
versions, you may want to start reading how old formats are written by reading
older version of Cassandra.
SSTable API is in
[o.a.c.io.sstable.format|https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/io/sstable/format].
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: gsoc2016, mentor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172917#comment-15172917 ] Kaide Mu commented on CASSANDRA-8928: - Hello Apache Cassandra community, I'm Kaide Mu, currently pursuing BSc of Computer Science at Polytechnic University of Valencia, Spain. I'd like to work in this issue as my project of GSoC 2016. About this project, I had been looking in a minor detail for SSTable architecture and some of different available SSTable source code, as I see, there's some different implementations of SSTable, therefore as the issue description indicates, I guess we aim to add a downgradesstable functionality to all existing SSTables, e.g downgrade from a 3.X version SSTable to 2.X SSTable or 1.X ones, please correct if my understanding is wrong. In addition, would you mind providing me any suggestion or references which I should take a look in order to clarify with this issue, I'd very appreciate it. Thank you so much and best regards, Kaide Mu > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370782#comment-14370782
]
Aleksey Yeschenko commented on CASSANDRA-8928:
--
FWIW, {{SSTableDowngrader}} would be the most straightforward way to implement
CASSANDRA-8110, so I wouldn't rush and close this just because it's 'hard'. And
unless I'm misunderstanding the compatibility policy that would naturally
follow from our latest release process announcements, CASSANDRA-8110 is
something we'll need soon.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368951#comment-14368951 ] Benedict commented on CASSANDRA-8928: - Data translations are inherently dangerous acts, and very hard to vet (see CASSANDRA-8993 which seems to be a result of upgrade corrupting index state so that records are silently not returned). Having a bidirectional translation seems particularly problematic. If we are to go down this route, we need to ensure HUGE effort is put in to exhaustively testing the resulting outputs, after a double cycle of upgrade/downgrade. This effort wouldn't be wasted though, as it would also help vet upgrades, which are essential acts. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368947#comment-14368947 ] Jeremy Hanna commented on CASSANDRA-8928: - [~benedict] would you like to bring up the points you made about viability of this feature? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
