[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[ https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13610045#comment-13610045 ] Marcus Eriksson commented on CASSANDRA-5195: looks good to me and verified it works as expected bug very similar to CASSANDRA-5061 > Offline scrub does not migrate the directory structure on migration from > 1.0.x to 1.1.x and causes the keyspace to disappear > > > Key: CASSANDRA-5195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5195 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.9 >Reporter: Omid Aladini > Fix For: 1.1.11 > > Attachments: 0001-Flush-newly-migrated-system-CFs.patch, 5195.patch > > > Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing > LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x > is started. But Cassandra 1.1.x uses a new directory structure > (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate. > How to reproduce: > 1- Run cassandra 1.0.12. > 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually. > 3- Stop cassandra 1.0.12 > 4- Run ./bin/sstablescrub Keyspace1 Standard1 > which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and > notice the data directory isn't migrated. > 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't > try to migrate the directory structure. Also commitlog entries get skipped: > "Skipped X mutations from unknown (probably removed) CF with id 1000" > Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the > Keyspace correctly. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[ https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13609018#comment-13609018 ] Jonathan Ellis commented on CASSANDRA-5195: --- [~krummas], can you review? > Offline scrub does not migrate the directory structure on migration from > 1.0.x to 1.1.x and causes the keyspace to disappear > > > Key: CASSANDRA-5195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5195 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.9 >Reporter: Omid Aladini > Fix For: 1.1.11 > > Attachments: 0001-Flush-newly-migrated-system-CFs.patch, 5195.patch > > > Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing > LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x > is started. But Cassandra 1.1.x uses a new directory structure > (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate. > How to reproduce: > 1- Run cassandra 1.0.12. > 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually. > 3- Stop cassandra 1.0.12 > 4- Run ./bin/sstablescrub Keyspace1 Standard1 > which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and > notice the data directory isn't migrated. > 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't > try to migrate the directory structure. Also commitlog entries get skipped: > "Skipped X mutations from unknown (probably removed) CF with id 1000" > Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the > Keyspace correctly. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[
https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604295#comment-13604295
]
Omid Aladini commented on CASSANDRA-5195:
-
I think upon loading the schema via OfflineScrubber, DefsTable.loadFromStorage
migrates the old system tables to the new format. Therefore it drops the old
ones but doesn't flush the commitlogs of new ones. OfflineScrubber exits and on
the next start, schema_{keyspaces, columnfamilies, columns} have no persisted
sstables, and the commitlog gets replayed after Cassandra tries to load CF
schemas, therefore finds none. This explains why after second restart, column
families appear again (something force-flushes system CFs?).
A new patch (0001-Flush-newly-migrated-system-CFs.patch) is attached to fix the
problem (although I'm not sure if there is a more proper fix for this or if
there is a better place to put the foced flush.)
> Offline scrub does not migrate the directory structure on migration from
> 1.0.x to 1.1.x and causes the keyspace to disappear
>
>
> Key: CASSANDRA-5195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5195
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Affects Versions: 1.1.9
>Reporter: Omid Aladini
> Fix For: 1.1.11
>
> Attachments: 0001-Flush-newly-migrated-system-CFs.patch, 5195.patch
>
>
> Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing
> LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x
> is started. But Cassandra 1.1.x uses a new directory structure
> (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate.
> How to reproduce:
> 1- Run cassandra 1.0.12.
> 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually.
> 3- Stop cassandra 1.0.12
> 4- Run ./bin/sstablescrub Keyspace1 Standard1
> which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and
> notice the data directory isn't migrated.
> 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't
> try to migrate the directory structure. Also commitlog entries get skipped:
> "Skipped X mutations from unknown (probably removed) CF with id 1000"
> Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the
> Keyspace correctly.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[ https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566142#comment-13566142 ] Ryan McGuire commented on CASSANDRA-5195: - One other interesting aspect: Removing the patch, resetting /var/lib/cassandra to the 1.0.12 state, re-running sstablescrub, restarting cassandra TWICE allows the keyspace to be read, but the table is empty! : * 11:03 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh Connected to Test Cluster at localhost:9160. [cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol 19.33.0] Use HELP for help. cqlsh> use Keyspace1; cqlsh:Keyspace1> select count(*) from Standard1; count --- 0 > Offline scrub does not migrate the directory structure on migration from > 1.0.x to 1.1.x and causes the keyspace to disappear > > > Key: CASSANDRA-5195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5195 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.9 >Reporter: Omid Aladini >Assignee: Jonathan Ellis > Fix For: 1.1.9 > > Attachments: 5195.patch > > > Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing > LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x > is started. But Cassandra 1.1.x uses a new directory structure > (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate. > How to reproduce: > 1- Run cassandra 1.0.12. > 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually. > 3- Stop cassandra 1.0.12 > 4- Run ./bin/sstablescrub Keyspace1 Standard1 > which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and > notice the data directory isn't migrated. > 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't > try to migrate the directory structure. Also commitlog entries get skipped: > "Skipped X mutations from unknown (probably removed) CF with id 1000" > Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the > Keyspace correctly. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[ https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566128#comment-13566128 ] Ryan McGuire commented on CASSANDRA-5195: - I have reproduced this issue. Omid's patch works as he described: it does not fully fix the issue, but does allow the keyspace to be loaded on the 2nd restart of cassandra. Below is my verification workflow: * Checkout/build 1.0.12 ** cd $CASSANDRA_DIR ** git checkout -b 5195-1.0.12 ** git reset --hard cassandra-1.0.12 ** git clean -f -d ** ant build * Run 1.0.12 test: ** sudo rm -rf /var/lib/cassandra ** sudo cassandra ** cd tool/stress ** ant build ** ./bin/stress ** sudo pkill -f CassandraDaemon * Verify the keyspace/cf was created by stress: ** 10:20 PM:~/git/datastax/cassandra/tools/stress[5195-1.0.12*]$ cqlsh Connected to Test Cluster at localhost:9160. [cqlsh 2.0.0 | Cassandra unknown | CQL spec unknown | Thrift protocol 19.20.0] Use HELP for help. cqlsh> use Keyspace1 ; cqlsh:Keyspace1> select count(*) from Standard1; count --- 1 * Checkout/build 1.1.9 ** cd $CASSANDRA_DIR ** git checkout -b 5195-1.1.9 ** git reset --hard cassandra-1.1.9 ** git clean -f -d ** ant build * Run 1.1.9 test: ** sudo ./bin/sstablescrub Keyspace1 Standard1 *** stdout: Unknown keyspace/columnFamily Keyspace1.Standard1 ** sudo cassandra *** log: INFO [main] 2013-01-29 22:28:44,800 CommitLogReplayer.java (line 103) Skipped 585748 mutations from unknown (probably removed) CF with id 1000 * Verify that Keyspace1 does or does not exist: ** 10:30 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh Connected to Test Cluster at localhost:9160. [cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol 19.33.0] Use HELP for help. cqlsh> use Keyspace1 ; Bad Request: Keyspace 'Keyspace1' does not exist * Run 1.1.9 test again without the sstablescrub (restoring /var/lib/cassandra from before): ** sudo pkill -f CassandraDaemon ** sudo cassandra *** log: INFO 22:33:01,240 Replaying /var/lib/cassandra/commitlog/CommitLog-1359515707503.log, /var/lib/cassandra/commitlog/CommitLog-1359515946450.log INFO 22:33:01,244 Replaying /var/lib/cassandra/commitlog/CommitLog-1359515707503.log INFO 22:33:02,318 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 4.55084790673026 (just-counted was 4.55084790673026). calculation took 866ms for 4590 columns INFO 22:33:02,930 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 5.226616220760892 (just-counted was 5.226616220760892). calculation took 357ms for 11635 columns INFO 22:33:04,186 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 5.094053078093754 (just-counted was 4.9614899354266155). calculation took 859ms for 26720 columns * Verify that Keyspace1 does or does not exist: ** 10:36 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh Connected to Test Cluster at localhost:9160. [cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol 19.33.0] Use HELP for help. cqlsh> use Keyspace1; cqlsh:Keyspace1> select count(*) from Standard1; count --- 1 * Apply patch and retest: ** cd $CASSANDRA_DIR ** git apply ~/Downloads/5195.patch ** ant clean build ** sudo rm -rf /var/lib/cassandra ** (restore /var/lib/cassandra from 1.0.12) ** sudo pkill -f CassandraDaemon ** sudo ./bin/sstablescrub Keyspace1 Standard1 *** stdout: Pre-scrub sstables snapshotted into snapshot pre-scrub-1359517364042 Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-17-Data.db') Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-17-Data.db') complete: 63608 rows in new sstable and 0 empty (tombstoned) rows dropped Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-10-Data.db') Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-10-Data.db') complete: 258153 rows in new sstable and 0 empty (tombstoned) rows dropped Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-18-Data.db') Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-18-Data.db') complete: 65207 rows in new sstable and 0 empty (tombstoned) rows dropped Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-15-Data.db') Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-15-Data.db') complete: 254487 rows in new sstable and 0 empty (tombstoned) rows dropped Scrubbing SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-5-Data.db') Scrub of SSTableReader(path='/var/lib/cassandra/data/Keyspace1/
[jira] [Commented] (CASSANDRA-5195) Offline scrub does not migrate the directory structure on migration from 1.0.x to 1.1.x and causes the keyspace to disappear
[ https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565540#comment-13565540 ] Jonathan Ellis commented on CASSANDRA-5195: --- Ryan, can you reproduce? > Offline scrub does not migrate the directory structure on migration from > 1.0.x to 1.1.x and causes the keyspace to disappear > > > Key: CASSANDRA-5195 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5195 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.9 >Reporter: Omid Aladini > Fix For: 1.1.9 > > > Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing > LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x > is started. But Cassandra 1.1.x uses a new directory structure > (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate. > How to reproduce: > 1- Run cassandra 1.0.12. > 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually. > 3- Stop cassandra 1.0.12 > 4- Run ./bin/sstablescrub Keyspace1 Standard1 > which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and > notice the data directory isn't migrated. > 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't > try to migrate the directory structure. Also commitlog entries get skipped: > "Skipped X mutations from unknown (probably removed) CF with id 1000" > Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the > Keyspace correctly. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
