Re: cassandra upgrade to 1.1 - migration problem
On Tue, May 15, 2012 at 5:41 PM, Dave Brosius wrote: > The replication factor for a keyspace is stored in the > system.schema_keyspaces column family. > > Since you can't view this with cli as the server won't start, the only way > to look at it, that i know of is to use the > > sstable2json tool on the *.db file for that column family... > > So for instance on my machine i do > > ./sstable2json /var/lib/cassandra/data/**system/schema_keyspaces/** > system-schema_keyspaces-ia-1-**Data.db > > and get > > > { > "7374726573735f6b73": [["durable_writes","true",**1968197311980145], > ["name","stress_ks",**1968197311980145], ["strategy_class","org.apache.** > cassandra.locator.**SimpleStrategy",**1968197311980145], > ["strategy_options","{\"**replication_factor\":\"3\"}",** > 1968197311980145]] > > It's likely you don't have a entry from replication_factor. > > Yep, I checked the system.schema_keyspaces ColumnFamily, and there was no replication_factor value, as you suspected. But the dev cluster that worked after upgrade did have that value, so it started up okay. Apparently pre-1.1 was less picky about its presence. > Theoretically i suppose you could embellish the output, and use > json2sstable to fix it, but I have no experience here, and would get the > blessings of datastax fellas, before proceeding. Actually, I went ahead and took a chance because I had already completely offline for several hours and wanted to get things back up. I did what you suggested and added the replication_factor value to the json returned from sstable2json and imported it using json2sstable. Fortunately I had the dev cluster values to use as a basis. I started things up, and it worked like a champ. Thanks! Casey
Re: cassandra upgrade to 1.1 - migration problem
The replication factor for a keyspace is stored in the system.schema_keyspaces column family. Since you can't view this with cli as the server won't start, the only way to look at it, that i know of is to use the sstable2json tool on the *.db file for that column family... So for instance on my machine i do ./sstable2json /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-1-Data.db and get { "7374726573735f6b73": [["durable_writes","true",1968197311980145], ["name","stress_ks",1968197311980145], ["strategy_class","org.apache.cassandra.locator.SimpleStrategy",1968197311980145], ["strategy_options","{\"replication_factor\":\"3\"}",1968197311980145]] It's likely you don't have a entry from replication_factor. Theoretically i suppose you could embellish the output, and use json2sstable to fix it, but I have no experience here, and would get the blessings of datastax fellas, before proceeding. On 05/15/2012 07:02 PM, Casey Deccio wrote: Sorry to reply to my own message (again). I took a closer look at the logs and realized that the partitioner errors aren't what kept the daemon to stop; those errors are in the logs even before I upgraded. This one seems to be the culprit. java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: SimpleStrategy requires a replication_factor strategy option. at org.apache.cassandra.db.Table.(Table.java:275) at org.apache.cassandra.db.Table.open(Table.java:114) at org.apache.cassandra.db.Table.open(Table.java:97) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:204) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) ... 5 more Caused by: org.apache.cassandra.config.ConfigurationException: SimpleStrategy requires a replication_factor strategy option. at org.apache.cassandra.locator.SimpleStrategy.validateOptions(SimpleStrategy.java:71) at org.apache.cassandra.locator.AbstractReplicationStrategy.createReplicationStrategy(AbstractReplicationStrategy.java:218) at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:295) at org.apache.cassandra.db.Table.(Table.java:271) ... 9 more Cannot load daemon I'm not sure how to check the replication_factor and/or update it without using cassandra-cli, which requires the daemon to be running. Casey
Re: cassandra upgrade to 1.1 - migration problem
Sorry to reply to my own message (again). I took a closer look at the logs and realized that the partitioner errors aren't what kept the daemon to stop; those errors are in the logs even before I upgraded. This one seems to be the culprit. java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: SimpleStrategy requires a replication_factor strategy option. at org.apache.cassandra.db.Table.(Table.java:275) at org.apache.cassandra.db.Table.open(Table.java:114) at org.apache.cassandra.db.Table.open(Table.java:97) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:204) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) ... 5 more Caused by: org.apache.cassandra.config.ConfigurationException: SimpleStrategy requires a replication_factor strategy option. at org.apache.cassandra.locator.SimpleStrategy.validateOptions(SimpleStrategy.java:71) at org.apache.cassandra.locator.AbstractReplicationStrategy.createReplicationStrategy(AbstractReplicationStrategy.java:218) at org.apache.cassandra.db.Table.createReplicationStrategy(Table.java:295) at org.apache.cassandra.db.Table.(Table.java:271) ... 9 more Cannot load daemon I'm not sure how to check the replication_factor and/or update it without using cassandra-cli, which requires the daemon to be running. Casey
Re: cassandra upgrade to 1.1 - migration problem
cassandra.yaml on all nodes had ByteOrderedPartitioner with both the previous version and upgraded version. That being said, when I first started up cassandra after upgrading (with the updated .yaml, including ByteOrderedPartitioner) all nodes in the ring appeared to be up. But the load they carried was minimal (KB, as opposed to GB in the previous version), and the keyspace didn't exist. Then when I attempted to restart the daemon on each to see if it would help, but starting up failed on each with the partition error. Casey On Tue, May 15, 2012 at 12:59 PM, Oleg Dulin wrote: > Did you check cassandra.yaml to make sure partitioner there matches what > was in your old cluster ? > > Regards, > Oleg Dulin > Please note my new office #: 732-917-0159 > > On May 15, 2012, at 3:22 PM, Casey Deccio wrote: > > Here's something new in the logs: > > ERROR 12:21:09,418 Exception in thread Thread[SSTableBatchOpen:2,5,main] > java.lang.RuntimeException: Cannot open > /var/lib/cassandra/data/system/Versions/system-Versions-hc-35 because > partitioner does not match org.apache.cassandra.dht.ByteOrderedPartitioner > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:164) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Casey > > On Tue, May 15, 2012 at 12:08 PM, Casey Deccio wrote: > >> I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine >> in one environment, but after I upgraded in another, I can't find my >> keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me >> that the keyspace doesn't exist. In the log I see this: >> >> ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 >> AbstractCassandraDaemon.java (line 134) Exception in thread >> Thread[MigrationStage:1,5,main]java.lang.AssertionError >> at >> org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) >> at >> org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) >> at >> org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) >> at >> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) >> at >> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at >> java.util.concurrent.FutureTask.run(FutureTask.java:166)at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:636) >> >> I can see that the data I would expect still seems to be in the new place >> (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. >> >> What am I missing? >> >> Thanks, >> Casey >> > > >
Re: cassandra upgrade to 1.1 - migration problem
Did you check cassandra.yaml to make sure partitioner there matches what was in your old cluster ? Regards, Oleg Dulin Please note my new office #: 732-917-0159 On May 15, 2012, at 3:22 PM, Casey Deccio wrote: > Here's something new in the logs: > > ERROR 12:21:09,418 Exception in thread Thread[SSTableBatchOpen:2,5,main] > java.lang.RuntimeException: Cannot open > /var/lib/cassandra/data/system/Versions/system-Versions-hc-35 because > partitioner does not match org.apache.cassandra.dht.ByteOrderedPartitioner > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:164) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Casey > > On Tue, May 15, 2012 at 12:08 PM, Casey Deccio wrote: > I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine in > one environment, but after I upgraded in another, I can't find my keyspace. > When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me that the > keyspace doesn't exist. In the log I see this: > > ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 AbstractCassandraDaemon.java > (line 134) Exception in thread > Thread[MigrationStage:1,5,main]java.lang.AssertionError > at > org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) > at > org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) > at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) > at > org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) > at > org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166)at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > I can see that the data I would expect still seems to be in the new place > (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. > > What am I missing? > > Thanks, > Casey >
Re: cassandra upgrade to 1.1 - migration problem
Here's something new in the logs: ERROR 12:21:09,418 Exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.RuntimeException: Cannot open /var/lib/cassandra/data/system/Versions/system-Versions-hc-35 because partitioner does not match org.apache.cassandra.dht.ByteOrderedPartitioner at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:164) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Casey On Tue, May 15, 2012 at 12:08 PM, Casey Deccio wrote: > I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine > in one environment, but after I upgraded in another, I can't find my > keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me > that the keyspace doesn't exist. In the log I see this: > > ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[MigrationStage:1,5,main]java.lang.AssertionError > at > org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) > at > org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) > at > org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) > at > org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) > at > org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > I can see that the data I would expect still seems to be in the new place > (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. > > What am I missing? > > Thanks, > Casey >
cassandra upgrade to 1.1 - migration problem
I recently upgraded from cassandra 1.0.10 to 1.1. Everything worked fine in one environment, but after I upgraded in another, I can't find my keyspace. When I run, e.g., cassandra-cli with 'use KeySpace;' It tells me that the keyspace doesn't exist. In the log I see this: ERROR [MigrationStage:1] 2012-05-15 11:39:48,216 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]java.lang.AssertionError at org.apache.cassandra.db.DefsTable.updateKeyspace(DefsTable.java:441) at org.apache.cassandra.db.DefsTable.mergeKeyspaces(DefsTable.java:339) at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:269) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248) at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) I can see that the data I would expect still seems to be in the new place (/var/lib/cassandra/data/App/ColFamily/App-DomainName-*) on all nodes. What am I missing? Thanks, Casey