I don’t know what it is but I also saw “empty” keyspaces via CQL while migrating an existing test cluster from 2.0.9 to 2.1.0 (final release bits prior to labelling). Since I was doing this manually (and had cqlsh problems due to python change) I figured it might have been me.
My observation was the same - this wasn’t actual data corruption but some nodes’ in memory state was incorrect - I didn’t investigate the problem too much as it soon went away, however it seemed to happen when there was schema disagreement in the cluster. Fixing that and restarting the nodes solved it. > On Dec 1, 2014, at 7:24 PM, Mark Greene <green...@gmail.com> wrote: > > I'm running Cassandra 2.1.0. > > I was attempting to drop two keyspaces via cqlsh and encountered an error in > the CLI as well as the appearance of losing all my keyspaces. Below is the > output from my cqlsh session: > > > ---------------------------------------------------------------------------- > > $ cqlsh > Connected to Production Cluster at 127.0.0.1:9042 <http://127.0.0.1:9042/>. > [cqlsh 5.0.1 | Cassandra 2.1.0 | CQL spec 3.2.0 | Native protocol v3] > Use HELP for help. > cqlsh> desc keyspaces; > > contacts_index contacts_testing contacts system "OpsCenter" system_traces > > > cqlsh> drop keyspace contacts_index; > > cqlsh> drop keyspace contacts; > <ErrorMessage code=0000 [Server error] message="java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.NullPointerException"> > > cqlsh> drop keyspace contacts; > <ErrorMessage code=0000 [Server error] message="java.lang.RuntimeException: > java.util.concurrent.ExecutionException: java.lang.IllegalStateException: One > row required, 0 found"> > > cqlsh> desc keyspaces; > > <empty> <------ OH SHIT > > ------------------------------------------------------------------------------ > > After it appeared that I had lost all my keyspaces, I looked at the > system.log and found this: (full log attached) > > ERROR [MigrationStage:1] 2014-12-01 23:54:05,622 CassandraDaemon.java:166 - > Exception in thread Thread[MigrationStage:1,5,main] > java.lang.IllegalStateException: One row required, 0 found > at > org.apache.cassandra.cql3.UntypedResultSet$FromResultSet.one(UntypedResultSet.java:78) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:275) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at org.apache.cassandra.db.DefsTables.mergeKeyspaces(DefsTables.java:230) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:186) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:164) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.0.jar:2.1.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > > At this point I wasn't sure quite what to do about this and did a rolling > restart of the entire ring. After which, the keyspaces that were not > attempted to be deleted returned when running 'desc keyspaces' and my > intended keyspaces to be deleted had been removed as expected. > > Strangely enough, because we run OpsCenter, we lost the dashboards we had > configured. Not a total deal breaker, but concerning that data loss occurred > here assuming it's related. > > > Anyone run into something like this before? > <system.log>
smime.p7s
Description: S/MIME cryptographic signature
