RE: Problem joining new node to cluster in 1.1.1

2012-06-08 Thread Bryce Godfrey 
https://issues.apache.org/jira/browse/CASSANDRA-4323

Not sure if it's a dupe of what Brandon sent (4251), so created the bug anyway.

-Original Message-
From: Sylvain Lebresne [mailto:sylv...@datastax.com] 
Sent: Friday, June 08, 2012 9:08 AM
To: user@cassandra.apache.org
Subject: Re: Problem joining new node to cluster in 1.1.1

That very much look like a bug. Would you mind opening a ticket on 
https://issues.apache.org/jira/browse/CASSANDRA with those stack traces and 
maybe a little bit more precision on what you were doing when that happened?

--
Sylvain

On Fri, Jun 8, 2012 at 12:28 AM, Bryce Godfrey  
wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of 
> Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-D
> ata.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:
> 115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.ja
> va:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:3
> 61)
>
>     at 
> org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248
> )
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThro
> w(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30
> )
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown 
> Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 
> ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamil
> ySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbH
> andler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.j
> ava:59)
>
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamil
> ySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(
> RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
>

Re: Problem joining new node to cluster in 1.1.1

2012-06-08 Thread Brandon Williams 
This sounds related to https://issues.apache.org/jira/browse/CASSANDRA-4251

On Thu, Jun 7, 2012 at 5:28 PM, Bryce Godfrey  wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
>
>     at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,813 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1020
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>  

Re: Problem joining new node to cluster in 1.1.1

2012-06-08 Thread Sylvain Lebresne 
That very much look like a bug. Would you mind opening a ticket on
https://issues.apache.org/jira/browse/CASSANDRA with those stack
traces and maybe a little bit more precision on what you were doing
when that happened?

--
Sylvain

On Fri, Jun 8, 2012 at 12:28 AM, Bryce Godfrey
 wrote:
> As the new node starts up I get this error before boostrap starts:
>
>
>
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0
> serialized/live bytes, 1 ops)
>
> INFO 08:20:51,589 Completed flushing
> /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db
> (61 bytes)
>
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
>
> java.lang.IllegalArgumentException: value already present: 1015
>
>     at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
>
>     at
> com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
>
>     at
> com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
>
>     at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
>
>     at org.apache.cassandra.config.Schema.load(Schema.java:385)
>
>     at
> org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
>
>     at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
>
>     at
> org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
>
>     at
> org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
>
>     at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>     at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>     at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
>
>     at java.util.concurrent.FutureTask.run(Unknown Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> INFO 08:20:51,931 Enqueuing flush of
> Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
>
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178
> serialized/live bytes, 20 ops)
>
>
>
>
>
> Then it starts spewing these errors nonstop until I kill it.
>
>
>
> ERROR 08:21:45,959 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,814 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1019
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
>
>     at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
>
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
>
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>     at java.lang.Thread.run(Unknown Source)
>
> ERROR 08:21:45,813 Error in row mutation
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=1020
>
>     at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
>
>     at
> org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
>
>     at
> org.apache.cassandra.db

Problem joining new node to cluster in 1.1.1

2012-06-07 Thread Bryce Godfrey 
As the new node starts up I get this error before boostrap starts:

INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0 
serialized/live bytes, 1 ops)
INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0 
serialized/live bytes, 1 ops)
INFO 08:20:51,589 Completed flushing 
/opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db 
(61 bytes)
ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
java.lang.IllegalArgumentException: value already present: 1015
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
at 
com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
at org.apache.cassandra.config.Schema.load(Schema.java:385)
at org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
at 
org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
at 
org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
at 
org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
INFO 08:20:51,931 Enqueuing flush of 
Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178 
serialized/live bytes, 20 ops)


Then it starts spewing these errors nonstop until I kill it.

ERROR 08:21:45,959 Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at 
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR 08:21:45,814 Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at 
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR 08:21:45,813 Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1020
at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at 
org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at 
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at 
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR 08:21:45,813 Error in row mutation


I'm guessing the first error caused some column families to not be created?  
But