[ https://issues.apache.org/jira/browse/CASSANDRA-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388615#comment-14388615 ]
Philip Thompson commented on CASSANDRA-8387: -------------------------------------------- Unfortunately, this is as expected. Multiple clients should not be trying to create the same table. If you really want some sort of Create Table via LWT, please open a new ticket marked "Improvement" for that feature request. > Schema inconsistency (cached vs schema_columnfamilies) > ------------------------------------------------------ > > Key: CASSANDRA-8387 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8387 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: C* 2.1.1 3-node cluster > Reporter: Marcus Olsson > > While running some tests on a 3-node cluster running C* 2.1.1 we encountered > a problem creating the same table schema twice(on different nodes). One thing > to note is that one of the nodes clock was ~4 seconds behind the others, but > I don't think that's the problem since the exception was reproduced here > aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html. > While running the same create table statement more than once(on different > clients) the logs outputted this on one of the nodes: > {noformat} > (node x.x.x.1): > 2014-11-25T16:11:44.651+0100 INFO [SharedPool-Worker-2] > MigrationManager.java:248 Create new ColumnFamily: > org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, > componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, > kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, > kind=COMPACT_VALUE, componentIndex=null, indexName=null, > indexType=null}],compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", > > "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] > ... > 2014-11-25T16:11:44.667+0100 INFO [MigrationStage:1] DefsTables.java:373 > Loading > org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, > componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, > kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, > kind=COMPACT_VALUE, componentIndex=null, indexName=null, > indexType=null}],compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", > > "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] > ... > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) > ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > 2014-11-25T16:11:44.859+0100 ERROR [SharedPool-Worker-2] > ErrorMessage.java:243 Unexpected exception during request > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4- > b1b6-017ad0689f5d; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:397) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:374) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:249) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:114) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:80) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:481) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:133) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) > [apache-cassandra-2.1.1.jar:2.1.1] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_51] > at > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > [apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > [apache-cassandra-2.1.1.jar:2.1.1] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d > ; expected 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > ~[na:1.7.0_51] > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:393) > ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 17 common frames omitted > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_51] > ... 1 common frames omitted > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) > ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > In my understanding this means that it first created the table(with UUID > 5e..) and then got a schema update from another node with UUID 5b.. . > And the node who created the 5b.. table had this: > {noformat} > (node x.x.x.2): > 2014-11-25T16:11:40.580+0100 INFO [SharedPool-Worker-1] > MigrationManager.java:248 Create new ColumnFamily: > org.apache.cassandra.config.CFMetaData@225c1975[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, > componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, > kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, > kind=COMPACT_VALUE, componentIndex=null, indexName=null, > indexType=null}],compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", > > "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] > ... > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) > ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > And the last node had this: > {noformat} > (node x.x.x.3): > 2014-11-25T16:11:44.363+0100 INFO [MigrationStage:1] DefsTables.java:373 > Loading > org.apache.cassandra.config.CFMetaData@767512bf[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id, > type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, > componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, > kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, > ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, > kind=COMPACT_VALUE, componentIndex=null, indexName=null, > indexType=null}],compactionStrategyClass=class > org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL", > > "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true] > ... > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException: Column family ID > mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:295) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:49) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Column > family ID mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected > 5bc7c980-74b5-11e4-9131-d9b94a3d8927) > at > org.apache.cassandra.config.CFMetaData.validateCompatility(CFMetaData.java:1254) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:1186) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1167) > ~[apache-cassandra-2.1.1.jar:2.1.1] > ... 11 common frames omitted > {noformat} > Not a problem at all, since they all seemed to agree that 5b.. was the schema > to go with and everything went fine with inserting data into the table. The > data directory contained the 5b.. directory and not 5e.. (on all nodes). The > problem occured later when node x.x.x.3 was restarted it created the > directory test/test-5e.. in the data directory. When performing a repair on > the node it gave these kinds of exceptions: > {noformat} > (node x.x.x.3): > 2014-11-25T16:21:31.726+0100 WARN [STREAM-IN-/x.x.x.2] > StreamSession.java:590 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] > Retrying for following error > java.io.IOException: CF 5bc7c980-74b5-11e4-9131-d9b94a3d8927 was dropped > during streaming > at > org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:71) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) > [apache-cassandra-2.1.1.jar:2.1.1] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > ... > 2014-11-25T16:21:31.726+0100 ERROR [STREAM-IN-/x.x.x.2] > StreamSession.java:470 [Stream #bc054793-74b6-11e4-baf6-fbb25862655d] > Streaming error occurred > java.lang.IllegalArgumentException: Unknown type 0 > at > org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:89) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > ... > 2014-11-25T16:21:31.738+0100 ERROR [AntiEntropySessions:1] > RepairSession.java:303 [repair #bbf34630-74b6-11e4-baf6-fbb25862655d] session > completed with the following error > org.apache.cassandra.exceptions.RepairException: [repair > #bbf34630-74b6-11e4-baf6-fbb25862655d on test/test, > (3708140006327119811,3721360916800446611]] Sync failed between /x.x.x.3 and > /x.x.x.2 > at > org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:389) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:126) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_51] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] > {noformat} > And the other nodes had exceptions like: > {noformat} > (node x.x.x.2) > 2014-11-25T16:21:28.145+0100 ERROR [STREAM-OUT-/x.x.x.3] > StreamSession.java:470 [Stream #bc309d50-74b6-11e4-baf6-fbb25862655d] > Streaming error occurred > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_51] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > ~[na:1.7.0_51] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > ~[na:1.7.0_51] > at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_51] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) > ~[na:1.7.0_51] > at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) > ~[na:1.7.0_51] > at java.nio.channels.Channels.writeFully(Channels.java:98) > ~[na:1.7.0_51] > at java.nio.channels.Channels.access$000(Channels.java:61) > ~[na:1.7.0_51] > at java.nio.channels.Channels$1.write(Channels.java:174) > ~[na:1.7.0_51] > at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.7.0_51] > at java.nio.channels.Channels$1.write(Channels.java:155) > ~[na:1.7.0_51] > at > org.apache.cassandra.io.util.DataOutputStreamPlus.write(DataOutputStreamPlus.java:45) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.AbstractDataOutput.writeLong(AbstractDataOutput.java:232) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:406) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata$ChunkSerializer.serialize(CompressionMetadata.java:402) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:60) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.serialize(CompressionInfo.java:47) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:135) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:120) > ~[apache-cassandra-2.1.1.jar:2.1.1] > {noformat} > In System.schema_columnfamilies the id for test.test is 5e.. on all nodes, > but the nodes that hadn't restarted still used the 5b.. id for the table > (atleast in the data directory, probably a cached id). > The problem seems to be that the schema change for 5e.. is rejected on all > nodes (java.lang.RuntimeException: > org.apache.cassandra.exceptions.ConfigurationException thrown), while the > mutation for it still gets applied in schema_columnfamilies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)