Saurabh a vague suggestion when you are dropping can you wait for sometime
to let the change propagate to other node.Also I see a replication factor 1
but you have 3 nodes....
On Jan 31, 2015 6:28 AM, "Saurabh Sethi" <saurabh_se...@symantec.com> wrote:
> Thanks Adam. I upgraded to 2.1.2 but still seeing this issue. Let me go in
> a bit more detail as to what I am seeing – When I create a keyspace for the
> first time in a 3 node cluster, it works fine but if I drop the keyspace
> and try to recreate it, I see that the node received the request to create
> it but it didn’t delegate the request to other nodes because of which the
> keyspace didn’t get created on other two nodes.
>
> But the request for creating a table does get propagated to other nodes
> and since they can’t find that keyspace they throw an exception.
>
> I have 2 nodes as seeds, so the seeds property in cassandra.yaml for all 3
> nodes looks like – seeds: "<node1ip>,<node3ip>"
>
> Following is that I see in system.log file when I enable DEBUG mode:
>
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,363 Message.java:437 -
> Received: STARTUP {CQL_VERSION=3.0.0}, v=3
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,364 Message.java:452 -
> Responding: READY, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,365 Message.java:437 -
> Received: QUERY select cluster_name from system.local, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,366 StorageProxy.java:1534
> - Estimated result rows per range: 20.057144; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Message.java:452 -
> Responding: ROWS [cluster_name(system, local),
> org.apache.cassandra.db.marshal.UTF8Type]
> | CASSANDRAEDPDEVCLUSTER
> ---, v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,386 Message.java:437 -
> Received: QUERY CREATE KEYSPACE IF NOT EXISTS TestMaxColumnsKeySpace WITH
> REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };, v=3
> INFO [SharedPool-Worker-1] 2015-01-30 16:39:37,387
> MigrationManager.java:229 - Create new Keyspace:
> KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
> strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
> userTypes=org.apache.cassandra.config.UTMetaData@5485ec68}
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,388 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 262698
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,389 FileCacheService.java:102
> - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-3-Data.db
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 394047
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:102
> - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1-Data.db
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> INFO [MigrationStage:1] 2015-01-30 16:39:37,392
> ColumnFamilyStore.java:840 - Enqueuing flush of schema_keyspaces: 496 (0%)
> on-heap, 0 (0%) off-heap
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,392
> ColumnFamilyStore.java:166 - scheduling flush in 3600000 ms
> INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,394 Memtable.java:325 -
> Writing Memtable-schema_keyspaces@922450609(138 serialized bytes, 3 ops,
> 0%/0% of on/off-heap limit)
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Statistics.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Statistics.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Filter.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Filter.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-TOC.txt
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-TOC.txt
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Digest.sha1
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Digest.sha1
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-CompressionInfo.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-CompressionInfo.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,411 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Index.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Index.db
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,412 FileUtils.java:161
> - Renaming
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Data.db
> to
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
> INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,413 Memtable.java:364 -
> Completed flushing
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db
> (175 bytes) for commitlog position ReplayPosition(segmentId=1422663061374,
> position=3040)
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413
> DataTracker.java:440 - adding
> bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10
> to list of files tracked for system.schema_keyspaces
> DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,413
> CompactionManager.java:179 - Scheduling a background task check for
> system.schema_keyspaces with SizeTieredCompactionStrategy
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:265 -
> discard completed log segments for ReplayPosition(segmentId=1422663061374,
> position=3040), column family b0f22357-4458-3cdb-9631-c43e59ce3676
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,414
> CompactionManager.java:218 - Checking system.schema_keyspaces
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,414 CommitLog.java:283 -
> Not safe to delete active commit log segment
> CommitLogSegment(bin/../data/commitlog/CommitLog-4-1422663061374.log);
> dirty is peers (37f71aca-7dc2-383b-a706-72528af04d4f), compaction_history
> (b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca), local
> (7ad54392-bcdd-35a6-8417-4e047860b377),
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_columnfamilies
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> SizeTieredCompactionStrategy.java:86 - Compaction buckets are
> [[SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-10-Data.db'),
> SSTableReader(path='/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-9-Data.db')]]
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_columns
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> SizeTieredCompactionStrategy.java:86 - Compaction buckets are []
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,415
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_triggers
> DEBUG [CompactionExecutor:13] 2015-01-30 16:39:37,415
> CompactionManager.java:229 - No tasks available
> DEBUG [MemtablePostFlush:24] 2015-01-30 16:39:37,416
> ColumnFamilyStore.java:879 - forceFlush requested but everything is clean
> in schema_usertypes
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,416 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,417 FileCacheService.java:150
> - Estimated memory usage is 1182141 compared to actual usage 525396
> DEBUG [MigrationStage:1] 2015-01-30 16:39:37,425 MigrationManager.java:418
> - Gossiping my schema version b5291c1d-6635-3627-928f-f5a0f0c27ec1
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,425 Message.java:452 -
> Responding: RESULT schema change CREATED KEYSPACE testmaxcolumnskeyspace,
> v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,502 Message.java:437 -
> Received: QUERY SELECT peer, rpc_address, schema_version FROM system.peers,
> v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,502 Message.java:437 -
> Received: QUERY SELECT schema_version FROM system.local WHERE key='local',
> v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,503 StorageProxy.java:1534
> - Estimated result rows per range: 23.2875; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,504
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 394047
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,505 StorageProxy.java:1271
> - Read: 2 ms.
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,505 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,506 Message.java:452 -
> Responding: ROWS [peer(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][rpc_address(system,
> peers),
> org.apache.cassandra.db.marshal.InetAddressType][schema_version(system,
> peers), org.apache.cassandra.db.marshal.UUIDType]
> | 10.65.119.193 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> | 10.65.58.24 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,506 Message.java:452 -
> Responding: ROWS [schema_version(system, local),
> org.apache.cassandra.db.marshal.UUIDType]
> | b5291c1d-6635-3627-928f-f5a0f0c27ec1
> ---, v=3
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,509 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_keyspaces WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,509 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_usertypes WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,510
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,510 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_columnfamilies WHERE
> keyspace_name = 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,510
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 394047
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,510 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,510 Message.java:437 -
> Received: QUERY SELECT * FROM system.schema_columns WHERE keyspace_name =
> 'testmaxcolumnskeyspace', v=3
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,510 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-4] 2015-01-30 16:39:37,511
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,511 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_keyspaces),
> org.apache.cassandra.db.marshal.UTF8Type][durable_writes(system,
> schema_keyspaces),
> org.apache.cassandra.db.marshal.BooleanType][strategy_class(system,
> schema_keyspaces),
> org.apache.cassandra.db.marshal.UTF8Type][strategy_options(system,
> schema_keyspaces), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,511
> FileCacheService.java:102 - Evicting cold readers for
> /usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-3-Data.db
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,511 StorageProxy.java:1271
> - Read: 0 ms.
> DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,511 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_usertypes),
> org.apache.cassandra.db.marshal.UTF8Type][type_name(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.UTF8Type][field_names(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)][field_types(system,
> schema_usertypes),
> org.apache.cassandra.db.marshal.ListType(org.apache.cassandra.db.marshal.UTF8Type)]
> ---, v=3
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,513
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 656745
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,513 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][bloom_filter_fp_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][caching(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][cf_id(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UUIDType][column_aliases(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][comment(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_class(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compaction_strategy_options(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][comparator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][compression_parameters(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][default_time_to_live(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][default_validator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][dropped_columns(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.LongType)][gc_grace_seconds(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][is_dense(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.BooleanType][key_aliases(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][key_validator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][local_read_repair_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][max_compaction_threshold(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][max_index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][memtable_flush_period_in_ms(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][min_compaction_threshold(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][min_index_interval(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.Int32Type][read_repair_chance(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.DoubleType][speculative_retry(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][subcomparator(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][type(system,
> schema_columnfamilies),
> org.apache.cassandra.db.marshal.UTF8Type][value_alias(system,
> schema_columnfamilies), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 StorageProxy.java:1271
> - Read: 2 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,514 Message.java:452 -
> Responding: ROWS [keyspace_name(system, schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][columnfamily_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][column_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][component_index(system,
> schema_columns),
> org.apache.cassandra.db.marshal.Int32Type][index_name(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][index_options(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][index_type(system,
> schema_columns), org.apache.cassandra.db.marshal.UTF8Type][type(system,
> schema_columns),
> org.apache.cassandra.db.marshal.UTF8Type][validator(system,
> schema_columns), org.apache.cassandra.db.marshal.UTF8Type]
> ---, v=3
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,516 Message.java:437 -
> Received: QUERY SELECT * FROM system.local WHERE key='local', v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 Message.java:437 -
> Received: QUERY SELECT * FROM system.peers, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,516 StorageProxy.java:1534
> - Estimated result rows per range: 23.2875; requested rows: 2147483647,
> ranges.size(): 1; concurrent range requests: 1
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517
> FileCacheService.java:150 - Estimated memory usage is 1050792 compared to
> actual usage 525396
> DEBUG [SharedPool-Worker-5] 2015-01-30 16:39:37,517
> FileCacheService.java:150 - Estimated memory usage is 1182141 compared to
> actual usage 656745
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 StorageProxy.java:1271
> - Read: 1 ms.
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-6] 2015-01-30 16:39:37,518 Message.java:452 -
> Responding: ROWS [key(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][bootstrapped(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][cluster_name(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][cql_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][data_center(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][gossip_generation(system, local),
> org.apache.cassandra.db.marshal.Int32Type][host_id(system, local),
> org.apache.cassandra.db.marshal.UUIDType][native_protocol_version(system,
> local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system,
> local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][schema_version(system, local),
> org.apache.cassandra.db.marshal.UUIDType][thrift_version(system, local),
> org.apache.cassandra.db.marshal.UTF8Type][tokens(system, local),
> org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system,
> local),
> org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]
> | local | COMPLETED | CASSANDRAEDPDEVCLUSTER | 3.2.0 | DC1 | 1422663061 |
> 19dbb3ec-ada4-4af0-9d39-95eb89a0a8e7 | 3 |
> org.apache.cassandra.dht.Murmur3Partitioner | RAC1 | 2.1.2 |
> b5291c1d-6635-3627-928f-f5a0f0c27ec1 | 19.39.0 | …
> ---, v=3
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Tracing.java:157 -
> request complete
> DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Message.java:452 -
> Responding: ROWS [peer(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][data_center(system,
> peers), org.apache.cassandra.db.marshal.UTF8Type][host_id(system, peers),
> org.apache.cassandra.db.marshal.UUIDType][preferred_ip(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][rack(system, peers),
> org.apache.cassandra.db.marshal.UTF8Type][release_version(system, peers),
> org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, peers),
> org.apache.cassandra.db.marshal.InetAddressType][schema_version(system,
> peers), org.apache.cassandra.db.marshal.UUIDType][tokens(system, peers),
> org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)]
> | 10.65.119.193 | DC1 | 12bdc695-1647-462c-8167-e386603643c9 | null |
> RAC1 | 2.1.2 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
> | 10.65.58.24 | DC1 | a1cf7f2c-fa1b-4d7d-95c9-381bcc2b36f9 | null | RAC1
> | 2.1.2 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
> ---, v=3
>
> Thanks,
> Saurabh
> From: Adam Holmberg <adam.holmb...@datastax.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Friday, January 30, 2015 at 9:25 AM
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Re: Unable to create a keyspace
>
> I would first ask if you could upgrade to the latest version of Cassandra
> 2.1.x (presently 2.1.2).
>
> If the issue still occurs consistently, it would be interesting to turn up
> logging on the client side and see if something is causing the client to
> disconnect during the metadata refresh following the schema change. If this
> yields further information, please raise the issue on the driver's user
> mailing list.
>
> Adam Holmberg
>
> On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi <saurabh_se...@symantec.com
> > wrote:
>
>> I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4
>> driver to create a keyspace followed by creating a column family within
>> that keyspace from my unit test.
>>
>> But I do not see the keyspace getting created and the code for creating
>> column family fails because it cannot find the keyspace. I see the
>> following in the system.log file:
>>
>> INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472
>> MigrationManager.java:229 - Create new Keyspace:
>> KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
>> strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
>> userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
>> INFO [MigrationStage:1] 2015-01-28 17:59:08,476
>> ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%)
>> on-heap, 0 (0%) off-heap
>> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326
>> - Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3
>> ops, 0%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360
>> - Completed flushing
>> /usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db
>> (175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803,
>> position=10514)
>>
>> This issue doesn’t happen always. My test runs fine sometimes but once it
>> gets into this state, it remains there for a while and I can constantly
>> reproduce this.
>>
>> Also, when this issue happens for the first time, I also see the
>> following error message in system.log file:
>>
>> ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 -
>> Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_05]
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_05]
>> at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>> ~[na:1.8.0_05]
>> at
>> io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at
>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>> ~[netty-all-4.0.20.Final.jar:4.0.20.Final]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
>>
>>
>> Anyone has any idea what might be going on here?
>>
>> Thanks,
>> Saurabh
>>
>
>
