Re: Unable to create a keyspace
Thanks Carlos for pointing that out. The clock on one of the nodes was not in sync and fixing that solved the issue. From: Jan cne...@yahoo.commailto:cne...@yahoo.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org, Jan cne...@yahoo.commailto:cne...@yahoo.com Date: Saturday, January 31, 2015 at 9:59 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Unable to create a keyspace Saurabh; a) How exactly are the three nodes hosted. b) Can you take down node 2 and create the keyspace from node 1 c) Can you take down node 1 and create the keyspace from node2 d) Do the nodes see each other with 'nodetool status' cheers Jan/ C* Architect On Saturday, January 31, 2015 5:40 AM, Carlos Rolo r...@pythian.commailto:r...@pythian.com wrote: Something that can cause weird behavior is the machine clocks not being properly synced. I didn't read the thread in full detail, so disregard this if it is not the case. --
Re: Unable to create a keyspace
Saurabh; a) How exactly are the three nodes hosted. b) Can you take down node 2 and create the keyspace from node 1c) Can you take down node 1 and create the keyspace from node2d) Do the nodes see each other with 'nodetool status' cheersJan/ C* Architect On Saturday, January 31, 2015 5:40 AM, Carlos Rolo r...@pythian.com wrote: Something that can cause weird behavior is the machine clocks not being properly synced. I didn't read the thread in full detail, so disregard this if it is not the case. --
Re: Unable to create a keyspace
I would first ask if you could upgrade to the latest version of Cassandra
2.1.x (presently 2.1.2).
If the issue still occurs consistently, it would be interesting to turn up
logging on the client side and see if something is causing the client to
disconnect during the metadata refresh following the schema change. If this
yields further information, please raise the issue on the driver's user
mailing list.
Adam Holmberg
On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi saurabh_se...@symantec.com
wrote:
I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4
driver to create a keyspace followed by creating a column family within
that keyspace from my unit test.
But I do not see the keyspace getting created and the code for creating
column family fails because it cannot find the keyspace. I see the
following in the system.log file:
INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472
MigrationManager.java:229 - Create new Keyspace:
KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
INFO [MigrationStage:1] 2015-01-28 17:59:08,476
ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%)
on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326 -
Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3 ops,
0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360 -
Completed flushing
/usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db
(175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803,
position=10514)
This issue doesn’t happen always. My test runs fine sometimes but once it
gets into this state, it remains there for a while and I can constantly
reproduce this.
Also, when this issue happens for the first time, I also see the following
error message in system.log file:
ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 -
Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.8.0_05]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.8.0_05]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
~[na:1.8.0_05]
at
io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:311)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:878)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:225)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:114)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_05]
Anyone has any idea what might be going on here?
Thanks,
Saurabh
Re: Unable to create a keyspace
][native_protocol_version(system,
local), org.apache.cassandra.db.marshal.UTF8Type][partitioner(system,
local), org.apache.cassandra.db.marshal.UTF8Type][rack(system, local),
org.apache.cassandra.db.marshal.UTF8Type][release_version(system, local),
org.apache.cassandra.db.marshal.UTF8Type][schema_version(system, local),
org.apache.cassandra.db.marshal.UUIDType][thrift_version(system, local),
org.apache.cassandra.db.marshal.UTF8Type][tokens(system, local),
org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)][truncated_at(system,
local),
org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UUIDType,org.apache.cassandra.db.marshal.BytesType)]
| local | COMPLETED | CASSANDRAEDPDEVCLUSTER | 3.2.0 | DC1 | 1422663061 |
19dbb3ec-ada4-4af0-9d39-95eb89a0a8e7 | 3 |
org.apache.cassandra.dht.Murmur3Partitioner | RAC1 | 2.1.2 |
b5291c1d-6635-3627-928f-f5a0f0c27ec1 | 19.39.0 | …
---, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Tracing.java:157 -
request complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,519 Message.java:452 -
Responding: ROWS [peer(system, peers),
org.apache.cassandra.db.marshal.InetAddressType][data_center(system,
peers), org.apache.cassandra.db.marshal.UTF8Type][host_id(system, peers),
org.apache.cassandra.db.marshal.UUIDType][preferred_ip(system, peers),
org.apache.cassandra.db.marshal.InetAddressType][rack(system, peers),
org.apache.cassandra.db.marshal.UTF8Type][release_version(system, peers),
org.apache.cassandra.db.marshal.UTF8Type][rpc_address(system, peers),
org.apache.cassandra.db.marshal.InetAddressType][schema_version(system,
peers), org.apache.cassandra.db.marshal.UUIDType][tokens(system, peers),
org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type)]
| 10.65.119.193 | DC1 | 12bdc695-1647-462c-8167-e386603643c9 | null |
RAC1 | 2.1.2 | 10.65.119.193 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
| 10.65.58.24 | DC1 | a1cf7f2c-fa1b-4d7d-95c9-381bcc2b36f9 | null | RAC1
| 2.1.2 | 10.65.58.24 | b5291c1d-6635-3627-928f-f5a0f0c27ec1 | …
---, v=3
Thanks,
Saurabh
From: Adam Holmberg adam.holmb...@datastax.com
Reply-To: user@cassandra.apache.org user@cassandra.apache.org
Date: Friday, January 30, 2015 at 9:25 AM
To: user@cassandra.apache.org user@cassandra.apache.org
Subject: Re: Unable to create a keyspace
I would first ask if you could upgrade to the latest version of Cassandra
2.1.x (presently 2.1.2).
If the issue still occurs consistently, it would be interesting to turn up
logging on the client side and see if something is causing the client to
disconnect during the metadata refresh following the schema change. If this
yields further information, please raise the issue on the driver's user
mailing list.
Adam Holmberg
On Wed, Jan 28, 2015 at 8:19 PM, Saurabh Sethi saurabh_se...@symantec.com
wrote:
I have a 3 node Cassandra 2.1.0 cluster and I am using datastax 2.1.4
driver to create a keyspace followed by creating a column family within
that keyspace from my unit test.
But I do not see the keyspace getting created and the code for creating
column family fails because it cannot find the keyspace. I see the
following in the system.log file:
INFO [SharedPool-Worker-1] 2015-01-28 17:59:08,472
MigrationManager.java:229 - Create new Keyspace:
KSMetaData{name=testmaxcolumnskeyspace, strategyClass=SimpleStrategy,
strategyOptions={replication_factor=1}, cfMetaData={}, durableWrites=true,
userTypes=org.apache.cassandra.config.UTMetaData@370ad1d3}
INFO [MigrationStage:1] 2015-01-28 17:59:08,476
ColumnFamilyStore.java:856 - Enqueuing flush of schema_keyspaces: 512 (0%)
on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,477 Memtable.java:326
- Writing Memtable-schema_keyspaces@1664717092(138 serialized bytes, 3
ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:22] 2015-01-28 17:59:08,486 Memtable.java:360
- Completed flushing
/usr/share/apache-cassandra-2.1.0/bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-118-Data.db
(175 bytes) for commitlog position ReplayPosition(segmentId=1422485457803,
position=10514)
This issue doesn’t happen always. My test runs fine sometimes but once it
gets into this state, it remains there for a while and I can constantly
reproduce this.
Also, when this issue happens for the first time, I also see the
following error message in system.log file:
ERROR [SharedPool-Worker-1] 2015-01-28 15:08:24,286 ErrorMessage.java:218 -
Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_05]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.8.0_05]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.8.0_05]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_05
Re: Unable to create a keyspace
I repeat the test after verifying that the keyspace has been dropped.
Also, I do not want any replication as of now that’s why replication factor is
1. But I don’t think that should be the reason for the command not propagating
to other nodes.
One more thing that I observed right now is that if I create the keyspace from
node 1 and node 2, it doesn’t get created but if I try to create it from node
3, it does get created. Node1 and Node 2 are Vms hosted on the same physical
hardware, not sure if that has anything to do.
Thanks,
Saurabh
From: Asit KAUSHIK
asitkaushikno...@gmail.commailto:asitkaushikno...@gmail.com
Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Date: Friday, January 30, 2015 at 5:33 PM
To: user@cassandra.apache.orgmailto:user@cassandra.apache.org
user@cassandra.apache.orgmailto:user@cassandra.apache.org
Subject: Re: Unable to create a keyspace
Saurabh a vague suggestion when you are dropping can you wait for sometime to
let the change propagate to other node.Also I see a replication factor 1 but
you have 3 nodes
On Jan 31, 2015 6:28 AM, Saurabh Sethi
saurabh_se...@symantec.commailto:saurabh_se...@symantec.com wrote:
Thanks Adam. I upgraded to 2.1.2 but still seeing this issue. Let me go in a
bit more detail as to what I am seeing – When I create a keyspace for the first
time in a 3 node cluster, it works fine but if I drop the keyspace and try to
recreate it, I see that the node received the request to create it but it
didn’t delegate the request to other nodes because of which the keyspace didn’t
get created on other two nodes.
But the request for creating a table does get propagated to other nodes and
since they can’t find that keyspace they throw an exception.
I have 2 nodes as seeds, so the seeds property in cassandra.yaml for all 3
nodes looks like – seeds: node1ip,node3ip
Following is that I see in system.log file when I enable DEBUG mode:
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,363 Message.java:437 -
Received: STARTUP {CQL_VERSION=3.0.0}, v=3
DEBUG [SharedPool-Worker-3] 2015-01-30 16:39:37,364 Message.java:452 -
Responding: READY, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,365 Message.java:437 -
Received: QUERY select cluster_name from system.local, v=3
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,366 StorageProxy.java:1534 -
Estimated result rows per range: 20.057144; requested rows: 2147483647,
ranges.size(): 1; concurrent range requests: 1
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Tracing.java:157 - request
complete
DEBUG [SharedPool-Worker-2] 2015-01-30 16:39:37,369 Message.java:452 -
Responding: ROWS [cluster_name(system, local),
org.apache.cassandra.db.marshal.UTF8Type]
| CASSANDRAEDPDEVCLUSTER
---, v=3
DEBUG [SharedPool-Worker-1] 2015-01-30 16:39:37,386 Message.java:437 -
Received: QUERY CREATE KEYSPACE IF NOT EXISTS TestMaxColumnsKeySpace WITH
REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };, v=3
INFO [SharedPool-Worker-1] 2015-01-30 16:39:37,387 MigrationManager.java:229 -
Create new Keyspace: KSMetaData{name=testmaxcolumnskeyspace,
strategyClass=SimpleStrategy, strategyOptions={replication_factor=1},
cfMetaData={}, durableWrites=true,
userTypes=org.apache.cassandra.config.UTMetaData@5485ec68}
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,388 FileCacheService.java:150 -
Estimated memory usage is 1182141 compared to actual usage 262698
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,389 FileCacheService.java:102 -
Evicting cold readers for
/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-3-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 -
Estimated memory usage is 1182141 compared to actual usage 394047
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:102 -
Evicting cold readers for
/usr/share/apache-cassandra-2.1.2/bin/../data/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1-Data.db
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,390 FileCacheService.java:150 -
Estimated memory usage is 1182141 compared to actual usage 525396
INFO [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:840 -
Enqueuing flush of schema_keyspaces: 496 (0%) on-heap, 0 (0%) off-heap
DEBUG [MigrationStage:1] 2015-01-30 16:39:37,392 ColumnFamilyStore.java:166 -
scheduling flush in 360 ms
INFO [MemtableFlushWriter:10] 2015-01-30 16:39:37,394 Memtable.java:325 -
Writing Memtable-schema_keyspaces@922450609(138 serialized bytes, 3 ops, 0%/0%
of on/off-heap limit)
DEBUG [MemtableFlushWriter:10] 2015-01-30 16:39:37,410 FileUtils.java:161 -
Renaming
bin/../data/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-tmp-ka-10-Statistics.db
to
bin/../data/data/system/schema_keyspaces
