Re: terrible read/write latency fluctuation

2015-11-02 Thread
the bottleneck. > > On Sun, Oct 25, 2015 at 3:47 PM, 曹志富 <cao.zh...@gmail.com> wrote: > >> I will try to trace a read that take > 20msec >> . >> >> just HDD.no delete just 60days ttl.value size is small ,max length is 140. >> >> >> My

Re: terrible read/write latency fluctuation

2015-10-25 Thread
I will try to trace a read that take > 20msec . just HDD.no delete just 60days ttl.value size is small ,max length is 140. My data like Time Series . date of 90% reads which timestamp < 7days. data almost just insert ,with a lit update.

CqlOutputFormat with auth

2015-10-23 Thread
Hadoop2.6,cassandra2.1.6. Here is the exception stack: Error: java.lang.RuntimeException: InvalidRequestException(why:You have not logged in) at org.apache.cassandra.hadoop.cql3.CqlRecordWriter.(CqlRecordWriter.java:121) at

Re: unusual GC log

2015-10-20 Thread
y special settings in > cassandra.yaml; are you running with stock GC settings in cassandra-env.sh? > what JDK/OS? > > On Oct 19, 2015, at 11:40 PM, 曹志富 <cao.zh...@gmail.com> wrote: > > INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - > ParNew G

unusual GC log

2015-10-19 Thread
INFO [Service Thread] 2015-10-20 10:42:47,854 GCInspector.java:252 - ParNew GC in 476ms. CMS Old Gen: 4288526240 -> 4725514832; Par Eden Space: 671088640 -> 0; INFO [Service Thread] 2015-10-20 10:42:50,870 GCInspector.java:252 - ParNew GC in 423ms. CMS Old Gen: 4725514832 -> 5114687560; Par

Re: abnormal log after remove a node

2015-09-01 Thread
ip state recover from this. > > C*heers, > > Alain > > 2015-06-19 11:40 GMT+02:00 曹志富 <cao.zh...@gmail.com>: > >> I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from >> this cluster using nodetool decommission. >> >> But tody

Re: node join ant decommission exception

2015-06-29 Thread
] -- Ranger Tsao 2015-06-29 21:33 GMT+08:00 曹志富 cao.zh...@gmail.com: error when node join cluster: WARN 13:30:18 UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=91748db0-9af4

Re: node join ant decommission exception

2015-06-29 Thread
(IncomingTcpConnection.java:170) ~[apache-cassandra-2.1.6.jar:2.1.6] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:88) ~[apache-cassandra-2.1.6.jar:2.1.6] -- Ranger Tsao 2015-06-29 18:01 GMT+08:00 曹志富 cao.zh...@gmail.com: Hi

Range not found after nodetool decommission

2015-06-24 Thread
ERROR [OptionalTasks:1] 2015-06-25 08:56:19,156 CassandraDaemon.java:223 - Exception in thread Thread[OptionalTasks:1,5,main] java.lang.AssertionError: -110036444293069784 not found in -- Ranger Tsao

Re: system-hints compaction all the time

2015-06-22 Thread
this is unusual. -- Ranger Tsao 2015-06-22 16:06 GMT+08:00 Jason Wee peich...@gmail.com: what's your questions? On Mon, Jun 22, 2015 at 12:05 AM, 曹志富 cao.zh...@gmail.com wrote: the logger like this : INFO [CompactionExecutor:501] 2015-06-21 21:42

system-hints compaction all the time

2015-06-21 Thread
the logger like this : INFO [CompactionExecutor:501] 2015-06-21 21:42:36,306 CompactionTask.java:140 - Compacting [SSTableReader(path='/home/ant/apache-cassandra-2.1.6/bin/../data/data/system/hints/system-hints-ka-365-Data.db')] INFO [CompactionExecutor:501] 2015-06-21 21:42:37,782

exception when anti-entropy repair

2015-06-20 Thread
ERROR [RepairJobTask:3] 2015-06-21 00:18:02,187 RepairJob.java:145 - Error occurred during snapshot phase java.lang.RuntimeException: Could not create snapshot at /172.19.104.107 at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:78)

cfstats ERROR

2015-06-20 Thread
error: /home/ant/apache-cassandra-2.1.6/bin/../data/data/blogger/edgestore/blogger-edgestore-tmplink-ka-146100-Data.db -- StackTrace -- java.lang.AssertionError: /home/ant/apache-cassandra-2.1.6/bin/../data/data/blogger/edgestore/blogger-edgestore-tmplink-ka-146100-Data.db at

abnormal log after remove a node

2015-06-19 Thread
I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from this cluster using nodetool decommission. But tody I find some log like this: INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - InetAddress /172.19.105.41 is now DOWN INFO [GossipStage:1] 2015-06-19

seems a doc error

2015-06-10 Thread
In ths doc :Enabling JMX authentication http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureJmxAuthentication.html In the chapter : $ nodetool status -u cassandra -pw cassandra should change to $ nodetool -u cassandra -pw cassandra status

auto clear data with ttl

2015-06-08 Thread
I have C* 2.1.5,store some data with ttl.Reduce the gc_grace_seconds to zero. But it seems has no effect. Did I miss something? -- Ranger Tsao

Re: auto clear data with ttl

2015-06-08 Thread
it. Sent from my iPhone On Jun 8, 2015, at 7:26 PM, 曹志富 cao.zh...@gmail.com wrote: I have C* 2.1.5,store some data with ttl.Reduce the gc_grace_seconds to zero. But it seems has no effect. Did I miss something? -- Ranger Tsao

Re: what this error mean

2015-05-29 Thread
, May 29, 2015 at 9:39 AM, 曹志富 cao.zh...@gmail.com wrote: I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split brain many times。 check the log I found this: INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176 Memtable.java:378 - Completed flushing /home/ant

what this error mean

2015-05-28 Thread
I have a 25 noedes C* cluster with C* 2.1.3. These days a node occur split brain many times。 check the log I found this: INFO [MemtableFlushWriter:118] 2015-05-29 08:07:39,176 Memtable.java:378 - Completed flushing

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

2015-05-03 Thread
Hi guys: I havle C* 2.1.3 cluster,25 nodes ,running in JDK_1.7.0_71, CentOS 2.6.32-220.el6.x86_64,4 Core,32GB RAM. Today one of the nodes,has some error like this: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code at

Re: how to clear data from disk

2015-03-09 Thread
nodetool clearsnapshot -- Ranger Tsao 2015-03-10 10:47 GMT+08:00 鄢来琼 laiqiong@gtafe.com: Hi ALL, After drop table, I found the data is not removed from disk, I should reduce the gc_grace_seconds before the drop operation. I have to wait for 10

C* 2.0.9 Compaction Error

2015-03-09 Thread
Hi,every one: I have a 12 nodes C* 2.0.9 cluster for titan.I found some error when doing compaction,the exception stack: java.lang.AssertionError: Added column does not sort as the last column at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:115) at

Run witch repair cmd when increase replication factor

2015-03-06 Thread
I want fo increase replication factor in my C* 2.1.3 cluster(rf chang from 2 to 3 for some keyspaces). I read the doc of Updating the replication factor http://www.datastax.com/documentation/cql/3.1/cql/cql_using/update_ks_rf_t.html . The step two is run the nodetool repair.But as I know nodetool

Re: Input/Output Error

2015-03-04 Thread
:18 AM, 曹志富 cao.zh...@gmail.com wrote: Some times My C* 2.1.3 cluster compaction or streaming occur this error ,do this because of disk or filesystem problem?? Thanks All. -- Ranger Tsao

Input/Output Error

2015-03-03 Thread
Some times My C* 2.1.3 cluster compaction or streaming occur this error ,do this because of disk or filesystem problem?? Thanks All. -- Ranger Tsao

Why nodetool netstats show this

2015-02-21 Thread
is the lastest joined in my cluster,his status is Up Normal. I want to know why this message?Shoul I remove this node? -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/

Re: can't delete tmp file

2015-02-19 Thread
Thanks you Roland -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer r.etzenham...@t-online.de: Hi, try 2.1.3 - with 2.1.2 this is normal. From the changelog: * Make sure we don't add

Node joining take a long time

2015-02-19 Thread
total I want know ,is there some problem with my cluster? -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/

Re: can't delete tmp file

2015-02-19 Thread
Just upgrade my cluster to 2.1.3??? -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-02-19 20:32 GMT+08:00 Roland Etzenhammer r.etzenham...@t-online.de: Hi, try 2.1.3 - with 2.1.2 this is normal. From the changelog: * Make

Re: Node joining take a long time

2015-02-19 Thread
So ,what can I do???Waiting for 2.1.4 or upgrade to 2.1.3?? -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli rc...@eventbrite.com: On Thu, Feb 19, 2015 at 7:34 AM, Mark Reddy mark.l.re

Re: Node joining take a long time

2015-02-19 Thread
First thank all of you. Almost three days,till right now the status is still Joining. My cluster per 650G a node. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-02-20 3:16 GMT+08:00 Robert Coli rc...@eventbrite.com: On Thu

Re: How to deal with too many sstables

2015-02-02 Thread
Just run nodetool repair. The nodes witch has many sstables are newest in my cluster.Before add these nodes to my cluster ,my cluster have not compaction automaticly because my cluster is an only write cluster. thanks. -- 曹志富 手机:18611121927 邮箱:caozf.zh

Re: How to deal with too many sstables

2015-02-02 Thread
You are right.I have already change cold_reads_to_omit to 0.0. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-02-03 14:15 GMT+08:00 Roland Etzenhammer r.etzenham...@t-online.de: Hi, maybe you are running into an issue

How to deal with too many sstables

2015-02-02 Thread
Hi,all: I have 18 nodes C* cluster with cassandra2.1.2.Some nodes have aboud 40,000+ sstables. my compaction strategy is STCS. Could someone give me some solution to deal with this situation. Thanks. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http

Re: SStables can't compat automaticly

2015-01-26 Thread
Your are right,currently My cluster only write ,when My cluster build after two month ,I will change to the default thresholds. Thanks for your reply. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-01-26 22:40 GMT+08:00 Eric

Re: SStables can't compat automaticly

2015-01-26 Thread
I read the CQL table properties again.This property could contorl the compaction . Right now My C* cluster only write without any read. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-01-26 17:41 GMT+08:00 Roland Etzenhammer

Re: SStables can't compat automaticly

2015-01-26 Thread
Yes I use cassandra 2.1.2 JDK is 1.7.0_71. I will try your solution. Thank you Roland Etzenhammer!!! -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-01-26 17:41 GMT+08:00 Roland Etzenhammer r.etzenham...@t-online.de: Hi

Re: SStables can't compat automaticly

2015-01-26 Thread
No,to confirm this I have run the command all my nodes:bin/nodetool enableautocompaction -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http://weibo.com/boliza/ 2015-01-26 16:49 GMT+08:00 Jason Wee peich...@gmail.com: Did you disable auto compaction

SStables can't compat automaticly

2015-01-25 Thread
to compact the sstables , it cost a long long time .Also the new sstables can't compat automatic. I trace the log , the CMS GC too often,almost 30 minute onetime. Could someone help me to solve this problem. -- 曹志富 手机:18611121927 邮箱:caozf.zh...@gmail.com 微博:http