Re: AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-18 Thread aaron morton 
> Double check the stack size is to set 100K see 
> https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.sh#L187
Sorry that was a late night typo, should have been 180K like in the link. 

Cheers

-
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 18/07/2013, at 10:19 AM, Dave Brosius  wrote:

> 
> What is your -Xss set to. If it's below 256m, set it there, and see if you 
> still have the issues.
> 
> 
> 
> - Original Message -
> From: "Julio Quierati"  
> Sent: Tue, July 16, 2013 14:26
> Subject: AbstractCassandraDaemon.java (line 134) Exception in thread
> 
> Hello,
>  
> At least 2 times a day I'm having hundreds of log entry related to exception 
> below, the network bottleneck seems, anyone know how to solve, or encountered 
> this problem
>  
> Using
>   C* version 1.1.4
> &n bsp;  and
> Java version "1.7.0_21"
> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> AWS m1.xlarge
>  
> ERROR [Thrift:967696] 2013-07-16 12:36:12,514 AbstractCassandraDaemon.java 
> (line 134) Exception in thread Thread[Thrift:967696,5,main]
> java.lang.StackOverflowError
>   at java.net.SocketInputStream.socketRead0(Native Method)
>   at java.net.SocketInputStream.read(SocketInputStream.java:150)
>   at java.net.SocketInputStream.read(SocketInputStream.java:121)
>   at java.io.BufferedInputStream.fill(Bu fferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>   at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at org.apache.thrift.protocol.TBinaryProtocol.readAll(T 
> BinaryProtocol.java:378)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> ERROR [Thrift:966717] 2013-07-16 12:36:17,152 AbstractCassandraDaem on.java 
> (line 134) Exception in thread Thread[Thrift:966717,5,main]
> java.lang.StackOverflowError
>   at java.net.SocketInputStream.socketRead0(Native Method)
>   at java.net.SocketInputStream.read(SocketInputStream.java:150)
>   at java.net.SocketInputStream.read(SocketInputStream.java:121)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>   at org.apache.thrift.transport.T Transport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>   at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>   at org.apache.c 
> assandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
>  INFO [OptionalTasks:1] 2013-07-16 12:38:07,847 MeteredFlusher.java (line 62) 
> flushing high-traffic column family CFS

Re: AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-17 Thread Dave Brosius 
 What is your -Xss set to. If it's below 256m, set it there, and see if you 
still have the issues.  - Original Message -From: "Julio 
Quierati" >;julio.quier...@gmail.com

Re: AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-17 Thread Julio Quierati 
System up again xD

nodetool removetoken 129020446491903175361975561488312102412
nodetool removetoken force
nodetool removetoken 42514072861213193928274919594412671125
nodetool removetoken force
nodetool removetoken 99246027852756676471313599864858497886
nodetool removetoken force




--
Julio Quierati
User Linux #492973

OpenPGP Key: 0xC9A064FA578E0D60
8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60


2013/7/17 Julio Quierati 

> Hi Aaron,
>
> My cassandra-env is by setting "-Xss160k" changed to "-Xss100k".
>
> if [ "`uname`" = "Linux" ] ; then
> # reduce the per-thread stack size to minimize the impact of Thrift
> # thread-per-client.  (Best practice is for client connections to
> # be pooled anyway.) Only do so on Linux where it is known to be
> # supported.
> if startswith "$JVM_VERSION" '1.7.'
> then
> JVM_OPTS="$JVM_OPTS -Xss160k"
> else
> JVM_OPTS="$JVM_OPTS -Xss128k"
> fi
> fi
>
> cassandra error after restart.
>
> The stack size specified is too small, Specify at least 160k
> Cannot create Java VM
> Service exit with a return value of 1
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-4275
> I set 256k.
>
> A new problem, after restart, the nodetool ring, this viewing old nodes
> removed a 1 months ago, these nodesnot exist anymore :<
>
>
> nodetool ring
> Note: Ownership information does not include topology, please specify a
> keyspace.
> Address DC  RackStatus State   Load
>  OwnsToken
>
>  134691819273918816419698471612174905938
> 172.16.1.48 datacenter1 rack1   Down   Normal  ?
> 45.82%  42514072861213193928274919594412671125
> 172.16.4.212datacenter1 rack1   Up Normal  21.59 GB
>  4.18%   49621227543684200553854819754232853074
> 172.16.3.221datacenter1 rack1   Down   Normal  ?
> 29.17%  99246027852756676471313599864858497886
> 172.16.5.57 datacenter1 rack1   Down   Normal  ?
> 17.50%  129020446491903175361975561488312102412
> 172.16.3.93 datacenter1 rack1   Up Normal  21.96 GB
>  3.33%   134691819273918816419698471612174905938
>
>
> I have 2 nodes, now errors, system is down now :<
> 172.16.3.93  datacenter1 rack1   Up Normal  21.96 GB
>  3.33%   134691819273918816419698471612174905938
> 172.16.4.212datacenter1 rack1   Up Normal  21.59 GB
>  4.18%   49621227543684200553854819754232853074
>
>
>
> --
> Julio Quierati
> User Linux #492973
>
> OpenPGP Key: 0xC9A064FA578E0D60
> 8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60
>
>
> 2013/7/17 aaron morton 
>
>> Double check the stack size is to set 100K see
>> https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.sh#L187
>>
>> Cheers
>>
>> -
>> Aaron Morton
>> Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 17/07/2013, at 6:26 AM, Julio Quierati 
>> wrote:
>>
>> > Hello,
>> >
>> > At least 2 times a day I'm having hundreds of log entry related to
>> exception below, the network bottleneck seems, anyone know how to solve, or
>> encountered this problem
>> >
>> > Using
>> >   C* version 1.1.4
>> >and
>> > Java version "1.7.0_21"
>> > Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>> > AWS m1.xlarge
>> >
>> > ERROR [Thrift:967696] 2013-07-16 12:36:12,514
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[Thrift:967696,5,main]
>> > java.lang.StackOverflowError
>> >   at java.net.SocketInputStream.socketRead0(Native Method)
>> >   at java.net.SocketInputStream.read(SocketInputStream.java:150)
>> >   at java.net.SocketInputStream.read(SocketInputStream.java:121)
>> >   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> >   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> >   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> >   at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> >   at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >   at
>> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTranspo

Re: AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-17 Thread Julio Quierati 
Hi Aaron,

My cassandra-env is by setting "-Xss160k" changed to "-Xss100k".

if [ "`uname`" = "Linux" ] ; then
# reduce the per-thread stack size to minimize the impact of Thrift
# thread-per-client.  (Best practice is for client connections to
# be pooled anyway.) Only do so on Linux where it is known to be
# supported.
if startswith "$JVM_VERSION" '1.7.'
then
JVM_OPTS="$JVM_OPTS -Xss160k"
else
JVM_OPTS="$JVM_OPTS -Xss128k"
fi
fi

cassandra error after restart.

The stack size specified is too small, Specify at least 160k
Cannot create Java VM
Service exit with a return value of 1


https://issues.apache.org/jira/browse/CASSANDRA-4275
I set 256k.

A new problem, after restart, the nodetool ring, this viewing old nodes
removed a 1 months ago, these nodesnot exist anymore :<


nodetool ring
Note: Ownership information does not include topology, please specify a
keyspace.
Address DC  RackStatus State   LoadOwns
   Token

   134691819273918816419698471612174905938
172.16.1.48 datacenter1 rack1   Down   Normal  ?
45.82%  42514072861213193928274919594412671125
172.16.4.212datacenter1 rack1   Up Normal  21.59 GB
 4.18%   49621227543684200553854819754232853074
172.16.3.221datacenter1 rack1   Down   Normal  ?
29.17%  99246027852756676471313599864858497886
172.16.5.57 datacenter1 rack1   Down   Normal  ?
17.50%  129020446491903175361975561488312102412
172.16.3.93 datacenter1 rack1   Up Normal  21.96 GB
 3.33%   134691819273918816419698471612174905938


I have 2 nodes, now errors, system is down now :<
172.16.3.93  datacenter1 rack1   Up Normal  21.96 GB
 3.33%   134691819273918816419698471612174905938
172.16.4.212datacenter1 rack1   Up Normal  21.59 GB
 4.18%   49621227543684200553854819754232853074



--
Julio Quierati
User Linux #492973

OpenPGP Key: 0xC9A064FA578E0D60
8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60


2013/7/17 aaron morton 

> Double check the stack size is to set 100K see
> https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.sh#L187
>
> Cheers
>
> -
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 17/07/2013, at 6:26 AM, Julio Quierati 
> wrote:
>
> > Hello,
> >
> > At least 2 times a day I'm having hundreds of log entry related to
> exception below, the network bottleneck seems, anyone know how to solve, or
> encountered this problem
> >
> > Using
> >   C* version 1.1.4
> >and
> > Java version "1.7.0_21"
> > Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
> > Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> > AWS m1.xlarge
> >
> > ERROR [Thrift:967696] 2013-07-16 12:36:12,514
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[Thrift:967696,5,main]
> > java.lang.StackOverflowError
> >   at java.net.SocketInputStream.socketRead0(Native Method)
> >   at java.net.SocketInputStream.read(SocketInputStream.java:150)
> >   at java.net.SocketInputStream.read(SocketInputStream.java:121)
> >   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> >   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> >   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> >   at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> >   at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> >   at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> >   at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> >   at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> >   at
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> >   at
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> >   at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> >   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
> >   at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
> >   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav

Re: AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-17 Thread aaron morton 
Double check the stack size is to set 100K see 
https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.sh#L187

Cheers

-
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 17/07/2013, at 6:26 AM, Julio Quierati  wrote:

> Hello,
> 
> At least 2 times a day I'm having hundreds of log entry related to exception 
> below, the network bottleneck seems, anyone know how to solve, or encountered 
> this problem
> 
> Using
>   C* version 1.1.4
>and
> Java version "1.7.0_21"
> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> AWS m1.xlarge
> 
> ERROR [Thrift:967696] 2013-07-16 12:36:12,514 AbstractCassandraDaemon.java 
> (line 134) Exception in thread Thread[Thrift:967696,5,main]
> java.lang.StackOverflowError
>   at java.net.SocketInputStream.socketRead0(Native Method)
>   at java.net.SocketInputStream.read(SocketInputStream.java:150)
>   at java.net.SocketInputStream.read(SocketInputStream.java:121)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>   at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> ERROR [Thrift:966717] 2013-07-16 12:36:17,152 AbstractCassandraDaemon.java 
> (line 134) Exception in thread Thread[Thrift:966717,5,main]
> java.lang.StackOverflowError
>   at java.net.SocketInputStream.socketRead0(Native Method)
>   at java.net.SocketInputStream.read(SocketInputStream.java:150)
>   at java.net.SocketInputStream.read(SocketInputStream.java:121)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>   at 
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>   at 
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>   at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>   at 
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>   at 
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
>  INFO [OptionalTasks:1] 2013-07-16 12:38:07,847 MeteredFlusher.java (line 62) 
> flushing high-traffic column family CFS(Keyspace='zm', 
> ColumnFamily='deviceConnectionLog') (estimated 200441950 bytes)
>  INFO [OptionalTasks:1] 2013-07-16 12:38:07,848 ColumnFamilyStore.java (line 
> 659) Enqueuing flush of 
> Memtable-deviceConnectionLog@64568169(20293642/200441950 serialized/live 
> bytes, 114490 ops)
>  INFO [FlushWriter:4306] 2013-07-16 12:38:07,848 Memtable.java (line 264) 
> Writing Memtable-d

AbstractCassandraDaemon.java (line 134) Exception in thread

2013-07-16 Thread Julio Quierati 
Hello,

At least 2 times a day I'm having hundreds of log entry related to
exception below, the network bottleneck seems, anyone know how to solve, or
encountered this problem

Using
  C* version 1.1.4
   and
Java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
AWS m1.xlarge

ERROR [Thrift:967696] 2013-07-16 12:36:12,514 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[Thrift:967696,5,main]
java.lang.StackOverflowError
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
 at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
 at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
 at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
 at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:722)
ERROR [Thrift:966717] 2013-07-16 12:36:17,152 AbstractCassandraDaemon.java
(line 134) Exception in thread Thread[Thrift:966717,5,main]
java.lang.StackOverflowError
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
 at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
 at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
 at
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
 at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
 at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:722)
 INFO [OptionalTasks:1] 2013-07-16 12:38:07,847 MeteredFlusher.java (line
62) flushing high-traffic column family CFS(Keyspace='zm',
ColumnFamily='deviceConnectionLog') (estimated 200441950 bytes)
 INFO [OptionalTasks:1] 2013-07-16 12:38:07,848 ColumnFamilyStore.java
(line 659) Enqueuing flush of
Memtable-deviceConnectionLog@64568169(20293642/200441950
serialized/live bytes, 114490 ops)
 INFO [FlushWriter:4306] 2013-07-16 12:38:07,848 Memtable.java (line 264)
Writing Memtable-deviceConnectionLog@64568169(20293642/200441950
serialized/live bytes, 114490 ops)
 INFO [FlushWriter:4306] 2013-07-16 12:38:09,154 Memtable.java (line 305)
Completed flushing
/mnt/cassandra/data/zm/deviceConnectionLog/zm-deviceConnectionLog-he-5083-Data.db
(5747695 bytes) for commitlog position
ReplayPosition(segmentId=1282719983369784, position=28288612)
 INFO [OptionalTasks:1] 2013-07-16 12:40:35,610 ColumnFamilyStore.java
(line 659) Enqueuing flush of
Memtable-userNotificationsHistory@1933122113(554/13847
serialized/live bytes, 10 ops)
 INFO [FlushWriter:4307] 2013-07-16 12:40:35,611 Memtable.java (line 264)
Writing Memtable-userNotificationsHistory@1933122113(554/13847
serialized/live bytes, 10 ops)
 INFO [OptionalTasks:1] 2013-07-16 12:40:35,611 ColumnFamilyStore.java
(line 659) Enqueuing flush of
Memtable-userPendingNotifications@1553457921(647/8729
serialized/live bytes, 10 ops)
 INFO [FlushWriter:4307] 2013-07-16 12:40:35,717 Memtable.java