Re: Thrift version and OOM errors
Hello, Thanks for the help. There was a problem in the code actually... The connection object was not thread safe. That is why the messages were so big. After fixing that we do not get any errors. The cluster seems stable. Thanks again for all the help. Regards, Vasilis On Thu, Jul 5, 2012 at 11:32 PM, aaron morton wrote: > agree. > > It's a good idea to remove as many variables and possible and get to a > stable/known state. Use a clean install and a well known client and see if > the problems persist. > > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 5/07/2012, at 4:58 PM, Tristan Seligmann wrote: > > On Jul 4, 2012 2:02 PM, "Vasileios Vlachos" > wrote: > > > > Any ideas what could be causing strange message lengths? > > One cause of this that I've seen is a client using unframed Thrift > transport while the server expects framed, or vice versa. I suppose a > similar cause could be something that is not a Thrift client at all > mistakenly connecting to Cassandra's Thrift port. > > >
Re: Thrift version and OOM errors
agree. It's a good idea to remove as many variables and possible and get to a stable/known state. Use a clean install and a well known client and see if the problems persist. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 5/07/2012, at 4:58 PM, Tristan Seligmann wrote: > On Jul 4, 2012 2:02 PM, "Vasileios Vlachos" > wrote: > > > > Any ideas what could be causing strange message lengths? > > One cause of this that I've seen is a client using unframed Thrift transport > while the server expects framed, or vice versa. I suppose a similar cause > could be something that is not a Thrift client at all mistakenly connecting > to Cassandra's Thrift port. >
Re: Thrift version and OOM errors
On Jul 4, 2012 2:02 PM, "Vasileios Vlachos" wrote: > > Any ideas what could be causing strange message lengths? One cause of this that I've seen is a client using unframed Thrift transport while the server expects framed, or vice versa. I suppose a similar cause could be something that is not a Thrift client at all mistakenly connecting to Cassandra's Thrift port.
Re: Thrift version and OOM errors
We also get negative message lengths occasionally... Please see below:
ERROR 12:49:00,777 Thrift error occurred during processing of message.
org.apache.thrift.TException: Negative length: -2147483634
at
org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:388)
at
org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.cassandra.thrift.Column.read(Column.java:528)
at
org.apache.cassandra.thrift.ColumnOrSuperColumn.read(ColumnOrSuperColumn.java:507)
at org.apache.cassandra.thrift.Mutation.read(Mutation.java:353)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
-
Any ideas what could be causing strange message lengths?
Thanks,
Vasilis
On Wed, Jul 4, 2012 at 12:55 PM, Vasileios Vlachos <
vasileiosvlac...@gmail.com> wrote:
> Hello Aaron, thanks for your email.
>
> - That's pretty small, try m1.xlarge.
>
> Yes, this is small. We are aware of that, but that doesn't seem to be the
> actual problem. But we cannot see any reason why this shouldn't work as a
> test environment. After we get a fair understanding we are going to invest
> on proper hardware.
>
> - 1.0.7 ships with thrift 0.6
> - What client are you using ? If you have rolled your own client try using
> one of
> - the pre-built ones to rule out errors in your code.
>
> So, we are now using the right thrift version I guess, unless there are
> significant changes between 0.6.1 and 0.6. But if that's the case, why are
> we still getting 'old-client' errors???
>
> At the moment we use thrift directly. We might start developing our own
> client using C#.
>
> - mmm 1.83 GB message size. Something is not right there.
>
> Do you have any ideas what could be causing that? We are definitely not
> trying to store such a large message.
>
> - 208 MB message size which is too big (max is 16MB) followed by out of
> memory.
>
> We cannot figure out why messages appear to be so large. We are aware of
> the 16MB limit and we are not even close to that limit. What could be
> causing such a large message size?
>
> - Do you get these errors with a stock 1.0.X install and a pre-built client ?
>
> We have not tested it with a higher level client yet. Do you think we
> should not be using thrift alone? Could that be what causes all these
> errors?
>
> Thanks in advance for your help,
>
> Regards,
>
> Vasilis
>
>
>
> On Wed, Jul 4, 2012 at 11:54 AM, aaron morton wrote:
>
>> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
>>
>> That's pretty small, try m1.xlarge.
>>
>> e are still not sure what version of thrift to use with Cassandra 1.0.7
>> (we are still getting the same message regarding the 'old client').
>>
>> 1.0.7 ships with thrift 0.6
>> What client are you using ? If you have rolled your own client try using
>> one of the pre-built ones to rule out errors in your code.
>>
>> org.apache.thrift.TException: Message length exceeded: 1970238464
>>
>> mmm 1.83 GB message size. Something is not right there.
>>
>>
>> org.apache.thrift.TException: Message length exceeded: 218104076
>>
>> 208 MB message size which is too big (max is 16MB) followed by out of
>> memory.
>>
>> Do you get these errors with a stock 1.0.X install and a pre-built client
>> ?
>>
>> Cheers
>>
>>
>> -
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 3/07/2012, at 9:57 AM, Vasileios Vlachos wrote:
>>
>> Hello All,
>>
>> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1
>> Core), running Ubuntu 12.04. We have three nodes in the cluster and we hit
>> only one node from our application. Thrift version is 0.6.1 (we changed
>> from 0.8 because we thought there was a compatibility problem between
>> thrift and Cassandra ('old client' according to the output.log). We are
>> still not sure what version of thrift to use with Cassandra 1.0.7 (we are
>> still getting the same message regarding the 'old client'). I would
>> appreciate any help on that please.
>>
>> Below, I am sharing the errors we are getting from the output.log file.
>> First three errors are not responsible for the crash, only the OOM error
>> is, but something seems to be really wr
Re: Thrift version and OOM errors
Hello Aaron, thanks for your email.
- That's pretty small, try m1.xlarge.
Yes, this is small. We are aware of that, but that doesn't seem to be the
actual problem. But we cannot see any reason why this shouldn't work as a
test environment. After we get a fair understanding we are going to invest
on proper hardware.
- 1.0.7 ships with thrift 0.6
- What client are you using ? If you have rolled your own client try
using one of
- the pre-built ones to rule out errors in your code.
So, we are now using the right thrift version I guess, unless there are
significant changes between 0.6.1 and 0.6. But if that's the case, why are
we still getting 'old-client' errors???
At the moment we use thrift directly. We might start developing our own
client using C#.
- mmm 1.83 GB message size. Something is not right there.
Do you have any ideas what could be causing that? We are definitely
not trying to store such a large message.
- 208 MB message size which is too big (max is 16MB) followed by out of memory.
We cannot figure out why messages appear to be so large. We are aware of
the 16MB limit and we are not even close to that limit. What could be
causing such a large message size?
- Do you get these errors with a stock 1.0.X install and a pre-built client ?
We have not tested it with a higher level client yet. Do you think we
should not be using thrift alone? Could that be what causes all these
errors?
Thanks in advance for your help,
Regards,
Vasilis
On Wed, Jul 4, 2012 at 11:54 AM, aaron morton wrote:
> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
>
> That's pretty small, try m1.xlarge.
>
> e are still not sure what version of thrift to use with Cassandra 1.0.7
> (we are still getting the same message regarding the 'old client').
>
> 1.0.7 ships with thrift 0.6
> What client are you using ? If you have rolled your own client try using
> one of the pre-built ones to rule out errors in your code.
>
> org.apache.thrift.TException: Message length exceeded: 1970238464
>
> mmm 1.83 GB message size. Something is not right there.
>
>
> org.apache.thrift.TException: Message length exceeded: 218104076
>
> 208 MB message size which is too big (max is 16MB) followed by out of
> memory.
>
> Do you get these errors with a stock 1.0.X install and a pre-built client ?
>
> Cheers
>
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 3/07/2012, at 9:57 AM, Vasileios Vlachos wrote:
>
> Hello All,
>
> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
> running Ubuntu 12.04. We have three nodes in the cluster and we hit only
> one node from our application. Thrift version is 0.6.1 (we changed from 0.8
> because we thought there was a compatibility problem between thrift and
> Cassandra ('old client' according to the output.log). We are still not sure
> what version of thrift to use with Cassandra 1.0.7 (we are still getting
> the same message regarding the 'old client'). I would appreciate any help
> on that please.
>
> Below, I am sharing the errors we are getting from the output.log file.
> First three errors are not responsible for the crash, only the OOM error
> is, but something seems to be really wrong there...
>
> Error #1
>
> ERROR 14:00:12,057 Thrift error occurred during processing of message.
> org.apache.thrift.TException: Message length exceeded: 1970238464
> at
> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
> at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #2
>
> ERROR 14:03:48,004 Error occurred during processing of message.
> java.lang.StringIndexOutOfBoundsException: String index out of range: -
> 2147418111
> at java.lang.String.checkBounds(String.java:397)
> at java.lang.String.(String.java:442)
> at
> org.
Re: Thrift version and OOM errors
> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
That's pretty small, try m1.xlarge.
> e are still not sure what version of thrift to use with Cassandra 1.0.7 (we
> are still getting the same message regarding the 'old client').
1.0.7 ships with thrift 0.6
What client are you using ? If you have rolled your own client try using one of
the pre-built ones to rule out errors in your code.
> org.apache.thrift.TException: Message length exceeded: 1970238464
mmm 1.83 GB message size. Something is not right there.
> org.apache.thrift.TException: Message length exceeded: 218104076
208 MB message size which is too big (max is 16MB) followed by out of memory.
Do you get these errors with a stock 1.0.X install and a pre-built client ?
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 3/07/2012, at 9:57 AM, Vasileios Vlachos wrote:
> Hello All,
>
> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
> running Ubuntu 12.04. We have three nodes in the cluster and we hit only one
> node from our application. Thrift version is 0.6.1 (we changed from 0.8
> because we thought there was a compatibility problem between thrift and
> Cassandra ('old client' according to the output.log). We are still not sure
> what version of thrift to use with Cassandra 1.0.7 (we are still getting the
> same message regarding the 'old client'). I would appreciate any help on that
> please.
>
> Below, I am sharing the errors we are getting from the output.log file. First
> three errors are not responsible for the crash, only the OOM error is, but
> something seems to be really wrong there...
>
> Error #1
>
> ERROR 14:00:12,057 Thrift error occurred during processing of message.
> org.apache.thrift.TException: Message length exceeded: 1970238464
> at
> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
> at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #2
>
> ERROR 14:03:48,004 Error occurred during processing of message.
> java.lang.StringIndexOutOfBoundsException: String index out of range:
> -2147418111
> at java.lang.String.checkBounds(String.java:397)
> at java.lang.String.(String.java:442)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:210)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #3
>
> ERROR 14:07:24,415 Thrift error occurred during processing of message.
> org.apache.thrift.protocol.TProtocolException: Missing version in
> readMessageBegin, old client?
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #4
>
> ERROR 16:07:10,168 Thrift error occurred during processing of message.
> org.apache.thrift.TException: Message length exceeded: 218104076
> at
> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> at
Re: Thrift version and OOM errors
Just an update to correct something...
The application hits 10.128.16.111. The last lines of Error #4 suggest
that 10.128.16.110 and 10.128.16.112 where down because Cassandra service
was down on 10.128.16.111 and it could not detect the cluster (I think it
must be gossip related, right???).
Thanks,
Vasilis
On Mon, Jul 2, 2012 at 10:57 PM, Vasileios Vlachos <
vasileiosvlac...@gmail.com> wrote:
> Hello All,
>
> We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
> running Ubuntu 12.04. We have three nodes in the cluster and we hit only
> one node from our application. Thrift version is 0.6.1 (we changed from 0.8
> because we thought there was a compatibility problem between thrift and
> Cassandra ('old client' according to the output.log). We are still not sure
> what version of thrift to use with Cassandra 1.0.7 (we are still getting
> the same message regarding the 'old client'). I would appreciate any help
> on that please.
>
> Below, I am sharing the errors we are getting from the output.log file.
> First three errors are not responsible for the crash, only the OOM error
> is, but something seems to be really wrong there...
>
> Error #1
>
> ERROR 14:00:12,057 Thrift error occurred during processing of message.
> org.apache.thrift.TException: Message length exceeded: 1970238464
> at
> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121)
> at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
> at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #2
>
> ERROR 14:03:48,004 Error occurred during processing of message.
> java.lang.StringIndexOutOfBoundsException: String index out of range: -
> 2147418111
> at java.lang.String.checkBounds(String.java:397)
> at java.lang.String.(String.java:442)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:210)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #3
>
> ERROR 14:07:24,415 Thrift error occurred during processing of message.
> org.apache.thrift.protocol.TProtocolException: Missing version in
> readMessageBegin, old client?
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Error #4
>
> ERROR 16:07:10,168 Thrift error occurred during processing of message.
> org.apache.thrift.TException: Message length exceeded: 218104076
> at
> org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:352)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:347)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
> at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
> at
> java.u
Thrift version and OOM errors
Hello All,
We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
running Ubuntu 12.04. We have three nodes in the cluster and we hit only
one node from our application. Thrift version is 0.6.1 (we changed from 0.8
because we thought there was a compatibility problem between thrift and
Cassandra ('old client' according to the output.log). We are still not sure
what version of thrift to use with Cassandra 1.0.7 (we are still getting
the same message regarding the 'old client'). I would appreciate any help
on that please.
Below, I am sharing the errors we are getting from the output.log file.
First three errors are not responsible for the crash, only the OOM error
is, but something seems to be really wrong there...
Error #1
ERROR 14:00:12,057 Thrift error occurred during processing of message.
org.apache.thrift.TException: Message length exceeded: 1970238464
at
org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #2
ERROR 14:03:48,004 Error occurred during processing of message.
java.lang.StringIndexOutOfBoundsException: String index out of range:
-2147418111
at java.lang.String.checkBounds(String.java:397)
at java.lang.String.(String.java:442)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:210)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #3
ERROR 14:07:24,415 Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in
readMessageBegin, old client?
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #4
ERROR 16:07:10,168 Thrift error occurred during processing of message.
org.apache.thrift.TException: Message length exceeded: 218104076
at
org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:352)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:347)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/lib/cassandra/java_1341224307.hprof ...
INFO 16:07:18,882 GC for Copy: 886 ms for 1 collections, 2242700896 used;
max is 2670985216
Java HotSpot(TM) 64-Bit Server VM warning: record is too large
Heap dump file created [4429997807 bytes in 95.755 secs]
INFO 16:08:54,749 GC for ConcurrentMarkSweep: 1157 ms for 4 collections,
224685
