Hello Aaron, thanks for your email. - That's pretty small, try m1.xlarge.
Yes, this is small. We are aware of that, but that doesn't seem to be the actual problem. But we cannot see any reason why this shouldn't work as a test environment. After we get a fair understanding we are going to invest on proper hardware. - 1.0.7 ships with thrift 0.6 - What client are you using ? If you have rolled your own client try using one of - the pre-built ones to rule out errors in your code. So, we are now using the right thrift version I guess, unless there are significant changes between 0.6.1 and 0.6. But if that's the case, why are we still getting 'old-client' errors??? At the moment we use thrift directly. We might start developing our own client using C#. - mmm 1.83 GB message size. Something is not right there. Do you have any ideas what could be causing that? We are definitely not trying to store such a large message. - 208 MB message size which is too big (max is 16MB) followed by out of memory. We cannot figure out why messages appear to be so large. We are aware of the 16MB limit and we are not even close to that limit. What could be causing such a large message size? - Do you get these errors with a stock 1.0.X install and a pre-built client ? We have not tested it with a higher level client yet. Do you think we should not be using thrift alone? Could that be what causes all these errors? Thanks in advance for your help, Regards, Vasilis On Wed, Jul 4, 2012 at 11:54 AM, aaron morton <aa...@thelastpickle.com>wrote: > We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core), > > That's pretty small, try m1.xlarge. > > e are still not sure what version of thrift to use with Cassandra 1.0.7 > (we are still getting the same message regarding the 'old client'). > > 1.0.7 ships with thrift 0.6 > What client are you using ? If you have rolled your own client try using > one of the pre-built ones to rule out errors in your code. > > org.apache.thrift.TException: Message length exceeded: 1970238464 > > mmm 1.83 GB message size. Something is not right there. > > > org.apache.thrift.TException: Message length exceeded: 218104076 > > 208 MB message size which is too big (max is 16MB) followed by out of > memory. > > Do you get these errors with a stock 1.0.X install and a pre-built client ? > > Cheers > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 3/07/2012, at 9:57 AM, Vasileios Vlachos wrote: > > Hello All, > > We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core), > running Ubuntu 12.04. We have three nodes in the cluster and we hit only > one node from our application. Thrift version is 0.6.1 (we changed from 0.8 > because we thought there was a compatibility problem between thrift and > Cassandra ('old client' according to the output.log). We are still not sure > what version of thrift to use with Cassandra 1.0.7 (we are still getting > the same message regarding the 'old client'). I would appreciate any help > on that please. > > Below, I am sharing the errors we are getting from the output.log file. > First three errors are not responsible for the crash, only the OOM error > is, but something seems to be really wrong there... > > Error #1 > > ERROR 14:00:12,057 Thrift error occurred during processing of message. > org.apache.thrift.TException: Message length exceeded: 1970238464 > at > org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393) > at > org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121) > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60) > at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355) > at > org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Error #2 > > ERROR 14:03:48,004 Error occurred during processing of message. > java.lang.StringIndexOutOfBoundsException: String index out of range: - > 2147418111 > at java.lang.String.checkBounds(String.java:397) > at java.lang.String.<init>(String.java:442) > at > org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339) > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:210) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Error #3 > > ERROR 14:07:24,415 Thrift error occurred during processing of message. > org.apache.thrift.protocol.TProtocolException: Missing version in > readMessageBegin, old client? > at > org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > Error #4 > > ERROR 16:07:10,168 Thrift error occurred during processing of message. > org.apache.thrift.TException: Message length exceeded: 218104076 > at > org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393) > at > org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:352) > at > org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:347) > at > org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > java.lang.OutOfMemoryError: Java heap space > Dumping heap to /var/lib/cassandra/java_1341224307.hprof ... > INFO 16:07:18,882 GC for Copy: 886 ms for 1 collections, 2242700896 used; > max is 2670985216 > Java HotSpot(TM) 64-Bit Server VM warning: record is too large > Heap dump file created [4429997807 bytes in 95.755 secs] > INFO 16:08:54,749 GC for ConcurrentMarkSweep: 1157 ms for 4 collections, > 2246857528 used; max is 2670985216 > WARN 16:08:54,761 Heap is 0.8412092715978552 full. You may need to reduce > memtable and/or cache sizes. > Cassandra will now flush up to the two largest memtables to free up > memory. > Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't > want Cassandra to do this automatically > ERROR 16:08:54,761 Fatal exception in thread Thread[Thrift:446,5,main] > java.lang.OutOfMemoryError: Java heap space > at java.util.HashMap.<init>(HashMap.java:187) > at java.util.HashMap.<init>(HashMap.java:199) > at > org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18953) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441) > at > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > INFO 16:08:54,760 InetAddress /10.128.16.110 is now dead. > INFO 16:08:54,764 InetAddress /10.128.16.112 is now dead. > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > First three errors appear a lot of times before error #4, which actually > causes the crash. 10.128.16.110 is the node our application hits. Although > the log suggests that 10.128.16.112 died, it did not. We ran 'nodetool > ring' on 10.128.16.112 and only 10.128.16.110 appeared to be down. > > Proper hardware might solve some of our problems, but we need a fair > understanding before we move on. At the moment we cannot get a stable > cluster for more than 12 hours. After that, 10.128.16.110 dies and the > output.log has the same errors. > > Any help would be much appreciated. Please, let me know if you need more > information in order to figure out what is going on. > > Thank you in advance. > > -- > Kind Regards, > > Vasilis > > >
