Re: java.lang.OutOfMemoryError: Java heap space

[Tobin Landricombe]

Hi Ben,

We're using the akka persistence layer which doesn't give me much scope for 
remodelling data.

So, on the assumption that the guys who wrote the persistence layer knew what 
they were doing, I followed your suggestion to increase RAM (still only to a 
miserly 8gig, which the startup script has decided means the JVM should be 
started with -Xms1968M, -Xmx1968M, -Xmn200M) and now the new nodes are coming 
up.

Thanks for your help,
Tobin

> On 12 Jun 2016, at 06:52, Ben Slater  wrote:
> 
> I should add - there is probably an option (c) of fiddling with a bunch of 
> tuning parameters to try to nurse things through with your current config but 
> I’m not sure that’s useful unless you really need to make the current set up 
> work for some reason.
> 
> On Sun, 12 Jun 2016 at 15:23 Ben Slater  wrote:
> Hi Tobin,
> 
> 4G RAM is a pretty small machine to be using to run Cassandra. As I 
> mentioned, 8G of heap is the normal recommendation for a production machine 
> which means you need at least 14-16G total (and can get performance benefit 
> from more).
> 
> I agree disk space doesn’t look to really be an issue here and I’m not sure 
> what impact degraded mode has but it doesn’t really sound good :-) (I think 
> it’s caused by Is swap disabled? : false - ie you have swap enabled which is 
> not recommended). 
> 
> In this case, I would expect that the relatively large partition(s) (175MB in 
> the warning) in conjunction with the low heap allocation is what is causing 
> C* to run out of heap. Heap exhaustion often manifests when C* has to compact 
> a large partition. When you add a new node the data that gets streamed across 
> has to be compacted which is why you’ll see it on the new node but node the 
> existing nodes (yet).
> 
> So, I’d say your options are either (a) get more memory and increase heap 
> space or (b) remodel your data with a partition key that does not create such 
> large partitions (generally, smaller is better if it meets your functional 
> needs and stay under 10MB to avoid having to tune specifically to meet the 
> needs of large partitions). And, there is a fair chance you need to do (b) 
> for a healthy cluster in the long run.
> 
> Cheers
> Ben
> 
> On Sat, 11 Jun 2016 at 20:52 Tobin Landricombe  wrote:
> Hi Ben,
> 
> I think the degraded mode is caused by one or both of these...
> • WARN  [main] 2016-06-10 14:23:01,690 StartupChecks.java:118 - 
> jemalloc shared library could not be preloaded to speed up memory allocations
> • WARN  [main] 2016-06-10 14:23:01,691 StartupChecks.java:150 - JMX 
> is not enabled to receive remote connections. Please see cassandra-env.sh for 
> more info.
> ...neither of which should cause the heap issue.
> 
> The disk space isn't low for our (test) usage and again shouldn't cause the 
> heap issue.
> 
> Which leaves the large partition. I couldn't find what is considered a large 
> partition. Is it possible that syncing the large partition is causing 
> problems? Why would it only affect the new node, not the running ones?
> 
> I looked at increasing the heap space but after reviewing the docs, the 
> current settings look correct for the machines.
> 
> All the nodes are running on VMs with 2 cores and 4gig RAM. Neither they nor 
> the hypervisor are showing much load.
> 
> Thanks for your help,
> Tobin
> 
> > On 10 Jun 2016, at 22:18, Ben Slater  wrote:
> >
> > The short-term fix is probably to try increasing heap space (in 
> > cassandra-env.sh). 8GB in the most standard but more may help in some 
> > circumstances.
> >
> > That said, your logs are pointing to a number of other issues which won’t 
> > be helping and probably need to be fixed for long-term stability:
> > - swap enabled ( Cassandra server running in degraded mode. Is swap 
> > disabled? : false,  Address space adequate? : true,  nofile limit adequate? 
> > : true, nproc limit adequate? : true)
> > - low disk space ( Only 36948 MB free across all data volumes. Consider 
> > adding more capacity to your cluster or removing obsolete snapshots)
> > - large partitions ( Writing large partition feed/messages:MANAGER:0 
> > (175811867 bytes))
> >
> > Cheers
> > Ben
> >
> > On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe  wrote:
> > Hi,
> >
> > I've been googling various parts of this all day but none of the 
> > suggestions seem to fit.
> >
> > I have 2 nodes, one of which is a seed. I'm trying to add a third node but, 
> > after a few minutes in the UJ state, the node dies with the above error 
> > (http://pastebin.com/iRvYfuAu).
> >
> > Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
> >
> > I've googled them but nothing seems appropriate.
> >
> > Debug log part 1: http://pastebin.com/b8ZSYtqV
> > Debug log part 2: http://pastebin.com/1Bbb7Vf8
> >
> > Thanks for any suggestions,
> > Tobin
> >
> > --
> > 
> > Ben 

Re: java.lang.OutOfMemoryError: Java heap space

[Ben Slater]

I should add - there is probably an option (c) of fiddling with a bunch of
tuning parameters to try to nurse things through with your current config
but I’m not sure that’s useful unless you really need to make the current
set up work for some reason.

On Sun, 12 Jun 2016 at 15:23 Ben Slater  wrote:

> Hi Tobin,
>
> 4G RAM is a pretty small machine to be using to run Cassandra. As I
> mentioned, 8G of heap is the normal recommendation for a production machine
> which means you need at least 14-16G total (and can get performance benefit
> from more).
>
> I agree disk space doesn’t look to really be an issue here and I’m not
> sure what impact degraded mode has but it doesn’t really sound good :-) (I
> think it’s caused by Is swap disabled? : false - ie you have swap enabled
> which is not recommended).
>
> In this case, I would expect that the relatively large partition(s) (175MB
> in the warning) in conjunction with the low heap allocation is what is
> causing C* to run out of heap. Heap exhaustion often manifests when C* has
> to compact a large partition. When you add a new node the data that gets
> streamed across has to be compacted which is why you’ll see it on the new
> node but node the existing nodes (yet).
>
> So, I’d say your options are either (a) get more memory and increase heap
> space or (b) remodel your data with a partition key that does not create
> such large partitions (generally, smaller is better if it meets your
> functional needs and stay under 10MB to avoid having to tune specifically
> to meet the needs of large partitions). And, there is a fair chance you
> need to do (b) for a healthy cluster in the long run.
>
> Cheers
> Ben
>
> On Sat, 11 Jun 2016 at 20:52 Tobin Landricombe  wrote:
>
>> Hi Ben,
>>
>> I think the degraded mode is caused by one or both of these...
>> • WARN  [main] 2016-06-10 14:23:01,690 StartupChecks.java:118 -
>> jemalloc shared library could not be preloaded to speed up memory
>> allocations
>> • WARN  [main] 2016-06-10 14:23:01,691 StartupChecks.java:150 -
>> JMX is not enabled to receive remote connections. Please see
>> cassandra-env.sh for more info.
>> ...neither of which should cause the heap issue.
>>
>> The disk space isn't low for our (test) usage and again shouldn't cause
>> the heap issue.
>>
>> Which leaves the large partition. I couldn't find what is considered a
>> large partition. Is it possible that syncing the large partition is causing
>> problems? Why would it only affect the new node, not the running ones?
>>
>> I looked at increasing the heap space but after reviewing the docs, the
>> current settings look correct for the machines.
>>
>> All the nodes are running on VMs with 2 cores and 4gig RAM. Neither they
>> nor the hypervisor are showing much load.
>>
>> Thanks for your help,
>> Tobin
>>
>> > On 10 Jun 2016, at 22:18, Ben Slater 
>> wrote:
>> >
>> > The short-term fix is probably to try increasing heap space (in
>> cassandra-env.sh). 8GB in the most standard but more may help in some
>> circumstances.
>> >
>> > That said, your logs are pointing to a number of other issues which
>> won’t be helping and probably need to be fixed for long-term stability:
>> > - swap enabled ( Cassandra server running in degraded mode. Is swap
>> disabled? : false,  Address space adequate? : true,  nofile limit adequate?
>> : true, nproc limit adequate? : true)
>> > - low disk space ( Only 36948 MB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots)
>> > - large partitions ( Writing large partition feed/messages:MANAGER:0
>> (175811867 bytes))
>> >
>> > Cheers
>> > Ben
>> >
>> > On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe 
>> wrote:
>> > Hi,
>> >
>> > I've been googling various parts of this all day but none of the
>> suggestions seem to fit.
>> >
>> > I have 2 nodes, one of which is a seed. I'm trying to add a third node
>> but, after a few minutes in the UJ state, the node dies with the above
>> error (http://pastebin.com/iRvYfuAu).
>> >
>> > Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
>> >
>> > I've googled them but nothing seems appropriate.
>> >
>> > Debug log part 1: http://pastebin.com/b8ZSYtqV
>> > Debug log part 2: http://pastebin.com/1Bbb7Vf8
>> >
>> > Thanks for any suggestions,
>> > Tobin
>> >
>> > --
>> > 
>> > Ben Slater
>> > Chief Product Officer, Instaclustr
>> > +61 437 929 798
>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>
-- 

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Re: java.lang.OutOfMemoryError: Java heap space

[Ben Slater]

Hi Tobin,

4G RAM is a pretty small machine to be using to run Cassandra. As I
mentioned, 8G of heap is the normal recommendation for a production machine
which means you need at least 14-16G total (and can get performance benefit
from more).

I agree disk space doesn’t look to really be an issue here and I’m not sure
what impact degraded mode has but it doesn’t really sound good :-) (I think
it’s caused by Is swap disabled? : false - ie you have swap enabled which
is not recommended).

In this case, I would expect that the relatively large partition(s) (175MB
in the warning) in conjunction with the low heap allocation is what is
causing C* to run out of heap. Heap exhaustion often manifests when C* has
to compact a large partition. When you add a new node the data that gets
streamed across has to be compacted which is why you’ll see it on the new
node but node the existing nodes (yet).

So, I’d say your options are either (a) get more memory and increase heap
space or (b) remodel your data with a partition key that does not create
such large partitions (generally, smaller is better if it meets your
functional needs and stay under 10MB to avoid having to tune specifically
to meet the needs of large partitions). And, there is a fair chance you
need to do (b) for a healthy cluster in the long run.

Cheers
Ben

On Sat, 11 Jun 2016 at 20:52 Tobin Landricombe  wrote:

> Hi Ben,
>
> I think the degraded mode is caused by one or both of these...
> • WARN  [main] 2016-06-10 14:23:01,690 StartupChecks.java:118 -
> jemalloc shared library could not be preloaded to speed up memory
> allocations
> • WARN  [main] 2016-06-10 14:23:01,691 StartupChecks.java:150 -
> JMX is not enabled to receive remote connections. Please see
> cassandra-env.sh for more info.
> ...neither of which should cause the heap issue.
>
> The disk space isn't low for our (test) usage and again shouldn't cause
> the heap issue.
>
> Which leaves the large partition. I couldn't find what is considered a
> large partition. Is it possible that syncing the large partition is causing
> problems? Why would it only affect the new node, not the running ones?
>
> I looked at increasing the heap space but after reviewing the docs, the
> current settings look correct for the machines.
>
> All the nodes are running on VMs with 2 cores and 4gig RAM. Neither they
> nor the hypervisor are showing much load.
>
> Thanks for your help,
> Tobin
>
> > On 10 Jun 2016, at 22:18, Ben Slater  wrote:
> >
> > The short-term fix is probably to try increasing heap space (in
> cassandra-env.sh). 8GB in the most standard but more may help in some
> circumstances.
> >
> > That said, your logs are pointing to a number of other issues which
> won’t be helping and probably need to be fixed for long-term stability:
> > - swap enabled ( Cassandra server running in degraded mode. Is swap
> disabled? : false,  Address space adequate? : true,  nofile limit adequate?
> : true, nproc limit adequate? : true)
> > - low disk space ( Only 36948 MB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots)
> > - large partitions ( Writing large partition feed/messages:MANAGER:0
> (175811867 bytes))
> >
> > Cheers
> > Ben
> >
> > On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe 
> wrote:
> > Hi,
> >
> > I've been googling various parts of this all day but none of the
> suggestions seem to fit.
> >
> > I have 2 nodes, one of which is a seed. I'm trying to add a third node
> but, after a few minutes in the UJ state, the node dies with the above
> error (http://pastebin.com/iRvYfuAu).
> >
> > Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
> >
> > I've googled them but nothing seems appropriate.
> >
> > Debug log part 1: http://pastebin.com/b8ZSYtqV
> > Debug log part 2: http://pastebin.com/1Bbb7Vf8
> >
> > Thanks for any suggestions,
> > Tobin
> >
> > --
> > 
> > Ben Slater
> > Chief Product Officer, Instaclustr
> > +61 437 929 798
>
> --

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Re: java.lang.OutOfMemoryError: Java heap space

[Tobin Landricombe]

Hi Ben,

I think the degraded mode is caused by one or both of these...
• WARN  [main] 2016-06-10 14:23:01,690 StartupChecks.java:118 - 
jemalloc shared library could not be preloaded to speed up memory allocations
• WARN  [main] 2016-06-10 14:23:01,691 StartupChecks.java:150 - JMX is 
not enabled to receive remote connections. Please see cassandra-env.sh for more 
info.
...neither of which should cause the heap issue.

The disk space isn't low for our (test) usage and again shouldn't cause the 
heap issue.

Which leaves the large partition. I couldn't find what is considered a large 
partition. Is it possible that syncing the large partition is causing problems? 
Why would it only affect the new node, not the running ones?

I looked at increasing the heap space but after reviewing the docs, the current 
settings look correct for the machines.

All the nodes are running on VMs with 2 cores and 4gig RAM. Neither they nor 
the hypervisor are showing much load.

Thanks for your help,
Tobin

> On 10 Jun 2016, at 22:18, Ben Slater  wrote:
> 
> The short-term fix is probably to try increasing heap space (in 
> cassandra-env.sh). 8GB in the most standard but more may help in some 
> circumstances.
> 
> That said, your logs are pointing to a number of other issues which won’t be 
> helping and probably need to be fixed for long-term stability:
> - swap enabled ( Cassandra server running in degraded mode. Is swap disabled? 
> : false,  Address space adequate? : true,  nofile limit adequate? : true, 
> nproc limit adequate? : true)
> - low disk space ( Only 36948 MB free across all data volumes. Consider 
> adding more capacity to your cluster or removing obsolete snapshots)
> - large partitions ( Writing large partition feed/messages:MANAGER:0 
> (175811867 bytes))
> 
> Cheers
> Ben
> 
> On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe  wrote:
> Hi,
> 
> I've been googling various parts of this all day but none of the suggestions 
> seem to fit.
> 
> I have 2 nodes, one of which is a seed. I'm trying to add a third node but, 
> after a few minutes in the UJ state, the node dies with the above error 
> (http://pastebin.com/iRvYfuAu).
> 
> Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
> 
> I've googled them but nothing seems appropriate.
> 
> Debug log part 1: http://pastebin.com/b8ZSYtqV
> Debug log part 2: http://pastebin.com/1Bbb7Vf8
> 
> Thanks for any suggestions,
> Tobin
> 
> -- 
> 
> Ben Slater
> Chief Product Officer, Instaclustr
> +61 437 929 798



smime.p7s
Description: S/MIME cryptographic signature

Re: java.lang.OutOfMemoryError: Java heap space

[Ben Slater]

The short-term fix is probably to try increasing heap space (in
cassandra-env.sh). 8GB in the most standard but more may help in some
circumstances.

That said, your logs are pointing to a number of other issues which won’t
be helping and probably need to be fixed for long-term stability:
- swap enabled ( Cassandra server running in degraded mode. Is swap
disabled? : false,  Address space adequate? : true,  nofile limit adequate?
: true, nproc limit adequate? : true)
- low disk space ( Only 36948 MB free across all data volumes. Consider
adding more capacity to your cluster or removing obsolete snapshots)
- large partitions ( Writing large partition feed/messages:MANAGER:0
(175811867 bytes))

Cheers
Ben

On Sat, 11 Jun 2016 at 01:11 Tobin Landricombe  wrote:

> Hi,
>
> I've been googling various parts of this all day but none of the
> suggestions seem to fit.
>
> I have 2 nodes, one of which is a seed. I'm trying to add a third node
> but, after a few minutes in the UJ state, the node dies with the above
> error (http://pastebin.com/iRvYfuAu).
>
> Here are the warnings from the logs: http://pastebin.com/vYLvsHrv
>
> I've googled them but nothing seems appropriate.
>
> Debug log part 1: http://pastebin.com/b8ZSYtqV
> Debug log part 2: http://pastebin.com/1Bbb7Vf8
>
> Thanks for any suggestions,
> Tobin
>
> --

Ben Slater
Chief Product Officer, Instaclustr
+61 437 929 798

Re: java.lang.OutOfMemoryError: Java heap space

[Gary Zhao]

BTW, the CPU usage on this node is pretty high, but data size is pretty
small.

  PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIMECPU COMMAND
28674 cassandr   89  250 9451M 8970M  525M sleep  32.1H   329% java

UN  8.92 GB256 35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
UN  5.32 GB256 33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
DN  339.94 MB  256 31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1


On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote:

 Hello

 I have a three nodes cluster. I noticed one node was always down.
 Restarting Cassandra fixes it but it will go down again after a couple of
 days. I'm pretty new to Cassandra so I'm wondering how I should
 troubleshoot it. Logs is as below.

  INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091
 ThriftServer.java (line 141) Stop listening to thrift clients
 ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:4,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 CassandraDaemon.java
 (line 196) Exception in thread Thread[ACCEPT-/10.20.132.44,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:3,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:2,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196)
 Exception in thread Thread[Thread-222,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.IllegalThreadStateException
 at java.lang.Thread.start(Thread.java:704)
  at
 org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
 at
 org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
  at
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java
 (line 181) Stop listening for CQL clients
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java
 (line 1251) Announcing shutdown
  INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524
 MessagingService.java (line 667) Waiting for messaging service to quiesce

 Thanks
 Gary

Re: java.lang.OutOfMemoryError: Java heap space

[Prem Yadav]

Are the virtual machines? The last time I had this issues was because of
VMWare ballooning.
If not, what versions of Cassandra and Java are you using?


On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao garyz...@gmail.com wrote:

 BTW, the CPU usage on this node is pretty high, but data size is pretty
 small.

   PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIMECPU COMMAND
 28674 cassandr   89  250 9451M 8970M  525M sleep  32.1H   329% java

 UN  8.92 GB256 35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
 UN  5.32 GB256 33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
 DN  339.94 MB  256 31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1


 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote:

 Hello

 I have a three nodes cluster. I noticed one node was always down.
 Restarting Cassandra fixes it but it will go down again after a couple of
 days. I'm pretty new to Cassandra so I'm wondering how I should
 troubleshoot it. Logs is as below.

  INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091
 ThriftServer.java (line 141) Stop listening to thrift clients
 ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:4,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261
 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/
 10.20.132.44,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:3,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:2,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line
 196) Exception in thread Thread[Thread-222,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.IllegalThreadStateException
 at java.lang.Thread.start(Thread.java:704)
  at
 org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
 at
 org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
  at
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java
 (line 181) Stop listening for CQL clients
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java
 (line 1251) Announcing shutdown
  INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524
 MessagingService.java (line 667) Waiting for messaging service to quiesce

 Thanks
 Gary

Re: java.lang.OutOfMemoryError: Java heap space

[Gary Zhao]

Yes, they are virtual machines, but we are using KVM. Is there any
solutions for this issue or we should use physical machines?


On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav ipremya...@gmail.com wrote:

 Are the virtual machines? The last time I had this issues was because of
 VMWare ballooning.
 If not, what versions of Cassandra and Java are you using?


 On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao garyz...@gmail.com wrote:

 BTW, the CPU usage on this node is pretty high, but data size is pretty
 small.

   PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIMECPU COMMAND
 28674 cassandr   89  250 9451M 8970M  525M sleep  32.1H   329% java

 UN  8.92 GB256 35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
 UN  5.32 GB256 33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
 DN  339.94 MB  256 31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1


 On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao garyz...@gmail.com wrote:

 Hello

 I have a three nodes cluster. I noticed one node was always down.
 Restarting Cassandra fixes it but it will go down again after a couple of
 days. I'm pretty new to Cassandra so I'm wondering how I should
 troubleshoot it. Logs is as below.

  INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091
 ThriftServer.java (line 141) Stop listening to thrift clients
 ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:4,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261
 CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/
 10.20.132.44,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:3,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipStage:2,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line
 196) Exception in thread Thread[Thread-222,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line
 196) Exception in thread Thread[GossipTasks:1,5,main]
 java.lang.IllegalThreadStateException
 at java.lang.Thread.start(Thread.java:704)
  at
 org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
 at
 org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
  at
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
 at
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
  at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java
 (line 181) Stop listening for CQL clients
  INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java
 (line 1251) Announcing shutdown
  INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524
 MessagingService.java (line 667) Waiting for messaging service to quiesce

 Thanks
 Gary

RE: java.lang.OutOfMemoryError: Java heap space

[Arindam Barua]

If you don’t have row caching turned on, you can try changing settings to get 
your memtables to flush faster (read more at the datastax documention at [1]). 
Btw, if you are using a pre-1.2 version, your bloom filter might come into play 
as well.

All said and done, if your write request rate is too high for your ring to 
handle, you just can’t flush fast enough, and you’ll need to move to physical 
machines and/or add nodes to your ring.

-Arindam

[1] 
http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootDyingNodes_r.html


From: Gary Zhao [mailto:garyz...@gmail.com]
Sent: Monday, April 28, 2014 5:34 PM
To: user@cassandra.apache.org
Subject: Re: java.lang.OutOfMemoryError: Java heap space

Yes, they are virtual machines, but we are using KVM. Is there any solutions 
for this issue or we should use physical machines?

On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav 
ipremya...@gmail.commailto:ipremya...@gmail.com wrote:
Are the virtual machines? The last time I had this issues was because of VMWare 
ballooning.
If not, what versions of Cassandra and Java are you using?

On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao 
garyz...@gmail.commailto:garyz...@gmail.com wrote:
BTW, the CPU usage on this node is pretty high, but data size is pretty small.

  PID USERNAME  THR PRI NICE  SIZE   RES   SHR STATE   TIMECPU COMMAND
28674 cassandr   89  250 9451M 8970M  525M sleep  32.1H   329% java

UN  8.92 GB256 35.4%  c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b  rack1
UN  5.32 GB256 33.1%  874a269f-96bc-4db6-9d15-7cac6771a2ea  rack1
DN  339.94 MB  256 31.5%  fba5b071-aee1-451f-ae42-49c3bbe70bbe  rack1

On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao 
garyz...@gmail.commailto:garyz...@gmail.com wrote:
Hello

I have a three nodes cluster. I noticed one node was always down. Restarting 
Cassandra fixes it but it will go down again after a couple of days. I'm pretty 
new to Cassandra so I'm wondering how I should troubleshoot it. Logs is as 
below.

 INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 ThriftServer.java 
(line 141) Stop listening to thrift clients
ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line 196) 
Exception in thread Thread[GossipStage:4,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [ACCEPT-/10.20.132.44http://10.20.132.44] 2014-04-28 13:06:10,261 
CassandraDaemon.java (line 196) Exception in thread 
Thread[ACCEPT-/10.20.132.44http://10.20.132.44,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line 196) 
Exception in thread Thread[GossipStage:3,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line 196) 
Exception in thread Thread[GossipStage:2,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line 196) 
Exception in thread Thread[Thread-222,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line 196) 
Exception in thread Thread[GossipTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line 196) 
Exception in thread Thread[GossipTasks:1,5,main]
java.lang.IllegalThreadStateException
at java.lang.Thread.start(Thread.java:704)
at 
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202)
at 
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220)
at 
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
 INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java (line 
181) Stop listening for CQL clients
 INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java (line 
1251) Announcing shutdown
 INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 
MessagingService.java (line 667) Waiting for messaging service to quiesce

Thanks
Gary

Re: java.lang.OutOfMemoryError: Java heap space

[Jonathan Ellis]

http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors

On Wed, Nov 24, 2010 at 4:38 PM, zangds zan...@ihep.ac.cn wrote:
 Hi,
 I'm using apache-cassandra-0.7.0-beta3 , when I did some insertions into
 cassandra,I got errors that stop cassandra from working, anyone have a look
 on this?

 WARN [ScheduledTasks:1] 2010-11-24 22:38:33,067 MessagingService.java (line 515) Dropped 39 messages in the
  last 1000ms
 ERROR [ReadStage:3] 2010-11-24 22:38:33,075 AbstractCassandraDaemon.java (line 89) Fatal exception in thread
  Thread[ReadStage:3,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ReadStage:7] 2010-11-24 22:38:49,581 AbstractCassandraDaemon.java (line 89) Fatal exception in thread
  Thread[ReadStage:7,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ReadStage:4] 2010-11-24 22:38:49,581 AbstractCassandraDaemon.java (line 89) Fatal exception in thread
  Thread[ReadStage:4,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [ReadStage:6] 2010-11-24 22:38:49,580 AbstractCassandraDaemon.java (line 89) Fatal exception in thread
  Thread[ReadStage:6,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:89)
 at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:342)
 at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:291)
 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
 at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108
 )
 at org.apache.commons.collections.iterators.CollatingIterator.anyHasNext(CollatingIterator.java:364)
 at org.apache.commons.collections.iterators.CollatingIterator.hasNext(CollatingIterator.java:217)
 at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:63)
 at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
 at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
 at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:122)
 at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:146)
 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1208)
 at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1068)
 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1092)
 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1055)
 at org.apache.cassandra.db.Table.getRow(Table.java:359)
 at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
 at org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:722)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
 at java.util.concurrent.FutureTask.run(FutureTask.java:166)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
 at java.lang.Thread.run(Thread.java:636)
 ..
 ERROR [ReadStage:2] 2010-11-24 22:38:49,580 AbstractCassandraDaemon.java (line 89) Fatal exception in thread
  Thread[ReadStage:2,5,main]
 java.lang.OutOfMemoryError: Java heap space
 ERROR [pool-1-thread-2427] 2010-11-24 22:38:49,580 Cassandra.java (line 2792) Internal error processing mult
 iget_slice
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap s
 pace
 at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:280)
 at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:223)
 at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:131)
 at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:227)
 at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:305)
 at org.apache.cassandra.thrift.CassandraServer.multiget_slice(CassandraServer.java:277)
 at org.apache.cassandra.thrift.Cassandra$Processor$multiget_slice.process(Cassandra.java:2784)
 at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
 at 

Re: java.lang.OutofMemoryerror: Java heap space

[Jonathan Ellis]

if you are reading 500MB per thrift request from each of 3 threads,
then yes, simple arithmetic indicates that 1GB heap is not enough.

On Mon, Jun 14, 2010 at 6:13 PM, Caribbean410 caribbean...@gmail.com wrote:
 Hi,

 I wrote 200k records to db with each record 5MB. Get this error when I uses
 3 threads (each thread tries to read 200k record totally, 100 records a
 time) to read data from db. The write is OK, the error comes from read.
 Right now the Xmx of JVM is 1GB. I changed it to 2GB, still not working. If
 the record size is under 4K, I will not get this error. Any clues to avoid
 this error?

 Thx




-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

RE: java.lang.OutofMemoryerror: Java heap space

[caribbean410]

Sorry, the record size should be 5KB not 5MB. Coz 4KB is still OK. I will
try Benjamin's suggestion.

-Original Message-
From: Jonathan Ellis [mailto:jbel...@gmail.com] 
Sent: Tuesday, June 15, 2010 8:09 AM
To: user@cassandra.apache.org
Subject: Re: java.lang.OutofMemoryerror: Java heap space

if you are reading 500MB per thrift request from each of 3 threads,
then yes, simple arithmetic indicates that 1GB heap is not enough.

On Mon, Jun 14, 2010 at 6:13 PM, Caribbean410 caribbean...@gmail.com
wrote:
 Hi,

 I wrote 200k records to db with each record 5MB. Get this error when I
uses
 3 threads (each thread tries to read 200k record totally, 100 records a
 time) to read data from db. The write is OK, the error comes from read.
 Right now the Xmx of JVM is 1GB. I changed it to 2GB, still not working.
If
 the record size is under 4K, I will not get this error. Any clues to avoid
 this error?

 Thx




-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: java.lang.OutofMemoryerror: Java heap space

[Benjamin Black]

You should only have to restart once per node to pick up config changes.

On Tue, Jun 15, 2010 at 9:41 AM, caribbean410 caribbean...@gmail.com wrote:
 Today I retry the 2GB heap now it's working. No that out of memory error.
 Looks like I have to restart Cassandra several times before the new changes
 take effect.

 -Original Message-
 From: Benjamin Black [mailto:b...@b3k.us]
 Sent: Monday, June 14, 2010 7:46 PM
 To: user@cassandra.apache.org
 Subject: Re: java.lang.OutofMemoryerror: Java heap space

 My guess: you are outrunning your disk I/O.  Each of those 5MB rows
 gets written to the commitlog, and the memtable is flushed when it
 hits the configured limit, which you've probably left at 128MB.  Every
 25 rows or so you are getting memtable flushed to disk.  Until these
 things complete, they are in RAM.

 If this is actually representative of your production use, you need a
 dedicated commitlog disk, several drives in RAID0 or RAID10 for data,
 a lot more RAM, and much larger memtable flush size.


 b

 On Mon, Jun 14, 2010 at 6:13 PM, Caribbean410 caribbean...@gmail.com
 wrote:
 Hi,

 I wrote 200k records to db with each record 5MB. Get this error when I
 uses
 3 threads (each thread tries to read 200k record totally, 100 records a
 time) to read data from db. The write is OK, the error comes from read.
 Right now the Xmx of JVM is 1GB. I changed it to 2GB, still not working.
 If
 the record size is under 4K, I will not get this error. Any clues to avoid
 this error?

 Thx

Re: java.lang.OutofMemoryerror: Java heap space

[Benjamin Black]

My guess: you are outrunning your disk I/O.  Each of those 5MB rows
gets written to the commitlog, and the memtable is flushed when it
hits the configured limit, which you've probably left at 128MB.  Every
25 rows or so you are getting memtable flushed to disk.  Until these
things complete, they are in RAM.

If this is actually representative of your production use, you need a
dedicated commitlog disk, several drives in RAID0 or RAID10 for data,
a lot more RAM, and much larger memtable flush size.


b

On Mon, Jun 14, 2010 at 6:13 PM, Caribbean410 caribbean...@gmail.com wrote:
 Hi,

 I wrote 200k records to db with each record 5MB. Get this error when I uses
 3 threads (each thread tries to read 200k record totally, 100 records a
 time) to read data from db. The write is OK, the error comes from read.
 Right now the Xmx of JVM is 1GB. I changed it to 2GB, still not working. If
 the record size is under 4K, I will not get this error. Any clues to avoid
 this error?

 Thx