RE: Cassandra memory problem

We are running into the same issue on some of our machines. Still haven't tracked down what is causing it. From: William Oberman [mailto:ober...@civicscience.com] Sent: Thursday, July 07, 2011 7:19 AM To: user@cassandra.apache.org Subject: Re: Cassandra m

Re: Cassandra memory problem

I think I had (and have) a similar problem: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/OOM-or-what-settings-to-use-on-AWS-large-td6504060.html My memory usage grew slowly until I ran out of mem and the OS killed my process (due to no swap). I'm still on 0.7.4, but I'm rolling

Re: Cassandra memory problem

Hm - had to digg deeper and it totally looks like a native mem leak to me: We are still growing with res += 100MB a day. Cassandra is > 8G now I checked the cassandra process with pmap -x Here's the human readable (aggregated) output: Format is thingy: RSS in KB Summary: Total SST: 1961616 An

Re: Cassandra memory problem

Hi Daniel, Yes we do see it, since I've added the JNA libraries, it takes a bit more time at that step and locks all the memory. We're using JNA 3.3.0 we've downloaded from there: https://github.com/twall/jna#readme Our servers currently have 32GB of memory a

Re: Cassandra memory problem

Hi Sebastian, one question: do you use jna.jar and do you see JNA mlockall successful in your logs. There's that wild theory here that our problem might be related to mlockall and no swap. Maybe the JVM does some realloc stuff and the pinned pages are not cleared ... but that's really only wi

Re: Cassandra memory problem

Ok - thanks but maybe some kernel guy can help or point to some good resource to get educated because I don'r really get it. The following is from our other small log cluster with 2 nodes with 8GM ram cassandra has 4GB max heap - We have disabled swap on all cassandra servers - On the machine w

Re: Cassandra memory problem

Yes thank you. I have read about the OpenJDK issue but unfortunately we are already on Sun JDK. On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote: > It was among one of the issues we had. One of our hosts was using OpenJDK and > we've switched it to Sun and this part of the issue stabilized. T

Re: Cassandra memory problem

It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched it to Sun and this part of the issue stabilized. The other issues we had were Heap going through the roof and then OOM under load. On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday wrote: > Just to make

Re: Cassandra memory problem

Just to make sure: You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings? Note that I am not having a heap / gc problem. The VM itself thinks everything is golden. On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote: > We had an issu

Re: Cassandra memory problem

We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, wri

Re: Cassandra memory problem

mmap'd data will be attributed to res, but the OS can page it out instead of killing the process. On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday wrote: > Hi all, > we have a mem problem with cassandra. res goes up without bounds (well until > the os kills the process because we dont have swap)

Cassandra memory problem

Hi all, we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap) I found a thread that's about the same problem but on OpenJDK: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-n