Re: Continuously increasing RAM usage

2010-06-02 Thread Paul Brown
FWIW, I'm seeing similar issues on a cluster. Three nodes, Cassandra 0.6.1, SUN JDK 1.6.0_b20. I will try to get some heap dumps to see what's building up. I've seen this sort of issue in systems that make heavy use of java.util.concurrent queues/executors, e.g.:

Re: Continuously increasing RAM usage

2010-06-02 Thread Torsten Curdt
We've also seen something like this. Will soon investigate and try again with 0.6.2 On Wed, Jun 2, 2010 at 20:27, Paul Brown paulrbr...@gmail.com wrote: FWIW, I'm seeing similar issues on a cluster.  Three nodes, Cassandra 0.6.1, SUN JDK 1.6.0_b20.  I will try to get some heap dumps to see

Re: Continuously increasing RAM usage

2010-06-02 Thread Jake Luciani
I've started seeing this issue as well. Running 0.6.2. One interesting thing I happened upon, I explicitly called the GC via jconsole and the heap dropped completely fixing the issue. When you explicitly call System.gc() it does a full sweep. I'm wondering if this issue is to do with the GC

Re: Continuously increasing RAM usage

2010-05-31 Thread Kyusik Chung
Hmm...heap usage slowly growing over time...the picture doesnt look like the problems we were running into, and I dont think they should be related to mmap-ing your data (since that is not counted in heap mem usage). Kyusik Chung On May 29, 2010, at 10:16 AM, James Golick wrote: Well, it's

Re: Continuously increasing RAM usage

2010-05-30 Thread Ian Soboroff
I'm also seeing the same thing. I need to do more log-staring to understand what's up... incompatible with holiday weekends ;-). tpstats are all 0. Ian On Sat, May 29, 2010 at 5:54 PM, Jonathan Ellis jbel...@gmail.com wrote: Did you manually invoke a GC? It doesn't look like you're using

Re: Continuously increasing RAM usage

2010-05-29 Thread James Golick
Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks to be about the same: http://skitch.com/jamesgolick/df46f/munin-fetlife.com-cassandra0.fetlife.com-cassandra-memory There's only one cache turned on, and it's a row cache, but the sizes of the rows are identical and it's

Re: Continuously increasing RAM usage

2010-05-29 Thread Jonathan Ellis
Did you manually invoke a GC? It doesn't look like you're using enough of the heap for it to care much on its own. On Sat, May 29, 2010 at 2:58 PM, James Golick jamesgol...@gmail.com wrote: Well, it's been a few days on 0.6.2 and the new jvm and the behaviour looks to be about the same:

Continuously increasing RAM usage

2010-05-27 Thread James Golick
We're seeing RAM usage continually climb until eventually, cassandra becomes unresponsive. The JVM isn't OOM'ing. It has only committed 14/24GB of memory. So, I am assuming that the memory usage is related to mmap'd IO. Fair assumption? I tried setting the IO mode to standard, but it seemed to

Re: Continuously increasing RAM usage

2010-05-27 Thread Ian Soboroff
A lot of folks have reported this issue, and there are a few JIRAs related to it. Post the output of nodetool tpstats. Also, are there lots of GCs in the system.log? If so, are they something besides ParNew? Ian On Thu, May 27, 2010 at 2:32 AM, James Golick jamesgol...@gmail.com wrote:

Re: Continuously increasing RAM usage

2010-05-27 Thread James Golick
into. *From:* James Golick [mailto:jamesgol...@gmail.com] *Sent:* Thursday, May 27, 2010 9:05 AM *To:* user@cassandra.apache.org *Subject:* Re: Continuously increasing RAM usage When I say unresponsive, I mean that latency becomes very high. Swap is turned off, but before I turned it on, it used

Re: Continuously increasing RAM usage

2010-05-27 Thread Kyusik Chung
Hi Philip, I think they chose to go with OpenJDK bc Sun's is not open source. Here's what we did on ubuntu 10.04 (if youre using a different debian distro, you can prob do something very similar): # this install gives us the convenient add-apt-repository command sudo apt-get install