Re: Heap is not released and streaming hangs at 0%
I do not see the out of heap errors but I am taking a bit of a performance hit. Take a look at nodetool cfhistograms to see how many SSTables are being touched per read and the local read latency. In general if you are hitting more than 4 it's not great. BloomFilterFalseRatio is 0.8367977262013025 this was the reason behind bumping bloom_filter_fp_chance. Not sure I understand the logic there. I also encountered similar problem. I dump the jvm heap and analyse it by eclipse mat. The eclipse plugin told me there are 10334 instances of SSTableReader, consuming 6.6G memory. I found the CompactionExecutor thread held 8000+ SSTalbeReader object. I wonder why there are so many SSTableReader in memory. I would guess you are using Levelled compaction strategy with the default 5MB file size. When analysing the heap, if bloom filters are taking up too much memory they will show up as long[][] Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 27/06/2013, at 2:36 AM, srmore comom...@gmail.com wrote: On Wed, Jun 26, 2013 at 12:16 AM, aaron morton aa...@thelastpickle.com wrote: bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only the new tables with have the 0.1 setting. I will try increasing the value to 0.7 and report my results. No need to, it will probably be something like Oh no, really, what, how, please make it stop :) 0.7 will mean reads will hit most / all of the SSTables for the CF. Changing the bloom_filter_fp_chance to 0.7 did seem to correct the problem in short run. I do not see the out of heap errors but I am taking a bit of a performance hit. Planning to run some more tests, also my BloomFilterFalseRatio is 0.8367977262013025 this was the reason behind bumping bloom_filter_fp_chance. I covered a high row situation in on of my talks at the summit this month, the slide deck is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass and the videos will soon be up at Planet Cassandra. This was/is extremely helpful Aaron, cannot thank you enough for sharing this with the community, eagerly looking forward for the video. Rebuild the sstables, then reduce the index_interval if you still need to reduce mem pressure. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 22/06/2013, at 1:17 PM, sankalp kohli kohlisank...@gmail.com wrote: I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot btal...@aeriagames.com wrote: bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.com wrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei From: Robert Coli rc...@eventbrite.com To: user@cassandra.apache.org Sent: Tuesday, June 18, 2013 10:43:13 AM Subject: Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a
Re: Heap is not released and streaming hangs at 0%
I also encountered similar problem. I dump the jvm heap and analyse it by eclipse mat. The eclipse plugin told me there are 10334 instances of SSTableReader, consuming 6.6G memory. I found the CompactionExecutor thread held 8000+ SSTalbeReader object. I wonder why there are so many SSTableReader in memory. On Wed, Jun 26, 2013 at 1:16 PM, aaron morton aa...@thelastpickle.comwrote: bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only the new tables with have the 0.1 setting. I will try increasing the value to 0.7 and report my results. No need to, it will probably be something like Oh no, really, what, how, please make it stop :) 0.7 will mean reads will hit most / all of the SSTables for the CF. I covered a high row situation in on of my talks at the summit this month, the slide deck is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass and the videos will soon be up at Planet Cassandra. Rebuild the sstables, then reduce the index_interval if you still need to reduce mem pressure. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 22/06/2013, at 1:17 PM, sankalp kohli kohlisank...@gmail.com wrote: I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot btal...@aeriagames.comwrote: bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.comwrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei -- *From: *Robert Coli rc...@eventbrite.com *To: *user@cassandra.apache.org *Sent: *Tuesday, June 18, 2013 10:43:13 AM *Subject: *Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
On Wed, Jun 26, 2013 at 12:16 AM, aaron morton aa...@thelastpickle.comwrote: bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only the new tables with have the 0.1 setting. I will try increasing the value to 0.7 and report my results. No need to, it will probably be something like Oh no, really, what, how, please make it stop :) 0.7 will mean reads will hit most / all of the SSTables for the CF. Changing the bloom_filter_fp_chance to 0.7 did seem to correct the problem in short run. I do not see the out of heap errors but I am taking a bit of a performance hit. Planning to run some more tests, also my BloomFilterFalseRatio is 0.8367977262013025 this was the reason behind bumping bloom_filter_fp_chance. I covered a high row situation in on of my talks at the summit this month, the slide deck is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass and the videos will soon be up at Planet Cassandra. This was/is extremely helpful Aaron, cannot thank you enough for sharing this with the community, eagerly looking forward for the video. Rebuild the sstables, then reduce the index_interval if you still need to reduce mem pressure. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 22/06/2013, at 1:17 PM, sankalp kohli kohlisank...@gmail.com wrote: I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot btal...@aeriagames.comwrote: bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.comwrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei -- *From: *Robert Coli rc...@eventbrite.com *To: *user@cassandra.apache.org *Sent: *Tuesday, June 18, 2013 10:43:13 AM *Subject: *Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. You need to re-write the sstables on disk using nodetool upgradesstables. Otherwise only the new tables with have the 0.1 setting. I will try increasing the value to 0.7 and report my results. No need to, it will probably be something like Oh no, really, what, how, please make it stop :) 0.7 will mean reads will hit most / all of the SSTables for the CF. I covered a high row situation in on of my talks at the summit this month, the slide deck is here http://www.slideshare.net/aaronmorton/cassandra-sf-2013-in-case-of-emergency-break-glass and the videos will soon be up at Planet Cassandra. Rebuild the sstables, then reduce the index_interval if you still need to reduce mem pressure. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 22/06/2013, at 1:17 PM, sankalp kohli kohlisank...@gmail.com wrote: I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot btal...@aeriagames.com wrote: bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.com wrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei From: Robert Coli rc...@eventbrite.com To: user@cassandra.apache.org Sent: Tuesday, June 18, 2013 10:43:13 AM Subject: Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei From: Robert Coli rc...@eventbrite.com To: user@cassandra.apache.org Sent: Tuesday, June 18, 2013 10:43:13 AM Subject: Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.comwrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei -- *From: *Robert Coli rc...@eventbrite.com *To: *user@cassandra.apache.org *Sent: *Tuesday, June 18, 2013 10:43:13 AM *Subject: *Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.comwrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei -- *From: *Robert Coli rc...@eventbrite.com *To: *user@cassandra.apache.org *Sent: *Tuesday, June 18, 2013 10:43:13 AM *Subject: *Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
I will take a heap dump and see whats in there rather than guessing. On Fri, Jun 21, 2013 at 4:12 PM, Bryan Talbot btal...@aeriagames.comwrote: bloom_filter_fp_chance = 0.7 is probably way too large to be effective and you'll probably have issues compacting deleted rows and get poor read performance with a value that high. I'd guess that anything larger than 0.1 might as well be 1.0. -Bryan On Fri, Jun 21, 2013 at 5:58 AM, srmore comom...@gmail.com wrote: On Fri, Jun 21, 2013 at 2:53 AM, aaron morton aa...@thelastpickle.comwrote: nodetool -h localhost flush didn't do much good. Do you have 100's of millions of rows ? If so see recent discussions about reducing the bloom_filter_fp_chance and index_sampling. Yes, I have 100's of millions of rows. If this is an old schema you may be using the very old setting of 0.000744 which creates a lot of bloom filters. bloom_filter_fp_chance value that was changed from default to 0.1, looked at the filters and they are about 2.5G on disk and I have around 8G of heap. I will try increasing the value to 0.7 and report my results. It also appears to be a case of hard GC failure (as Rob mentioned) as the heap is never released, even after 24+ hours of idle time, the JVM needs to be restarted to reclaim the heap. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 20/06/2013, at 6:36 AM, Wei Zhu wz1...@yahoo.com wrote: If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei -- *From: *Robert Coli rc...@eventbrite.com *To: *user@cassandra.apache.org *Sent: *Tuesday, June 18, 2013 10:43:13 AM *Subject: *Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Re: Heap is not released and streaming hangs at 0%
If you want, you can try to force the GC through Jconsole. Memory-Perform GC. It theoretically triggers a full GC and when it will happen depends on the JVM -Wei - Original Message - From: Robert Coli rc...@eventbrite.com To: user@cassandra.apache.org Sent: Tuesday, June 18, 2013 10:43:13 AM Subject: Re: Heap is not released and streaming hangs at 0% On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
Heap is not released and streaming hangs at 0%
I see an issues when I run high traffic to the Cassandra nodes, the heap gets full to about 94% (which is expected) but the thing that confuses me is that the heap usage never goes down after the traffic is stopped (at-least, it appears to be so) . I kept the nodes up for a day after stopping the traffic and the logs still tell me Heap is 0.9430032942657169 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically Things go back to normal when I restart Cassandra. nodetool netstats tells me the following: Mode: Normal Not sending streams and a bunch of keyspaces streaming from other nodes which are at 0% and this stays this way until I restart Cassandra. Also I see this at the bottom: Pool NameActive Pending Completed Commandsn/a 08267930 Responses n/a 0 15184810 Any ideas as to how I can speed up this up and reclaim the heap ? Thanks !
Re: Heap is not released and streaming hangs at 0%
On Tue, Jun 18, 2013 at 8:25 AM, srmore comom...@gmail.com wrote: I see an issues when I run high traffic to the Cassandra nodes, the heap gets full to about 94% (which is expected) Which is expected to cause GC failure? ;) But seriously, the reason your node is unable to GC is that you have filled your heap too fast for it to keep up. The JVM has seized up like Joe Namath with vapor lock. Any ideas as to how I can speed up this up and reclaim the heap ? Don't exhaust the ability of GC to C G. :) =Rob PS - What version of cassandra? If you nodetool -h localhost flush does it help?
Re: Heap is not released and streaming hangs at 0%
Thanks Rob, But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. nodetool -h localhost flush didn't do much good. the version I am running is 1.0.12 (I know its due for a upgrade but gotto work with this for now). On Tue, Jun 18, 2013 at 12:13 PM, Robert Coli rc...@eventbrite.com wrote: On Tue, Jun 18, 2013 at 8:25 AM, srmore comom...@gmail.com wrote: I see an issues when I run high traffic to the Cassandra nodes, the heap gets full to about 94% (which is expected) Which is expected to cause GC failure? ;) But seriously, the reason your node is unable to GC is that you have filled your heap too fast for it to keep up. The JVM has seized up like Joe Namath with vapor lock. Any ideas as to how I can speed up this up and reclaim the heap ? Don't exhaust the ability of GC to C G. :) =Rob PS - What version of cassandra? If you nodetool -h localhost flush does it help?
Re: Heap is not released and streaming hangs at 0%
On Tue, Jun 18, 2013 at 10:33 AM, srmore comom...@gmail.com wrote: But then shouldn't JVM C G it eventually ? I can still see Cassandra alive and kicking but looks like the heap is locked up even after the traffic is long stopped. No, when GC system fails this hard it is often a permanent failure which requires a restart of the JVM. nodetool -h localhost flush didn't do much good. This adds support to the idea that your heap is too full, and not full of memtables. You could try nodetool -h localhost invalidatekeycache, but that probably will not free enough memory to help you. =Rob
