Re: C* 2.1.2 invokes oom-killer

After couple of days it's still behaving fine. Case closed. On Thu, Feb 19, 2015 at 11:15 PM, Michał Łowicki wrote: > Upgrade to 2.1.3 seems to help so far. After ~12 hours total memory > consumption grew from 10GB to 10.5GB. > > On Thu, Feb 19, 2015 at 2:02 PM, Carlos Rolo wrote: > >> Then you

Re: C* 2.1.2 invokes oom-killer

Upgrade to 2.1.3 seems to help so far. After ~12 hours total memory consumption grew from 10GB to 10.5GB. On Thu, Feb 19, 2015 at 2:02 PM, Carlos Rolo wrote: > Then you are probably hitting a bug... Trying to find out in Jira. The bad > news is the fix is only to be released on 2.1.4. Once I fin

Re: C* 2.1.2 invokes oom-killer

Then you are probably hitting a bug... Trying to find out in Jira. The bad news is the fix is only to be released on 2.1.4. Once I find it out I will post it here. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com

Re: C* 2.1.2 invokes oom-killer

|trickle_fsync| has been enabled for long time in our settings (just noticed): trickle_fsync: true trickle_fsync_interval_in_kb: 10240 On Thu, Feb 19, 2015 at 12:12 PM, Michał Łowicki wrote: > > > On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo wrote: > >> Do you have trickle_fsync enabled? Try

Re: C* 2.1.2 invokes oom-killer

On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo wrote: > Do you have trickle_fsync enabled? Try to enable that and see if it solves > your problem, since you are getting out of non-heap memory. > > Another question, is always the same nodes that die? Or is 2 out of 4 that > die? > Always the same

Re: C* 2.1.2 invokes oom-killer

Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory. Another question, is always the same nodes that die? Or is 2 out of 4 that die? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo

Re: C* 2.1.2 invokes oom-killer

On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo wrote: > So compaction doesn't seem to be your problem (You can check with nodetool > compactionstats just to be sure). > pending tasks: 0 > > How much is your write latency on your column families? I had OOM related > to this before, and there was

Re: C* 2.1.2 invokes oom-killer

So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. -- --

Re: C* 2.1.2 invokes oom-killer

In all tables SSTable counts is below 30. On Thu, Feb 19, 2015 at 9:43 AM, Carlos Rolo wrote: > Can you check how many SSTables you have? It is more or less a know fact > that 2.1.2 has lots of problems with compaction so a upgrade can solve it. > But a high number of SSTables can confirm that i

Re: C* 2.1.2 invokes oom-killer

Can you check how many SSTables you have? It is more or less a know fact that 2.1.2 has lots of problems with compaction so a upgrade can solve it. But a high number of SSTables can confirm that indeed compaction is your problem not something else. Regards, Carlos Juzarte Rolo Cassandra Consultan

Re: C* 2.1.2 invokes oom-killer

We don't have other things running on these boxes and C* is consuming all the memory. Will try to upgrade to 2.1.3 and if won't help downgrade to 2.1.2.  — Michał On Thu, Feb 19, 2015 at 2:39 AM, Jacob Rhoden wrote: > Are you tweaking the "nice" priority on Cassandra? (Type: man nice) if

Re: C* 2.1.2 invokes oom-killer

I neglected to mention, I also adjust the oom score of cassandra, to tell the kernel to kill something else other than cassandra. (Like if one of your dev’s runs a script that uses a lot of memory, so it kills your dev’s script instead). http://lwn.net/Articles/317814/

Re: C* 2.1.2 invokes oom-killer

Are you tweaking the "nice" priority on Cassandra? (Type: man nice) if you don't know much about it. Certainly improving cassandra's nice score becomes important when you have other things running on the server like scheduled jobs of people logging in to the server and doing things. ___

Re: C* 2.1.2 invokes oom-killer

On Wed, Feb 18, 2015 at 10:28 AM, Michał Łowicki wrote: > Couple of times a day 2 out of 4 members cluster nodes are killed > This sort of issue is usually best handled/debugged interactively on IRC. But briefly : - 2.1.2 is IMO broken for production. Downgrade (officially unsupported but fine

C* 2.1.2 invokes oom-killer

Hi, Couple of times a day 2 out of 4 members cluster nodes are killed root@db4:~# dmesg | grep -i oom [4811135.792657] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [6559049.307293] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Nodes are us