Re: Cassandra nodes failing with OOM

2012-11-26 Thread Ивaн Cобoлeв
Hi, all, thank you very much for the help. Aaron was right - we had a multiget_count query, which depending on the app input would result in a calculation performed for ~40k keys. We've released the fix and ~100 GCInspector warnings per day per node went to ~1 per day per 30 nodes :) Thank you v

RE: Cassandra nodes failing with OOM

2012-11-19 Thread Viktor Jevdokimov
We've seen OOM in a situation, when OS was not properly prepared in production. http://www.datastax.com/docs/1.1/install/recommended_settings Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.com Phone: +370 5 212 3

Re: Cassandra nodes failing with OOM

2012-11-19 Thread Janne Jalkanen
Something that bit us recently was the size of bloom filters: we have a column family which is mostly written to, and only read sequentially, so we were able to free a lot of memory and decrease GC pressure by increasing bloom_filter_fp_chance for that particular CF. This on 1.0.12. /Janne O

Re: Cassandra nodes failing with OOM

2012-11-18 Thread aaron morton
> 1. How much GCInspector warnings per hour are considered 'normal'? None. A couple during compaction or repair is not the end of the world. But if you have enough to thinking about "per hour" it's too many. > 2. What should be the next thing to check? Try to determine if the GC activity correl