Sorry to hear about the frustration. How often are you deleting data/what are you setting for ttl on cols?
Jonathan Lacefield Solutions Architect, DataStax (404) 822 3487 <http://www.linkedin.com/in/jlacefield> <http://www.datastax.com/what-we-offer/products-services/training/virtual-training> On Tue, Mar 25, 2014 at 4:22 PM, Oleg Dulin <oleg.du...@gmail.com> wrote: > Sigh, so I am back to where I started from... > > I did lower gc_grace... > > jmap -histo:live shows heap is stuffed with DeletedColumn and > ExpiringColumn.... > > This is extremely frustrating. > > > On 2014-03-11 19:24:50 +0000, Oleg Dulin said: > > Good news is that since I lowered gc_grace period it collected over >> 100Gigs of tombstones and seems much happier now. >> >> Oleg >> >> On 2014-03-10 13:33:43 +0000, Jonathan Lacefield said: >> >> Hello, >>> >>> You have several options: >>> >>> 1) going forward lower gc_grace_seconds http://www.datastax.com/ >>> documentation/cassandra/1.2/cassandra/configuration/ >>> configStorage_r.html?pagename=docs&version=1.2&file= >>> configuration/storage_configuration#gc-grace-seconds >>> - this is very use case specific. Default is 10 days. Some >>> users will put this at 0 for specific use cases. >>> 2) you could also lower tombstone compaction threshold and interval to >>> get tombstone compaction to fire more often on your tables/cfs: >>> https://datastax.jira.com/wiki/pages/viewpage.action?pageId=54493436 >>> 3) to clean out old tombstones you could always run a manual >>> compaction, those these aren't typically recommended though: >>> http://www.datastax.com/documentation/cassandra/1.2/ >>> cassandra/tools/toolsNodetool_r.html >>> >>> For 1 and 2, be sure your disks can keep up with compaction to ensure >>> tombstone, or other, compaction fires regularly enough to clean out old >>> tombstones. Also, you probably want to ensure you are using Level >>> Compaction: http://www.datastax.com/dev/blog/when-to-use-leveled- >>> compaction. Again, this assumes your disk system can handle the >>> increased io from Leveled Compaction. >>> >>> Also, you may be running into this with the older version of >>> Cassandra: https://issues.apache.org/jira/browse/CASSANDRA-6541 >>> >>> Hope this helps. >>> >>> Jonathan >>> >>> >>> Jonathan Lacefield >>> Solutions Architect, DataStax >>> (404) 822 3487 >>> <image> >>> >>> >>> <image> >>> >>> >>> On Mon, Mar 10, 2014 at 6:41 AM, Oleg Dulin <oleg.du...@gmail.com> >>> wrote: >>> I get that :) >>> >>> What I'd like to know is how to fix that :) >>> >>> On 2014-03-09 20:24:54 +0000, Takenori Sato said: >>> >>> You have millions of org.apache.cassandra.db.DeletedColumn instances on >>> the snapshot. >>> >>> This means you have lots of column tombstones, and I guess, which are >>> read into memory by slice query. >>> >>> >>> On Sun, Mar 9, 2014 at 10:55 PM, Oleg Dulin <oleg.du...@gmail.com> >>> wrote: >>> I am trying to understand why one of my nodes keeps full GC. >>> >>> I have Xmx set to 8gigs, memtable total size is 2 gigs. >>> >>> Consider the top entries from jmap -histo:live @ >>> http://pastebin.com/UaatHfpJ >>> >>> -- >>> Regards, >>> Oleg Dulin >>> http://www.olegdulin.com >>> >>> >>> -- >>> Regards, >>> Oleg Dulin >>> http://www.olegdulin.com >>> >> >> S >> > > > -- > Regards, > Oleg Dulin > http://www.olegdulin.com > > >
