Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 12:44 PM, Mikhail Strebkov wrote: > I see, well that's what I expected, but it still should improve a read > latency, since it will reduce the number of disk seeks per row request, is > my assumption correct? > Yep. Also other things per-sstable, like bloom filters and in

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
I see, well that's what I expected, but it still should improve a read latency, since it will reduce the number of disk seeks per row request, is my assumption correct? On Wed, Dec 31, 2014 at 11:51 AM, Robert Coli wrote: > On Wed, Dec 31, 2014 at 11:35 AM, Mikhail Strebkov > wrote: > >> > How

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 11:35 AM, Mikhail Strebkov wrote: > > How effective are compactions in this CF? "grep % > /var/log/cassandra/system.log"? > I'm not sure what type of logs should I share, when I do "grep % > /var/log/cassandra/system.log" I don't see anything related to this > long-running

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
> You can also set this online w/ nodetool, fyi. Oh, thanks! Will use this approach from now on > If you have SSD-esque devices, it is common to be CPU bound on compaction. We use spinning disks, unfortunately we can not afford SSDs in EC2 yet > How effective are compactions in this CF? "grep % /

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Robert Coli
On Wed, Dec 31, 2014 at 12:01 AM, Mikhail Strebkov wrote: > I set compaction_throughput_mb_per_sec to 0 and restarted Cassandra. > You can also set this online w/ nodetool, fyi. > It looks to me that compaction thread is busy doing something that > produces quite a lot of garbage for GC to coll

Re: Manual compaction can't finish because of GC

2014-12-31 Thread Mikhail Strebkov
Hi Rob, Thanks for your response! Unthrottle compaction, that's an insane number of SSTables. I set compaction_throughput_mb_per_sec to 0 and restarted Cassandra. Somehow I don't see GC anymore but compaction is still very slow and IO is still not a bottleneck: iotop reports ~400 K/s for disk r

Re: Manual compaction can't finish because of GC

2014-12-30 Thread Robert Coli
On Tue, Dec 30, 2014 at 3:12 PM, Mikhail Strebkov wrote: > We have a table in our production Cassandra that is stored on 11369 > SSTables. The average SSTable count for the other tables is around 15, and > the read latency for them is much smaller. > Unthrottle compaction, that's an insane numbe

Manual compaction can't finish because of GC

2014-12-30 Thread Mikhail Strebkov
Hi, We have a table in our production Cassandra that is stored on 11369 SSTables. The average SSTable count for the other tables is around 15, and the read latency for them is much smaller. I tried to run manual compaction (nodetool compact my_keyspace my_table) but then the node starts spending ~