Re: Help diagnosing performance issue

Honestly 20ms for spinning disks is really good, so I think you're just dealing with the reality of having a certain percentage of your reads off disk and not in memory. If you're reading data that is on older SSTables and you're out of buffer cache I'm not sure how else you could improve that.

Re: Help diagnosing performance issue

Sebastian (and others, help is always appreciated), After 24h OK, read latencies started to degrade (up to 20ms) and I had to ramp down volumes again. The degradation is clearly linked to the number read IOPs which went up to 1.65k/s after 24h. If anybody can give me hints on what I should

Re: Help diagnosing performance issue

Sebastian, I tried to ramp up volume with this new setting and ran into the same problems. After that I restarted my nodes. This pretty much instantly got read latencies back to normal (< 5ms) on the 32G nodes. I am currently ramping up volumes again and here is what I am seeing on 32G

Re: Help diagnosing performance issue

Sebastian, I took into account your suggestion and set max_sstable_age_days to 1. I left the TTL at 432000 and the gc_grace_seconds at 172800. So, I expect SSTable older than 7 days to get deleted. Am I right ? I did not change dclocal_read_repair_chance because I have only one DC at this

Re: Help diagnosing performance issue

unsubscribe. > On Nov 19, 2015, at 3:58 PM, Antoine Bonavita wrote: > > Sebastian, > > I took into account your suggestion and set max_sstable_age_days to 1. > > I left the TTL at 432000 and the gc_grace_seconds at 172800. So, I expect > SSTable older than 7 days to get

Re: Help diagnosing performance issue

Yep, I think you've mixed up your DTCS levers. I would read, or re-read Marcus's post http://www.datastax.com/dev/blog/datetieredcompactionstrategy *TL;DR:* - *base_time_seconds* is the size of your initial window - *max_sstable_age_days* is the time after which you stop compacting

Re: Help diagnosing performance issue

Sebastian, Robet, First, a big thank you to both of you for your help. It looks like you were right. I used pcstat (awesome tool, thanks for that as well) and it appears some files I would not expect to be in cache actually are. Here is a sample of my output (edited for convenience, adding

Re: Help diagnosing performance issue

> > When you say drop you mean reduce the value (to 1 day for example), not > "don't set the value", right ? Yes. If I set max sstable age days to 1, my understanding is that SSTables with > expired data (5 days) are not going to be compacted ever. And therefore my > disk usage will keep

Re: Help diagnosing performance issue

Sebastian, Your help is very much appreciated. I re-read the blog post and also https://labs.spotify.com/2014/12/18/date-tiered-compaction/ but some things are still confusing me. Please see my questions inline below. On 11/18/2015 04:21 PM, Sebastian Estevez wrote: Yep, I think you've

Re: Help diagnosing performance issue

Hello, As I have not heard from anybody on the list, I guess I did not provide the right kind of information or I did not ask the right question. The things I forgot to mention in my previous email: * Checked the logs without noticing anything out of the ordinary. Memtables flushes occur

Re: Help diagnosing performance issue

Hi, You're sstables are probably falling out of page cache on the smaller nodes and your slow disks are killing your latencies. Check to see if this is the case with pcstat: https://github.com/tobert/pcstat All the best, [image: datastax_logo.png] Sebastián

Re: Help diagnosing performance issue

On Tue, Nov 17, 2015 at 11:08 AM, Sebastian Estevez < sebastian.este...@datastax.com> wrote: > You're sstables are probably falling out of page cache on the smaller > nodes and your slow disks are killing your latencies. > +1 most likely. Are the heaps the same size on both machines? =Rob

Help diagnosing performance issue

Hello, We have a performance problem when trying to ramp up cassandra (as a mongo replacement) on a very specific use case. We store a blob indexed by a key and expire it after a few days: CREATE TABLE views.views ( viewkey text PRIMARY KEY, value blob ) WITH bloom_filter_fp_chance =