Hey Garo,
I see you are using 2.2.x. Do you have compression enabled on commit logs by
any chance? If so, try to disable it and see if that helps.
Right after migrating from 2.1.x to 2.2.x, I remember having the behavior you
described with 10k SAS disks when commit log compression was enabled
Hi Garo,
I haven't had this issue on SSDs, but I have definitely seen it with
spinning drives. I would think that SSDs would have more than enough
bandwidth to keep up with requests, but you may be running into issues
with Cassandra calling fsync on the commitlog.
What are your settings for the f
Hi Mark.
I have an LVM volume which stripes the four ephemeral SSD:s in the system
and we use that for both data and commit log. I've used similar setup in
the past (but with EBS) and we didn't see this behavior. Each node gets
just around 250 writes per second. It is possible that the commit log
Hi Garo,
Did you put the commit log on its own drive? Spiking CPU during stalls
is a symptom of not doing that. The commitlog is very latency
sensitive, even under low load. Do be sure you're using the deadline
or noop scheduler for that reason, too.
-Mark
On Fri, Jul 22, 2016 at 4:44 PM, Juho M
>
> Are you using XFS or Ext4 for data?
We are using XFS. Many nodes have a couple large SSTables (in order of
20-50 GiB), but I havent cross checked if the load spikes happen only on
machines which have these tables.
> As an aside, for the amount of reads/writes you're doing, I've found
> usin
Unsubscribe me.
Thank you
From: Ryan Svihla [mailto:r...@foundev.pro]
Sent: viernes, 22 de julio de 2016 14:39
To: user@cassandra.apache.org
Subject: Re: My cluster shows high system load without any apparent reason
You aren't using counters by chance?
regards,
Ryan Svihla
On Jul 22, 20
You aren't using counters by chance?
regards,
Ryan Svihla
On Jul 22, 2016, 2:00 PM -0500, Mark Rose , wrote:
> Hi Garo,
>
> Are you using XFS or Ext4 for data? XFS is much better at deleting
> large files, such as may happen after a compaction. If you have 26 TB
> in just two tables, I bet you h
Hi Garo,
Are you using XFS or Ext4 for data? XFS is much better at deleting
large files, such as may happen after a compaction. If you have 26 TB
in just two tables, I bet you have some massive sstables which may
take a while for Ext4 to delete, which may be causing the stalls. The
underlying bloc
After a few days I've also tried disabling Linux kernel huge pages
defragement (echo never > /sys/kernel/mm/transparent_hugepage/defrag) and
turning coalescing off (otc_coalescing_strategy: DISABLED), but either did
do any good. I'm using LCS, there are no big GC pauses, and I have set
"concurrent_
I just recently upgraded our cluster to 2.2.7 and after turning the cluster
under production load the instances started to show high load (as shown by
uptime) without any apparent reason and I'm not quite sure what could be
causing it.
We are running on i2.4xlarge, so we have 16 cores, 120GB of ra
10 matches
Mail list logo
