Hi guys,
I have a DevOps related question - hope someone here could give some
ideas/pointers...
We are running a 3 nodes Cassandra cluster
Recently we realized we do have performance issues. And based on
investigation we took it seems our bottleneck is the Cassandra cluster.
The application layer is waiting a lot for Cassandra ops. So queries are
running slow on Cassandra side however due to our monitoring it looks
the Cassandra servers still have lots of free resources...
The Cassandra machines are virtual machines (we do own the physical
hosts too) built with kvm - with 6 CPU cores (3 physical) and 32GB RAM
dedicated to it.
We are using Ubuntu Linux 18.04 distro - everywhere the same version
(the physical and virtual host)
We are running Cassandra 4.0-alpha4
What we see is
* CPU load is around 20-25% - so we have lots of spare capacity
* iowait is around 2-5% - so disk bandwidth should be fine
* network load is around 50% of the full available bandwidth
* loadavg is max around 4 - 4.5 but typically around 3 (because of the
cpu count 6 should represent 100% load)
and still, query performance is slow ... and we do not understand what
could hold Cassandra back to fully utilize the server resources...
We are clearly missing something!
Anyone any idea / tip?
thanks!
--
Attila Wind
http://www.linkedin.com/in/attilaw <http://www.linkedin.com/in/attilaw>
Mobile: +49 176 43556932
