[
https://issues.apache.org/jira/browse/CASSANDRA-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jai Bheemsen Rao Dhanwada updated CASSANDRA-17355:
--------------------------------------------------
Description:
Hello Everyone,
I am noticing a huge perf drop (spike in latency and CPU utilization) for the
counter type tables when the data size grows. To better understand/simulate I
have done the following perf test with `cassandra-stress` instead of my
use-case and I can reproduce the performance issue consistently. When using the
counter type tables, when the datasize grows the read latency and cpu spikes to
very high number.
*Test Setup:*
# Setup a cluster with 3 nodes.
# Run a test with cassandra-stress and I see the latency and CPU are okay
without much spike.
# Send a lot of counter traffic using `cassandra-stress` tool (Replication
Factory = 3)
# Now the data size on the cluster is ~300G.
# Now run another test with cassandra-stress with 3:1 read write mixed
workload.
# At this point I see the CPU spikes to double (32 on a 16 core CPU) and the
latency reaches ~1 seconds (which earlier was < 5ms).
# Another interesting observation is the disk reads goes to a higher number
and it keeps going higher with the increase in the disk size.
# It pretty much looked like a disk bottleneck issue but the same result shows
very low disk reads, cpu, latency with less amount of data.
# Below is the configuration I have used for testing this.
{quote}C* Version: 3.11.9
CPU: 16
Memory: 64G
Heap: 16G
GC: G1GC
Disk: 500G GCP Persistent disk
{quote}
I understand that, with growth in disk the number of lookup grows high, but
this looked to be a big performance drop.
Please let me know if you need more details. Also let me know this is known
limitation with the counter type and if there is a work around.
was:
Hello Everyone,
I am noticing a huge perf drop (spike in latency and CPU utilization) for the
counter type tables when the data size grows. To better understand/simulate I
have done the following perf test with `cassandra-stress` instead of my
use-case and I can reproduce the performance issue consistently. When using the
counter type tables, when the datasize grows the read latency and cpu spikes to
very high number.
*Test Setup:*
# Setup a cluster with 3 nodes.
# Run a test with cassandra-stress and I see the latency and CPU are okay
without much spike.
# Send a lot of counter traffic using `cassandra-stress` tool (Replication
Factory = 3)
# Now the data size on the cluster is ~300G.
# Now run another test with cassandra-stress with 3:1 read write mixed
workload.
# At this point I see the CPU spikes to double (32 on a 16 core CPU) and the
latency reaches ~1 seconds (which earlier was < 5ms).
# Another interesting observation is the disk reads goes to a higher number
and it keeps going higher with the increase in the disk size.
# It pretty much looked like a disk bottleneck issue but the same result shows
very low disk reads, cpu, latency with less amount of data.
# Below is the configuration I have used for testing this.
```
C* Version: 3.11.9
CPU: 16
Memory: 64G
Heap: 16G
GC: G1GC
Disk: 500G GCP Persistent disk
```
I understand that, with growth in disk the number of lookup grows high, but
this looked to be a big performance drop.
Please let me know if you need more details. Also let me know this is known
limitation with the counter type and if there is a work around.
> Performance degradation with Counter tables when the data size grows
> --------------------------------------------------------------------
>
> Key: CASSANDRA-17355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17355
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jai Bheemsen Rao Dhanwada
> Priority: Normal
>
> Hello Everyone,
> I am noticing a huge perf drop (spike in latency and CPU utilization) for the
> counter type tables when the data size grows. To better understand/simulate I
> have done the following perf test with `cassandra-stress` instead of my
> use-case and I can reproduce the performance issue consistently. When using
> the counter type tables, when the datasize grows the read latency and cpu
> spikes to very high number.
>
> *Test Setup:*
> # Setup a cluster with 3 nodes.
> # Run a test with cassandra-stress and I see the latency and CPU are okay
> without much spike.
> # Send a lot of counter traffic using `cassandra-stress` tool (Replication
> Factory = 3)
> # Now the data size on the cluster is ~300G.
> # Now run another test with cassandra-stress with 3:1 read write mixed
> workload.
> # At this point I see the CPU spikes to double (32 on a 16 core CPU) and the
> latency reaches ~1 seconds (which earlier was < 5ms).
> # Another interesting observation is the disk reads goes to a higher number
> and it keeps going higher with the increase in the disk size.
> # It pretty much looked like a disk bottleneck issue but the same result
> shows very low disk reads, cpu, latency with less amount of data.
> # Below is the configuration I have used for testing this.
>
> {quote}C* Version: 3.11.9
> CPU: 16
> Memory: 64G
> Heap: 16G
> GC: G1GC
> Disk: 500G GCP Persistent disk
>
> {quote}
> I understand that, with growth in disk the number of lookup grows high, but
> this looked to be a big performance drop.
> Please let me know if you need more details. Also let me know this is known
> limitation with the counter type and if there is a work around.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
