[ https://issues.apache.org/jira/browse/CASSANDRA-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821383#comment-17821383 ]
Dipietro Salvatore commented on CASSANDRA-19429: ------------------------------------------------ 1. Test without compaction after writes. Cmd used: {code:java} sbin/cqlsh -e 'drop table if exists keyspace1.standard1;' && bin/cqlsh -e 'drop keyspace if exists keyspace1;' && bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write n=10000000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log -graph file=cload.html && sleep 30s && tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m cl=ONE -rate threads=100 -node localhost -log file=result.log -graph file=graph.html {code} Results using Ubuntu22.04 on r8g.24xlarge with stress test colocated on the same instance : - 4.1.3 released: {code:java} Results: Op rate : 167,478 op/s [READ: 150,727 op/s, WRITE: 16,750 op/s] Partition rate : 167,478 pk/s [READ: 150,727 pk/s, WRITE: 16,750 pk/s] Row rate : 167,478 row/s [READ: 150,727 row/s, WRITE: 16,750 row/s] Latency mean : 0.6 ms [READ: 0.6 ms, WRITE: 0.2 ms] Latency median : 0.4 ms [READ: 0.5 ms, WRITE: 0.1 ms] Latency 95th percentile : 1.6 ms [READ: 1.6 ms, WRITE: 0.2 ms] Latency 99th percentile : 2.2 ms [READ: 2.2 ms, WRITE: 0.3 ms] Latency 99.9th percentile : 7.1 ms [READ: 7.3 ms, WRITE: 1.2 ms] Latency max : 48.3 ms [READ: 48.3 ms, WRITE: 46.0 ms] Total partitions : 100,600,039 [READ: 90,538,533, WRITE: 10,061,506] Total errors : 0 [READ: 0, WRITE: 0] Total GC count : 1,515 Total GC memory : 2411.407 GiB Total GC time : 8.3 seconds Avg GC time : 5.5 ms StdDev GC time : 2.4 ms Total operation time : 00:10:00{code} - 4.1.3 with patch: {code:java} Results: Op rate : 435,180 op/s [READ: 391,655 op/s, WRITE: 43,525 op/s] Partition rate : 435,180 pk/s [READ: 391,655 pk/s, WRITE: 43,525 pk/s] Row rate : 435,180 row/s [READ: 391,655 row/s, WRITE: 43,525 row/s] Latency mean : 0.2 ms [READ: 0.2 ms, WRITE: 0.2 ms] Latency median : 0.2 ms [READ: 0.2 ms, WRITE: 0.2 ms] Latency 95th percentile : 0.3 ms [READ: 0.3 ms, WRITE: 0.2 ms] Latency 99th percentile : 0.4 ms [READ: 0.4 ms, WRITE: 0.3 ms] Latency 99.9th percentile : 6.7 ms [READ: 6.7 ms, WRITE: 6.2 ms] Latency max : 1057.5 ms [READ: 1,057.5 ms, WRITE: 47.7 ms] Total partitions : 261,410,615 [READ: 235,265,301, WRITE: 26,145,314] Total errors : 0 [READ: 0, WRITE: 0] Total GC count : 4,543 Total GC memory : 7225.988 GiB Total GC time : 25.7 seconds Avg GC time : 5.7 ms StdDev GC time : 3.1 ms Total operation time : 00:10:00{code} Patch seems to have huge benefit on this case as well (2.6x). Noticed also huge benefit in P99 latency (5.5x decrease). > Remove lock contention generated by getCapacity function in SSTableReader > ------------------------------------------------------------------------- > > Key: CASSANDRA-19429 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19429 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable > Reporter: Dipietro Salvatore > Assignee: Dipietro Salvatore > Priority: Normal > Fix For: 4.0.x, 4.1.x > > Attachments: Screenshot 2024-02-26 at 10.27.10.png, > asprof_cass4.1.3__lock_20240216052912lock.html > > Time Spent: 20m > Remaining Estimate: 0h > > Profiling Cassandra 4.1.3 on large AWS instances, a high number of lock > acquires is measured in the `getCapacity` function from > `org/apache/cassandra/cache/InstrumentingCache` (1.9M lock acquires per 60 > seconds). Based on our tests on r8g.24xlarge instances (using Ubuntu 22.04), > this limits the CPU utilization of the system to under 50% when testing at > full load and therefore limits the achieved throughput. > Removing the lock contention from the SSTableReader.java file by replacing > the call to `getCapacity` with `size` achieves up to 2.95x increase in > throughput on r8g.24xlarge and 2x on r7i.24xlarge: > |Instance type|Cass 4.1.3|Cass 4.1.3 patched| > |r8g.24xlarge|168k ops|496k ops (2.95x)| > |r7i.24xlarge|153k ops|304k ops (1.98x)| > > Instructions to reproduce: > {code:java} > ## Requirements for Ubuntu 22.04 > sudo apt install -y ant git openjdk-11-jdk > ## Build and run > CASSANDRA_USE_JDK11=true ant realclean && CASSANDRA_USE_JDK11=true ant jar && > CASSANDRA_USE_JDK11=true ant stress-build && rm -rf data && bin/cassandra -f > -R > # Run > bin/cqlsh -e 'drop table if exists keyspace1.standard1;' && \ > bin/cqlsh -e 'drop keyspace if exists keyspace1;' && \ > bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write > n=10000000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log > -graph file=cload.html && \ > bin/nodetool compact keyspace1 && sleep 30s && \ > tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m > cl=ONE -rate threads=406 -node localhost -log file=result.log -graph > file=graph.html > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org