Hi All, On debugging the cluster for performance dip seen while using 4.1.4, i found high speculation retries Value in nodetool tablestats during read operation.
I ran the below tablestats command and checked its output after every few secs and noticed that retries are on rising side. Also there is one open ticket (https://issues.apache.org/jira/browse/CASSANDRA-18766) similar to this. /usr/share/cassandra/bin/nodetool -u <username> -pw <pwd> -p <port> tablestats <keyspace> | grep -i 'Speculative retries' Speculative retries: 11633 .. .. Speculative retries: 13727 Speculative retries: 14256 Speculative retries: 14855 Speculative retries: 14858 Speculative retries: 14859 Speculative retries: 14873 Speculative retries: 14875 Speculative retries: 14890 Speculative retries: 14893 Speculative retries: 14896 Speculative retries: 14901 Speculative retries: 14905 Speculative retries: 14946 Speculative retries: 14948 Speculative retries: 14957 Suspecting this could be performance dip cause. Please add in case anyone knows more about it. Regards On Wed, Mar 27, 2024 at 10:43 PM Subroto Barua via user < user@cassandra.apache.org> wrote: > we are seeing similar perf issues with counter writes - to reproduce: > > cassandra-stress counter_write n=100000 no-warmup cl=LOCAL_QUORUM -rate > threads=50 -mode native cql3 user=<user> password=<pw> -name <cluster_name> > > > op rate: 39,260 ops (4.1) and 63,689 ops (4.0) > latency 99th percentile: 7.7ms (4.1) and 1.8ms (4.0) > Total GC count: 750 (4.1) and 744 (4.0) > Avg GC time: 106 ms (4.1) and 110.1 ms (4.0) > > > On Wednesday, March 27, 2024 at 12:18:50 AM PDT, ranju goel < > goel.ra...@gmail.com> wrote: > > > Hi All, > > Was going through this mail chain > (https://www.mail-archive.com/user@cassandra.apache.org/msg63564.html) > and was wondering that if this could cause a performance degradation in > 4.1 without changing compactionThroughput. > > As seeing performance dip in Read/Write after upgrading from 4.0 to 4.1. > > Regards > Ranju >
