I have a two node cluster hosting a 45 gig dataset. I periodically have to read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a time and then processing them.
This used to result in about 300-500 reads a second which seemed quite good. Recently that number has plummeted to 20-50 reads a second. The obvious question is what did I change? I certainly added more data....bringing my total load from 38 or so gig to 45 or so gig but its hard to imagine that causing this problem. The shape of my data has not changed and I haven't changed any cassandra configuration. Running nodetool tpstats I'm for the first time ever seeing entries under ReadStage Active and Pending which correlates with slow reads. Running iostat I'm seeing a significant (10-50%) of iowait where I previously never saw higher than 1-2% I ran a full compaction on the relevant CF (which took 3.5 hours) to no avail. Any suggestions on where I can look next? Thanks.
