did the amount of data finally exceed your per machine RAM capacity? is it the same 20% each time you read? or do your periodic reads eventually work through the entire dataset?
if you are essentially table scanning your data set, and the size exceeds available RAM, then a degradation like that isn't crazy. and this is indicated with your iowait %% On Fri, Oct 12, 2012 at 6:33 AM, Brian Tarbox <tar...@cabotresearch.com> wrote: > I have a two node cluster hosting a 45 gig dataset. I periodically have to > read a high fraction (20% or so) of my 'rows', grabbing a few thousand at a > time and then processing them. > > This used to result in about 300-500 reads a second which seemed quite good. > Recently that number has plummeted to 20-50 reads a second. The obvious > question is what did I change? I certainly added more data....bringing my > total load from 38 or so gig to 45 or so gig but its hard to imagine that > causing this problem. The shape of my data has not changed and I haven't > changed any cassandra configuration. > > Running nodetool tpstats I'm for the first time ever seeing entries under > ReadStage Active and Pending which correlates with slow reads. > Running iostat I'm seeing a significant (10-50%) of iowait where I > previously never saw higher than 1-2% > > I ran a full compaction on the relevant CF (which took 3.5 hours) to no > avail. > > Any suggestions on where I can look next? > > Thanks.
