Well... I think that pretty much is showing the problem. The problem I'd say is a bad data model. Your read query is perfect, it hits a single partition and that's the best situation, but on the other hand, it turns out that there are one or two huge partitions and fully reading them is extremely expensive. The tail of all you cfhistograms are very long. ie: the difference between 99% and the max is very big
As you can see, the 99% of your partitions are within a reasonable 454Kb, but it turns out that the maximum goes above 10Mb. Reviewing your data model I have two ideas: 1. Either you have very wide rows (ie, lots of records for the same partition key) 2. Some of your blob data is huge. I kind of think the problem is here, as the difference in cells between the 99% and the max is very small while the data size difference is huge, so partitions are not very big (ie, not very wide rows), but very heavy weighting (lots of data). Does that make sense? Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> On 21 October 2015 at 08:35, Brice Figureau < brice+cassan...@daysofwonder.com> wrote: > Hi, > > On 20/10/2015 19:48, Carlos Alonso wrote: > > I think also having the output of cfhistograms could help. I'd like to > > know how many sstables are being hit during reads. > > Ooops, my bad I thought you mentioned proxyhistograms. > Here's the worst node cfhistograms: > > nodetool cfhistograms akka messages > akka/messages histograms > Percentile SSTables Write Latency Read Latency Partition > Size Cell Count > (micros) (micros) (bytes) > 50% 1.00 60.00 310.00 > 535 1 > 75% 2.00 103.00 9887.00 > 11864 17 > 95% 3.00 14237.00 126934.00 > 73457 215 > 98% 3.00 51012.00 219342.00 > 263210 446 > 99% 3.00 61214.00 1131752.00 > 454826 924 > Min 0.00 6.00 18.00 > 73 0 > Max 6.00 263210.00 5839588.00 > 10090808 1109 > > > > Also, which CL are you reading with? > > For the test request in cqlsh, I'm using ONE, but the application in which > I first observed the issue was using QUORUM. > > > cfstats is a local command, so maybe that node you've printed is working > > fine but there's another that is causing the latency. Can you check that > > command in all nodes? > > I checked and on all nodes, the read latency and read local latency are > within 15 to 40ms. > > I also noticed that C* was taking a fair bit of CPU on some of the nodes > (ranging from 60% to 200%), looking at ttop output it was mostly taken by > SharedPool-Worker threads, which I assume are the thread that are doing the > real query work. > > Well, I'm puzzled, and I'll keep searching, thanks for your help! > -- > Brice Figureau >
