Well...

I think that pretty much is showing the problem. The problem I'd say is a
bad data model. Your read query is perfect, it hits a single partition and
that's the best situation, but on the other hand, it turns out that there
are one or two huge partitions and fully reading them is extremely
expensive. The tail of all you cfhistograms are very long. ie: the
difference between 99% and the max is very big

As you can see, the 99% of your partitions are within a reasonable 454Kb,
but it turns out that the maximum goes above 10Mb. Reviewing your data
model I have two ideas:

1. Either you have very wide rows (ie, lots of records for the same
partition key)
2. Some of your blob data is huge. I kind of think the problem is here, as
the difference in cells between the 99% and the max is very small while the
data size difference is huge, so partitions are not very big (ie, not very
wide rows), but very heavy weighting (lots of data).

Does that make sense?

Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>

On 21 October 2015 at 08:35, Brice Figureau <
brice+cassan...@daysofwonder.com> wrote:

> Hi,
>
> On 20/10/2015 19:48, Carlos Alonso wrote:
> > I think also having the output of cfhistograms could help. I'd like to
> > know how many sstables are being hit during reads.
>
> Ooops, my bad I thought you mentioned proxyhistograms.
> Here's the worst node cfhistograms:
>
> nodetool cfhistograms akka messages
> akka/messages histograms
> Percentile  SSTables     Write Latency      Read Latency    Partition
> Size        Cell Count
>                               (micros)          (micros)           (bytes)
> 50%             1.00             60.00            310.00
>  535                 1
> 75%             2.00            103.00           9887.00
>  11864                17
> 95%             3.00          14237.00         126934.00
>  73457               215
> 98%             3.00          51012.00         219342.00
> 263210               446
> 99%             3.00          61214.00        1131752.00
> 454826               924
> Min             0.00              6.00             18.00
> 73                 0
> Max             6.00         263210.00        5839588.00
> 10090808              1109
>
>
> > Also, which CL are you reading with?
>
> For the test request in cqlsh, I'm using ONE, but the application in which
> I first observed the issue was using QUORUM.
>
> > cfstats is a local command, so maybe that node you've printed is working
> > fine but there's another that is causing the latency. Can you check that
> > command in all nodes?
>
> I checked and on all nodes, the read latency and read local latency are
> within 15 to 40ms.
>
> I also noticed that C* was taking a fair bit of CPU on some of the nodes
> (ranging from 60% to 200%), looking at ttop output it was mostly taken by
> SharedPool-Worker threads, which I assume are the thread that are doing the
> real query work.
>
> Well, I'm puzzled, and I'll keep searching, thanks for your help!
> --
> Brice Figureau
>

Reply via email to