Re: timeouts on counter tables

[Rudi Bruchez]

It can happen on any of the nodes. We can have a large number of pending on ReadStage and CounterMutationStage. We'll try to increase concurrent_counter_writes to see how it changes things

Likely. I believe counter mutations are a tad more expensive than a normal mutation. If you're doing a lot of counter updates that probably doesn't help. Regardless, high amounts of pending reads/mutations is generally not good and indicates the node being overloaded. Are you just seeing this on the 1 node with IO issues or do other nodes have this problem as well?

On 3 September 2017 at 20:25, Rudi Bruchez <r...@babaluga.com <mailto:r...@babaluga.com>> wrote:

    Le 30/08/2017 à 05:33, Erick Ramirez a écrit :

    Is it possible at all that you may have a data hotspot if it's
    not hardware-related?

    It does not seem so, The partition key seems well distributed and
    the queries update different keys.

    We have dropped counter_mutation messages in the log :

    COUNTER_MUTATION messages were dropped in last 5000 ms: 0 internal
    and 2 cross node. Mean internal dropped latency: 0 ms and Mean
    cross-node dropped latency: 5960 ms


    ReadStage    32       503 7481787         0                 0


    The load could be too high ?

    Thanks

Pool Name Active Pending Completed Blocked All Time Blocked CounterMutationStage 32 221 5722101 0 0

<<attachment: rudi.vcf>>

smime.p7s
Description: Signature cryptographique S/MIME