[
https://issues.apache.org/jira/browse/CASSANDRA-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115065#comment-18115065
]
Benedict Elliott Smith commented on CASSANDRA-21571:
----------------------------------------------------
With Accord this shouldn't be a problem, as transactions against the same key
can proceed in parallel. So, the first question is how much work we want to do
to solve this for Paxos.
However, there is an approach which has broader benefits by providing a QoS
mechanism, which is to inspect the client request queue so that we can
prioritise work, by e.g. client (IP, user), partition key, type of request
(read, write, lwt, transaction). There is no reason to discard work, we just
need to make sure that alternative work can arrive and be submitted in the
meantime. Request queue pruning kicks in when the queue is overloaded, which
can throw/discard work as appropriate/configured.
> Guardrail to limit LWT per-partition concurrency
> ------------------------------------------------
>
> Key: CASSANDRA-21571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21571
> Project: Apache Cassandra
> Issue Type: Improvement
> Reporter: Elliott Sims
> Priority: Normal
>
> I'm actually not sure if this should be a Guardrail exactly, or something
> else, but:
> Cassandra's threading model combined with LWT behavior means that it's very
> easy for contention on a single partition to tie up all of the threads on the
> hosts that own that partition, which causes cascading failures across the
> cluster and any clients. It's possible to set up rate-limits at the
> application level, but that can be challenging in a distributed system.
> Especially one that involves multiple applications or products.
> So, it would be very nice to have some way to say "only allow N (or X% of a
> pool?) threads to contend LWTs on a partition, and reject any further
> queries" to limit the blast radius to only that partition.
> Even a per-host limit on concurrent LWTs would help a fair amount even if it
> wouldn't completely isolate the problem partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
