[ https://issues.apache.org/jira/browse/CASSANDRA-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb Rackliffe updated CASSANDRA-16663: ---------------------------------------- Test and Documentation Plan: There are two new {{cassandra.yaml}} options in this patch: {{native_transport_rate_limiting_enabled}} - Whether or not to apply a rate limit on CQL requests. (Default: false) {{native_transport_max_requests_per_second}} - The limit itself. (Default: Double.MAX_VALUE) In terms of testing, there is a new suite of overload tests in {{RateLimitingTest}}, and this provides at least basic coverage along the major axes of message size, back-pressure vs. throw-on-overload, and behavior in the face of configuration changes at runtime (although part of that last bit is in {{ClientResourceLimitsTest}}). As this moves into review, I've also used {{SimpleClientPerfTest}} across small/large messages and all 4 active protocol versions to make sure the limiter maintains throughput in a reasonably tight band around the configured limit. More stress testing, likely w/ tlp-stress and with multi-node clusters, will happen concurrently w/ review. Finally, from an operational perspective, this is a coordinator-level limit, and assuming RF=N, any time a replica is unavailable, cluster-wide capacity to query a given partition (assuming a token-aware client) decreases by a factor of 1/N. was: There are two new {{cassandra.yaml}} options in this patch: {{native_transport_rate_limiting_enabled}} - Whether or not to apply a rate limit on CQL requests. (Default: false) {{native_transport_requests_per_second}} - The limit itself. (Default: Double.MAX_VALUE) In terms of testing, there is a new suite of overload tests in {{RateLimitingTest}}, and this provides at least basic coverage along the major axes of message size, back-pressure vs. throw-on-overload, and behavior in the face of configuration changes at runtime (although part of that last bit is in {{ClientResourceLimitsTest}}). As this moves into review, I've also used {{SimpleClientPerfTest}} across small/large messages and all 4 active protocol versions to make sure the limiter maintains throughput in a reasonably tight band around the configured limit. More stress testing, likely w/ tlp-stress and with multi-node clusters, will happen concurrently w/ review. Finally, from an operational perspective, this is a coordinator-level limit, and assuming RF=N, any time a replica is unavailable, cluster-wide capacity to query a given partition (assuming a token-aware client) decreases by a factor of 1/N. > Request-Based Native Transport Rate-Limiting > -------------------------------------------- > > Key: CASSANDRA-16663 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16663 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client > Reporter: Caleb Rackliffe > Assignee: Caleb Rackliffe > Priority: Normal > Fix For: 4.x > > Time Spent: 9h 20m > Remaining Estimate: 0h > > Together, CASSANDRA-14855, CASSANDRA-15013, and CASSANDRA-15519 added support > for a runtime-configurable, per-coordinator limit on the number of bytes > allocated for concurrent requests over the native protocol. It supports > channel back-pressure by default, and optionally supports throwing > OverloadedException if that is requested in the relevant connection’s STARTUP > message. > This can be an effective tool to prevent the coordinator from running out of > memory, but it may not correspond to how expensive a queries are or provide a > direct conceptual mapping to how users think about request capacity. I > propose adding the option of request-based (or perhaps more correctly > message-based) back-pressure, coexisting with (and reusing the logic that > supports) the current bytes-based back-pressure. > _We can roll this forward in phases_, where the server’s cost accounting > becomes more accurate, we segment limits by operation type/keyspace/etc., and > the client/driver reacts more intelligently to (especially non-back-pressure) > overload, _but something minimally viable could look like this_: > 1.) Reuse most of the existing logic in Limits, et al. to support a simple > per-coordinator limit only on native transport requests per second. Under > this limit will be CQL reads and writes, but also auth requests, prepare > requests, and batches. This is obviously simplistic, and it does not account > for the variation in cost between individual queries, but even a fixed cost > model should be useful in aggregate. > * If the client specifies THROW_ON_OVERLOAD in its STARTUP message at > connection time, a breach of the per-node limit will result in an > OverloadedException being propagated to the client, and the server will > discard the request. > * If THROW_ON_OVERLOAD is not specified, the server will stop consuming > messages from the channel/socket, which should back-pressure the client, > while the message continues to be processed. > 2.) This limit is infinite by default (or simply disabled), and can be > enabled via the YAML config or JMX at runtime. (It might be cleaner to have a > no-op rate limiter that's used when the feature is disabled entirely.) > 3.) The current value of the limit is available via JMX, and metrics around > coordinator operations/second are already available to compare against it. > 4.) Any interaction with existing byte-based limits will intersect. (i.e. A > breach of any limit, bytes or request-based, will actuate back-pressure or > OverloadedExceptions.) > In this first pass, explicitly out of scope would be any work on the > client/driver side. > In terms of validation/testing, our biggest concern with anything that adds > overhead on a very hot path is performance. In particular, we want to fully > understand how the client and server perform along two axes constituting 4 > scenarios. Those are a.) whether or not we are breaching the request limit > and b.) whether the server is throwing on overload at the behest of the > client. Having said that, query execution should dwarf the cost of limit > accounting. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org