Ray Mattingly created HBASE-29351:
-------------------------------------
Summary: Quotas: adaptive wait intervals
Key: HBASE-29351
URL: https://issues.apache.org/jira/browse/HBASE-29351
Project: HBase
Issue Type: Improvement
Reporter: Ray Mattingly
Assignee: Ray Mattingly
We've been using Quotas heavily at my day job, particularly user throttles. You
can read more about our setup
[here|https://product.hubspot.com/blog/hbase-share-resources].
We have hundreds of clusters with a wide variety of usage patterns; we've found
it difficult to find a `hbase.quota.rate.limiter.refill.interval.ms` that
perfectly blends optimistic enough retries (to make fully utilizing the quota
easy) and sufficiently pessimistic backoffs (to avoid hotspotting RPC layers
with doomed retries) across the board and out-of-the-box.
For some background, when a request is throttled our RateLimiters will return
the backoff millis that must be waited in order for the RateLimiter to service
the given request's estimated workload in a single threaded environment. This
falls apart for two reasons: clients are rarely single threaded, and estimated
workloads are often wrong.
To make usage and configuration easier, I'm suggesting that we add support for
"adaptive wait intervals" to the FixedIntervalRateLimiter. This would involve
introducing a multiplier for the wait interval which grows when we throttle
multiple times in a given refill interval, or shrinks when we go many refill
intervals without throttling. The result would be wait interval suggestions
that provide smarter backoffs designed to fully utilize, but not oversubscribe,
the allowance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
