[
https://issues.apache.org/jira/browse/HBASE-29351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Mattingly resolved HBASE-29351.
-----------------------------------
Fix Version/s: 4.0.0-alpha-1
2.7.0
3.0.0-beta-2
Release Note:
Added new FeedbackAdaptiveRateLimiter to the quota system to improve quota
utilization and reduce client contention under heavy load.
The FeedbackAdaptiveRateLimiter is an adaptive rate limiter that dynamically
adjusts its behavior based on observed usage patterns. It addresses two key
challenges in quota management:
1. Adaptive Backpressure for Contention Management: When multiple threads
compete for limited resources, the limiter detects contention and dynamically
extends wait intervals to prevent thundering herd
behavior where many threads wake simultaneously. The backoff multiplier
increases when contention is detected and decreases when no contention occurs,
converging toward optimal throughput.
2. Oversubscription for Full Utilization: In practice, synchronization
overhead and timing variations often prevent clients from consuming their full
quota allowance, resulting in consistent
under-utilization. The limiter tracks utilization via an exponentially
weighted moving average (EWMA) and gradually increases oversubscription when
average utilization falls below target, allowing
slightly more resources per interval to compensate for inefficiencies and
achieve stable throughput closer to configured quotas.
This rate limiter can be enabled by setting `hbase.quota.rate.limiter.class`
to `org.apache.hadoop.hbase.quotas.FeedbackAdaptiveRateLimiter`. The algorithm
is tunable via several configuration parameters for
backoff multipliers, oversubscription limits, and utilization error budgets
to adapt to different workload characteristics.
Resolution: Fixed
> Quotas: adaptive wait intervals
> -------------------------------
>
> Key: HBASE-29351
> URL: https://issues.apache.org/jira/browse/HBASE-29351
> Project: HBase
> Issue Type: Improvement
> Components: Quotas
> Reporter: Ray Mattingly
> Assignee: Ray Mattingly
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2
>
>
> We've been using Quotas heavily at my day job, particularly user throttles.
> You can read more about our setup
> [here|https://product.hubspot.com/blog/hbase-share-resources].
> We have hundreds of clusters with a wide variety of usage patterns; we've
> found it difficult to find a `hbase.quota.rate.limiter.refill.interval.ms`
> that perfectly blends optimistic enough retries (to make fully utilizing the
> quota easy) and sufficiently pessimistic backoffs (to avoid hotspotting RPC
> layers with doomed retries) across the board and out-of-the-box.
> For some background, when a request is throttled our RateLimiters will return
> the backoff millis that must be waited in order for the RateLimiter to
> service the given request's estimated workload in a single threaded
> environment. This falls apart for two reasons: clients are rarely single
> threaded, and estimated workloads are often wrong.
> To make usage and configuration easier, I'm suggesting that we add support
> for "adaptive wait intervals" to the FixedIntervalRateLimiter. This would
> involve introducing a multiplier for the wait interval which grows when we
> throttle multiple times in a given refill interval, or shrinks when we go
> many refill intervals without throttling. The result would be wait interval
> suggestions that provide smarter backoffs designed to fully utilize, but not
> oversubscribe, the allowance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
