[
https://issues.apache.org/jira/browse/HBASE-30551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ronald Macmaster updated HBASE-30551:
-------------------------------------
Attachment: (was: orig-2.5.patch.txt)
> BucketCache: Move offsetLock ObjectPool purge off the read path
> ---------------------------------------------------------------
>
> Key: HBASE-30551
> URL: https://issues.apache.org/jira/browse/HBASE-30551
> Project: HBase
> Issue Type: Improvement
> Components: BucketCache
> Affects Versions: 3.0.0, 2.5.16, 2.6.7
> Environment: Discovered on HBase 2.5.11, but also appears unpatched
> on the 2.6 + 3.0 + master branches
> Reporter: Ronald Macmaster
> Assignee: Ronald Macmaster
> Priority: Major
> Labels: pull-request-available
> Attachments: after-patch-1.png, after-patch-2.png, after-patch-3.png,
> before-patch-zgc-1.png, profile1.png, profile2.png
>
>
> While experimenting with JDK 21 and the new ZGC on the server side, we
> noticed a performance issue from the BucketCache. GC Threads can contend with
> the reader threads and significantly degrade P99 / Max Latencies.
> Every [{{offsetLock.getLock()}}
> call|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLockWithObjectPool.java#L76-L81]
> on the read path runs {{{}ObjectPool.purge(){}}}, which [attempts cleanup on
> the pool's stale-reference
> queue|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L113-L128].
> Most of these attempts fall through the {{{}tryLock(){}}}, but one unlucky
> read req thread gets saddled with a large amount of reference queue cleanup.
> This is a standard cleanup convention aligned with the JDK docs for
> [ReferenceQueues|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/package-summary.html].
> And it's a reasonable approach under old G1GC/CMS assumptions
> (stop-the-world JDK8-17).
> However, it breaks down under ZGC. The GC threads clean up soft refs
> concurrently while app threads continue to run, and they continuously add
> more stale refs to the queue that the reader has paused to clean up. [The
> cleanup while loop is
> infinite|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L116-L123],
> so reader threads from the BucketCache livelock until ZGC finishes. It
> manifests as significantly worse tail-end performance (P99 / Max Get Latency).
> {quote}While some programs will choose to dedicate a thread to removing
> reference objects from one or more queues and processing them, this is by no
> means necessary. A tactic that often works well is to examine a reference
> queue in the course of performing some other fairly-frequent action. For
> example, a hashtable that uses weak references to implement weak keys could
> poll its reference queue each time the table is accessed. This is how the
> WeakHashMap class works. Because the ReferenceQueue.poll method simply checks
> an internal data structure, this check will add little overhead to the
> hashtable access methods.
> {quote}
> We should consider moving the ObjectPool clean-up off the read path. The docs
> state a separate cleanup thread is also a valid alternative approach.
> See the async-profiler and grafana attachments below for more details.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)