[ 
https://issues.apache.org/jira/browse/HBASE-30551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ronald Macmaster updated HBASE-30551:
-------------------------------------
    Attachment:     (was: orig-2.5.patch.txt)

> BucketCache: Move offsetLock ObjectPool purge off the read path
> ---------------------------------------------------------------
>
>                 Key: HBASE-30551
>                 URL: https://issues.apache.org/jira/browse/HBASE-30551
>             Project: HBase
>          Issue Type: Improvement
>          Components: BucketCache
>    Affects Versions: 3.0.0, 2.5.16, 2.6.7
>         Environment: Discovered on HBase 2.5.11, but also appears unpatched 
> on the 2.6 + 3.0 + master branches
>            Reporter: Ronald Macmaster
>            Assignee: Ronald Macmaster
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: after-patch-1.png, after-patch-2.png, after-patch-3.png, 
> before-patch-zgc-1.png, profile1.png, profile2.png
>
>
> While experimenting with JDK 21 and the new ZGC on the server side, we 
> noticed a performance issue from the BucketCache. GC Threads can contend with 
> the reader threads and significantly degrade P99 / Max Latencies.
> Every [{{offsetLock.getLock()}} 
> call|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLockWithObjectPool.java#L76-L81]
>  on the read path runs {{{}ObjectPool.purge(){}}}, which [attempts cleanup on 
> the pool's stale-reference 
> queue|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L113-L128].
>  Most of these attempts fall through the {{{}tryLock(){}}}, but one unlucky 
> read req thread gets saddled with a large amount of reference queue cleanup.
> This is a standard cleanup convention aligned with the JDK docs for 
> [ReferenceQueues|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/ref/package-summary.html].
>  And it's a reasonable approach under old G1GC/CMS assumptions 
> (stop-the-world JDK8-17).
> However, it breaks down under ZGC. The GC threads clean up soft refs 
> concurrently while app threads continue to run, and they continuously add 
> more stale refs to the queue that the reader has paused to clean up. [The 
> cleanup while loop is 
> infinite|https://github.com/apache/hbase/blob/dcd112b85972660312476e0b46d093b61e19f566/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ObjectPool.java#L116-L123],
>  so reader threads from the BucketCache livelock until ZGC finishes. It 
> manifests as significantly worse tail-end performance (P99 / Max Get Latency).
> {quote}While some programs will choose to dedicate a thread to removing 
> reference objects from one or more queues and processing them, this is by no 
> means necessary. A tactic that often works well is to examine a reference 
> queue in the course of performing some other fairly-frequent action. For 
> example, a hashtable that uses weak references to implement weak keys could 
> poll its reference queue each time the table is accessed. This is how the 
> WeakHashMap class works. Because the ReferenceQueue.poll method simply checks 
> an internal data structure, this check will add little overhead to the 
> hashtable access methods.
> {quote}
> We should consider moving the ObjectPool clean-up off the read path. The docs 
> state a separate cleanup thread is also a valid alternative approach.
> See the async-profiler and grafana attachments below for more details.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to