Zhiwen Deng created HBASE-30560:
-----------------------------------

             Summary: WAL: add a minimum sync interval (sync linger) to avoid 
per-drain hflush on high-QPS small-write workloads
                 Key: HBASE-30560
                 URL: https://issues.apache.org/jira/browse/HBASE-30560
             Project: HBase
          Issue Type: Improvement
          Components: wal
    Affects Versions: 3.0.0
            Reporter: Zhiwen Deng


h3. Current state

WAL sync is decided in {{AbstractFSWAL#appendAndSync()}} / {{#consume()}}, by 
two triggers only:

# siize: {{(writer.getLength() - fleLengthAtLastSync) >= hbase.wal.batch.size}} 
(default 64KB, a byte threshold rather than an entry count);
# empty buffer: the consumer has drained the ring buffer and a SyncFuture is 
pending -> issue {{sync(writer)}} immediately and stop scheduling the consumer 
(see "sync because buffer size limit" in {{appendAndSync()}}).

Between the two, {{appendAndSync()}} deliberately defers sync while more 
appends may arrive, so batching works as long as writes keep coming. The 
empty-buffer path has no grace period.

h3. Problem

The empty-buffer trigger makes the sync rate independent of QPS but tightly 
coupled to the hflush round trip: as soon as the ring buffer drains, a sync is 
issued, so sync/s approaches 1/hflush-RTT (~5000/s on local NVMe pipelines) no 
matter how few bytes each sync carries.

With single-row 1KB puts this is the dominant cost: one sync per drain, each 
covering a few KB. The single WAL consumer thread ends up spending essentially 
all of its time inside sync, leaving no headroom, so throughput saturates and 
p99 write latency becomes unstable under pipeline jitter. Tuning 
{{hbase.wal.batch.size}} does not help here: it only decides "enough bytes -> 
sync now", and never sees the "not enough bytes but buffer empty" case.

h3. Proposed change

Add an opt-in minimum interval between two WAL syncs, default 0 (disabled, 
current behaviour unchanged):

{{hbase.regionserver.wal.sync.linger.ms}}

When a sync would fire only because the ring buffer is empty, and the previous 
sync finished less than N ms ago, postpone it. Size trigger, forceSync 
requests, a pending roll and shutdown always bypass the linger.

Default 0 keeps behaviour byte-identical to today. The setting is a 
latency/throughput trade: write p99 can grow by up to the linger value and the 
SYNC_WAL durability window widens by the same amount, so it should stay opt-in 
for latency-sensitive workloads.

h3. Result

With {{hbase.regionserver.wal.sync.linger.ms}} = 1, on a single RegionServer 
(NVMe, 3 replicas), 100% single-row puts with one 1KB cell per RPC:

* throughput: 160,000 - 170,000 ops/s, single WAL;
* write p99: under 10 ms;

the sync rate is now bounded by the window instead of by hflush RTT, so the 
consumer thread keeps headroom and stops being the limiting factor at high QPS.

I'm not sure whether this is the right approach for the WAL sync path - any 
feedback would be welcome. If the direction looks reasonable, I'd be happy to 
prepare and submit a patch for it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to