[
https://issues.apache.org/jira/browse/HBASE-30560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiwen Deng updated HBASE-30560:
--------------------------------
Description:
h3. Current state
WAL sync is decided in {{AbstractFSWAL#appendAndSync()}} / {{{}#consume(){}}},
by two triggers only:
# size: {{(writer.getLength() - fleLengthAtLastSync) >= hbase.wal.batch.size}}
(default 64KB, a byte threshold rather than an entry count);
# empty buffer: the consumer has drained the ring buffer and a SyncFuture is
pending -> issue {{sync(writer)}} immediately and stop scheduling the consumer
(see "sync because buffer size limit" in {{{}appendAndSync(){}}}).
Between the two, {{appendAndSync()}} deliberately defers sync while more
appends may arrive, so batching works as long as writes keep coming. The
empty-buffer path has no grace period.
h3. Problem
The empty-buffer trigger makes the sync rate independent of QPS but tightly
coupled to the hflush round trip: as soon as the ring buffer drains, a sync is
issued, so sync/s approaches 1/hflush-RTT (~5000/s on local NVMe pipelines) no
matter how few bytes each sync carries.
With single-row 1KB puts this is the dominant cost: one sync per drain, each
covering a few KB. The single WAL consumer thread ends up spending essentially
all of its time inside sync, leaving no headroom, so throughput saturates and
p99 write latency becomes unstable under pipeline jitter. Tuning
{{hbase.wal.batch.size}} does not help here: it only decides "enough bytes ->
sync now", and never sees the "not enough bytes but buffer empty" case.
h3. Proposed change
Add an opt-in minimum interval between two WAL syncs, default 0 (disabled,
current behaviour unchanged):
{{hbase.regionserver.wal.sync.linger.ms}}
When a sync would fire only because the ring buffer is empty, and the previous
sync finished less than N ms ago, postpone it. Size trigger, forceSync
requests, a pending roll and shutdown always bypass the linger.
Default 0 keeps behaviour byte-identical to today. The setting is a
latency/throughput trade: write p99 can grow by up to the linger value and the
SYNC_WAL durability window widens by the same amount, so it should stay opt-in
for latency-sensitive workloads.
h3. Result
With {{hbase.regionserver.wal.sync.linger.ms}} = 2, on a single RegionServer
(NVMe, 3 replicas), 100% single-row puts with one 1KB cell per RPC:
* throughput: 160,000 - 170,000 ops/s, single WAL;
* write p99: under 10 ms;
the sync rate is now bounded by the window instead of by hflush RTT, so the
consumer thread keeps headroom and stops being the limiting factor at high QPS.
I'm not sure whether this is the right approach for the WAL sync path - any
feedback would be welcome. If the direction looks reasonable, I'd be happy to
prepare and submit a patch for it.
was:
h3. Current state
WAL sync is decided in {{AbstractFSWAL#appendAndSync()}} / {{{}#consume(){}}},
by two triggers only:
# size: {{(writer.getLength() - fleLengthAtLastSync) >= hbase.wal.batch.size}}
(default 64KB, a byte threshold rather than an entry count);
# empty buffer: the consumer has drained the ring buffer and a SyncFuture is
pending -> issue {{sync(writer)}} immediately and stop scheduling the consumer
(see "sync because buffer size limit" in {{{}appendAndSync(){}}}).
Between the two, {{appendAndSync()}} deliberately defers sync while more
appends may arrive, so batching works as long as writes keep coming. The
empty-buffer path has no grace period.
h3. Problem
The empty-buffer trigger makes the sync rate independent of QPS but tightly
coupled to the hflush round trip: as soon as the ring buffer drains, a sync is
issued, so sync/s approaches 1/hflush-RTT (~5000/s on local NVMe pipelines) no
matter how few bytes each sync carries.
With single-row 1KB puts this is the dominant cost: one sync per drain, each
covering a few KB. The single WAL consumer thread ends up spending essentially
all of its time inside sync, leaving no headroom, so throughput saturates and
p99 write latency becomes unstable under pipeline jitter. Tuning
{{hbase.wal.batch.size}} does not help here: it only decides "enough bytes ->
sync now", and never sees the "not enough bytes but buffer empty" case.
h3. Proposed change
Add an opt-in minimum interval between two WAL syncs, default 0 (disabled,
current behaviour unchanged):
{{hbase.regionserver.wal.sync.linger.ms}}
When a sync would fire only because the ring buffer is empty, and the previous
sync finished less than N ms ago, postpone it. Size trigger, forceSync
requests, a pending roll and shutdown always bypass the linger.
Default 0 keeps behaviour byte-identical to today. The setting is a
latency/throughput trade: write p99 can grow by up to the linger value and the
SYNC_WAL durability window widens by the same amount, so it should stay opt-in
for latency-sensitive workloads.
h3. Result
With {{hbase.regionserver.wal.sync.linger.ms}} = 1, on a single RegionServer
(NVMe, 3 replicas), 100% single-row puts with one 1KB cell per RPC:
* throughput: 160,000 - 170,000 ops/s, single WAL;
* write p99: under 10 ms;
the sync rate is now bounded by the window instead of by hflush RTT, so the
consumer thread keeps headroom and stops being the limiting factor at high QPS.
I'm not sure whether this is the right approach for the WAL sync path - any
feedback would be welcome. If the direction looks reasonable, I'd be happy to
prepare and submit a patch for it.
> WAL: add a minimum sync interval (sync linger) to avoid per-drain hflush on
> high-QPS small-write workloads
> ----------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30560
> URL: https://issues.apache.org/jira/browse/HBASE-30560
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Affects Versions: 3.0.0
> Reporter: Zhiwen Deng
> Priority: Major
>
> h3. Current state
> WAL sync is decided in {{AbstractFSWAL#appendAndSync()}} /
> {{{}#consume(){}}}, by two triggers only:
> # size: {{(writer.getLength() - fleLengthAtLastSync) >=
> hbase.wal.batch.size}} (default 64KB, a byte threshold rather than an entry
> count);
> # empty buffer: the consumer has drained the ring buffer and a SyncFuture is
> pending -> issue {{sync(writer)}} immediately and stop scheduling the
> consumer (see "sync because buffer size limit" in {{{}appendAndSync(){}}}).
> Between the two, {{appendAndSync()}} deliberately defers sync while more
> appends may arrive, so batching works as long as writes keep coming. The
> empty-buffer path has no grace period.
> h3. Problem
> The empty-buffer trigger makes the sync rate independent of QPS but tightly
> coupled to the hflush round trip: as soon as the ring buffer drains, a sync
> is issued, so sync/s approaches 1/hflush-RTT (~5000/s on local NVMe
> pipelines) no matter how few bytes each sync carries.
> With single-row 1KB puts this is the dominant cost: one sync per drain, each
> covering a few KB. The single WAL consumer thread ends up spending
> essentially all of its time inside sync, leaving no headroom, so throughput
> saturates and p99 write latency becomes unstable under pipeline jitter.
> Tuning {{hbase.wal.batch.size}} does not help here: it only decides "enough
> bytes -> sync now", and never sees the "not enough bytes but buffer empty"
> case.
> h3. Proposed change
> Add an opt-in minimum interval between two WAL syncs, default 0 (disabled,
> current behaviour unchanged):
> {{hbase.regionserver.wal.sync.linger.ms}}
> When a sync would fire only because the ring buffer is empty, and the
> previous sync finished less than N ms ago, postpone it. Size trigger,
> forceSync requests, a pending roll and shutdown always bypass the linger.
> Default 0 keeps behaviour byte-identical to today. The setting is a
> latency/throughput trade: write p99 can grow by up to the linger value and
> the SYNC_WAL durability window widens by the same amount, so it should stay
> opt-in for latency-sensitive workloads.
> h3. Result
> With {{hbase.regionserver.wal.sync.linger.ms}} = 2, on a single RegionServer
> (NVMe, 3 replicas), 100% single-row puts with one 1KB cell per RPC:
> * throughput: 160,000 - 170,000 ops/s, single WAL;
> * write p99: under 10 ms;
> the sync rate is now bounded by the window instead of by hflush RTT, so the
> consumer thread keeps headroom and stops being the limiting factor at high
> QPS.
> I'm not sure whether this is the right approach for the WAL sync path - any
> feedback would be welcome. If the direction looks reasonable, I'd be happy to
> prepare and submit a patch for it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)