[
https://issues.apache.org/jira/browse/CASSANDRA-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Lightfoot updated CASSANDRA-21679:
--------------------------------------
Change Category: Performance
Complexity: Normal
Status: Open (was: Triage Needed)
> Implement trickle_fsync with sync_file_range
> --------------------------------------------
>
> Key: CASSANDRA-21679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21679
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Local/SSTable
> Reporter: Sam Lightfoot
> Assignee: Sam Lightfoot
> Priority: Normal
>
> h4. Summary
> `trickle_fsync` paces SSTable writeback with `fdatasync`, which also commits
> the ext4 journal, flushes the device cache on drives that have one, walks the
> whole file's dirty set, and most notably blocks the flush writer with no
> overlap between it and the device. None of that is required, durability is
> the commitlog's job and `SequentialWriter` fsyncs at `close()` regardless.
> The `sync_file_range` syscall does only the pacing work. This changes the
> implementation beneath `trickle_fsync: true`; it does not add an option.
> h4. What changes
> At each interval, on Linux, two calls:
> {code:java}
> sync_file_range(fd, off_N, len, SYNC_FILE_RANGE_WRITE); // submit N, returns
> sync_file_range(fd, off_N-1, len, SYNC_FILE_RANGE_WAIT_BEFORE); // wait for
> N-1{code}
> Lagging the wait by one interval is the point. `fdatasync` waits on data it
> submitted in the same call, so producing and draining never overlap and every
> interval costs both. Waiting one behind offsets the wait by exactly one
> interval's generation time, N−1 has been in flight since the previous call,
> so an interval costs the larger of the two and the stall is `max(0, drain −
> produce)`.
>
> That is zero whenever the device drains an interval faster than the writer
> generates one, which is the case here at 24–33% device utilisation: 15.3 s →
> 1.0 s below, 93% of flush-writer stall removed, throughput and burst shaping
> unchanged. Where the device is the limit instead, the writer is paced by it,
> the backpressure `trickle_fsync` exists to provide. The writer runs at most
> one interval ahead either way, so the bound holds.
> `sync_file_range` writes no metadata, so the close sync becomes the writer's
> only metadata commit.
> h4. Measured
> Single node, 10 MiB interval, ~411 MiB memtable flushes. Paired A/B, both
> arms in the same invocation, n=5, arms matched to 0.0001% on read count.
> | |trickle off|`fdatasync`|this change|
> |flush-writer stall|0.4 s|15.3 s|1.0 s|
> |ext4 journal transactions|3493|6574 (+3081)|3497 |
> |read p99 during flush|24.3 ms|11.4 ms|9.0 ms|
> |system flush throughput|116.4 MiB/s|116.4|116.5|
> |dirty-window peak|439 MiB|50.8|48.6|
> Stall drops 14.8× with no overlap between arms (14.2–17.0 s vs 0.8–1.4 s).
> Flush-writer CPU is unchanged (69.7 vs 70.1 s), so that is time blocked, not
> work removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
