[
https://issues.apache.org/jira/browse/CASSANDRA-21679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Lightfoot updated CASSANDRA-21679:
--------------------------------------
Description:
h4. Summary
`trickle_fsync` paces SSTable writeback with `fdatasync`, which also commits
the ext4 journal, flushes the device cache on drives that have one, walks the
whole file's dirty set, and most notably blocks the flush writer with no
overlap between it and the device. None of that is required, durability is the
commitlog's job and `SequentialWriter` fsyncs at `close()` regardless.
`sync_file_range` does only the pacing. Aiming its wait one interval behind the
write overlaps producing with draining, so the writer blocks only when the
device is genuinely behind: 93% of flush-writer stall goes away (15.3 s → 1.0
s) with throughput, burst shaping and the bound on outstanding writeback
unchanged. This changes the implementation beneath `trickle_fsync: true`; it
adds no option.
h4. What changes
At each interval, on Linux, two calls:
{code:java}
sync_file_range(fd, off_N, len, SYNC_FILE_RANGE_WRITE); // submit N, returns
sync_file_range(fd, off_N-1, len, SYNC_FILE_RANGE_WAIT_BEFORE); // wait for
N-1{code}
Lagging the wait by one interval is the point. `fdatasync` waits on data it
submitted in the same call, so producing and draining never overlap and every
interval costs both. Waiting one behind offsets the wait by exactly one
interval's generation time, N−1 has been in flight since the previous call, so
an interval costs the larger of the two and the stall is `max(0, drain −
produce)`.
That is zero whenever the device drains an interval faster than the writer
generates one, which is the case here at 24–33% device utilisation: 15.3 s →
1.0 s below, 93% of flush-writer stall removed, throughput and burst shaping
unchanged. Where the device is the limit instead, the writer is paced by it,
the backpressure `trickle_fsync` exists to provide. The writer runs at most one
interval ahead either way, so the bound holds.
`sync_file_range` writes no metadata, so the close sync becomes the writer's
only metadata commit.
h4. Measured
Single node, 10 MiB interval, ~411 MiB memtable flushes. Paired A/B, both arms
in the same invocation, n=5, arms matched to 0.0001% on read count.
| |trickle off|`fdatasync`|this change|
|flush-writer stall|0.4 s|15.3 s|1.0 s|
|ext4 journal transactions|3493|6574 (+3081)|3497 |
|read p99 during flush|24.3 ms|11.4 ms|9.0 ms|
|system flush throughput|116.4 MiB/s|116.4|116.5|
|dirty-window peak|439 MiB|50.8|48.6|
Stall drops 14.8× with no overlap between arms (14.2–17.0 s vs 0.8–1.4 s).
Flush-writer CPU is unchanged (69.7 vs 70.1 s), so that is time blocked, not
work removed.
was:
h4. Summary
`trickle_fsync` paces SSTable writeback with `fdatasync`, which also commits
the ext4 journal, flushes the device cache on drives that have one, walks the
whole file's dirty set, and most notably blocks the flush writer with no
overlap between it and the device. None of that is required, durability is the
commitlog's job and `SequentialWriter` fsyncs at `close()` regardless.
The `sync_file_range` syscall does only the pacing work. This changes the
implementation beneath `trickle_fsync: true`; it does not add an option.
h4. What changes
At each interval, on Linux, two calls:
{code:java}
sync_file_range(fd, off_N, len, SYNC_FILE_RANGE_WRITE); // submit N, returns
sync_file_range(fd, off_N-1, len, SYNC_FILE_RANGE_WAIT_BEFORE); // wait for
N-1{code}
Lagging the wait by one interval is the point. `fdatasync` waits on data it
submitted in the same call, so producing and draining never overlap and every
interval costs both. Waiting one behind offsets the wait by exactly one
interval's generation time, N−1 has been in flight since the previous call, so
an interval costs the larger of the two and the stall is `max(0, drain −
produce)`.
That is zero whenever the device drains an interval faster than the writer
generates one, which is the case here at 24–33% device utilisation: 15.3 s →
1.0 s below, 93% of flush-writer stall removed, throughput and burst shaping
unchanged. Where the device is the limit instead, the writer is paced by it,
the backpressure `trickle_fsync` exists to provide. The writer runs at most one
interval ahead either way, so the bound holds.
`sync_file_range` writes no metadata, so the close sync becomes the writer's
only metadata commit.
h4. Measured
Single node, 10 MiB interval, ~411 MiB memtable flushes. Paired A/B, both arms
in the same invocation, n=5, arms matched to 0.0001% on read count.
| |trickle off|`fdatasync`|this change|
|flush-writer stall|0.4 s|15.3 s|1.0 s|
|ext4 journal transactions|3493|6574 (+3081)|3497 |
|read p99 during flush|24.3 ms|11.4 ms|9.0 ms|
|system flush throughput|116.4 MiB/s|116.4|116.5|
|dirty-window peak|439 MiB|50.8|48.6|
Stall drops 14.8× with no overlap between arms (14.2–17.0 s vs 0.8–1.4 s).
Flush-writer CPU is unchanged (69.7 vs 70.1 s), so that is time blocked, not
work removed.
> Implement trickle_fsync with sync_file_range
> --------------------------------------------
>
> Key: CASSANDRA-21679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21679
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Local/SSTable
> Reporter: Sam Lightfoot
> Assignee: Sam Lightfoot
> Priority: Normal
>
> h4. Summary
> `trickle_fsync` paces SSTable writeback with `fdatasync`, which also commits
> the ext4 journal, flushes the device cache on drives that have one, walks the
> whole file's dirty set, and most notably blocks the flush writer with no
> overlap between it and the device. None of that is required, durability is
> the commitlog's job and `SequentialWriter` fsyncs at `close()` regardless.
> `sync_file_range` does only the pacing. Aiming its wait one interval behind
> the write overlaps producing with draining, so the writer blocks only when
> the device is genuinely behind: 93% of flush-writer stall goes away (15.3 s →
> 1.0 s) with throughput, burst shaping and the bound on outstanding writeback
> unchanged. This changes the implementation beneath `trickle_fsync: true`; it
> adds no option.
> h4. What changes
> At each interval, on Linux, two calls:
> {code:java}
> sync_file_range(fd, off_N, len, SYNC_FILE_RANGE_WRITE); // submit N, returns
> sync_file_range(fd, off_N-1, len, SYNC_FILE_RANGE_WAIT_BEFORE); // wait for
> N-1{code}
> Lagging the wait by one interval is the point. `fdatasync` waits on data it
> submitted in the same call, so producing and draining never overlap and every
> interval costs both. Waiting one behind offsets the wait by exactly one
> interval's generation time, N−1 has been in flight since the previous call,
> so an interval costs the larger of the two and the stall is `max(0, drain −
> produce)`.
>
> That is zero whenever the device drains an interval faster than the writer
> generates one, which is the case here at 24–33% device utilisation: 15.3 s →
> 1.0 s below, 93% of flush-writer stall removed, throughput and burst shaping
> unchanged. Where the device is the limit instead, the writer is paced by it,
> the backpressure `trickle_fsync` exists to provide. The writer runs at most
> one interval ahead either way, so the bound holds.
> `sync_file_range` writes no metadata, so the close sync becomes the writer's
> only metadata commit.
> h4. Measured
> Single node, 10 MiB interval, ~411 MiB memtable flushes. Paired A/B, both
> arms in the same invocation, n=5, arms matched to 0.0001% on read count.
> | |trickle off|`fdatasync`|this change|
> |flush-writer stall|0.4 s|15.3 s|1.0 s|
> |ext4 journal transactions|3493|6574 (+3081)|3497 |
> |read p99 during flush|24.3 ms|11.4 ms|9.0 ms|
> |system flush throughput|116.4 MiB/s|116.4|116.5|
> |dirty-window peak|439 MiB|50.8|48.6|
> Stall drops 14.8× with no overlap between arms (14.2–17.0 s vs 0.8–1.4 s).
> Flush-writer CPU is unchanged (69.7 vs 70.1 s), so that is time blocked, not
> work removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
