[
https://issues.apache.org/jira/browse/HDDS-17399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-17399:
--------------------------------
Description:
h3. Problem
* {{BlockInputStream.readPositioned}} and
{{StreamBlockInputStream.readPositioned}} take the stream lock on each call.
* The sequential read of each class keeps the same lock while it waits for the
datanode.
* Thus a positioned read waits for a sequential read in progress on the same
block.
h3. Changes
After the one-time setup of a block stream, a positioned read takes no lock.
* {{BlockInputStream}}: after the volatile {{initialized}} is true, read
{{chunkOffsets}}, {{chunkStreams}} and {{length}} without the lock.
{{initialize()}} sets them first, and nothing changes them after that.
* {{StreamBlockInputStream}}: make {{needUpdateBlockLength}} volatile, and call
{{updateBlockLength()}} only while it is set. Make {{xceiverClientFactory}}
final.
* The block streams do not check that they are open. {{MultipartInputStream}},
their only caller, checks its volatile {{closed}} flag first.
* Sequential reads do not change.
h3. Tests
* For both classes, a positioned read completes while a sequential read on the
same block holds the lock. Without the change, it times out.
* After {{close()}}, a positioned read fails and does not reach the block
stream.
h3. Benchmark
{{ozone freon ocpr}} (HDDS-17433) on one host: 3 datanodes on one NVMe, warm
page cache, a 256 MiB key (one block). All positioned-read threads share one
stream with 1 sequential reader of 1 MiB reads. Each run takes 20 s. Each value
is the median of 3 runs.
{noformat}
ozone freon ocpr -v vol1 -b bucket1 -k key1 -t 8 --read-size 65536
--sequential-readers 1 --duration 20s
{noformat}
The stream rows add {{\-D ozone.client.stream.readblock.enable=true}} before
{{ocpr}}.
||Read path||Positioned reads||Before (ops/s, p50, p99)||After (ops/s, p50,
p99)||
|Chunk (default)|1 thread x 64 KiB|63, 4.0 ms, 267 ms|1,488, 0.45 ms, 1.6 ms|
|Chunk (default)|8 threads x 64 KiB|83, 41 ms, 908 ms|6,762, 0.89 ms, 2.4 ms|
|Stream ({{ozone.client.stream.readblock.enable}})|1 thread x 64 KiB|22, 9.2
ms, 459 ms|463, 1.8 ms, 5.4 ms|
|Stream ({{ozone.client.stream.readblock.enable}})|8 threads x 64 KiB|51, 44
ms, 1.17 s|1,975, 3.5 ms, 18.8 ms|
* Without the change, longer runs give lower rates. Stream, 1 thread: 24 ops/s
in 20 s, 12 ops/s in 60 s.
* With positioned reads only (1 to 32 threads), throughput does not change
beyond run-to-run noise (about ±10%).
* The sequential reader alone has the same speed with and without the change.
Next to 8 positioned-read threads on the stream path, it is about 5% slower
(median of 9 pairs). The positioned reads now run on the same datanode and do
not wait. Thus the total bytes read increase by about 11%.
Follow-up to HDDS-16983 and HDDS-16395.
was:
h3. Problem
* {{BlockInputStream.readPositioned}} and
{{StreamBlockInputStream.readPositioned}} take the stream lock on each call.
* The sequential read of each class keeps the same lock while it waits for the
datanode.
* Thus a positioned read waits for a sequential read in progress on the same
block.
h3. Changes
After the one-time setup of a block stream, a positioned read takes no lock.
* {{BlockInputStream}}: after the volatile {{initialized}} is true, read
{{chunkOffsets}}, {{chunkStreams}} and {{length}} without the lock.
{{initialize()}} sets them first, and nothing changes them after that.
* {{StreamBlockInputStream}}: make {{needUpdateBlockLength}} volatile, and call
{{updateBlockLength()}} only while it is set. Make {{xceiverClientFactory}}
final.
* The block streams do not check that they are open. {{MultipartInputStream}},
their only caller, checks its volatile {{closed}} flag first.
* Sequential reads do not change.
h3. Tests
* For both classes, a positioned read completes while a sequential read on the
same block holds the lock. Without the change, it times out.
* After {{close()}}, a positioned read fails and does not reach the block
stream.
h3. Benchmark
One host, 3 datanodes, warm page cache. One shared stream on a 256 MiB key: 1
sequential reader and N positioned-read threads. Median of 3 runs.
||Read path||Positioned reads||Before (ops/s, p50, p99)||After (ops/s, p50,
p99)||
|Chunk (default)|1 thread x 64 KiB|2, 680 ms, 1.55 s|1,811, 0.41 ms, 1.36 ms|
|Chunk (default)|8 threads x 64 KiB|9, 721 ms, 4.89 s|7,398, 0.78 ms, 2.04 ms|
|Stream ({{ozone.client.stream.readblock.enable}})|1 thread x 64 KiB|3, 309 ms,
641 ms|586, 1.3 ms, 5.6 ms|
|Stream ({{ozone.client.stream.readblock.enable}})|8 threads x 64 KiB|10, 653
ms, 3.56 s|2,060, 3.3 ms, 9.4 ms|
* With positioned reads only (1 to 32 threads), throughput does not change
beyond run-to-run noise (about ±10%).
Follow-up to HDDS-16983 and HDDS-16395.
> Make positioned reads lock-free after initialization in BlockInputStream and
> StreamBlockInputStream
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-17399
> URL: https://issues.apache.org/jira/browse/HDDS-17399
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> * {{BlockInputStream.readPositioned}} and
> {{StreamBlockInputStream.readPositioned}} take the stream lock on each call.
> * The sequential read of each class keeps the same lock while it waits for
> the datanode.
> * Thus a positioned read waits for a sequential read in progress on the same
> block.
> h3. Changes
> After the one-time setup of a block stream, a positioned read takes no lock.
> * {{BlockInputStream}}: after the volatile {{initialized}} is true, read
> {{chunkOffsets}}, {{chunkStreams}} and {{length}} without the lock.
> {{initialize()}} sets them first, and nothing changes them after that.
> * {{StreamBlockInputStream}}: make {{needUpdateBlockLength}} volatile, and
> call {{updateBlockLength()}} only while it is set. Make
> {{xceiverClientFactory}} final.
> * The block streams do not check that they are open.
> {{MultipartInputStream}}, their only caller, checks its volatile {{closed}}
> flag first.
> * Sequential reads do not change.
> h3. Tests
> * For both classes, a positioned read completes while a sequential read on
> the same block holds the lock. Without the change, it times out.
> * After {{close()}}, a positioned read fails and does not reach the block
> stream.
> h3. Benchmark
> {{ozone freon ocpr}} (HDDS-17433) on one host: 3 datanodes on one NVMe, warm
> page cache, a 256 MiB key (one block). All positioned-read threads share one
> stream with 1 sequential reader of 1 MiB reads. Each run takes 20 s. Each
> value is the median of 3 runs.
> {noformat}
> ozone freon ocpr -v vol1 -b bucket1 -k key1 -t 8 --read-size 65536
> --sequential-readers 1 --duration 20s
> {noformat}
> The stream rows add {{\-D ozone.client.stream.readblock.enable=true}} before
> {{ocpr}}.
> ||Read path||Positioned reads||Before (ops/s, p50, p99)||After (ops/s, p50,
> p99)||
> |Chunk (default)|1 thread x 64 KiB|63, 4.0 ms, 267 ms|1,488, 0.45 ms, 1.6 ms|
> |Chunk (default)|8 threads x 64 KiB|83, 41 ms, 908 ms|6,762, 0.89 ms, 2.4 ms|
> |Stream ({{ozone.client.stream.readblock.enable}})|1 thread x 64 KiB|22, 9.2
> ms, 459 ms|463, 1.8 ms, 5.4 ms|
> |Stream ({{ozone.client.stream.readblock.enable}})|8 threads x 64 KiB|51, 44
> ms, 1.17 s|1,975, 3.5 ms, 18.8 ms|
> * Without the change, longer runs give lower rates. Stream, 1 thread: 24
> ops/s in 20 s, 12 ops/s in 60 s.
> * With positioned reads only (1 to 32 threads), throughput does not change
> beyond run-to-run noise (about ±10%).
> * The sequential reader alone has the same speed with and without the change.
> Next to 8 positioned-read threads on the stream path, it is about 5% slower
> (median of 9 pairs). The positioned reads now run on the same datanode and do
> not wait. Thus the total bytes read increase by about 11%.
> Follow-up to HDDS-16983 and HDDS-16395.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
