yandrey321 opened a new pull request, #11501:
URL: https://github.com/apache/ozone/pull/11501
## What changes were proposed in this pull request?
On the healthy EC read path, `ECBlockInputStream` reads one cell at a time,
so a read that spans `N` cells costs
`N` serial datanode round trips, even though the cells are on different
datanodes. With this change, a read that
spans more than one cell reads the cells of different data indexes in
parallel. The reconstruction path is already
parallel (HDDS-5952), and HDFS does the same in `StripeReader.readStripe`.
Key decisions:
- **Off by default.** The new flag `ozone.client.ec.parallel.read.enabled`
defaults to `false`, and with it off,
behavior is unchanged. Reads within a single cell always stay on the
existing serial path.
- **Dedicated pool.** Parallel reads use a new pool
(`ozone.client.ec.parallel.read.pool.limit`, default 30), not
the reconstruct pool, so healthy and degraded reads do not compete for
threads. The pool is created lazily.
When it is saturated, the calling thread reads the remaining cells itself.
- **One task per data index.** That index's cells from consecutive stripes
are contiguous in its stream, so the
fan-out is at most the number of data units.
- **Cannot hang.** The caller runs the first task itself and then any task
the pool has not started. It waits for
every task before returning, so nothing writes into the caller's buffer
after `read` returns.
- **Failure semantics.** A failure throws, as the serial path does, instead
of returning the good prefix:
`ECBlockInputStreamProxy` treats each inner read as all-or-nothing. Every
failed index of the call is reported
in one `BadDataLocationException`, so spare-location retry and the
failover to reconstruction each happen once.
- **Compatible API.** Existing constructors and factory methods are
unchanged and read serially; overloads take
the pool.
Out of scope: reading ahead to the end of the stripe (HDFS `curStripeBuf`).
Byte-array reads on encrypted (TDE)
keys do not benefit, because `CryptoInputStream` reads through an 8 KB
buffer, which never spans more than one
cell.
Generated-by: Claude Code (Claude Opus 5.5)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16867
## How was this patch tested?
- **Both modes in every run:** the existing EC read tests now run with cells
read one at a time and in parallel,
with no extra configuration. `TestECBlockInputStream` and
`TestOzoneECClient` use `@ParameterizedClass`, the
two EC tests in `TestOzoneFSInputStream` are parameterized, and
`TestKeyInputStream` reads through both kinds of
client. All pass.
- **New unit tests:** they check byte-exact data for rs-3-2, rs-6-3 and
rs-10-4. A latch-based test proves reads
are in flight on all indexes at once, and it fails with `Timed out waiting
for 2 concurrent reads` when the
executor hand-off is removed. Other tests cover multi-index failure
reporting and spare-location retry.
- **Regression:** `TestOzoneIntegrationNonHA` (340 run, 0 failures),
`TestECKeyOutputStream`, `TestKeyInputStream`.
**Benchmark** (`TestEcParallelReadBenchmark`, tagged `benchmark` so it does
not run in CI): mini cluster, 48 MB
keys, 1 MB cells, 7 rounds after warm-up, run twice. Speed-up of parallel
over serial throughput:
| Cluster | Policy | 1 reader, buffer of 1 stripe or more | 4 clients x 1
thread each | 1 MB buffer (one cell, control) |
|---|---|---|---|---|
| 5 DN | rs-3-2 | 1.75-1.87x | 1.25-1.32x | 0.99-1.02x |
| 9 DN | rs-3-2 | 1.71-1.83x | 1.16-1.23x | 0.96-1.06x |
| 9 DN | rs-6-3 | 2.17-2.30x | 1.19-1.22x | 0.99-1.03x |
Observations:
- The gain grows with the number of data units (rs-6-3 over rs-3-2). The
one-cell control stays within noise
(about +-8%).
- With 4 clients, median latency drops 17-24% both per file and per `read()`
call. p99 mostly drops too, but 7
rounds are too few to support tail claims.
- These numbers are a lower bound. The datanodes are in-JVM and on loopback,
so the network round trips the change
removes are nearly free here.
- The 4-client gain is capped by one machine's CPUs. That is also why serial
throughput is lower on 9 datanodes
than on 5, so the 5 vs 9 difference reflects CPU load, not how cells are
spread over datanodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
