yandrey321 opened a new pull request, #11971:
URL: https://github.com/apache/ozone/pull/11971
## What changes were proposed in this pull request?
**Change.** `getFileStatus` on ofs and o3fs sends a head op, so OM skips the
pipeline refresh for each block of the
file. Like HDFS, the result is a plain `FileStatus` without block locations.
Most callers only read the length, type
or times; HBase calls it on every store file reopen. `getFileBlockLocations`
fetches the locations from OM when it is
given a status from `getFileStatus`. Given a path, it still makes one OM
call. A head op ignores the block locations
returned by an OM without HDDS-15678, which come without pipelines and used
to fail the client with a
`NullPointerException`.
**Key decisions.**
- **Only a status from `getFileStatus` makes `getFileBlockLocations` call
OM.** It is marked by a package-private
`FileStatus` subclass. A status from `listStatus` keeps today's
placeholder location without an OM call. Spark lists
non-HDFS file systems with `listStatus` and asks for the block locations
of each file, so a call per file would
slow down query planning.
- **The subclass keeps the encrypted and erasure coded flags,** which
Hadoop's `FileStatus` copy constructor drops,
and works with the Hadoop 2 jar.
- **No configuration to restore the old behaviour**, and no OM, protocol or
version change.
**Side effects.**
- `getFileStatus` no longer returns a `LocatedFileStatus`; code that casts
it must use `getFileBlockLocations`.
- `getFileStatus` followed by `getFileBlockLocations(status)` takes 2 OM
calls instead of 1, and returns the
locations of the file at the time of the second call, as on HDFS.
Co-authored-by: Claude Opus 5.5 <[email protected]>
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17714
## How was this patch tested?
Unit and integration tests.
- **Benchmark** (not included):
- setup: mini cluster with 5 datanodes; 8 files of 24 MB, EC rs-3-2 (4
block groups) and Ratis/THREE (12
blocks);
- workload: 4,000 `getFileStatus` calls per run, sequential and with 10
threads;
- arms: master vs this branch, alternating, 3 runs each.
Median of 3 runs, master → this branch:
| file | threads | mean latency (µs) | p99 (µs) | throughput (calls/s) |
OM handler CPU per call (µs) | SCM handler CPU per call (µs) |
|---|---|---|---|---|---|---|
| EC rs-3-2 | 1 | 273 → 127 | 377 → 204 | 3,663 → 7,887 (2.2x) | 107 → 68
| 47 → 6 |
| EC rs-3-2 | 10 | 729 → 402 | 1,084 → 768 | 13,565 → 24,677 (1.8x) | 275
→ 152 | 110 → 5 |
| Ratis/THREE | 1 | 298 → 122 | 412 → 180 | 3,358 → 8,159 (2.4x) | 115 →
67 | 45 → 1 |
| Ratis/THREE | 10 | 739 → 405 | 1,068 → 750 | 13,423 → 24,474 (1.8x) |
273 → 155 | 98 → 1 |
- Both arms make one OM call per `getFileStatus`. Throughput varies by at
most 5% between the runs of an arm.
- The SCM cost on master is the pipeline lookup of the refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
