Andrey Yarovoy created HDDS-16948:
-------------------------------------
Summary: Remove the per-call InfoBucket from rooted-ofs open
Key: HDDS-16948
URL: https://issues.apache.org/jira/browse/HDDS-16948
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
`BasicRootedOzoneClientAdapterImpl.readFile` (behind `FileSystem.open`) calls
`getBucket()`, which is an
`InfoBucket` RPC. For a link bucket it adds an `InfoVolume` and an `InfoBucket`
per hop. It then calls
`OzoneBucket.readFile`. Every `open` therefore costs at least 2 OM requests. On
read-heavy file system
workloads (Hive, Impala, Spark scans) `open` is one of the most frequent
metadata operations, so this is
the largest single `InfoBucket` source left after HDDS-15925 and HDDS-16760.
The fetch does one job: `OzoneFSUtils.validateBucketLayout`, which rejects
OBJECT_STORE. `readFile`
reads nothing else from the bucket.
### Why HDDS-15951's lookupFile check does not cover this
`RpcClient.readFile` uses `lookupFile` only for an OM older than
`OPTIMIZED_GET_KEY_INFO` (3). For every
current OM it sends **`GetKeyInfo`**, then checks `isFile()` on the client. Its
read-retry path
(`RpcClient.getKeyInfo`) also sends `GetKeyInfo`. So the OBJECT_STORE rejection
that HDDS-15951 added to
`lookupFile` is never reached from `open`.
### Why the rejection is opt-in
`GetKeyInfo` is the object store read RPC. It sits behind
`OzoneBucket.readKey`, `getKeyDetails`,
`headObject` and the S3 key lookups (`assumeS3Context=true`), which the S3
Gateway uses for GetObject and
HeadObject. `cli-shell`, `freon` and `cli-debug` call it as well. An
unconditional OBJECT_STORE rejection
would break all of them. As in HDDS-16760, the client asks for the check:
{code}
message GetKeyInfoRequest {
required KeyArgs keyArgs = 1;
optional bool assumeS3Context = 2;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 3;
}
{code}
### Proposed change
*OM.*
* `OzoneManagerRequestHandler.getKeyInfo` copies the field into `OmKeyArgs`.
The `requireFileSystemLayout`
field added there by HDDS-16760 is reused, and the builder already survives
`ResolvedBucket.update`.
* `OmMetadataReader.getKeyInfo` already holds the link-resolved
`ResolvedBucket`. When the field is set,
it calls the existing `validateFileSystemLayout(bucket)` before the ACL check.
This is the same
placement as `listStatus` in HDDS-16760, so it adds no lookup.
*Client.*
* `RpcClient.readFile` sets the field on the `OmKeyArgs` it builds for
`GetKeyInfo`. The read-retry
`getKeyInfo` call also sets it, so a retried read is checked too.
* `OzoneManagerProtocolClientSideTranslatorPB.getKeyInfo` builds `KeyArgs`
through
`OmKeyArgs.toProtobuf()` but sets the request-level field explicitly. Do not
set it unconditionally:
`readKey` / `getKeyDetails` share this translator method.
* The adapter needs a way to ask for it, mirroring HDDS-16760's
`ListStatusLightOptions`. Options are a
`readFile` overload or an options object on `ClientProtocol`, or
`OzoneBucket.readFile` setting it
itself. `OzoneBucket.readFile` is a file-oriented API, so it could always set
it; decide on review.
* The adapter skips `getBucket()` when `proxy.getOmVersion() >=` a new
`OzoneManagerVersion` constant.
Place it after `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the
number is assigned at
merge. It maps `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`. Against
any older OM, including
one that reports ZDU, it keeps the current path.
* The `lookupFile` branch for OMs older than version 3 is unchanged, and such
OMs take the fallback anyway.
### Rejected alternative
`GetKeyInfoResponse` already returns `bucketLayout` (HDDS-16362), so the client
could check that instead
and need no new request field. Two problems:
* It only works when the key exists. A missing key on an OBJECT_STORE bucket
would then report
`FileNotFoundException` instead of today's `IllegalArgumentException`.
* The field was added without a version gate, so an older OM leaves it unset,
and the client cannot
tell "not OBJECT_STORE" from "not reported".
### Compatibility
| | Behavior |
|---|---|
| New client, new OM | 1 RPC (`GetKeyInfo`); OM rejects OBJECT_STORE |
| New client, older OM (incl. ZDU) | `getBucket()` + `GetKeyInfo`, as today |
| Old client / S3 Gateway / `readKey` / `getKeyDetails`, new OM | field absent
⇒ unchanged |
There is one new optional proto field, no new result code and no new
configuration.
## Testing
* Adapter (Mockito):
*`open` makes exactly one OM call, carrying the field.
*`NOT_SUPPORTED_OPERATION` maps to `IllegalArgumentException`; `KEY_NOT_FOUND`
/ `FILE_NOT_FOUND` /
`NOT_A_FILE` map to `FileNotFoundException`, as today.
* An older OM, including `ZDU`, falls back to `getBucket()`.
* `TestOMMetadataReader`:
* With the field set, `getKeyInfo` rejects OBJECT_STORE without reaching
`KeyManager` and serves FSO
and **LEGACY**.
* Without the field, it still serves OBJECT_STORE: the S3 Gateway case, and the
guard against anyone
making the check unconditional.
* The same holds with `assumeS3Context=true`.
* `TestOzoneManagerRequestHandler`: the field is forwarded into `OmKeyArgs`.
* Existing S3 and object-store read suites pass unchanged.
* `AbstractRootedOzoneFileSystemTest`: `open` on an OBJECT_STORE bucket fails
with
`IllegalArgumentException`. Do RPC counting in the adapter test, not with exact
OMMetrics deltas
(HDDS-16513).
* Benchmark `TestOfsOpenRpcBenchmark`, same shape as
`TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets, open + close of a small file, single thread and 10 threads.
* n=3 per arm, built and run strictly one after the other.
* Invariant: exactly 1 OM RPC per `open` instead of 2. `open` alone, with no
data read, so DN reads do
not mask the OM saving.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
