[
https://issues.apache.org/jira/browse/HDDS-16953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16953:
-------------------------------------
Assignee: Andrey Yarovoy
> Remove the per-call InfoBucket from rooted-ofs create by resolving file
> system replication in OM
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16953
> URL: https://issues.apache.org/jira/browse/HDDS-16953
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> `BasicRootedOzoneClientAdapterImpl.createFile` and `createStreamFile` (behind
> `FileSystem.create` / `createNonRecursive`) call `getBucket(ofsPath,
> recursive)` and then `OzoneBucket.createFile`. So every
> `create` sends at least 2 OM requests: an `InfoBucket` followed by a
> `CreateFile`, and a link bucket adds an `InfoVolume` and an `InfoBucket` for
> each hop. On write-heavy workloads such as ingest, ETL output and Hive
> inserts, `create` is the hottest write metadata operation.
> The fetch does three jobs:
> 1. **Layout check.** `OzoneFSUtils.validateBucketLayout` rejects
> OBJECT_STORE. `OMFileCreateRequest` is registered for
> `BucketLayout.OBJECT_STORE`, so OM does not reject it.
> 2. **Replication.** The adapter passes `bucket.getReplicationConfig()` to
> `OzoneClientUtils.resolveClientSideReplicationConfig`. **If the bucket
> default is EC, the bucket wins**, because "there is no way for file system
> interfaces to pass EC replication". Otherwise the FS-supplied factor wins, as
> RATIS or as the client-configured type adjusted to that factor.
> 3. **Auto-create.** With `recursive=true` (always set by `CopyCommands`), a
> missing volume or bucket is created.
> ### Proposal: opt-in fields that let OM apply the file system rule
> {code}
> message CreateFileRequest {
> required KeyArgs keyArgs = 1;
> required bool isRecursive = 2;
> required bool isOverwrite = 3;
> optional uint64 clientID = 4;
> // Set by file system clients: OM rejects the request with
> // NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
> optional bool requireFileSystemLayout = 5;
> // Set by file system clients, which cannot express EC replication: if the
> // bucket default replication is EC, OM uses it instead of the replication
> // in keyArgs.
> optional bool preferBucketEcReplication = 6;
> }
> {code}
> The second field reproduces the client rule exactly. The non-EC branch of
> `resolveClientSideReplicationConfig` does not depend on the bucket. So the
> client can compute it with
> `bucketReplConfig = null` and send the result. Then:
> * if the field is set and the bucket default is EC, OM uses the bucket
> default;
> * otherwise OM uses the client value. If the client sent nothing, OM falls
> back to the bucket default,
> then the server default, through the existing logic.
> That is the same decision the client makes today, with the inputs on the
> other side of the wire.
> *OM.* `OMFileCreateRequest.preExecute` already reads `bucketInfo`
> (`ozoneManager.getBucketInfo`) to feed
> `resolveReplicationConfigPreference`, before it calls `allocateBlock`. Both
> checks go there, on the
> leader, before Ratis, and **before `allocateBlock`**. Otherwise a rejected
> `create` would still allocate
> SCM blocks. Things to settle in review:
> * whether `getBucketInfo` returns the *source* bucket's default replication
> for a link bucket (UNVERIFIED);
> * whether the layout check can use the factory's already-resolved layout
> instead of re-deriving it.
> This adds no extra lookup either way. Do not put either check in
> `BucketLayoutAwareOMKeyRequestFactory.createRequest`, because that also runs
> at apply time on every OM.
> *Client.*
> * When `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant, the
> adapter sends `CreateFile` with
> both fields set and does not call `getBucket()` first. Place the constant
> after
> `LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is
> assigned at merge. Against any
> older OM, including ZDU, keep today's path.
> * `RpcClient.createFile` builds the output stream from the replication
> returned by the server. Check
> that nothing between `createFile` and the stream relies on the client-side
> value. `RpcClient` rejects
> EC client-side for OM < 2; that is irrelevant here, because such an OM takes
> the fallback.
> * Auto-create on the error path, as in the mkdirs Jira. On `VOLUME_NOT_FOUND`
> / `BUCKET_NOT_FOUND` with
> `recursive=true`, call `getBucket(ofsPath, true)` and retry once, using
> today's replication path
> because the bucket is now in hand. With `recursive=false`, propagate as today.
> * Map `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`.
> `FILE_ALREADY_EXISTS` / `NOT_A_FILE` keep
> mapping to `FileAlreadyExistsException`.
> ### Other callers
> `CreateFile` has no S3 Gateway caller (S3 uses `CreateKey`). `freon
> OmMetadataGenerator` calls it on any
> layout. Neither sets the new fields, so their behavior is unchanged.
> ### Compatibility
> | | Behavior |
> |---|---|
> | New client, new OM, bucket exists | 1 RPC; OBJECT_STORE rejected; EC bucket
> ⇒ EC, as today |
> | New client, new OM, bucket missing (recursive) | failed `CreateFile` →
> `getBucket(…, true)` → `CreateFile` |
> | New client, older OM (incl. ZDU) | as today |
> | Old client / freon / direct `createFile`, new OM | fields absent ⇒
> unchanged |
> There are two new optional proto fields, no new result code and no new
> configuration.
> ## Testing
> * **Replication decision table.** In both an OM request test and the adapter,
> cover these inputs:
> * bucket default: EC, RATIS, or none;
> * client-configured `ozone.replication`: unset, RATIS, or EC;
> * FS factor: 1, 3, or unsupported;
> * the field: set or unset.
> For every combination, assert that the new path's effective replication
> equals the old client-side
> path's. Keep a copy of `resolveClientSideReplicationConfig` in the test as
> the oracle.
> * Adapter (Mockito):
> * Existing bucket: exactly one `CreateFile` with both fields set, and no
> `getBucket()`.
> * Missing bucket with recursive: one failed call, then `getBucket(…, true)`,
> then a retry.
> * Missing bucket without recursive: the error propagates.
> * Older OMs (including `ZDU`) fall back.
> * OM request: with the field set, OBJECT_STORE is rejected **without**
> `allocateBlock` being called;
> FSO and **LEGACY** succeed. Without the field, OBJECT_STORE still succeeds. A
> link to an EC bucket
> resolves to EC.
> * `AbstractRootedOzoneFileSystemTest`:
> * create on an EC bucket writes EC;
> * create into a missing bucket auto-creates it;
> * create on an OBJECT_STORE bucket fails with `IllegalArgumentException`.
> Count RPCs in the adapter test, not with OMMetrics deltas (HDDS-16513).
> * Benchmark `TestOfsCreateRpcBenchmark`, same shape as
> `TestOfsListStatusRpcBenchmark`:
> * 200 FSO buckets; create + close of empty files, single-threaded and 10
> threads;
> * n=3 per arm, built and run strictly one after the other;
> * invariant: one fewer OM RPC per `create` (CreateFile + CommitKey; no
> InfoBucket).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
