Andrey Yarovoy created HDDS-16953:
-------------------------------------
Summary: Remove the per-call InfoBucket from rooted-ofs create by
resolving file system replication in OM
Key: HDDS-16953
URL: https://issues.apache.org/jira/browse/HDDS-16953
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
`BasicRootedOzoneClientAdapterImpl.createFile` and `createStreamFile` (behind
`FileSystem.create` / `createNonRecursive`) call `getBucket(ofsPath,
recursive)` and then `OzoneBucket.createFile`. So every
`create` sends at least 2 OM requests: an `InfoBucket` followed by a
`CreateFile`, and a link bucket adds an `InfoVolume` and an `InfoBucket` for
each hop. On write-heavy workloads such as ingest, ETL output and Hive inserts,
`create` is the hottest write metadata operation.
The fetch does three jobs:
1. **Layout check.** `OzoneFSUtils.validateBucketLayout` rejects OBJECT_STORE.
`OMFileCreateRequest` is registered for `BucketLayout.OBJECT_STORE`, so OM does
not reject it.
2. **Replication.** The adapter passes `bucket.getReplicationConfig()` to
`OzoneClientUtils.resolveClientSideReplicationConfig`. **If the bucket default
is EC, the bucket wins**, because "there is no way for file system interfaces
to pass EC replication". Otherwise the FS-supplied factor wins, as RATIS or as
the client-configured type adjusted to that factor.
3. **Auto-create.** With `recursive=true` (always set by `CopyCommands`), a
missing volume or bucket is created.
### Proposal: opt-in fields that let OM apply the file system rule
{code}
message CreateFileRequest {
required KeyArgs keyArgs = 1;
required bool isRecursive = 2;
required bool isOverwrite = 3;
optional uint64 clientID = 4;
// Set by file system clients: OM rejects the request with
// NOT_SUPPORTED_OPERATION if the resolved bucket layout is OBJECT_STORE.
optional bool requireFileSystemLayout = 5;
// Set by file system clients, which cannot express EC replication: if the
// bucket default replication is EC, OM uses it instead of the replication
// in keyArgs.
optional bool preferBucketEcReplication = 6;
}
{code}
The second field reproduces the client rule exactly. The non-EC branch of
`resolveClientSideReplicationConfig` does not depend on the bucket. So the
client can compute it with
`bucketReplConfig = null` and send the result. Then:
* if the field is set and the bucket default is EC, OM uses the bucket default;
* otherwise OM uses the client value. If the client sent nothing, OM falls back
to the bucket default,
then the server default, through the existing logic.
That is the same decision the client makes today, with the inputs on the other
side of the wire.
*OM.* `OMFileCreateRequest.preExecute` already reads `bucketInfo`
(`ozoneManager.getBucketInfo`) to feed
`resolveReplicationConfigPreference`, before it calls `allocateBlock`. Both
checks go there, on the
leader, before Ratis, and **before `allocateBlock`**. Otherwise a rejected
`create` would still allocate
SCM blocks. Things to settle in review:
* whether `getBucketInfo` returns the *source* bucket's default replication for
a link bucket (UNVERIFIED);
* whether the layout check can use the factory's already-resolved layout
instead of re-deriving it.
This adds no extra lookup either way. Do not put either check in
`BucketLayoutAwareOMKeyRequestFactory.createRequest`, because that also runs at
apply time on every OM.
*Client.*
* When `proxy.getOmVersion() >=` a new `OzoneManagerVersion` constant, the
adapter sends `CreateFile` with
both fields set and does not call `getBucket()` first. Place the constant after
`LIST_STATUS_REJECTS_OBS(101)` and never below `ZDU(100)`; the number is
assigned at merge. Against any
older OM, including ZDU, keep today's path.
* `RpcClient.createFile` builds the output stream from the replication returned
by the server. Check
that nothing between `createFile` and the stream relies on the client-side
value. `RpcClient` rejects
EC client-side for OM < 2; that is irrelevant here, because such an OM takes
the fallback.
* Auto-create on the error path, as in the mkdirs Jira. On `VOLUME_NOT_FOUND` /
`BUCKET_NOT_FOUND` with
`recursive=true`, call `getBucket(ofsPath, true)` and retry once, using today's
replication path
because the bucket is now in hand. With `recursive=false`, propagate as today.
* Map `NOT_SUPPORTED_OPERATION` to `IllegalArgumentException`.
`FILE_ALREADY_EXISTS` / `NOT_A_FILE` keep
mapping to `FileAlreadyExistsException`.
### Other callers
`CreateFile` has no S3 Gateway caller (S3 uses `CreateKey`). `freon
OmMetadataGenerator` calls it on any
layout. Neither sets the new fields, so their behavior is unchanged.
### Compatibility
| | Behavior |
|---|---|
| New client, new OM, bucket exists | 1 RPC; OBJECT_STORE rejected; EC bucket ⇒
EC, as today |
| New client, new OM, bucket missing (recursive) | failed `CreateFile` →
`getBucket(…, true)` → `CreateFile` |
| New client, older OM (incl. ZDU) | as today |
| Old client / freon / direct `createFile`, new OM | fields absent ⇒ unchanged |
There are two new optional proto fields, no new result code and no new
configuration.
## Testing
* **Replication decision table.** In both an OM request test and the adapter,
cover these inputs:
* bucket default: EC, RATIS, or none;
* client-configured `ozone.replication`: unset, RATIS, or EC;
* FS factor: 1, 3, or unsupported;
* the field: set or unset.
For every combination, assert that the new path's effective replication equals
the old client-side
path's. Keep a copy of `resolveClientSideReplicationConfig` in the test as the
oracle.
* Adapter (Mockito):
* Existing bucket: exactly one `CreateFile` with both fields set, and no
`getBucket()`.
* Missing bucket with recursive: one failed call, then `getBucket(…, true)`,
then a retry.
* Missing bucket without recursive: the error propagates.
* Older OMs (including `ZDU`) fall back.
* OM request: with the field set, OBJECT_STORE is rejected **without**
`allocateBlock` being called;
FSO and **LEGACY** succeed. Without the field, OBJECT_STORE still succeeds. A
link to an EC bucket
resolves to EC.
* `AbstractRootedOzoneFileSystemTest`:
* create on an EC bucket writes EC;
* create into a missing bucket auto-creates it;
* create on an OBJECT_STORE bucket fails with `IllegalArgumentException`.
Count RPCs in the adapter test, not with OMMetrics deltas (HDDS-16513).
* Benchmark `TestOfsCreateRpcBenchmark`, same shape as
`TestOfsListStatusRpcBenchmark`:
* 200 FSO buckets; create + close of empty files, single-threaded and 10
threads;
* n=3 per arm, built and run strictly one after the other;
* invariant: one fewer OM RPC per `create` (CreateFile + CommitKey; no
InfoBucket).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
