[
https://issues.apache.org/jira/browse/HDDS-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16774:
------------------------------
Attachment: TestBugCR10DotComponents.java
> CreateDirectory and CreateFile accept '.' and '..' path components on FSO
> buckets
> ---------------------------------------------------------------------------------
>
> Key: HDDS-16774
> URL: https://issues.apache.org/jira/browse/HDDS-16774
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-10-normalize-key-name-in-create-directory-and-create-file.patch,
> TestBugCR10DotComponents.java
>
>
> h3. Mechanism
> {{OMDirectoryCreateRequest.preExecute}} and
> {{OMFileCreateRequest.preExecute}} neither normalize nor validate the key
> name, so the FSO path walk ({{Paths.get(keyName)}} in
> {{verifyDirectoryKeysInPath}}) treats {{.}} and {{..}} as ordinary components
> and the request creates them as {{directoryTable}} rows. Delete, rename,
> commit and the filesystem client all normalize them away, so most single key
> operations cannot address the entries afterwards. {{OMKeyCreateRequest}} does
> call {{validateAndNormalizeKey}}.
> h3. Trigger
> A client with CREATE permission calls
> {{OzoneBucket.createDirectory("a/..")}}, {{createDirectory(".")}} or
> {{createFile("d/./f")}} on an FSO bucket. This needs the Java object client
> or another raw OM client; Hadoop filesystem clients normalize paths before
> sending. The S3 gateway also calls {{createDirectory}} for directory style
> PUTs; whether S3 clients can trigger it was not checked.
> h3. Impact
> * {{BasicOzoneFileSystem.listStatus}} ({{o3fs}}) on the parent never returns
> (about 190,000 {{ListStatus}} RPCs in 30 s) or throws
> {{InvalidPathException}}. {{ofs}} has the same loop shape but was not run.
> * Single key delete and rename cannot address the entry (except a {{..}}
> directly under the bucket root, which {{deleteKey("..")}} reaches). The
> parent cannot be deleted non recursively, and a bucket holding a root {{.}}
> cannot be deleted.
> * {{createFile}} fails at commit but leaves the dot directories and an open
> file row behind.
> Batch {{deleteKeys}} (which does not normalize) or a recursive delete of the
> parent removes the entries. Nothing removes them automatically.
> h3. Reproduction
> PASS, deterministic: [^TestBugCR10DotComponents.java]
> ({{hadoop-ozone/integration-test}}), public client API on a mini cluster.
> h3. Suggested fix
> Normalize the key name in both {{preExecute}} methods and write it back into
> {{KeyArgs}}. CreateFile can use {{validateAndNormalizeKey}} as
> {{OMKeyCreateRequest}} does. CreateDirectory should resolve {{.}} and {{..}}
> and reject a path that leaves the bucket, without the character check
> (HDDS-12911 deliberately keeps {{:}} creatable there). The snapshot reserved
> name check has to run on the normalized name. Separately, as a follow up,
> make the {{o3fs}} and {{ofs}} {{listStatus}} loops end when a page does not
> advance {{startKey}}.
> h3. Patch
> [^CR-10-normalize-key-name-in-create-directory-and-create-file.patch],
> against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master
> at 9c6bee807c8.
> {{OMDirectoryCreateRequest.preExecute}} normalizes the key name with
> {{OmUtils.normalizeKey}} when {{BucketLayout.shouldNormalizePaths}} says so
> (always for FSO, for LEGACY only with {{ozone.om.enable.filesystem.paths}},
> never for OBS), rejects a result that still contains a {{..}} component with
> {{INVALID_KEY_NAME}}, and runs the snapshot reserved name check on the
> result. {{OMFileCreateRequest.preExecute}} calls the existing
> {{validateAndNormalizeKey}}. Both write the result into the {{KeyArgs}} that
> go into the Raft log, so followers apply the leader's name and no proto
> change is needed. A name that normalizes to the empty string follows the
> existing empty key handling.
> Behaviour changes to be aware of: on a LEGACY bucket with
> {{ozone.om.enable.filesystem.paths=true}} the two requests now normalize as
> {{OMKeyCreateRequest}} already does, and an FSO CreateFile with an invalid
> name is rejected at create instead of at commit. The client side
> {{listStatus}} guard is not part of this patch. Existing {{.}} and {{..}}
> rows are not cleaned up.
> Covered by new cases in the existing {{TestOMDirectoryCreateRequestWithFSO}},
> {{TestOMFileCreateRequestWithFSO}} and {{TestOMDirectoryCreateRequest}}; 17
> of them fail without the change (for example "expected: \"a/b\" but was:
> \"a/./b\"" and "expected: CANNOT_CREATE_DIRECTORY_AT_ROOT but was: OK"). The
> file, key create and key commit request unit suites (307 tests) and
> {{TestO3FSWithFSO}}, {{TestOFSWithFSO}}, {{TestO3FSWithFSPaths}} (202 tests)
> pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
