Siyao Meng created HDDS-16774:
---------------------------------
Summary: CreateDirectory and CreateFile accept '.' and '..' path
components on FSO buckets
Key: HDDS-16774
URL: https://issues.apache.org/jira/browse/HDDS-16774
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-10-normalize-key-name-in-create-directory-and-create-file.patch,
TestBugCR10DotComponents.java
h3. Mechanism
{{OMDirectoryCreateRequest.preExecute}} and {{OMFileCreateRequest.preExecute}}
neither normalize nor validate the key name, so the FSO path walk
({{Paths.get(keyName)}} in {{verifyDirectoryKeysInPath}}) treats {{.}} and
{{..}} as ordinary components and the request creates them as
{{directoryTable}} rows. Delete, rename, commit and the filesystem client all
normalize them away, so most single key operations cannot address the entries
afterwards. {{OMKeyCreateRequest}} does call {{validateAndNormalizeKey}}.
h3. Trigger
A client with CREATE permission calls {{OzoneBucket.createDirectory("a/..")}},
{{createDirectory(".")}} or {{createFile("d/./f")}} on an FSO bucket. This
needs the Java object client or another raw OM client; Hadoop filesystem
clients normalize paths before sending. The S3 gateway also calls
{{createDirectory}} for directory style PUTs; whether S3 clients can trigger it
was not checked.
h3. Impact
* {{BasicOzoneFileSystem.listStatus}} ({{o3fs}}) on the parent never returns
(about 190,000 {{ListStatus}} RPCs in 30 s) or throws {{InvalidPathException}}.
{{ofs}} has the same loop shape but was not run.
* Single key delete and rename cannot address the entry (except a {{..}}
directly under the bucket root, which {{deleteKey("..")}} reaches). The parent
cannot be deleted non recursively, and a bucket holding a root {{.}} cannot be
deleted.
* {{createFile}} fails at commit but leaves the dot directories and an open
file row behind.
Batch {{deleteKeys}} (which does not normalize) or a recursive delete of the
parent removes the entries. Nothing removes them automatically.
h3. Reproduction
PASS, deterministic: [^TestBugCR10DotComponents.java]
({{hadoop-ozone/integration-test}}), public client API on a mini cluster.
h3. Suggested fix
Normalize the key name in both {{preExecute}} methods and write it back into
{{KeyArgs}}. CreateFile can use {{validateAndNormalizeKey}} as
{{OMKeyCreateRequest}} does. CreateDirectory should resolve {{.}} and {{..}}
and reject a path that leaves the bucket, without the character check
(HDDS-12911 deliberately keeps {{:}} creatable there). The snapshot reserved
name check has to run on the normalized name. Separately, as a follow up, make
the {{o3fs}} and {{ofs}} {{listStatus}} loops end when a page does not advance
{{startKey}}.
h3. Patch
[^CR-10-normalize-key-name-in-create-directory-and-create-file.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
{{OMDirectoryCreateRequest.preExecute}} normalizes the key name with
{{OmUtils.normalizeKey}} when {{BucketLayout.shouldNormalizePaths}} says so
(always for FSO, for LEGACY only with {{ozone.om.enable.filesystem.paths}},
never for OBS), rejects a result that still contains a {{..}} component with
{{INVALID_KEY_NAME}}, and runs the snapshot reserved name check on the result.
{{OMFileCreateRequest.preExecute}} calls the existing
{{validateAndNormalizeKey}}. Both write the result into the {{KeyArgs}} that go
into the Raft log, so followers apply the leader's name and no proto change is
needed. A name that normalizes to the empty string follows the existing empty
key handling.
Behaviour changes to be aware of: on a LEGACY bucket with
{{ozone.om.enable.filesystem.paths=true}} the two requests now normalize as
{{OMKeyCreateRequest}} already does, and an FSO CreateFile with an invalid name
is rejected at create instead of at commit. The client side {{listStatus}}
guard is not part of this patch. Existing {{.}} and {{..}} rows are not cleaned
up.
Covered by new cases in the existing {{TestOMDirectoryCreateRequestWithFSO}},
{{TestOMFileCreateRequestWithFSO}} and {{TestOMDirectoryCreateRequest}}; 17 of
them fail without the change (for example "expected: \"a/b\" but was:
\"a/./b\"" and "expected: CANNOT_CREATE_DIRECTORY_AT_ROOT but was: OK"). The
file, key create and key commit request unit suites (307 tests) and
{{TestO3FSWithFSO}}, {{TestOFSWithFSO}}, {{TestO3FSWithFSPaths}} (202 tests)
pass, checkstyle is clean.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
