Siyao Meng created HDDS-16775:
---------------------------------
Summary: Renaming an ancestor of an hsync'ed file breaks its lease
expiry auto commit
Key: HDDS-16775
URL: https://issues.apache.org/jira/browse/HDDS-16775
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-commit-expired-hsync-key-at-current-path-after-parent-rename.patch
h3. Mechanism
{{OMKeyRenameRequestWithFSO}} rejects renaming an hsync'ed file (HDDS-8545) but
not a directory above one, and does not touch the open file table. The open row
keeps the create time full path in {{keyName}}.
{{OmMetadataManagerImpl.getExpiredOpenKeys}} copies that path into the lease
expiry auto commit request, and {{OMKeyCommitRequestWithFSO}} resolves it by
path.
h3. Trigger
With hsync enabled on an FSO bucket: create and hsync {{/v/b/src/sub/f}},
rename {{/v/b/src}} to {{/v/b/dst}}, and the writer does not close before
{{ozone.om.lease.hard.limit}}.
h3. Impact
Every {{OpenKeyCleanupService}} run submits a {{CommitKey}} for {{src/sub/f}}
that fails with {{DIRECTORY_NOT_FOUND}}, and the service takes no action on the
failed response (the failure is logged at ERROR by the commit request on every
run). The entry is returned by every scan, so it also uses up part of each
run's {{ozone.om.open.key.cleanup.limit.per.task}} budget. The file keeps its
hsync client ID and its open row, so it stays an open file after lease expiry
and renaming it is rejected with {{RENAME_OPEN_FILE}}. No data is lost. A
manual {{recoverLease}} on the new path clears it.
h3. Reproduction
PASS, with a test added to the existing hsync mini cluster suite
({{TestHSync}}): the three steps above, then run the cleanup service and assert
the file is still open. It also reproduces at OM level through the real write
path without a mini cluster.
h3. Suggested fix
Make the auto commit independent of the stored path: rebuild the current path
from the parent ID in {{getExpiredOpenKeys}}, or locate the file by parent
object ID in the commit. Rejecting the rename of a directory that has hsync'ed
open descendants is not a good alternative: it needs a subtree check on every
directory rename, has no precedent (HDDS-10650 lets a recursive delete through
and flags the descendants' open keys), and HBase WAL splitting renames a
directory whose WALs are still open.
h3. Patch
[^CR-4-commit-expired-hsync-key-at-current-path-after-parent-rename.patch],
against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
In {{getExpiredOpenKeys}}, for an FSO hsync'ed key that is about to be auto
committed, the stored path is first checked with {{OMFileRequest.getParentID}};
if it still leads to the open key's parent ID it is used as before. Only when
it does not, the parent's current path is rebuilt with the existing
{{FSODirectoryPathResolver}} and the commit request is built with that path.
The commit request, the proto and the apply path are unchanged. Directory
renames stay allowed, and the file is now closed at its current path once the
lease hard limit passes.
Cost: one directory table lookup per path component for each expired hsync'ed
key, and for a key whose stored path is stale a breadth first walk of the
bucket's directory tree per run. Not addressed: the writer's own hsync or close
after a parent rename still fails with {{DIRECTORY_NOT_FOUND}}, because the
client sends the path it opened.
Covered by {{testCommitExpiredHsyncKeyAfterParentRename}} in the existing
{{TestOpenKeyCleanupService}}, which fails without the change because the scan
asks to commit {{src/sub/file}} where {{dst/sub/file}} is expected.
{{TestOpenKeyCleanupService}}, {{TestOmMetadataManager}} and
{{TestFSODirectoryPathResolver}} (46 tests) and {{TestHSync}} (42 tests) pass,
checkstyle is clean.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
