Siyao Meng created HDDS-16793:
---------------------------------
Summary: Quota repair adds deletedTable entries without the
committed flag to the pending deletion usage of the bucket, and the purge never
releases them
Key: HDDS-16793
URL: https://issues.apache.org/jira/browse/HDDS-16793
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-2-count-only-committed-deleted-keys-in-quota-repair.patch,
TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java
h3. Mechanism
{{QuotaRepairTask.countDeletedKey}} (HDDS-15204) adds the replicated size of
every key info in the {{deletedTable}} entries of a bucket to
{{snapshotUsedBytes}}, and one unit per key info to {{snapshotUsedNamespace}}.
The requests that maintain these counters follow a narrower rule (HDDS-13756).
Only a key that leaves the key table after commit is added to them, and it is
marked with {{OmKeyInfo.withCommittedKeyDeletedFlag(true)}}: by
{{AbstractOMKeyDeleteResponse}} for DeleteKey and DeleteKeys, by
{{OMKeyCommitRequest}} for the key it overwrites, and by
{{OMDirectoriesPurgeResponseWithFSO}} for sub files. When the entry is purged,
{{KeyDeletingService}} and {{OMKeyPurgeRequest}} release only key infos that
have the flag.
Every other key info in {{deletedTable}} has no flag and was never added to the
counters: the key that {{S3MultipartUploadCompleteRequest}} overwrites, the
parts of an aborted multipart upload, an overwritten part and the unused parts
of a completed upload, expired open keys ({{OMOpenKeysDeleteRequest}}), the
uncommitted blocks of a key commit, and entries written before HDDS-13756. A
repair that runs while such an entry is waiting adds an amount to the counters
that no purge removes.
By history, not run on older code: the recount of {{deletedTable}} was added by
HDDS-15204 and is in no release tag up to 2.2.1, so only unreleased master is
affected.
h3. Trigger
# A bucket has a {{deletedTable}} entry without the committed flag. Run with a
key that is overwritten by a completed multipart upload. The other sources
listed above are from code reading, not run.
# Quota repair runs for the bucket before {{KeyDeletingService}} purges the
entry. An admin starts it with {{ozone repair om quota start}}
({{OzoneManager.startQuotaRepair}}). The service interval is 60s by default.
The entry of the overwritten key stays for as long as a snapshot references
that key (run separately on the same harness, not part of the attached test:
still present after 30 service runs with a snapshot taken before the overwrite).
# The entry is purged.
h3. Impact
* With {{deletedTable}} empty again, {{snapshotUsedBytes}} and
{{snapshotUsedNamespace}} of the bucket keep the size and one unit for each key
info that had no flag when the repair ran. Both are part of the usage that the
quota checks compare. Run: a bucket with a namespace quota of 2 that holds one
key rejects a second key with {{QUOTA_EXCEEDED}}. The space quota check reads
{{snapshotUsedBytes}} the same way (argued, not run).
* No data is affected. The counters stay wrong until quota repair runs again at
a time when no such entry is waiting (argued, not run).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java]
({{hadoop-ozone/ozone-manager}}) drives one real OM with its
{{KeyDeletingService}} through {{OzoneManagerProtocol}}, with the testing SCM
block client of the OM tests and no datanodes. Repair is started with {{new
QuotaRepairTask(om).repair(buckets)}}, the body of
{{OzoneManager.startQuotaRepair}}. The service interval is 100 ms and the
service is held with its own suspend and resume hook so that the repair runs
while the entry is waiting.
{{multipartOverwriteThenRepairLeavesPendingCountersStuck}} shows
{{snapshotUsedBytes=805306368}} and {{snapshotUsedNamespace=1}} after the
repair and unchanged after the purge, with an empty {{deletedTable}}, and the
rejected second key. {{keyOverwriteThenRepairReturnsToZero}} is the control
with a plain key overwrite, where the counters return to zero and the second
key is accepted. A passing first test means the defect is present.
h3. Patch
[^CR-2-count-only-committed-deleted-keys-in-quota-repair.patch], against
a6332fb59879b80615b9233e953d6813d03ae7a8. It also applies to master at
3a22acfaa21.
{{countDeletedKey}} counts a key info only if {{isDeletedKeyCommitted()}} is
set, which is the rule the purge uses to release it. The size formula is
unchanged (replicated size per key info, HDDS-16176 covers that formula). The
function is shared by the scan of the active {{deletedTable}} and the scans of
the snapshot {{deletedTable}}s, so both follow the rule. The patch does not
change which entries carry the flag.
Covered by {{testQuotaRepairCountsOnlyCommittedDeletedKeys}} in the existing
{{TestQuotaRepairTask}}: an entry with the flag, an entry without it and an
entry that holds one key info of each kind (as a key commit writes when it
overwrites a key and drops uncommitted blocks) must count as two keys. Without
the change it fails with "expected: 2000L but was: 5000L". The existing
fixtures ({{putDeletedKey}} in the test and {{OMRequestTestUtils.deleteKey}})
now write the flag, as a real delete of a committed key does. With the change
{{TestQuotaRepairTask}}, {{TestOMQuotaRepairRequest}},
{{TestOMKeyPurgeRequestAndResponse}},
{{TestOMDirectoriesPurgeRequestAndResponse}} and {{TestKeyDeletingService}} (55
tests) and {{TestOzoneRepairShell}} on a mini cluster (3 tests) pass,
checkstyle is clean. With the patch the first test of the reproduction fails at
the counters after the repair (0 instead of 805306368), the control still
passes.
Found by code review of the OM key deletion service and key purge paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
a6332fb59879b80615b9233e953d6813d03ae7a8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
