Siyao Meng created HDDS-17049:
---------------------------------
Summary: Recon counts keys of a recreated bucket under the deleted
bucket after a delete and recreate in one OM flush or behind a rebuild
Key: HDDS-17049
URL: https://issues.apache.org/jira/browse/HDDS-17049
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-7-keep-bucket-cache-in-line-with-recreate-and-rebuild.patch,
TestBugCR7BucketCacheAfterRecreate.java
h3. Mechanism
Since HDDS-15335 (2.2.0) the OBS and Legacy NSSummary sub-tasks cache each
bucket's {{OmBucketInfo}} ({{NSSummaryTaskDbEventHandler.lookupBucketCached}})
and drop the entry only on a bucketTable DELETE event, on the assumption that a
recreate is always preceded by a delete event. Two paths break that assumption:
* The OM double buffer writes a flush through one {{RDBBatchOperation}}, which
keeps only the last write per key. A DeleteBucket and a CreateBucket of the
same name in one flush reach Recon as a single bucketTable UPDATE, so the cache
keeps the deleted bucket.
* {{ReconTaskControllerImpl.reInitializeTasks}} rebuilds on a staged task
instance, swaps the DB in and calls {{init()}} on the live tasks.
{{NSSummaryTask}} does not override {{init()}}, so the live sub-tasks keep
their caches. A delete and recreate whose events were never delivered (the
reason Recon fell back to a rebuild) stays invisible to them.
In both cases every later key of the recreated bucket is counted under the
deleted bucket's objectID, while the namespace API ({{BucketEntityHandler}})
reads the summary of the current objectID.
h3. Trigger
* A client deletes a bucket and creates a bucket with the same name, both
applied in one OM double buffer flush, then writes keys to it.
* Or a bucket is deleted and recreated while Recon misses the events and
rebuilds (for example after a full snapshot fallback or an event buffer
overflow), then keys are written to it.
Affects releases since 2.2.0, which added the cache.
h3. Impact
The namespace summary and disk usage endpoints ({{/namespace/summary}},
{{/namespace/du}}) under report the recreated bucket, silently. New keys are
counted under the deleted bucket until Recon restarts or the bucket is deleted
again, and the keys already miscounted stay missing from the bucket until a
rebuild recomputes the summaries; a restart does not rebuild NSSummary unless a
task lags behind the last OM delta. After a restart, deleting one of those keys
even decrements the new bucket. A rebuild alone does not help either, as the
second path shows. The reproduction checks the summary response; disk usage
reads the same NSSummary entry of the bucket's current objectID
({{BucketEntityHandler.getDuResponse}}).
h3. Reproduction
PASS on unmodified source, deterministic, no fault injection.
[^TestBugCR7BucketCacheAfterRecreate.java] ({{hadoop-ozone/recon}}) writes real
OM batches (the bucket changes through {{OMBucketDeleteResponse}} and
{{OMBucketCreateResponse}}), serves them through {{getUpdatesSince}} and
applies them to the Recon OM DB as {{OzoneManagerServiceProviderImpl}} does.
* One flush: Recon receives {{[bucketTable UPDATE /vol/bucket, keyTable PUT
/vol/bucket/k2]}}. After one more key, the old objectID holds 2 files, the new
one 0, and the namespace summary reports 0 keys (2 expected).
* Rebuild: the delete and recreate are applied to the Recon OM DB without
delivering the events, then the real {{reInitializeTasks}} with the real
{{ReconDBProvider}} rebuilds (new bucket at 1 file, correct). The next key goes
to the old objectID; the namespace summary reports 1 key (2 expected).
A passing test means the defect is present.
h3. Patch
[^CR-7-keep-bucket-cache-in-line-with-recreate-and-rebuild.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
Both sub-tasks now keep the cache in line with every bucketTable event: a PUT
or UPDATE refreshes a cached entry with the event's {{OmBucketInfo}}, a DELETE
removes it. Buckets that are not cached stay uncached, so the cache does not
grow, and no extra DB read is needed. Invalidating on every bucketTable event
was not chosen: every key commit and delete rewrites the bucket row, so the
cache would be cold in almost every batch, and a missing cache entry is one way
to reach the NullPointerException of HDDS-17048 (a key event of a bucket
already deleted in the Recon OM DB). That issue needs its own fix and should
land with or before this one, since clearing the cache on rebuild adds a cold
cache window. {{NSSummaryTask.init()}} now clears the caches; Recon calls it at
startup ({{OzoneManagerServiceProviderImpl.start()}}), and the controller after
each successful rebuild on the event processing thread, between {{process()}}
calls.
Covered by {{testBucketCacheInvalidatedOnRecreateInOneFlush}} and
{{testBucketCacheClearedOnInit}} in the existing
{{TestNSSummaryTask.TestProcessBucketRecreate}}, which fail without the change.
With it, the NSSummary task suites, {{TestNSSummaryUnifiedControl}},
{{TestReconTaskControllerImpl}} and {{TestOMDBUpdatesHandler}} pass (90 tests,
1 skipped as before) and checkstyle is clean. With the patch both reproduction
tests fail because the new bucket counts both keys.
Not fixed: events of one OM flush are grouped per table, and bucketTable events
come before keyTable events. A key of the old bucket deleted in the same flush
that recreates the bucket is therefore subtracted from the new bucket, which
leaves it one key short when a key of the new bucket from that flush was
counted first (otherwise the delete finds no summary and only logs an error).
In a run with one flush that deletes the old bucket's last key, deletes and
recreates the bucket and writes one new key, followed by one more key, the new
bucket ends at 1 file instead of 2 for three of four key names tried, and at 2
for the fourth; before the patch it ends at 0 for all four. Before the patch
that flush and every later one charged new keys to the old bucket instead.
Found by code review of the Recon namespace summary and task sync paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
