[
https://issues.apache.org/jira/browse/HDDS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16766:
------------------------------
Attachment: TestBugMC4EmptinessCheckRace.java
> FSO directory delete emptiness check races with flush and cache eviction, so
> OMs apply the same delete differently
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16766
> URL: https://issues.apache.org/jira/browse/HDDS-16766
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: MC-4-recheck-db-in-fso-emptiness-check.patch,
> TestBugMC4EmptinessCheckRace.java
>
>
> h3. Mechanism
> {{OMFileRequest.checkSubDirectoryExists}} and {{checkSubFileExists}} create a
> RocksDB iterator, which is a point in time view, and then for each row treat
> it as deleted only if a tombstone is still in the table cache. The double
> buffer flush thread ({{OzoneManagerDoubleBuffer.flushBatch}}) and the cache
> eviction ({{PartialTableCache.evictCache}}, scheduled from the flush thread)
> do not take the bucket lock that {{validateAndUpdateCache}} holds.
> So a child's delete can be committed and its tombstone evicted after the
> iterator was created. The iterator still returns the row, the cache lookup
> returns null, and the row is counted as a live child.
> h3. Trigger
> A directory has one child. A client deletes the child and then the parent,
> non recursively. On one OM the flush of the child delete and the eviction of
> its tombstone land between iterator creation and the cache lookup in the
> apply of the parent delete. On the other OMs they land before or after.
> h3. Impact
> The OM that hit the window answers {{DIRECTORY_NOT_EMPTY}} and keeps the
> directory, the others answer OK and delete it. {{directoryTable}} and the
> bucket's {{usedNamespace}} diverge between OMs until the affected OM's DB is
> replaced. If the leader hit it, the client gets {{DIRECTORY_NOT_EMPTY}} for
> an empty directory. If a follower hit it, the directory reappears after a
> failover to that follower. On a single OM the consequence is only the
> spurious error.
> h3. Reproduction
> PASS with controlled timing. [^TestBugMC4EmptinessCheckRace.java]
> ({{hadoop-ozone/integration-test}}), 3 OM HA mini cluster, client API only,
> one follower's flush controlled with
> {{OzoneManagerDoubleBuffer.pause/unpause}}: hit in 1 of 150 rounds on
> unmodified source. With an 800 ms sleep inserted after iterator creation it
> hits in 9 of 9 rounds. 150 rounds without controlled timing did not trigger
> it, so the natural window is microseconds.
> h3. Suggested fix
> Replace the cache marker check with {{Table.isExist}} on the row's key, which
> looks in the cache first and then reads the DB again. Flush commits to
> RocksDB before eviction and the bucket write lock prevents the child being
> recreated meanwhile, so "no cache entry and row absent now" means deleted.
> HDDS-14600 added the cache marker check for the same user visible symptom,
> and HDDS-8526 already rechecks the DB this way for bucket and volume
> emptiness.
> h3. Patch
> [^MC-4-recheck-db-in-fso-emptiness-check.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> 9c6bee807c8.
> In {{checkSubDirectoryExists}} and {{checkSubFileExists}} the cache marker
> check becomes {{if (!table.isExist(dbKey)) continue;}}. The cost is one point
> read per row the iterator returns without a cache entry, and the loop returns
> at the first row that still exists, so a non empty directory pays one read.
> Covered by the existing
> {{TestOMKeyDeleteRequestWithFSO#testDeleteParentAfterChildDeleted}} (the
> HDDS-14600 test), now parameterized: the new variant flushes and evicts the
> child delete right after the emptiness check opens its iterator, using table
> spies, with no sleep and no product hook. Without the change it fails with
> "expected: <OK> but was: <DIRECTORY_NOT_EMPTY>". The delete, purge and rename
> unit suites (86 tests) and {{TestO3FSWithFSO}}, {{TestOFSWithFSO}},
> {{TestDirectoryDeletingServiceWithFSO}} (146 tests) pass, checkstyle is
> clean. The patch prevents new occurrences only.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
