Siyao Meng created HDDS-16853:
---------------------------------
Summary: OM snapshot install leaves the previous multi-tenant
manager and its Ranger background sync running
Key: HDDS-16853
URL: https://issues.apache.org/jira/browse/HDDS-16853
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-2-stop-previous-multi-tenant-manager-on-reload.patch,
TestBugCR2SnapshotInstallKeepsOldRangerSync.java
h3. Mechanism
When a follower OM installs a Ratis snapshot, {{OzoneManager.reloadOMState}}
calls {{instantiateServices}}. That creates a new {{OMMultiTenantManagerImpl}},
whose constructor starts a new {{OMRangerBGSyncService}}. The previous manager
is never stopped; {{OMMultiTenantManagerImpl.stop}} is only called from
{{OzoneManager.stop}}. The old sync thread stays scheduled, and holds the old
manager, its tenant cache, its authorizer lock and the old, closed
{{OmMetadataManagerImpl}}. It checks the live {{OzoneManager.isLeaderReady}},
so it does sync work whenever that OM is leader.
h3. Trigger
Multi-tenancy is enabled on an HA OM, and a follower falls behind the leader's
purged log and installs a snapshot. Each install adds one more sync service.
h3. Impact
* One leaked sync thread per snapshot install, keeping the old manager and old
metadata manager (with their table caches) reachable. The memory part is
argued, not measured.
* While that OM is leader and the Ranger and OM service versions differ, every
run of the old service reads from Ranger and then fails with "Rocks Database is
closed", logging a WARN stack trace.
* No stale writes to Ranger. The old service reaches its first Ranger write
only after iterating its tenant state table, and that always fails on the
closed DB, because the reload only happens after the old metadata manager was
stopped. This holds only because of that ordering.
h3. Reproduction
[^TestBugCR2SnapshotInstallKeepsOldRangerSync.java]
({{hadoop-ozone/integration-test}}). It uses an HA mini cluster with 3 OMs (one
inactive) and multi-tenancy with the in-tree in-memory Ranger stand-in
({{ozone.om.tenant.dev.skip.ranger}}). It installs the leader's checkpoint on
the follower with {{installCheckpoint}}. On the unmodified code it passes:
after the install the follower has a new manager, the old manager's sync
executor is not shut down, there are two sync threads, and a run of the old
service fails with "Rocks Database is closed". A passing test means the defect
is present.
h3. Patch
[^CR-2-stop-previous-multi-tenant-manager-on-reload.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{instantiateServices}} stops the previous multi-tenant manager, if there is
one, before creating the new one. This is the same place where the authorizer
instance is handled across installs (HDDS-11472). The manager is created and
started only there, so no restart is needed when an install is refused and the
old DB stays open. At first start nothing changes. In {{OzoneManager.restart}}
the manager was already stopped by {{OzoneManager.stop}}, and stopping it again
does nothing.
Covered in the existing {{TestOMRatisSnapshots}}, whose cluster now enables
multi-tenancy with the in-memory stand-in.
{{testInstallSnapshotWithClientWrite}} checks that the old manager's sync is
shut down and the new one runs. {{testInstallOldCheckpointFailure}} checks that
a refused install keeps the same manager running. Without the change the first
fails, because the old sync executor is still running. With it
{{TestOMRatisSnapshots}} (7 tests),
{{TestOMInstallSnapshotDuringBootstrapping}} and {{TestMultiTenantVolume}}
pass, and checkstyle is clean. With the patch the reproduction fails, because
the old sync is shut down.
Found by code review of the OM multi-tenancy and Ranger sync paths, as part of
the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
