[
https://issues.apache.org/jira/browse/HDDS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16853:
------------------------------
Attachment: TestBugCR2SnapshotInstallKeepsOldRangerSync.java
> OM snapshot install leaves the previous multi-tenant manager and its Ranger
> background sync running
> ---------------------------------------------------------------------------------------------------
>
> Key: HDDS-16853
> URL: https://issues.apache.org/jira/browse/HDDS-16853
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-2-stop-previous-multi-tenant-manager-on-reload.patch,
> TestBugCR2SnapshotInstallKeepsOldRangerSync.java
>
>
> h3. Mechanism
> When a follower OM installs a Ratis snapshot, {{OzoneManager.reloadOMState}}
> calls {{instantiateServices}}. That creates a new
> {{OMMultiTenantManagerImpl}}, whose constructor starts a new
> {{OMRangerBGSyncService}}. The previous manager is never stopped;
> {{OMMultiTenantManagerImpl.stop}} is only called from {{OzoneManager.stop}}.
> The old sync thread stays scheduled, and holds the old manager, its tenant
> cache, its authorizer lock and the old, closed {{OmMetadataManagerImpl}}. It
> checks the live {{OzoneManager.isLeaderReady}}, so it does sync work whenever
> that OM is leader.
> h3. Trigger
> Multi-tenancy is enabled on an HA OM, and a follower falls behind the
> leader's purged log and installs a snapshot. Each install adds one more sync
> service.
> h3. Impact
> * One leaked sync thread per snapshot install, keeping the old manager and
> old metadata manager (with their table caches) reachable. The memory part is
> argued, not measured.
> * While that OM is leader and the Ranger and OM service versions differ,
> every run of the old service reads from Ranger and then fails with "Rocks
> Database is closed", logging a WARN stack trace.
> * No stale writes to Ranger. The old service reaches its first Ranger write
> only after iterating its tenant state table, and that always fails on the
> closed DB, because the reload only happens after the old metadata manager was
> stopped. This holds only because of that ordering.
> h3. Reproduction
> [^TestBugCR2SnapshotInstallKeepsOldRangerSync.java]
> ({{hadoop-ozone/integration-test}}). It uses an HA mini cluster with 3 OMs
> (one inactive) and multi-tenancy with the in-tree in-memory Ranger stand-in
> ({{ozone.om.tenant.dev.skip.ranger}}). It installs the leader's checkpoint on
> the follower with {{installCheckpoint}}. On the unmodified code it passes:
> after the install the follower has a new manager, the old manager's sync
> executor is not shut down, there are two sync threads, and a run of the old
> service fails with "Rocks Database is closed". A passing test means the
> defect is present.
> h3. Patch
> [^CR-2-stop-previous-multi-tenant-manager-on-reload.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{instantiateServices}} stops the previous multi-tenant manager, if there is
> one, before creating the new one. This is the same place where the authorizer
> instance is handled across installs (HDDS-11472). The manager is created and
> started only there, so no restart is needed when an install is refused and
> the old DB stays open. At first start nothing changes. In
> {{OzoneManager.restart}} the manager was already stopped by
> {{OzoneManager.stop}}, and stopping it again does nothing.
> Covered in the existing {{TestOMRatisSnapshots}}, whose cluster now enables
> multi-tenancy with the in-memory stand-in.
> {{testInstallSnapshotWithClientWrite}} checks that the old manager's sync is
> shut down and the new one runs. {{testInstallOldCheckpointFailure}} checks
> that a refused install keeps the same manager running. Without the change the
> first fails, because the old sync executor is still running. With it
> {{TestOMRatisSnapshots}} (7 tests),
> {{TestOMInstallSnapshotDuringBootstrapping}} and {{TestMultiTenantVolume}}
> pass, and checkstyle is clean. With the patch the reproduction fails, because
> the old sync is shut down.
> Found by code review of the OM multi-tenancy and Ranger sync paths, as part
> of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
