[
https://issues.apache.org/jira/browse/HDDS-17296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17296:
------------------------------
Attachment: TestBugMC7StaleDirDeletingServiceRestart.java
> After an OM installs a checkpoint, reconfiguring the directory deleting
> interval restarts the replaced DirectoryDeletingService as well
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17296
> URL: https://issues.apache.org/jira/browse/HDDS-17296
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-restart-only-current-dir-deleting-service-on-reconfig.patch,
> TestBugMC7StaleDirDeletingServiceRestart.java
>
>
> h3. Mechanism
> Each {{DirectoryDeletingService}} registers a reconfiguration complete
> callback in its constructor ({{registerReconfigCallbacks}}), and the callback
> calls {{updateAndRestart}} on that instance. {{ReconfigurationHandler}} keeps
> these callbacks in a list for the life of the OM process and has no way to
> remove one. {{OzoneManager.installCheckpoint}} stops the {{KeyManager}} and
> then builds a new {{DirectoryDeletingService}}: {{reloadOMState}} creates a
> new metadata manager, a new {{OzoneManagerLock}} and a new
> {{KeyManagerImpl}}, and when the old DB is kept the same {{KeyManagerImpl}}
> is started again. The callback of the replaced service stays registered.
> The next reconfiguration that changes
> {{ozone.directory.deleting.service.interval}} or
> {{ozone.thread.number.dir.deletion}} therefore restarts the replaced service
> too. Its tasks check only the suspend flag and leader readiness, and they
> look up the key manager and metadata manager from the OM on every run, so the
> revived service does not fail on the closed DB: it works on the installed DB
> like the current one. It keeps two things from before the install: the
> bootstrap lock it took from the replaced metadata manager in the
> {{AbstractKeyDeletingService}} constructor, and that metadata manager's
> snapshot chain. {{OMDBCheckpointServlet}} resolves the current lock on every
> call, so it does not exclude the revived service. The purge request checks
> the expected previous snapshot against the current chain when it is applied,
> so the stale chain makes the revived service's purges fail once the chain has
> changed, rather than purge wrongly.
> h3. Trigger
> # An OM installs, or attempts to install, a checkpoint from the leader. This
> happens to any follower that lags behind the purged Raft log, and every
> attempt that gets as far as stopping the key manager counts, also when the
> checkpoint is rejected as not newer or pausing the state machine fails. No
> admin action is needed.
> # An admin changes the directory deleting interval or thread count in
> ozone-site.xml and runs {{ozone admin reconfig --service=OM start}} on that
> OM.
> # The replaced service now runs next to the current one whenever that OM is
> leader.
> Each further install attempt adds another stale callback, and nothing stops a
> revived service until the process exits.
> h3. Impact
> * Two {{DirectoryDeletingService}} instances submit directory purges on the
> leader. The replaced one is not excluded by the checkpoint servlet's
> bootstrap write lock, so it keeps changing the DB while a checkpoint for
> another OM or Recon is being taken. Both shown in the reproduction, before
> any snapshot change after the install.
> * Each install leaves the replaced service, and the lock and snapshot chain
> it references, reachable from the handler (code reading).
> * Argued, not run: two instances can read and submit a purge for the same
> deleted directory, and {{OMDirectoriesPurgeRequestWithFSO}} applies the quota
> change of a purge without checking that the paths still exist, so the bucket
> used bytes and namespace could be counted down twice. That needs both
> instances to read the same directory before the first purge is applied; it
> was not observed without an artificial delay. Once a snapshot is created or
> deleted after the install, the revived service's purges are rejected by the
> expected previous snapshot check instead.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC7StaleDirDeletingServiceRestart.java]
> ({{hadoop-ozone/ozone-manager}}) starts one OM through {{OmTestManagers}},
> installs a checkpoint of its own DB with {{installCheckpoint}} (presented
> with a newer transaction index so that it proceeds) and runs the
> reconfiguration task that {{ozone admin reconfig start}} starts. It suspends
> the current service with its own hook, adds a deleted directory, and shows
> that the replaced service was restarted, purged the directory from the
> installed DB, and still gets its bootstrap read lock while
> {{OMDBCheckpointServlet.Lock}} holds the bootstrap write lock (the current
> service does not). A passing test means the defect is present.
> h3. Patch
> [^MC-7-restart-only-current-dir-deleting-service-on-reconfig.patch], against
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
> 669912118cf.
> The per instance callback is removed. {{OzoneManager}} registers one callback
> that, when the interval or thread count changes, restarts the service that
> {{keyManager.getDirDeletingService()}} returns at that moment. A service
> created later reads the new values from the OM configuration, which the
> reconfiguration task has already updated.
> {{DirectoryDeletingService.shutdown}} now also marks the service closed, and
> {{updateAndRestart}} checks that mark under the service's monitor before it
> starts the executors again, so a restart that picked up the service just
> before an install shut it down leaves it stopped. The callback takes no new
> lock and does not wait for a running install. It also removes the
> {{registerCompleteCallback}} call that the install thread made on the
> handler's unsynchronized list while a reconfiguration task could be iterating
> it.
> Covered by {{testReconfigurationRestartsOnlyCurrentService}} in the existing
> {{TestDirectoryDeletingService}}: it replaces the service by stopping and
> starting the key manager, as an install does, runs the reconfiguration task
> and checks that the replaced service stays shut down, also when
> {{updateAndRestart}} is called on it directly, and that the current one has
> the new interval. Without the change the test fails because the replaced
> service is running again; with only the callback change and without the
> closed check it fails at the direct restart. With the change
> {{TestDirectoryDeletingService}} and {{TestOzoneIntegrationNonHA}} (including
> the OM reconfiguration and reconfig shell tests) pass, and checkstyle is
> clean. With the patch the reproduction fails at its first assertion, because
> the replaced service stays shut down.
> Found by TLA+ model checking and code review of the Ozone Manager lock
> ordering paths under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-12554 (added this callback), HDDS-11513 (extended it to the
> thread count), HDDS-15429 and HDDS-15405 (other {{updateAndRestart}} fixes),
> HDDS-17178 (another component of a replaced OM state left registered after an
> install), HDDS-17120 and HDDS-16878 (checkpoint servlets that keep a DB store
> an install replaced, the same pattern for another object). The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
