Siyao Meng created HDDS-17297:
---------------------------------
Summary: Reconfiguring the SST filtering interval while an OM
installs a checkpoint starts an SstFilteringService on the replaced KeyManager
Key: HDDS-17297
URL: https://issues.apache.org/jira/browse/HDDS-17297
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-5-serialize-sst-filtering-reconfig-with-checkpoint-install.patch,
TestBugCR5SstFilteringReconfigDuringInstall.java
h3. Mechanism
{{OzoneManager.reconfOzoneSnapshotSSTFilteringServiceInterval}} reads the
{{keyManager}} field several times and calls
{{KeyManagerImpl.startSnapshotSstFilteringService}} on it. It runs on the
reconfiguration task thread and nothing makes it wait for
{{OzoneManager.installCheckpoint}}, which stops the old {{KeyManager}},
replaces the DB and then, in {{reloadOMState}}, creates a new metadata manager
(with a new lock) and a new {{KeyManagerImpl}}. {{installSnapshotFromLeader}}
holds the OM monitor for the whole install, but the reconfiguration function
does not take it. A reconfiguration that runs after the old key manager was
stopped and before the new one is assigned starts a service on the old key
manager, which nothing references afterwards.
{{reconfOzoneKeyDeletingLimitPerTask}} has the same race: it calls
{{getKeyManager().getDeletingService()}} without the monitor.
That service took its bootstrap lock from the replaced metadata manager when it
was created, while {{OMDBCheckpointServlet}} resolves the current lock on every
call. Its task looks up the current metadata manager and snapshot manager on
every run, so it works on the live snapshots.
h3. Trigger
# SST filtering is disabled ({{ozone.snapshot.filtering.service.interval}} set
to 0) and the admin enables it again with {{ozone admin reconfig --service=OM
start}}.
# At the same time the OM installs a checkpoint from the leader (any follower
that lags behind the purged Raft log), and the reconfiguration task reaches
this property inside the install, between the stop of the old key manager and
the creation of the new one.
The window is the DB replacement and reload, so the race is narrow. Only an
admin can start the reconfiguration task.
h3. Impact
* An {{SstFilteringService}} keeps running on the replaced key manager. Setting
the interval back to 0 stops only the service of the current key manager, so
filtering continues although the admin disabled it. Shown in the reproduction.
* The orphan service is not excluded by the checkpoint servlet's bootstrap
write lock. Shown in the reproduction. That it actually deletes SST files of a
snapshot while a checkpoint is being taken is argued, not run.
* Argued from code, not run: with filtering enabled (the default), the same
race makes the {{Objects.requireNonNull}} check in the reconfiguration function
fail with {{NullPointerException}}, because the stopped key manager no longer
has a service. The reconfiguration of the property is then reported as failed
and no orphan is left.
* A reconfiguration of {{ozone.key.deleting.limit.per.task}} inside the same
window fails with {{NullPointerException}}, because the stopped key manager has
no {{KeyDeletingService}}, and is reported as failed. Shown in the regression
test. Argued from code, not run: one that runs just before the window sets the
limit on the service that the install then replaces, and the new service can
read the old value from the configuration, so the change can be lost silently.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR5SstFilteringReconfigDuringInstall.java]
({{hadoop-ozone/ozone-manager}}) starts one OM through {{OmTestManagers}},
disables filtering, and installs a checkpoint of its own DB with
{{installCheckpoint}} (presented with a newer transaction index so that it
proceeds). The existing {{checkpointBackupInjector}} hook holds the install
inside the window while the test calls
{{ReconfigurationHandler.reconfigurePropertyImpl}}, the call the
reconfiguration task makes for each changed property, on its own thread. It
shows that the reconfiguration does not wait for the install, that the old key
manager has a running service, that this service survives disabling filtering
again, and that it gets its bootstrap read lock while
{{OMDBCheckpointServlet.Lock}} holds the bootstrap write lock. A passing test
means the defect is present.
h3. Patch
[^CR-5-serialize-sst-filtering-reconfig-with-checkpoint-install.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
669912118cf.
{{reconfOzoneSnapshotSSTFilteringServiceInterval}},
{{reconfOzoneKeyDeletingLimitPerTask}} and {{installCheckpoint}} are now
{{synchronized}}. The install path from Ratis already holds the OM monitor in
{{installSnapshotFromLeader}}, so these reconfigurations now wait for a running
install and then apply to the services of the new key manager. Marking
{{installCheckpoint}} as well keeps that true for its direct callers; the lock
is reentrant, so the Ratis path is unchanged. {{installSnapshotFromLeader}}
holds the monitor for the whole install, including the checkpoint download, so
on a follower that is downloading a checkpoint the reconfiguration task, and
every property after this one in the same task, waits for the download. It
waits in the property loop, outside {{reconfigLock}}, so {{ozone admin reconfig
status}} keeps answering that the task is running.
Covered by {{testServiceReconfigurationWaitsForInstall}} in the existing
{{TestOMRatisSnapshots}}: on the active follower of the suite's HA cluster it
holds an install inside the window with the suite's existing fault injector,
runs both reconfigurations on their own threads, and checks that the replaced
key manager has no filtering service, the new one has a filtering service with
the new interval, and its key deleting service has the new limit. Without the
change the test fails at the limit reconfiguration with the
{{NullPointerException}}; with only the filtering part of the test it fails
because the replaced key manager has a filtering service. With it
{{TestOMRatisSnapshots}} and {{TestOzoneIntegrationNonHA}} (including the OM
reconfiguration and reconfig shell tests) pass, and checkstyle is clean. With
the patch the reproduction fails at the assertion that the old key manager got
a service.
Found by code review of the Ozone Manager lock ordering paths, as part of the
TLA+ verification effort under HDDS-15926, on commit
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-13464 (made this interval reconfigurable), HDDS-15103 (a race
between OM start and snapshot install), HDDS-17120 and HDDS-16878 (checkpoint
servlets that keep a DB store an install replaced, the same pattern for another
object). The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
