Siyao Meng created HDDS-16879:
---------------------------------
Summary: RevokedSTSTokenCleanupService keeps the metadata manager
closed by a checkpoint install, so every cleanup run fails while that OM is
leader
Key: HDDS-16879
URL: https://issues.apache.org/jira/browse/HDDS-16879
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-2-sts-token-cleanup-read-current-metadata-manager.patch,
TestBugCR2StsCleanupClosedMetadataManager.java
h3. Mechanism
{{RevokedSTSTokenCleanupService}} stores {{ozoneManager.getMetadataManager()}}
in a final field in its constructor, and the service is created once in
{{OzoneManager.start}}. {{OzoneManager.installCheckpoint}} stops the old
metadata manager, which closes its store, and {{reloadOMState}} creates a new
{{OmMetadataManagerImpl}}. The service is not stopped, recreated or updated on
that path, so each later run iterates {{s3RevokedStsTokenTable}} of the closed
store.
The service is not in a release yet (HDDS-14094), so only master is affected.
The OM checkpoint servlets have the same problem, see HDDS-16878.
h3. Trigger
# An OM installs a checkpoint from the leader through
{{installSnapshotFromLeader}}.
# That OM becomes leader (election or leadership transfer). The service only
runs on the leader.
h3. Impact
* While that OM is leader, every run fails with {{RocksDatabaseException: Rocks
Database is closed}}, logs "Failure while scanning s3RevokedStsTokenTable" at
ERROR with a stack trace, and deletes nothing, until the OM restarts. With the
default {{ozone.om.sts.token.cleanup.service.interval}} of 3h that is one ERROR
every 3 hours. The scan fails on an empty table too, so the ERROR appears
whether or not STS is used.
* Expired revocation entries stay in the table while that OM leads. They belong
to tokens past their maximum lifetime, and revocation checks read the current
table, so no authentication result changes. Another OM that has not installed a
checkpoint since it started cleans up the backlog once it becomes leader (code
reading).
h3. Reproduction
PASS on the unmodified source, deterministic.
[^TestBugCR2StsCleanupClosedMetadataManager.java]
({{hadoop-ozone/integration-test}}) starts three OMs with one held back and the
cleanup interval set to 1s, advances the leader past the Ratis log purge gap,
starts the late OM so that Ratis installs a checkpoint on it, and transfers
leadership to it. One revocation entry 13 hours old is then written directly
into the new leader's table, because a real revoke cannot be back dated; it
only makes the missed cleanup visible. Output:
{noformat}
After omNode-3 installed a checkpoint and became leader: scan failures logged =
4, 'Rocks Database is closed' in log = true, expired entry still present = true
ERROR service.RevokedSTSTokenCleanupService
(RevokedSTSTokenCleanupService.java:call(187)) - Failure while scanning
s3RevokedStsTokenTable. It will be retried in the next interval
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is closed
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:809)
{noformat}
A passing test means the defect is present. With the patch no scan fails, the
service logs "Found and removed 1 revoked STS token entries." and the test
fails at its first defect assertion.
h3. Patch
[^CR-2-sts-token-cleanup-read-current-metadata-manager.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
6818707698c.
The service no longer keeps the metadata manager and reads
{{ozoneManager.getMetadataManager()}} at the start of each run, as
{{OMUpgradeFinalizeService}} already does. Nothing changes for an OM that never
installed a checkpoint, and no lifecycle code is added to the install path.
Covered by the new
{{TestRevokedSTSTokenCleanupService#scansCurrentMetadataManagerAfterItIsReplaced}}:
after the service is created the OM returns a new metadata manager, and one
run must clean the expired entry of the new table and not the one of the
original table. Without the change it submits the original table's entry. With
it {{TestRevokedSTSTokenCleanupService}}, {{TestS3RevokeSTSTokenRequest}} and
{{TestS3SecurityUtil}} pass (38 tests) and checkstyle is clean.
Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
