Siyao Meng created HDDS-17295:
---------------------------------
Summary: OzoneManager read paths release their volume or bucket
read lock on a lock that a checkpoint install may have replaced
Key: HDDS-17295
URL: https://issues.apache.org/jira/browse/HDDS-17295
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-8-release-om-read-locks-on-the-acquired-lock.patch,
TestBugMC8VolumeOwnerLockAcrossInstall.java
h3. Mechanism
{{OzoneManager.getVolumeOwner(String)}} takes the VOLUME read lock on
{{metadataManager.getLock()}} and releases it in its {{finally}} block on a
second read of the {{metadataManager}} field.
{{OzoneManager.installCheckpoint}} replaces that field, and with it the lock,
in {{reloadOMState}}. If the replacement happens between the acquire and the
release, the release goes to the new lock and throws
{{IllegalMonitorStateException}}, and the read hold on the replaced lock is
never released. {{getBucketOwner}}, {{getLifecycleConfiguration}},
{{listUsersInTenant}}, {{getS3VolumeContext}} and
{{SstFilteringService.markSSTFilteredFlagForSnapshot}} use the same pattern.
h3. Trigger
A read with an ACL check (it calls {{getVolumeOwner}} or {{getBucketOwner}}
through {{OmMetadataReader.checkAcls}}), or one of the other methods above, is
in progress on an OM while that OM installs a checkpoint from the leader, and
it reaches the lock acquire before the swap and the release after it. A
follower installs a checkpoint when it lags behind the purged Raft log. On a
follower such reads are served only with follower reads (off by default on
client and server). {{installCheckpoint}} stops the Hadoop RPC server before
the swap, but not the OM gRPC server ({{ozone.om.s3.grpc.server_enabled}}, on
by default), so gRPC requests keep being served during the install. A Hadoop
RPC call that is already running when the server stops loses its connection
anyway. Any authenticated client can send such a read; it cannot make the
install happen. For {{SstFilteringService}} it needs a filtering task that
outlives the shutdown wait of {{KeyManager.stop}}.
h3. Impact
* A read over the OM gRPC server fails with {{IllegalMonitorStateException}}
(returned as {{Status.INTERNAL}}) although the volume exists. The failed call
is shown in the reproduction, in process. Argued from code, not run over a real
connection: the gRPC follower read client treats this error as a reason to try
another OM.
* The read hold on the replaced lock is never released. Nothing uses the
replaced lock after the install, so this holds no current request up.
* {{OzoneManagerLock}} updates its per thread lock tracker after the unlock, so
the failed release also leaves a stale bit on that thread. A stale VOLUME bit
only refuses {{S3_BUCKET_LOCK}}, which no OM code acquires. A stale BUCKET bit
(from {{getBucketOwner}} or {{getLifecycleConfiguration}}) also refuses
{{VOLUME_LOCK}} on that thread until the thread next releases a BUCKET lock.
Hadoop RPC handler threads exit with the stopped server; a gRPC handler thread
keeps the bit (code reading).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC8VolumeOwnerLockAcrossInstall.java] ({{hadoop-ozone/ozone-manager}})
starts one OM through {{OmTestManagers}} and creates a volume. It holds the
VOLUME write lock of the current lock, starts {{getVolumeOwner}} on another
thread so that it waits in its read lock acquire, installs a checkpoint of the
OM's own DB with {{installCheckpoint}} (presented with a newer transaction
index so that it proceeds), and then releases the write lock. The lookup fails
with {{IllegalMonitorStateException}}. A passing test means the defect is
present.
h3. Patch
[^MC-8-release-om-read-locks-on-the-acquired-lock.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
669912118cf.
The six methods read the lock once into a local variable and release on that
same lock. Nothing else changes: the same lock is taken as before, and the data
is read as before.
Covered by {{testOwnerLookupsAcrossInstallReleaseTheirLocks}} in the existing
{{TestOMRatisSnapshots}}: on the active follower of the suite's HA cluster it
puts a {{getVolumeOwner}} and a {{getBucketOwner}} call across an install in
the same way (the install fails at its first backup move with the suite's
existing fault injector and reloads the old DB, which also replaces the
metadata manager and the lock) and checks that both calls return the owner.
Without the change the test fails with {{IllegalMonitorStateException}}. With
it the test, {{TestOMRatisSnapshots}}, {{TestSstFilteringService}},
{{TestMultiTenantVolume}}, {{TestOMMetadataReader}} and
{{TestOzoneIntegrationNonHA}} pass, and checkstyle is clean. With the patch the
reproduction fails at its assertion, because the lookup returns the owner.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not duplicates: HDDS-4088 (added the owner lookup
for the authorizer), HDDS-9279 (follower reads, which make this reachable),
HDDS-6833 (caching to reduce lock contention in {{getVolumeOwner}}), HDDS-17120
and HDDS-16878 (checkpoint servlets that keep a DB store an install replaced,
the same pattern for another object). The attached patch is a proposal for
review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
