[
https://issues.apache.org/jira/browse/HDDS-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16858:
------------------------------
Attachment:
MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch
> Tenant requests release the authorizer write lock only in their apply, so a
> request that is not applied on this OM leaves the lock held and later tenant
> operations time out
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16858
> URL: https://issues.apache.org/jira/browse/HDDS-16858
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch
>
>
> h3. Mechanism
> Each of the six tenant requests ({{OMTenantCreateRequest}},
> {{OMTenantDeleteRequest}}, {{OMTenantAssignUserAccessIdRequest}},
> {{OMTenantRevokeUserAccessIdRequest}}, {{OMTenantAssignAdminRequest}},
> {{OMTenantRevokeAdminRequest}}) takes the OM wide authorizer write lock in
> {{preExecute}} through {{AuthorizerLock.tryWriteLockInOMRequest}}, changes
> Ranger, and releases the lock only in the {{finally}} of its
> {{validateAndUpdateCache}}. {{AuthorizerLockImpl}} records a single stamp and
> holder thread for this lock.
> {{OMExecutionFlow.submitExecutionToRatis}} has no step that releases the lock
> when the request is not applied on this OM. When
> {{OzoneManagerRatisServer.submitRequest}} throws {{ServiceException}} (for
> example from {{NotLeaderException}}, {{LeaderSteppingDownException}} or
> {{ResourceUnavailableException}}), it calls nothing, and when it returns a
> failed response it calls only {{OMClientRequest.handleRequestFailure}}, which
> is empty. The tenant requests stopped overriding {{handleRequestFailure}}
> when the lock was introduced in HDDS-6701.
> The OM is then left with the stamp recorded and no tenant request in flight.
> Each later tenant request on that OM fails in {{preExecute}} with {{TIMEOUT}}
> ("Timed out acquiring authorizer write lock. Another multi-tenancy request is
> in-progress. Try again later"), and the Ranger background sync cannot take
> its read lock either (from {{OMRangerBGSyncService}}, not run). Because
> {{unlockWriteInOMRequest}} releases whatever stamp is recorded, the lock is
> freed only when this OM applies a tenant log entry or restarts.
> h3. Impact
> * Tenant operations (create, delete, user assign and revoke, admin assign and
> revoke) fail with {{TIMEOUT}} on the affected OM while it is leader, until it
> applies a tenant log entry or restarts.
> * The Ranger background sync makes no progress on that OM in the meantime
> (code reading, not run).
> * No data loss. Key, bucket and S3 operations do not use this lock.
> h3. Workaround
> Move leadership to another OM ({{ozone admin om transfer}}) and run a tenant
> operation there that succeeds. Once the affected OM applies that entry, its
> lock is released. Restarting the affected OM also releases it.
> h3. Patch
> [^MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> The lock is released whatever the submit outcome. {{OMExecutionFlow}} calls a
> new hook, {{OMClientRequest.afterSubmit}}, in a {{finally}} after the submit
> to Ratis. It is a no op by default, and the six tenant requests override it
> to call the new
> {{AuthorizerLock.unlockWriteInOMRequestIfHeldByCurrentThread}}, which
> releases the lock only if the current handler thread is its recorded holder.
> If the request was applied, its apply has already cleared the holder before
> the reply was completed, so the call does nothing and a lock taken meanwhile
> by the next tenant request is left alone. Non tenant requests are not
> affected, and there is no protocol, config or apply change.
> The issue has no standalone reproduction. The regression tests are in the
> existing {{TestOMTenantDeleteRequest}}:
> {{lockNotHeldAfterSubmitWhateverTheOutcome}} (each submit outcome),
> {{responseAfterApplyKeepsLockOfNextRequest}} (the request was applied and the
> next request took the lock before the response was returned) and
> {{tenantRequestReleasesOnlyOwnLockAfterSubmit}} (each of the six tenant
> requests). Without the change the first and third fail because the lock is
> still held; the second guards against an unconditional unlock. With it the
> tenant and S3 secret request suites, {{TestOMExecutionFlow}},
> {{TestAuthorizerLockImpl}}, the multi tenant manager suites and the OM Ratis
> suites (165 tests) pass, as do {{TestMultiTenantVolume}} and
> {{TestOzoneTenantShell}} on a mini cluster (11 tests). Checkstyle is clean.
> HDDS-7184 reported the same symptom ("Timed out acquiring authorizer write
> lock" with no multi-tenancy request in progress) and was resolved in 2023 as
> probably fixed by HDDS-7755, after the failing test stopped failing. This
> lock leak is another way to reach that symptom.
> Suggested priority: Major. Tenant operations stop on one OM until it applies
> a tenant entry from another leader or restarts. No data is lost and the key
> and S3 paths are not affected.
> Found by TLA+ model checking and code review of the OM multi-tenancy and
> Ranger sync paths under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests before filing; no existing issue covers it.
> Related: HDDS-6701 (introduced the lock and removed the tenant
> {{handleRequestFailure}} overrides), HDDS-7184 (same symptom, resolved). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
