[
https://issues.apache.org/jira/browse/HDDS-16854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16854:
------------------------------
Attachment: TestBugCR3OrphanRoleCleanup.java
> Ranger background sync never retries a failed role delete of a removed
> tenant, so the tenant can never be created again
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16854
> URL: https://issues.apache.org/jira/browse/HDDS-16854
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-3-retry-failed-ranger-role-deletes-in-sync.patch,
> TestBugCR3OrphanRoleCleanup.java
>
>
> h3. Mechanism
> {{OMRangerBGSyncService}} removes the Ranger state of a tenant that is not in
> the OM DB by deleting its labeled policies in {{processAllPoliciesFromOMDB}}
> and then its roles in {{processAllRolesFromOMDB}}. A failed role delete is
> only logged ("Failed to delete role", without the exception). The sync learns
> about roles only from the role ACLs of the policies labeled {{OzoneTenant}}
> ({{loadAllPoliciesAndRoleNamesFromRanger}}), and those policies are already
> gone. So no later run sees the role again, and the run records the Ranger
> service version as in sync. {{createRole}} fails when a role with the same
> name exists (HDDS-15982 kept creation fail fast), so every later create of
> that tenant fails.
> HDDS-16188 describes the same end state reached through a partially failed
> tenant create. This report is about the cleanup path of the sync, which needs
> its own change.
> h3. Trigger
> # A tenant create leaves Ranger roles and policies without a tenant in OM.
> One way: {{OMTenantCreateRequest.preExecute}} creates them, then the apply
> fails, for example with {{VOLUME_ALREADY_EXISTS}} because the volume was
> created concurrently. {{preExecute}} expects the sync to clean this up.
> # The sync deletes both policies, then the delete of {{tenant1-UserRole}}
> fails once (HTTP 5xx, timeout, Ranger restart).
> # Later sync runs, also runs that execute because the Ranger service version
> changed, never delete the role.
> # Every {{ozone tenant create tenant1}} fails with
> {{TENANT_AUTHORIZER_ERROR}}, cause "Role already exists". {{ozone tenant
> delete tenant1}} cannot help because OM has no such tenant (argued, not run).
> h3. Impact
> The tenant id cannot be created again until an operator deletes the leftover
> roles in Ranger by hand. It needs two independent events: a tenant create
> that fails after its Ranger objects were created, and a failed Ranger call
> during the cleanup. No data is lost. Creating a tenant requires cluster admin
> privilege.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugCR3OrphanRoleCleanup.java]
> ({{hadoop-ozone/ozone-manager}}) runs the real {{OMTenantCreateRequest}}
> ({{preExecute}}, then {{validateAndUpdateCache}} after a volume of the same
> name was added to the OM DB), the real {{AuthorizerOp}}, the real
> {{OMRangerBGSyncService.triggerRangerSyncOnce}} and a real OM DB. Ranger is
> the in-memory controller that OM uses with
> {{ozone.om.tenant.dev.skip.ranger}}, wrapped in a Mockito spy that fails one
> delete of the user role. Before the second and third sync run the test
> changes the Ranger service version with an unrelated labeled policy, so that
> the sync executes. Output on the unmodified source:
> {noformat}
> create #1: VOLUME_ALREADY_EXISTS; Ranger roles [tenant1-UserRole,
> tenant1-AdminRole], policies [tenant1-BucketAccess, tenant1-VolumeAccess],
> tenant in OM DB: false
> sync run 1: completed true; Ranger roles [tenant1-UserRole], policies []
> sync run 2: completed true; Ranger roles [tenant1-UserRole], policies []
> sync run 3: completed true; Ranger roles [tenant1-UserRole], policies []
> retry 1: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
> retry 2: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
> {noformat}
> With real Ranger, the admin role delete is then refused too, because the user
> role still names it as delegated admin
> ({{RoleDBStore.ensureRoleDeleteAllowed}} refuses to delete a role referenced
> by a policy, another role or a security zone), so both roles are left. The
> in-memory controller does not check this, so only the user role is left here.
> A passing test means the defect is present. Not run against a real Ranger.
> h3. Suggested fix
> Remember the roles whose delete failed and retry them in the following runs,
> skipping any role that belongs to a tenant in OM again. Finding tenant roles
> in Ranger by name or description instead was rejected, because Ranger roles
> are not scoped to a service, so the sync of one cluster could delete the
> tenant roles of another cluster using the same Ranger. Persisting the pending
> deletes in the OM DB would survive a leader change, but needs a new DB entry.
> h3. Patch
> [^CR-3-retry-failed-ranger-role-deletes-in-sync.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> The sync keeps the names of roles whose delete failed in memory. At the start
> of each run, before the version check, it retries each of them once, under
> the authorizer write lock and against the tenant cache read inside that lock
> (tenant requests hold the lock until their cache update), so a role that
> belongs to a tenant again, for example one re-created after a manual cleanup,
> is dropped instead of deleted. A successful retry changes the Ranger service
> version, so the normal sync follows as today. The first failure of a delete
> is now logged with its exception, a failed retry with one line. Behaviour
> changes only while a role delete keeps failing, which includes a delete that
> Ranger refuses permanently (a leftover role that is referenced from another
> policy, role or security zone): each sync run (default every 600s) then makes
> one Ranger delete call per such role and logs one line, with no reload and no
> {{SetRangerServiceVersion}} request. The set lives on the leader only and is
> cleared when the sync task runs on an OM that is not the ready leader, so a
> leader change or restart before the retry succeeds loses it.
> Covered by two tests in the existing {{TestOMMultiTenantManagerImpl}}.
> {{testRangerSyncRetriesFailedRoleDelete}}: the user role delete fails in the
> first run and in the retry of the second run, which then submits no
> {{SetRangerServiceVersion}} request; the third run deletes the role and the
> fourth issues no further delete. Without the change it fails with
> "tenant2-UserRole left in Ranger".
> {{testRangerSyncKeepsRoleOfRecreatedTenant}}: after a failed delete, the role
> is deleted by hand and the tenant is created again; the next run keeps the
> new tenant's role. With the change the multi-tenancy unit suites in
> {{hadoop-ozone/ozone-manager}} (35 tests) and {{TestOzoneTenantShell}} and
> {{TestMultiTenantVolume}} on a mini cluster (11 tests) pass, checkstyle is
> clean. With the patch the reproduction fails after the second sync run,
> because no role is left.
> Found by code review of the OM multi-tenancy and Ranger sync paths, as part
> of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
