Siyao Meng created HDDS-16855:
---------------------------------
Summary: Tenant create that fails in Ranger after creating the
admin role leaves the role behind, so the tenant can never be created again
Key: HDDS-16855
URL: https://issues.apache.org/jira/browse/HDDS-16855
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-6-undo-ranger-changes-of-failed-tenant-create.patch,
TestBugMC6OrphanAdminRole.java
h3. Mechanism
{{OMMultiTenantManagerImpl.AuthorizerOp.createTenant}} makes four Ranger calls
in order: create the admin role, create the user role (with the admin role as
delegated admin), create the VolumeAccess policy, create the BucketAccess
policy. When a later call fails, the catch block only wraps the exception into
{{TENANT_AUTHORIZER_ERROR}}. Nothing created so far is removed, although the
Javadoc of the method says "In case of failure: Undo all Ranger State", and
{{OMTenantCreateRequest.preExecute}} relies on the Ranger background sync to
"clean up any leftover policies and roles".
The sync cannot do that for roles without policies.
{{OMRangerBGSyncService.loadAllPoliciesAndRoleNamesFromRanger}} learns about
Ranger roles only from the role ACLs of the policies labeled {{OzoneTenant}}. A
role that no such policy refers to is never seen, never deleted, and the sync
still records the Ranger service version as in sync. {{createRole}} fails when
a role with the same name exists (HDDS-15982 kept creation fail fast and made
only deletes idempotent), so every later create of that tenant stops at the
first {{createRole}}.
Related to HDDS-16188 (open, assigned), which covers recovery of tenant
creation in general. This issue covers only undoing the Ranger changes of a
failed create and does not resolve HDDS-16188: a create that timed out after
Ranger created the entity, and reconciliation with entities that already exist,
stay open there.
h3. Trigger
# {{ozone tenant create tenant1}}. Ranger creates {{tenant1-AdminRole}}, then
the create of {{tenant1-UserRole}} fails once (HTTP 5xx, timeout, Ranger
restart).
# The request fails with {{TENANT_AUTHORIZER_ERROR}} in {{preExecute}}. Nothing
reaches Ratis, OM has no tenant, Ranger keeps the admin role.
# Any number of sync runs leave the role in place. {{ozone tenant delete
tenant1}} fails with {{TENANT_NOT_FOUND}} (argued from
{{OMTenantDeleteRequest.preExecute}}, not run).
# Every retry of {{ozone tenant create tenant1}} fails with
{{TENANT_AUTHORIZER_ERROR}}, cause "Role already exists".
A failure of the VolumeAccess policy create leaves both roles behind in the
same way. A failure of the BucketAccess policy create leaves the VolumeAccess
policy, which refers to both roles, so the sync does clean that case up.
h3. Impact
The tenant id cannot be created again until an operator deletes the leftover
roles in Ranger by hand. No data is lost, and the leftover roles carry no users
and are not referenced by any policy, so they grant nothing. Creating a tenant
requires cluster admin privilege.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugMC6OrphanAdminRole.java]
({{hadoop-ozone/ozone-manager}}) runs the real
{{OMTenantCreateRequest.preExecute}}, the real {{AuthorizerOp}}, the real
{{OMRangerBGSyncService.triggerRangerSyncOnce}} and a real OM DB. Ranger is the
in-memory controller that OM uses with {{ozone.om.tenant.dev.skip.ranger}},
wrapped in a Mockito spy that fails the first create of the user role. Output
on the unmodified source:
{noformat}
create #1: TENANT_AUTHORIZER_ERROR; admin role in Ranger: true, user role in
Ranger: false, tenant in OM DB: false
sync run 1: completed true, admin role in Ranger: true
sync run 2: completed true, admin role in Ranger: true
retry 1: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
retry 2: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
{noformat}
A passing test means the defect is present. Not run against a real Ranger.
h3. Suggested fix
Undo in {{createTenant}} what this call created, as its Javadoc says. Roles
have to be deleted in reverse order of creation: Ranger refuses to delete a
role that a policy, another role or a security zone refers to
({{RoleDBStore.ensureRoleDeleteAllowed}}), and the user role names the admin
role as delegated admin. Only objects whose create returned successfully are
removed, so a role with the same name that existed before (another cluster
sharing the Ranger, a manual role, or a concurrent duplicate create) is never
deleted. Two alternatives were rejected: letting the sync find tenant roles by
name or description is unsafe because Ranger roles are not scoped to a service,
so the sync of one cluster could delete the tenant roles of another cluster
using the same Ranger; adopting an existing role in {{createTenant}} needs the
state aware validation that HDDS-16188 is about.
The undo narrows the problem but does not remove every case: when the undo
deletes fail too, or when a create timed out on the client but succeeded in
Ranger, roles can still be left behind. For example, if the user role create
times out but succeeds, Ranger refuses the undo's delete of the admin role, so
both roles stay, as without the patch, and the refused delete is logged at WARN.
h3. Patch
[^MC-6-undo-ranger-changes-of-failed-tenant-create.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
On failure, {{createTenant}} now deletes the policies and roles it has created,
best effort, policies first and then the user role before the admin role,
logging each delete that fails. The deletes are the idempotent ones from
HDDS-15982. The success path is unchanged. The comment in
{{OMTenantCreateRequest.preExecute}} that sends all leftovers to the sync is
updated accordingly.
Covered by two tests in the existing {{TestOMMultiTenantManagerImpl}}.
{{testCreateTenantUndoesRangerChangesOnFailure}} fails the user role create,
then the BucketAccess policy create, checks that nothing is left in Ranger, and
that the retry succeeds. Without the change it fails with "tenant2-AdminRole
left in Ranger". {{testCreateTenantKeepsExistingRangerRoleOnFailure}} starts
with a user role that existed before the call: the create fails at the user
role, the undo deletes the admin role it created and keeps the existing user
role. Without the change it also fails, because the admin role is left. With
the change the multi-tenancy unit suites in {{hadoop-ozone/ozone-manager}} (35
tests) and {{TestOzoneTenantShell}} and {{TestMultiTenantVolume}} on a mini
cluster (11 tests) pass, checkstyle is clean. With the patch the reproduction
fails at its first check, because the admin role is gone after the failed
create.
Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger
sync paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
