[
https://issues.apache.org/jira/browse/HDDS-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16859:
------------------------------
Attachment: CR-1-ranger-sync-push-current-role-members.patch
> Ranger background sync pushes the role members loaded at the start of the
> attempt instead of the current ones
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16859
> URL: https://issues.apache.org/jira/browse/HDDS-16859
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-1-ranger-sync-push-current-role-members.patch
>
>
> h3. Mechanism
> Each attempt of {{OMRangerBGSyncService.executeOMDBToRangerSync}} first loads
> the Ranger roles and the OM role members
> ({{OMMultiTenantManagerImpl.getAllRolesFromCache}}).
> {{processAllRolesFromOMDB}} then compares them, and for each role that
> differs {{pushOMDBRoleToRanger}} takes the write lock and writes the members
> from that load to Ranger. Tenant requests change Ranger and the tenant cache
> under the same lock, so by the time of the push the loaded members can be
> older than the tenant cache.
> The {{createRole}} branch of {{processAllRolesFromOMDB}} decides from the
> same load. If the tenant is deleted after the load, the branch creates its
> role again.
> h3. Impact
> * Right after a push, the members of the Ranger role can differ from the
> tenant cache: Ranger has the members of the load, not those of the last
> completed tenant request.
> * A role created again for a deleted tenant stays in Ranger. The sync finds
> Ranger roles only through the labeled policies, and the deleted tenant has
> none, so no later run deletes the role. A later tenant create with the same
> name then fails in Ranger on the existing role (code reading, not run against
> Ranger).
> h3. Reproduction
> There is no standalone reproduction. The two regression tests in the attached
> patch cover it and fail on unmodified source. Both use
> {{InMemoryMultiTenantAccessController}} in place of Ranger.
> * {{TestOMMultiTenantManagerImpl.testRangerSyncPushesCurrentRoleMembers}}:
> the OM members of a differing role change after the load. Right after the
> push Ranger must have the current members. Without the change it has
> {{["user2", "seed-user1"]}} instead of {{["seed-user1"]}}.
> *
> {{TestOMMultiTenantManagerImpl.testRangerSyncDoesNotCreateRoleOfTenantDeletedDuringSync}}:
> the tenant is deleted after the sync created its policies again and before
> it creates its roles. After the run Ranger must have neither role. Without
> the change both roles are there.
> h3. Patch
> [^CR-1-ranger-sync-push-current-role-members.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> A new private helper {{OMRangerBGSyncService.getCurrentRoleUsers}} reads the
> role members from {{getAllRolesFromCache}}, or returns the loaded members for
> other {{OMMultiTenantManager}} implementations, as {{loadAllRolesFromOM}}
> does. {{pushOMDBRoleToRanger}} calls it under its write lock and writes the
> members it returns. The {{createRole}} branch calls it under its own write
> lock. Both skip the role when its tenant has been deleted. Without concurrent
> tenant requests the sync writes the same data as before.
> Not changed: the orphan {{deleteRole}} branch also decides from the load. A
> role it deletes by mistake is created again by the next run (code reading,
> not run).
> With the patch the tenant request, manager and in-memory controller unit
> suites pass (25 tests), as do {{TestMultiTenantVolume}} and
> {{TestOzoneTenantShell}} (11 tests), and checkstyle on
> {{hadoop-ozone/ozone-manager}} is clean.
> Found by code review of the OM multi-tenancy and Ranger sync paths, as part
> of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not a
> duplicate: HDDS-7178 (introduced the optimistic read of the load). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
