[
https://issues.apache.org/jira/browse/HDDS-16857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16857:
------------------------------
Attachment: MC-5-ranger-sync-keep-policies-of-unflushed-tenants.patch
> Ranger background sync deletes the default policies of a tenant whose create
> is applied but not yet flushed to RocksDB
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16857
> URL: https://issues.apache.org/jira/browse/HDDS-16857
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-5-ranger-sync-keep-policies-of-unflushed-tenants.patch,
> TestBugMC5SyncDeletesUnflushedTenantPolicies.java
>
>
> h3. Mechanism
> {{OMRangerBGSyncService.processAllPoliciesFromOMDB}} decides which labeled
> Ranger policies to keep by iterating {{getTenantStateTable().iterator()}}.
> That iterator reads RocksDB only and skips the table cache. The role members,
> {{tenantCache}} and {{OMMultiTenantManagerImpl.getTenantVolumeName}} all see
> applied state, including entries still waiting in the double buffer. Every
> labeled policy starts on the delete list, and nothing in the sync waits for
> the flush.
> This gives two symptoms:
> * A tenant create that is applied but not flushed is missing from the keep
> set, so the sync deletes both default policies of that live tenant
> ({{<tenant>-VolumeAccess}} and {{<tenant>-BucketAccess}}).
> * A tenant delete that is applied but not flushed is still in RocksDB. Its
> policies are already gone from Ranger, so the sync tries to recreate them.
> {{attemptToCreateDefaultPolicy}} then gets {{TENANT_NOT_FOUND}} from
> {{getTenantVolumeName}}, and the whole sync run aborts ("Exception during
> Ranger Sync"), so no other tenant is reconciled in that run.
> h3. Trigger
> # An OM admin creates (or deletes) a tenant. The request updates Ranger in
> preExecute, updates the table cache and {{tenantCache}} in
> validateAndUpdateCache, and releases the authorizer lock.
> # The sync on the leader takes its optimistic read after that and finishes
> loading from Ranger before the double buffer has flushed the request.
> With normal flush latency the flush lands well before the sync reaches the
> policy phase, so this needs a slow flush (argued, not measured). Once the
> create is flushed, attempt 2 of the same run or the next run (up to
> {{ozone.om.multitenancy.ranger.sync.interval}}, default 10m, later) recreates
> the policies. Attempt 2 does not wait for the flush, so if it also runs
> before the flush, the policies stay missing until the next run.
> h3. Impact
> For a tenant that every OM reader sees as existing, Ranger has no default
> policies during the window, so its users lose the grants on the tenant volume
> and buckets. That access effect is argued; no Ranger plugin ran in the tests.
> A pending delete blocks the reconcile of all tenants for the same window. No
> OM state is lost, and the sync restores the policies by itself.
> h3. Reproduction
> [^TestBugMC5SyncDeletesUnflushedTenantPolicies.java]
> ({{hadoop-ozone/ozone-manager}}). It runs the real {{OMTenantCreateRequest}}
> (preExecute and validateAndUpdateCache), the manager's own
> {{OMRangerBGSyncService}} and a real {{OmMetadataManagerImpl}}. Ranger is the
> in-tree {{InMemoryMultiTenantAccessController}}, and the flush is done by
> hand. On the unmodified code it passes: after one sync run the tenant is in
> {{tenantCache}} but Ranger has no policies for it, and the policies come back
> only after the flush and a second run. The control test (flush before the
> sync) keeps them. A passing test means the defect is present.
> h3. Patch
> [^MC-5-ranger-sync-keep-policies-of-unflushed-tenants.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{processAllPoliciesFromOMDB}} builds the keep set from the same view as the
> rest of OM. It collects the tenant IDs from {{cacheIterator}} first and then
> from {{keyIterator}}, the same order
> {{OmMetadataManagerImpl.getMultipartUploadKeys}} uses. It reads each tenant
> through the cache and skips a tenant deleted in the cache. Once the double
> buffer is flushed, the result is the same as before. A keep set built only
> from {{tenantCache}} would also fix this, but it reads no table. After a
> snapshot install the previous sync service keeps running on the closed DB
> (HDDS-16853), and today it fails on that DB before its first Ranger write.
> With a {{tenantCache}} keep set it would no longer fail there, and would go
> on to write to Ranger from the tenants of the old DB.
> Covered by {{testRangerSyncKeepsPoliciesOfUnflushedTenant}} and
> {{testRangerSyncSkipsDeletedUnflushedTenant}} in the existing
> {{TestOMMultiTenantManagerImpl}}. Without the change the first finds only the
> policies of the flushed tenant, and the second finds the sync run failed.
> With it, those two tests and the tenant request, manager and in-memory
> controller unit suites pass (25 tests), as do {{TestMultiTenantVolume}} and
> {{TestOzoneTenantShell}} (11 tests), and checkstyle is clean. With the patch
> the reproduction fails, because the policies are kept.
> Found by TLA+ model checking and code review of the OM multi-tenancy and
> Ranger sync paths under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
