yuqi1129 commented on code in PR #14045:
URL: https://github.com/apache/gravitino/pull/14045#discussion_r4491055489


##########
server-common/src/main/java/org/apache/gravitino/server/authorization/jcasbin/JcasbinAuthorizer.java:
##########
@@ -1416,102 +1437,352 @@ private Set<Long> versionCheckAndLoadRoles(
 
     List<RoleUpdatedAt> staleRoleVersions = new ArrayList<>();
     for (RoleUpdatedAt rv : roleVersions) {
-      Optional<Long> cachedUpdatedAt = 
loadedRoles.getIfPresent(rv.getRoleId());
-      if (cachedUpdatedAt.isPresent() && cachedUpdatedAt.get() >= 
rv.getUpdatedAt()) {
-        continue;
-      }
-      // A role missing from loadedRoles because its last load was incomplete 
would otherwise be
-      // retried on every single request. Its partially loaded policies stay 
in the enforcer, so
-      // the privileges that did resolve keep working while the backoff is in 
effect.
-      if (!cachedUpdatedAt.isPresent()
-          && partialRoleLoadBackoff.getIfPresent(rv.getRoleId()).isPresent()) {
-        continue;
+      if (needsRolePolicyLoad(rv)) {
+        staleRoleVersions.add(rv);
       }
-      staleRoleVersions.add(rv);
     }
 
     if (staleRoleVersions.isEmpty()) {
       return Collections.emptySet();
     }
 
+    List<Long> leasedRoles = new ArrayList<>();
+    try {
+      return coalesceRolePolicyLoads(metalake, staleRoleVersions, 
requestContext, leasedRoles);
+    } finally {
+      releaseRolePolicyLoadStates(leasedRoles);
+    }
+  }
+
+  private Set<Long> coalesceRolePolicyLoads(
+      String metalake,
+      List<RoleUpdatedAt> staleRoleVersions,
+      AuthorizationRequestContext requestContext,
+      List<Long> leasedRoles) {
+    Map<RolePolicyLoadKey, CompletableFuture<Boolean>> requestLoads = new 
LinkedHashMap<>();
+    List<RolePolicyLoadKey> ownedLoads = new ArrayList<>();
+    try {
+      for (RolePolicyLoadKey key : 
retainRolePolicyLoadStates(staleRoleVersions, leasedRoles)) {
+        CompletableFuture<Boolean> result = new CompletableFuture<>();
+        CompletableFuture<Boolean> existing = rolePolicyLoads.putIfAbsent(key, 
result);
+        requestLoads.put(key, existing == null ? result : existing);
+        if (existing == null) {
+          ownedLoads.add(key);
+        }
+      }
+
+      // Claim every role before doing I/O, and finish every load we own 
before waiting for another
+      // request. Waiting first could deadlock requests owning opposite halves 
of a role batch.
+      List<RolePolicyLoadKey> pendingLoads = new ArrayList<>();
+      // Recheck after claiming: a request can observe a stale marker, be 
delayed, and become the
+      // owner after the previous owner has already completed and removed its 
flight.
+      for (RolePolicyLoadKey key : ownedLoads) {
+        if (needsRolePolicyLoad(key.roleVersion)) {
+          pendingLoads.add(key);
+        } else {
+          requestLoads.get(key).complete(true);
+        }
+      }
+      loadRolePolicies(metalake, pendingLoads, requestContext, requestLoads);
+    } catch (RuntimeException | Error e) {
+      for (RolePolicyLoadKey key : ownedLoads) {
+        requestLoads.get(key).completeExceptionally(e);
+      }
+      throw e;
+    } finally {
+      for (RolePolicyLoadKey key : ownedLoads) {
+        rolePolicyLoads.remove(key, requestLoads.get(key));
+      }
+    }
+
     Set<Long> unreadableRoleIds = new HashSet<>();
+    for (Map.Entry<RolePolicyLoadKey, CompletableFuture<Boolean>> load : 
requestLoads.entrySet()) {
+      if (!awaitRolePolicyLoad(load.getKey(), load.getValue())) {
+        unreadableRoleIds.add(load.getKey().roleVersion.getRoleId());
+      }
+    }
+    return unreadableRoleIds;
+  }
+
+  /**
+   * Captures the clear generation of every stale role and retains its version 
floor, all in one
+   * critical section. The write lock is shared with every enforce() call, so 
it is taken once per
+   * request rather than once per role. Pruning removes map entries before 
publishing its watermark
+   * under the same lock, so a captured generation never observes the gap 
between the two.
+   */
+  private List<RolePolicyLoadKey> retainRolePolicyLoadStates(
+      List<RoleUpdatedAt> staleRoleVersions, List<Long> leasedRoles) {
+    List<RolePolicyLoadKey> keys = new ArrayList<>(staleRoleVersions.size());
+    rolePolicyLock.writeLock().lock();
+    try {
+      for (RoleUpdatedAt rv : staleRoleVersions) {
+        long roleId = rv.getRoleId();
+        RolePolicyLoadKey key = new RolePolicyLoadKey(rv, 
roleClearGenerationWithoutLock(roleId));
+        RolePolicyLoadState state = rolePolicyLoadStates.get(roleId);
+        if (state == null) {
+          rolePolicyLoadStates.put(roleId, new 
RolePolicyLoadState(rv.getUpdatedAt()));
+        } else {
+          state.highestVersion = Math.max(state.highestVersion, 
rv.getUpdatedAt());
+          state.references++;
+        }
+        leasedRoles.add(roleId);
+        keys.add(key);
+      }
+    } finally {
+      rolePolicyLock.writeLock().unlock();
+    }
+    return keys;
+  }
+
+  private void releaseRolePolicyLoadStates(List<Long> leasedRoles) {
+    rolePolicyLock.writeLock().lock();
+    try {
+      for (Long roleId : leasedRoles) {
+        RolePolicyLoadState state = rolePolicyLoadStates.get(roleId);
+        if (--state.references == 0) {
+          rolePolicyLoadStates.remove(roleId);
+        }
+      }
+    } finally {
+      rolePolicyLock.writeLock().unlock();
+    }
+  }
+
+  private boolean needsRolePolicyLoad(RoleUpdatedAt roleVersion) {
+    Optional<Long> cachedUpdatedAt = 
loadedRoles.getIfPresent(roleVersion.getRoleId());
+    if (cachedUpdatedAt.isPresent() && cachedUpdatedAt.get() >= 
roleVersion.getUpdatedAt()) {
+      return false;
+    }
+    // Partially loaded policies stay usable during their existing retry 
backoff, even though
+    // they deliberately have no loadedRoles marker.
+    return cachedUpdatedAt.isPresent()
+        || 
!partialRoleLoadBackoff.getIfPresent(roleVersion.getRoleId()).isPresent();
+  }
+
+  private boolean awaitRolePolicyLoad(RolePolicyLoadKey key, 
CompletableFuture<Boolean> result) {
+    try {
+      // There is no separate waiter timeout. This wait depends on the owner's 
completion,
+      // including its DB pool maxWaitMillis and query execution time (and 
query timeouts, if set).
+      // One stuck owner can delay every request waiting on its roles.
+      return result.get();
+    } catch (InterruptedException e) {
+      // Interrupting a waiter must not cancel the load other requests still 
depend on.
+      Thread.currentThread().interrupt();
+      throw new IllegalStateException(
+          "Interrupted while waiting for role policy load " + 
key.roleVersion.getRoleId(), e);
+    } catch (ExecutionException e) {
+      Throwable cause = e.getCause();
+      if (cause instanceof RuntimeException) {
+        throw (RuntimeException) cause;
+      }
+      if (cause instanceof Error) {
+        throw (Error) cause;
+      }
+      throw new IllegalStateException("Failed to load role policies", cause);
+    }
+  }
+
+  private void loadRolePolicies(
+      String metalake,
+      List<RolePolicyLoadKey> pendingLoads,
+      AuthorizationRequestContext requestContext,
+      Map<RolePolicyLoadKey, CompletableFuture<Boolean>> requestLoads) {
+    if (pendingLoads.isEmpty()) {
+      return;
+    }
     EntityStore entityStore = GravitinoEnv.getInstance().entityStore();
     List<NameIdentifier> roleIdents =
-        staleRoleVersions.stream()
-            .map(rv -> NameIdentifierUtil.ofRole(metalake, rv.getRoleName()))
+        pendingLoads.stream()
+            .map(key -> NameIdentifierUtil.ofRole(metalake, 
key.roleVersion.getRoleName()))
             .collect(Collectors.toList());
     List<RoleEntity> roleEntities;
     try {
       roleEntities = entityStore.batchGet(roleIdents, Entity.EntityType.ROLE, 
RoleEntity.class);
     } catch (Exception e) {
-      LOG.warn("Failed to batch load stale role policies for roleIds {}", 
staleRoleVersions, e);
-      roleEntities = new ArrayList<>();
-    }
-    if (roleEntities == null) {
-      roleEntities = new ArrayList<>();
-    }
-    // Some EntityStore implementations don't support batchGet for ROLE and 
return empty;
-    // fall back to per-role get so policies still load.
-    if (roleEntities.isEmpty()) {
-      roleEntities = new ArrayList<>(staleRoleVersions.size());
-      for (RoleUpdatedAt rv : staleRoleVersions) {
-        try {
-          roleEntities.add(
-              entityStore.get(
-                  NameIdentifierUtil.ofRole(metalake, rv.getRoleName()),
-                  Entity.EntityType.ROLE,
-                  RoleEntity.class));
-        } catch (NoSuchEntityException e) {
-          LOG.debug("Role {} was dropped before its policies could be loaded", 
rv.getRoleId(), e);
-        } catch (Exception e) {
-          LOG.warn("Failed to load role policies for roleId {}", 
rv.getRoleId(), e);
-          unreadableRoleIds.add(rv.getRoleId());
+      LOG.warn("Failed to batch load stale role policies for roles {}", 
roleIdents, e);
+      roleEntities = Collections.emptyList();
+    }
+    Map<Long, RoleEntity> roleEntitiesById = new HashMap<>();
+    if (roleEntities != null) {
+      for (RoleEntity roleEntity : roleEntities) {
+        if (roleEntity != null) {
+          roleEntitiesById.put(roleEntity.id(), roleEntity);
         }
       }
     }
 
-    Map<Long, RoleUpdatedAt> staleRoleVersionById =
-        
staleRoleVersions.stream().collect(Collectors.toMap(RoleUpdatedAt::getRoleId, 
rv -> rv));
-    for (RoleEntity roleEntity : roleEntities) {
-      if (roleEntity == null) {
-        continue;
-      }
-      RoleUpdatedAt rv = staleRoleVersionById.get(roleEntity.id());
-      if (rv == null) {
-        continue;
+    for (RolePolicyLoadKey key : pendingLoads) {
+      CompletableFuture<Boolean> result = requestLoads.get(key);
+      try {
+        result.complete(
+            loadRolePolicy(
+                entityStore,
+                metalake,
+                key,
+                roleEntitiesById.get(key.roleVersion.getRoleId()),
+                requestContext));
+      } catch (RuntimeException | Error e) {
+        // A failed role must not fail another role's waiters, including roles 
still to be loaded.
+        // Settle each owned flight before the caller awaits results and 
propagates its failure.
+        result.completeExceptionally(e);
       }
-      long roleId = rv.getRoleId();
-      long dbUpdatedAt = rv.getUpdatedAt();
-
-      // Resolve every securable object to a metadata id before touching the 
enforcers, so the
-      // critical section below stays free of DB round-trips.
-      Optional<ResolvedRolePolicies> resolution = 
resolveRolePolicies(roleEntity, requestContext);
-      if (!resolution.isPresent()) {
-        unreadableRoleIds.add(roleId);
-        continue;
+    }
+  }
+
+  private boolean loadRolePolicy(
+      EntityStore entityStore,
+      String metalake,
+      RolePolicyLoadKey key,
+      @Nullable RoleEntity roleEntity,
+      AuthorizationRequestContext requestContext) {
+    RoleUpdatedAt rv = key.roleVersion;
+    long roleId = rv.getRoleId();
+    // Stores may not support batchGet for ROLE, or may return only part of a 
batch. Every owned
+    // role needs a concrete outcome so all its waiters receive the same read 
failure, if any.
+    if (roleEntity == null) {
+      try {
+        roleEntity =
+            entityStore.get(
+                NameIdentifierUtil.ofRole(metalake, rv.getRoleName()),
+                Entity.EntityType.ROLE,
+                RoleEntity.class);
+      } catch (NoSuchEntityException e) {
+        LOG.debug("Role {} was dropped before its policies could be loaded", 
roleId, e);
+        invalidateRolePolicies(roleId);
+        return true;
+      } catch (Exception e) {
+        LOG.warn("Failed to load role policies for roleId {}", roleId, e);
+        return false;
       }
-      ResolvedRolePolicies resolved = resolution.get();
+    }
+    if (roleEntity == null) {
+      return false;
+    }
+    if (roleEntity.id() != roleId) {
+      LOG.warn(
+          "Role {} ({}) was read with id {}: it was dropped and recreated 
after its version was"
+              + " probed, so its policies are not loaded and requests on it 
fail closed",
+          roleId,
+          rv.getRoleName(),
+          roleEntity.id());
+      return false;
+    }
 
-      if (!replaceRolePolicies(roleId, dbUpdatedAt, resolved)) {
-        // Another request completed this version (or a newer one) while this 
request was resolving
-        // metadata ids. Its policies and marker are authoritative, even if 
this request's older
-        // resolution result was partial.
-        continue;
+    // Resolve every securable object to a metadata id before touching the 
enforcers, so the
+    // critical section below stays free of DB round-trips.
+    Optional<ResolvedRolePolicies> resolution = 
resolveRolePolicies(roleEntity, requestContext);

Review Comment:
   [Nit] This shared load resolves objects with the owner's request context. 
That context remembers a normalization failure for a whole catalog, for this 
request only (see 
`AuthorizationRequestContext.computeNormalizedMetadataObjectIfAbsent` and the 
"Only this request remembers the failure" comment in 
`JcasbinAuthorizationLookups`).
   
   Timeline:
   1. Request A hits a transient normalization failure for catalog `c1`. A's 
context remembers it.
   2. Role R is stale. A owns its load, and request B joins the flight.
   3. A resolves R with its own context, so every object in `c1` stays 
unresolved. A publishes a partial load. If R has a DENY in `c1`, a 
catalog-level deny guard is installed instead.
   4. B uses this result. For the next 10 s (`PARTIAL_ROLE_LOAD_RETRY_MS`), 
every request skips the reload.
   
   Before this PR, B loaded R with its own context. Its complete load replaced 
A's partial one, because a partial load leaves no loaded marker. Now one 
request's failure becomes everyone's state for up to 10 s.
   
   This fails closed and is unlikely, so it is not a blocker. Please either 
note it in a comment and the PR body, or resolve shared loads with a fresh 
`AuthorizationRequestContext`. The global `metadataIdCache` still deduplicates 
lookups, and a shared result should not depend on one request's snapshot.



##########
server-common/src/main/java/org/apache/gravitino/server/authorization/jcasbin/JcasbinAuthorizer.java:
##########
@@ -235,6 +238,22 @@ public class JcasbinAuthorizer implements 
GravitinoAuthorizer {
    */
   private GravitinoCache<Long, Long> loadedRoles;
 
+  /**
+   * In-progress policy loads shared across requests. Versions and clear 
generations are part of the
+   * key: an older load must not satisfy a request that observed newer 
privileges or a policy
+   * invalidation. Entries are removed on completion, including failures; this 
is not another cache.
+   */
+  private final Map<RolePolicyLoadKey, CompletableFuture<Boolean>> 
rolePolicyLoads =
+      new ConcurrentHashMap<>();
+
+  /**
+   * Highest observed role versions while calls are loading or waiting. A 
partial load has no
+   * loadedRoles marker, so that cache alone cannot stop an older in-progress 
load from overwriting
+   * it. References keep this floor alive until all such calls finish, 
independently of the partial
+   * retry backoff. Guarded by the write lock of {@link #rolePolicyLock}; 
removed when idle.
+   */
+  private final Map<Long, RolePolicyLoadState> rolePolicyLoadStates = new 
HashMap<>();

Review Comment:
   [Nit] The version floor lives only while a participating call is active. The 
PR body says "a late older load cannot overwrite a newer partial policy state", 
which is broader than that.
   
   Timeline:
   1. Request A probes v1 and is then delayed.
   2. Request B loads v2. The load is partial, or the read fails (no backoff 
marker in that case). B finishes and releases the floor.
   3. After the 10 s backoff has expired, A retains with v1. The post-claim 
recheck says a load is needed, and A publishes v1.
   
   This also happens on main, and the next request that probes v2 reloads the 
role, so no code change is needed. Please add the limit to that sentence, for 
example "while the newer call is still active or within its partial backoff".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to