jarredhj0214 opened a new issue, #12169:
URL: https://github.com/apache/gravitino/issues/12169
### Version
main branch
### Describe what's wrong
We observed an intermittent Gravitino authorization failure where a user
that already had the required privilege was denied once, while later retries
succeeded without any privilege or metadata change.
The failed request was:
```text
Operation: loadCatalog
Metadata: <metalake>.<catalog>
Expression: ANY_USE_CATALOG || ANY(OWNER, METALAKE, CATALOG)
```
The authorization failure was not caused by a missing privilege in the
relational store:
- The user already had a role assigned.
- The assigned role already had `USE_CATALOG` and `USE_SCHEMA` on the
metalake.
- The role privilege had existed for days before the failure.
- No user/role/privilege changes happened around the failure time.
- Retrying the same operation succeeded.
The failure appears to happen at the boundary where
`JcasbinLoadedRolesCache` expires a loaded role entry. The removal listener
currently calls:
```java
allowEnforcer.deleteRole(String.valueOf(roleId));
denyEnforcer.deleteRole(String.valueOf(roleId));
```
However, in jCasbin, `deleteRole(role)` removes both:
```java
removeFilteredGroupingPolicy(1, role);
removeFilteredPolicy(0, role);
```
So expiring a loaded role policy cache entry can also remove `g(userId,
roleId)` grouping policies. If this happens during an authorization request
after `bindUserRoles(userId, roleIds)` but before `enforcer.enforce(...)`, the
current request can miss the user's role link and return `false`, even though
the role policy exists and the DB state is correct.
This looks related to, or a recurrence of, #10978. That issue showed a
similar symptom: after an idle period, one request failed with
`ForbiddenException`, and a later retry succeeded.
### Error message and/or stacktrace
Sanitized logs from the failed request:
```text
2026-07-24T05:02:01.317+08:00 DEBUG [Gravitino-webserver-65]
batchGetAuthSubjectsForUser:137 - <== Total: 3
```
The request did load the user and its roles from DB.
Immediately after that, the loaded role cache entry for the granting role
expired:
```text
2026-07-24T05:02:01.319+08:00 DEBUG [Gravitino-webserver-65]
JcasbinLoadedRolesCache:54 - Removed JCasbin loaded role cache entry,
roleId=<granting_role_id>, cause=EXPIRED
```
Then authorization failed:
```text
2026-07-24T05:02:02.571+08:00 INFO [Gravitino-webserver-65]
jcasbin:137 - Request:
[<user_id>, METALAKE, <metalake_id>, USE_CATALOG] ---> false
2026-07-24T05:02:02.571+08:00 INFO [Gravitino-webserver-65]
jcasbin:139 - Hit Policy: []
2026-07-24T05:02:02.586+08:00 WARN [Gravitino-webserver-65]
GravitinoInterceptionService$MetadataAuthorizationMethodInterceptor:253
- Authorization failed - User: <user>,
Operation: loadCatalog,
Metadata: <metalake>.<catalog>,
Expression: ANY_USE_CATALOG || ANY(OWNER, METALAKE, CATALOG)
2026-07-24T05:02:02.587+08:00 DEBUG [Gravitino-webserver-65]
COMMIT for /api/metalakes/<metalake>/catalogs/<catalog> ... 403 Forbidden
```
A later retry of the same operation succeeded and hit the expected role
policy:
```text
2026-07-24T05:08:23.838+08:00 INFO [Gravitino-webserver-53]
jcasbin:137 - Request:
[<user_id>, METALAKE, <metalake_id>, USE_CATALOG] ---> true
2026-07-24T05:08:23.838+08:00 INFO [Gravitino-webserver-53]
jcasbin:139 - Hit Policy:
[<granting_role_id>, METALAKE, <metalake_id>, USE_CATALOG, allow]
2026-07-24T05:08:23.838+08:00 INFO [Gravitino-webserver-53]
CatalogOperations:272 - Catalog loaded: <metalake>.<catalog>
2026-07-24T05:08:23.838+08:00 DEBUG [Gravitino-webserver-53]
COMMIT for /api/metalakes/<metalake>/catalogs/<catalog> ... 200 OK
```
The relational store state at the time of investigation:
```text
user_id = <user_id>
role_id = <granting_role_id>
role object = METALAKE <metalake>
privilege_names = ["USE_CATALOG","USE_SCHEMA"]
privilege_conditions = ["ALLOW","ALLOW"]
role updated time = several days before the failed request
```
### How to reproduce
This is intermittent, but the suspected sequence is:
1. Enable authorization with `JcasbinAuthorizer`.
2. Create a user and grant it a role.
3. Grant the role `USE_CATALOG` on a metalake.
4. Run a request that requires `ANY_USE_CATALOG`, such as `loadCatalog`, and
confirm it succeeds.
5. Wait until `gravitino.authorization.jcasbin.cacheExpirationSecs` expires
for the loaded role cache entry, or simulate `JcasbinLoadedRolesCache`
expiration/invalidation.
6. Run the same request at the expiration boundary.
The race window is:
1. `loadUserRoles` / `loadGroupRoles` calls `bindUserRoles(userId, roleIds)`.
2. `versionCheckAndLoadRoles` calls `loadedRoles.getIfPresent(roleId)`.
3. The Caffeine entry expires and the removal listener calls
`deleteRole(roleId)`.
4. jCasbin deletes `g(userId, roleId)` via `removeFilteredGroupingPolicy(1,
role)`.
5. Role policies may be reloaded, but the user-role grouping is not
necessarily rebound afterward.
6. `enforcer.enforce(userId, metadataType, metadataId, privilege)` can
return `false` with `Hit Policy: []`.
7. The next request re-runs `bindUserRoles`, so the issue disappears.
### Additional context
Current code in `JcasbinLoadedRolesCache`:
```java
if (roleId != null && cause != RemovalCause.REPLACED) {
allowEnforcer.deleteRole(String.valueOf(roleId));
denyEnforcer.deleteRole(String.valueOf(roleId));
}
```
jCasbin `Enforcer.deleteRole` in 1.99.0:
```java
public void deleteRole(String role) {
removeFilteredGroupingPolicy(1, role);
removeFilteredPolicy(0, role);
}
```
The loaded role cache seems to represent whether a role's permission
policies have been loaded, not whether user-role or group-role relations are
valid. Therefore, cache eviction should only remove role permission policies
(`p` rules), not grouping policies (`g` rules).
Suggested fix:
```java
if (roleId != null && cause != RemovalCause.REPLACED) {
String roleIdStr = String.valueOf(roleId);
allowEnforcer.removeFilteredPolicy(0, roleIdStr);
denyEnforcer.removeFilteredPolicy(0, roleIdStr);
}
```
This matches the existing `JcasbinAuthorizer.clearRolePolicies(roleId)`
behavior:
```java
private void clearRolePolicies(long roleId) {
String roleIdStr = String.valueOf(roleId);
allowEnforcer.removeFilteredPolicy(0, roleIdStr);
denyEnforcer.removeFilteredPolicy(0, roleIdStr);
}
```
This would keep the responsibilities separated:
- `loadedRoles` eviction manages role permission policies.
- `userRoleCache` / `groupRoleCache` and `bindUserRoles` manage grouping
relations.
A regression test could simulate loaded role cache expiration after
user-role grouping is bound and verify that a user with a valid role still
authorizes successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
