geyanggang opened a new issue, #11179:
URL: https://github.com/apache/gravitino/issues/11179
### Version
main branch
### Describe what's wrong
Spark Connector fails to start with `SocketTimeoutException` when calling
`listCatalogsInfo`. The Gravitino Server takes over 3 minutes to respond
because the authorization filtering on `listCatalogs` is too slow.
Root cause: `MetadataAuthzHelper.doFilter()` submits 298 concurrent
`CompletableFuture` tasks (one per catalog) to evaluate permissions. Each task
calls `loadRolePrivilege` which queries the DB for user's roles and policies.
With 1500+ roles in the system and multiple concurrent requests, the thread
pool and DB become saturated.
### Error message and/or stacktrace
org.apache.gravitino.exceptions.RESTException: Error occurred while
processing GET request
at org.apache.gravitino.client.HTTPClient.execute(HTTPClient.java:410)
at
org.apache.gravitino.client.GravitinoMetalake.listCatalogsInfo(GravitinoMetalake.java:202)
at
org.apache.gravitino.spark.connector.catalog.GravitinoCatalogManager.loadRelationalCatalogs(GravitinoCatalogManager.java:79)
at
org.apache.gravitino.spark.connector.plugin.GravitinoDriverPlugin.init(GravitinoDriverPlugin.java:118)
Caused by: java.net.SocketTimeoutException: Read timed out
### How to reproduce
- Metalake with 298 catalogs
- 1100 users, 1500 roles (test env)
- Authorization enabled
- Multiple Spark sessions starting concurrently, each calling
`listCatalogsInfo`
### Additional context
Server log shows `MetadataFilterHelper-ThreadPool` threads doing Casbin
`USE_CATALOG` checks for all 298 catalogs per request. Most return `false`. The
filtering approach is O(all catalogs) instead of O(user's authorized catalogs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
