yuqi1129 opened a new issue, #13299:
URL: https://github.com/apache/gravitino/issues/13299
### What would you like to be improved?
`GET /api/metalakes/{m}/catalogs/{c}/schemas/{s}/models` issues metadata
store queries per model instead of per request. With PostgreSQL logging every
statement, one list of 2,000 models executes 16,044 statements (2,001 × `SELECT
... FROM schema_meta WHERE catalog_id = ? AND schema_name = ?`, 2,001 × `SELECT
model_id FROM model_meta WHERE schema_id = ? AND model_name = ?`, each in its
own transaction). Listing 2,000 tables in the same metalake takes 56 statements.
Root cause: `MetadataAuthzHelper.filterByExpression` calls `preloadOwner`,
which goes through `OwnerMetaService.batchGetOwner` →
`EntityIdService.getEntityId` once per identifier. `MODEL` and `JOB_TEMPLATE`
are not cacheable (#12374), so every id resolution hits the store twice. The
batch-loaded owners are never used: since #12006 removed relation data from the
entity cache, `batchListEntitiesByRelation` results are discarded, and the
JCasbin authorizer loads owners through its own cache. `preloadToCache` also
issues a batch get for these types whose result the cache drops.
The same shape affects job templates (a metalake id + a template id lookup
per template: 4,064 statements for 500 templates) and model versions, which are
filtered one `filterByExpression` call per version and reload the caller's user
record each time (4,024 statements for 1,000 versions).
### How should we improve?
- Drop `preloadOwner`; it has no consumer.
- Only preload cacheable entity types into the entity cache.
- Register list short-circuits for `MODEL` (owner or `USE_MODEL` at
metalake/catalog/schema) and `JOB_TEMPLATE` (owner or `USE_JOB_TEMPLATE` at
metalake), like tables and schemas already have.
- Filter all versions of a model in one `filterByExpression` call.
Measured after the change (PostgreSQL 16, authorization enabled): 2,000
models 16,044 → 24 statements; 5,000 models 20 statements / 0.13 s; 1,000
versions 4,024 → 20; 500 templates 4,064 → 16.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
