LuciferYang opened a new issue, #14655:
URL: https://github.com/apache/gravitino/issues/14655
### What would you like to be improved?
`BaseCatalog.catalog()` calls
`GravitinoCatalogManager.getGravitinoCatalogInfo(getName())` every time, and
that sends `GET /api/metalakes/{metalake}/catalogs/{catalog}` because the Java
client does not cache catalogs. Nearly every method of the Flink catalog
(`databaseExists`, `getTable`, `listTables`, `createTable`, ...) calls
`catalog()` before its own schema or table request, so each of them costs two
REST calls instead of one.
Flink does not cache table lookups and resolves a table more than once while
planning. On Flink 1.20 and 2.1, planning `SELECT * FROM cat.db.t` calls
`databaseExists` twice and `getTable` twice, which sends 8 requests to the
Gravitino server where 4 are enough. `INSERT INTO cat.db.s SELECT * FROM
cat.db.t` sends 10 instead of 5. Each load also writes an INFO log line.
### How should we improve?
Load the catalog once per Flink catalog instance and reuse it. Flink keeps
the catalog instance for the whole session. Schema and table calls made through
the kept handle still go to the server every time, by catalog name, so they
still see changes made by other clients.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
