junmuz opened a new pull request, #10594: URL: https://github.com/apache/paimon/pull/10594
### Purpose `CatalogEnvironment#snapshotLoader()` creates a `SnapshotLoader` for every catalog-backed table. Only catalogs with version management (REST) implement `loadSnapshot` / `rollbackTo`. For other catalogs (JDBC, Hive, ...) each `latestSnapshot()` / `latestSnapshotId()` call loads a new catalog just to hit `UnsupportedOperationException` and fall back to the file system. For `JdbcCatalog` this is expensive: loading the catalog opens the JVM-wide shared connection pool (`client-pool-size` connections, kept until the JVM exits) and runs metadata queries. When this happens inside executor tasks, every executor opens its own connections. For example, a Spark job with 500 executors writing to a dynamic bucket table opened 1000 JDBC connections: - Dynamic bucket write: `HashBucketAssigner` -> `PartitionIndex.loadIndex` -> `IndexFileHandler.scanEntries` -> `latestSnapshot()` - Cross-partition upsert: `IndexBootstrap` plans a latest scan on executors - Any scan planned inside a task, e.g. Flink lookup join refreshes This PR only creates the snapshot loader when the catalog supports version management, the same check already used by `catalogTableRollback()` and `catalogSchemaRollback()`. Behavior is unchanged for REST catalogs. Other catalogs read snapshots from the file system directly, which is what they already ended up doing. ### Tests - `CatalogEnvironmentTest#testSnapshotLoaderWithoutVersionManagementReturnsNull`: no snapshot loader and no catalog load without version management. - `JdbcCatalogTest#testSnapshotLoaderDisabledWithoutVersionManagement`: tables of a JDBC catalog have no snapshot loader, and the latest snapshot is still resolved after a commit. - `JdbcCatalogExecutorConnectionsTest`: ships tables to a simulated executor through Java serialization and counts the JDBC connections opened. Without this fix, dynamic bucket assignment and scan planning on the executor each open 2 connections. With the fix, no executor-side operation opens any. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
