yuqi1129 opened a new pull request, #14643: URL: https://github.com/apache/gravitino/pull/14643
### What changes were proposed in this pull request? - Reload catalog configuration/credentials before physical table loading and scan/write planning; reject changed snapshots and failed reloads, requiring a Spark session restart. New instances bypass the catalog-info cache. - Make timestamp conversion provider-aware and pass preferTimestampNTZ to the JDBC reader; retain MySQL's existing mapping. - Keep predicate, aggregate, Top-N/limit, and offset evaluation in Spark for tables containing string, binary, UUID, or external-type columns. Preserve existing pushdown for compatible numeric/date/timestamp-only tables. - Materialize Trino's source column array once and return NOT_SUPPORTED for schema rename. Document extension hooks and behavior changes. ### Why are the changes needed? A long-lived Spark JDBC catalog retains its original connection configuration and credentials. Zone-free PostgreSQL timestamps are exposed as instants and shift with Spark's session time zone. String operations can use source collation after pushdown, changing query results; pushed binary literals can produce invalid SQL. Separately, Trino repeatedly materializes every column inside its column loop, and unsupported schema rename appears as an internal error. Fixes #14642 ### Does this PR introduce _any_ user-facing change? Affected Spark tables transfer more data: the conservative table-level policy also disables numeric pushdown on a table with a sensitive column. Catalog checks add driver-side server requests and do not revoke already planned/running executor work or cached results. Custom type converters must opt into NTZ handling, and custom table wrappers must pass the catalog guard. The standard JDBC wrapper and PostgreSQL were runtime-tested; other provider-specific adapters require their own validation. No API migration, new dependency, or source schema change is added. ### How was this patch tested? 800 cases discovered with zero failures/errors and one pre-existing disabled Trino test: Spark 3.5.3 full module task (260), targeted Spark 4.0.3 JDBC/catalog-manager cases (27), and Trino 478-479 full module task (513). Both Spark versions ran actual local Spark queries against PostgreSQL 16, verifying string aggregation/order, binary filters, zone-free timestamps (including year 1 and microseconds), and zoned instants under UTC, America/Los_Angeles, and Asia/Shanghai. Unit tests cover stale configuration/credential/recreation refusals, cached-table scan/write guards, numeric-only pushdown, and one column-array read for a 1000-column Trino table. Source SQL and the production JDBC table wrapper ran without a deployed Gravitino REST server. Scoped Spotless, compilation on both Spark versions, and git diff checks passed. Additional credential-rotation and explicit cache-reload regressions passed on both Spark versions. Temporarily restoring the old two-snapshot initialization made the rotation regression fail; the fixed implementation was restored and its tests passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
