yuqi1129 commented on PR #14643: URL: https://github.com/apache/gravitino/pull/14643#issuecomment-6659684620
@diqiu50 Addressed the NTZ and provider-scope points in your [comment](https://github.com/apache/gravitino/pull/14643#issuecomment-6658385986): the table wrapper checks whether the actual JDBC dialect compiles `LocalDateTime` into SQL. When it does not (Spark 3.5 PostgreSQL), NTZ tables use Spark evaluation; Spark 4 retains supported timestamp pushdown. Automatic NTZ conversion and the matching reader option now apply only to `jdbc-postgresql`; MySQL, Doris, StarRocks, OceanBase, and Oracle retain their existing defaults. I retained the conservative table-level pushdown policy for this bounded correctness fix. This also disables numeric expressions such as `id = 1` on affected tables and may increase data transfer; that tradeoff is now explicit in the documentation and PR description. Column/expression-level pushdown needs separate work to validate source collation, literal compilation, and interactions with aggregates/Top-N/limits. Validation: Spark 3.5.3 (265 cases), Spark 4.0.3 (260), and Trino 478-479 (513), with zero failures/errors and one pre-existing disabled Trino test. Both Spark versions executed real PostgreSQL 16 queries, including the timestamp-only filter under multiple session time zones. The new test also asserts the Spark 3.5 fallback, retained Spark 4 pushdown, and scan-option safety. Reversing the NTZ safety classification made the regression fail; restoring it passed. No deployed Gravitino REST server was used in these runtime tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
