andygrove opened a new pull request, #6348: URL: https://github.com/apache/datafusion-comet/pull/6348
## Which issue does this PR close? Closes #6333. ## Rationale for this change Spark's `Days` and `Hours` partition transforms are `Unevaluable`: evaluating either one in a projection throws `PARTITION_TRANSFORM_EXPRESSION_NOT_IN_PARTITIONED_BY`. Comet evaluates both natively, but it counted them differently: - `CometDays` cast the timestamp to a date in the session timezone. - `CometHours` divides the raw UTC microseconds. Iceberg's own `days` and `hours` transforms are UTC-based, and so are Comet's native versions of them. So in a non-UTC session, Comet's `days` gave a different day than `hours` and Iceberg imply for the same timestamp. For example, in `America/Los_Angeles`, `days(TIMESTAMP'2024-07-01T03:00:00Z')` returned 19904 (2024-06-30) instead of 19905. ## What changes are included in this PR? - `CometDays` casts a timestamp to a date in UTC instead of in the session timezone. Date inputs are unchanged. - The `days - timestamp input` and `days - literal edge cases` tests in `CometTemporalExpressionSuite` now compare against a UTC day count, `floor(unix_micros(ts) / 86400000000)`, like the `hours` test does. Before, they compared against `unix_date(CAST(ts AS DATE))`, which depends on the session timezone. ## How are these changes tested? - **Old serde, updated tests.** With the old serde, `days - timestamp input` fails against the new baseline in the `America/Los_Angeles` and `Asia/Tokyo` sessions it runs in. - **With this change.** All 35 tests in `CometTemporalExpressionSuite` pass on Spark 4.1, including the `spotless` and `scalastyle` checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
