andygrove opened a new pull request, #4308: URL: https://github.com/apache/datafusion-comet/pull/4308
## Which issue does this PR close? Closes #2013. ## Rationale for this change `from_utc_timestamp` is a commonly used Spark datetime function. The expression maps cleanly to the upstream `datafusion-spark::function::datetime::from_utc_timestamp::SparkFromUtcTimestamp` UDF, so wiring it through gives Comet acceleration with minimal native code. ## What changes are included in this PR? - Add `CometFromUTCTimestamp` serde in `spark/src/main/scala/org/apache/comet/serde/datetime.scala` and register it in `QueryPlanSerde`'s `temporalExpressions` map. - Register the upstream `SparkFromUtcTimestamp` UDF in `native/core/src/execution/jni_api.rs::register_datafusion_spark_function`. - Override `getIncompatibleReasons` to document the one known divergence: arrow's `Tz` parser does not accept Spark's legacy timezone forms (`GMT+1`, `UTC+1`, `PST` and similar). Such timezones surface a native parse error rather than a silent wrong result. IANA names and fixed offsets (`+HH:MM`) are fully supported. - Add a Comet SQL Test at `spark/src/test/resources/sql-tests/expressions/datetime/from_utc_timestamp.sql` covering column and literal arguments for both timestamp and timezone, IANA names, fixed offsets, summer and winter DST rows for `America/Los_Angeles`, and null handling. The file uses a `ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles` to verify the result is independent of session timezone. - Update `docs/source/contributor-guide/spark_expressions_support.md` to mark `from_utc_timestamp` as supported and record dated audit notes for Spark 3.4.3, 3.5.8, and 4.0.1. This PR was scaffolded with the `implement-comet-expression` project skill, which also drove the `audit-comet-expression` follow-up that produced the test matrix above. ## How are these changes tested? - `./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite from_utc_timestamp" -Dtest=none` (passes both `ConfigMatrix` runs locally). - `cd native && cargo clippy --all-targets --workspace -- -D warnings` passes clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
