andygrove opened a new issue, #6327:
URL: https://github.com/apache/datafusion-comet/issues/6327
### Describe the bug
`create_case_expr` (`native/core/src/execution/planner.rs:4727`) casts every
`THEN` and `ELSE` branch to the type DataFusion's
`get_coerce_type_for_case_expression` picks, and it builds those casts with
`SparkCastOptions::new_without_timezone`. When two timestamp branches differ
only in their Arrow timezone, for example `Timestamp(µs, "Etc/UTC")` and
`Timestamp(µs, "UTC")`, one of them gets a timestamp-to-timestamp cast with an
empty timezone, and `array_with_timezone` panics on
`assert!(!timezone.is_empty())` (`native/spark-expr/src/utils.rs:127`, or `:93`
when the branch is `Timestamp(µs, None)`). `coalesce` goes through the same
path.
The branches of a `TimestampType` expression shouldn't have different Arrow
types in the first place, but today two native expressions produce them:
`timestamp_seconds` in any session, and `date_trunc` in an `Etc/UTC` session.
Both are filed separately under the timezone EPIC. So this is reachable with
the default config.
### Steps to reproduce
On `main` at `764936187`, with the default config, on Spark 3.5 and 4.1:
```sql
CREATE TABLE secs USING parquet AS SELECT id, CAST(id * 3600 AS TIMESTAMP)
AS ts FROM range(4);
SELECT id, CASE WHEN id > 1 THEN timestamp_seconds(id * 3600) ELSE ts END
FROM secs;
```
```sql
CREATE TABLE events USING parquet AS
SELECT * FROM VALUES (TIMESTAMP'2024-01-15T18:30:45Z'),
(TIMESTAMP'2024-06-30T23:30:00Z') AS v(ts);
SET spark.sql.session.timeZone=Etc/UTC;
SELECT coalesce(date_trunc('HOUR', ts), ts) FROM events;
```
Both fail with `CometNativeException: native panic: assertion failed:
!timezone.is_empty()`. Spark returns the expected rows.
### Expected behavior
Spark's result. A cast that only changes a timestamp's timezone label is a
relabel and shouldn't need a session timezone at all.
### Additional context
Fixing the two producers makes this unreachable for now, but the panic stays
one label drift away. Could `create_case_expr` skip casts that only change the
timezone label (or pass `"UTC"`), and could `array_with_timezone` return an
error instead of asserting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
