tlfua opened a new issue, #7471:
URL: https://github.com/apache/datafusion-comet/issues/7471
### Describe the bug
Native casts to and from timestamps fail for values whose year is outside
chrono's range (about ±262,143), while Spark supports timestamps up to year
294,247 and returns a result. This happens for TRY casts too, so `try_cast`
raises instead of returning a value.
| Query | Comet |
| --- | --- |
| `CAST(ts AS DATE)` | `CometNativeException: Cast error: Cannot convert
TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports
dates between Jan 1, 262145 BCE and Dec 31, 262143 CE` |
| `CAST(ts AS STRING)` | same error |
| `try_cast(ts AS DATE)` | same error |
| `CAST(dt AS TIMESTAMP)` | native panic at
`spark-expr/src/conversion_funcs/temporal.rs:74:34` |
### Steps to reproduce
As a SQL file test (`spark/src/test/resources/sql-tests/...`), default Spark
4.1 profile:
```sql
statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet
statement
INSERT INTO tsr VALUES
(timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
(timestamp_micros(0), date_from_unix_date(0))
query
SELECT CAST(ts AS DATE) FROM tsr
query
SELECT CAST(ts AS STRING) FROM tsr
query
SELECT try_cast(ts AS DATE) FROM tsr
query
SELECT CAST(dt AS TIMESTAMP) FROM tsr
```
`timestamp_micros(9000000000000000000)` is in year 287,000 or so, and
`date_from_unix_date(100000000)` in year 275,000 or so; both are valid Spark
values.
### Expected behavior
The same results as Spark. Where the native kernel cannot represent the
value, the serde should fall back to Spark rather than fail or panic.
### Additional context
The error paths:
- `pre_timestamp_cast` in `native/spark-expr/src/utils.rs`
(`as_datetime(...).ok_or_else(datetime_cast_err)`), used by timestamp to string
and timestamp to date for every eval mode.
- `timestamp_ntz_to_timestamp` and `cast_timestamp_to_ntz` in the same file,
same pattern.
- Date to timestamp in `native/spark-expr/src/conversion_funcs/temporal.rs`:
`epoch + chrono::Duration::days(d)` panics. The comment above it says chrono's
range covers every `i32` day count, but it does not.
- `test_cast_unsupported_timestamp_to_date` in `conversion_funcs/cast.rs`
asserts the error for `i64::MAX`.
Found while auditing which expressions can safely run as TopK later keys in
https://github.com/apache/datafusion-comet/pull/7457 , which now treats every
timestamp cast as able to fail. Suggested triage: `priority:high` (native
panic), `area:expressions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
