ninglinLiu opened a new pull request, #7219:
URL: https://github.com/apache/datafusion-comet/pull/7219
## Which issue does this PR close?
Closes #7211.
## Rationale for this change
`CAST(timestamp_ntz AS TIMESTAMP)` fails the task for values outside
chrono's year range (-262144 to 262143), for example `Long.MAX` and `Long.MIN`
micros, with `Cannot convert TimestampMicrosecondType ... to datetime`. Spark
uses `java.time`, which covers the whole `i64` microsecond range, so it returns
a value. Because Spark inserts this cast implicitly, expressions such as
`date_trunc('MICROSECOND', ntz)` fail the same way.
## What changes are included in this PR?
`timestamp_ntz_to_timestamp` now goes through a small `local_to_utc` helper
(for both the microsecond and millisecond paths):
- Values chrono can represent are converted exactly as before.
- For a value outside chrono's range, the zone offset is looked up at the
same local time 100 Gregorian 400-year cycles (40,000 years) closer to the
epoch. Calendars and weekdays repeat every 400 years, so recurring zone rules
give the same offset there, and for fixed offsets and UTC the result is
trivially the same. That offset is then applied to the original value.
- If applying the offset overflows `i64` (for example `Long.MIN` local time
in a zone east of UTC), the conversion raises `long overflow`, the same message
Spark's `Math.subtractExact` raises, instead of the chrono range error.
In a UTC session, `Long.MAX`, `Long.MIN` and `0` now convert to the same
instants as in Spark.
## How are these changes tested?
New unit tests in `native/spark-expr/src/utils.rs`:
- `Long.MAX`, `Long.MIN` and `0` round-trip unchanged in UTC, and a `+02:00`
offset is applied to `Long.MAX`.
- A far-future `America/Los_Angeles` value gets the offset of the same local
time shifted into chrono's range.
- `Long.MIN` in `+02:00` raises `long overflow`.
The first and last tests fail without the change (chrono range error).
`cargo test -p datafusion-comet-spark-expr` (1301 tests), `cargo clippy
--all-targets -- -D warnings` and `cargo fmt` pass.
This PR was prepared with AI assistance and reviewed and tested by the
author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
