andygrove opened a new pull request, #6353:
URL: https://github.com/apache/datafusion-comet/pull/6353
## Which issue does this PR close?
Closes #6331.
## Rationale for this change
Spark converts between instants and local time using the JVM's timezone
rules (`tzdb.dat`). The version of those rules depends on the JDK build and on
whether TZUpdater has been run. Comet's native kernels use chrono-tz instead,
which compiles its own copy of the IANA database into libcomet: chrono-tz
0.10.4 carries 2025b.
When the two versions disagree about a zone, every local-time operation on
the affected timestamps returns a different answer, and nothing reports it. For
example, with AppleJDK 17.0.10, which ships tzdata 2023c, `hour(ts)` for
`2025-06-15T12:00:00Z` in `America/Asuncion` is 8 in Spark and 9 in Comet.
Matching Spark in every deployment would mean making native code use the
JVM's rules, which is a much larger change. This PR makes the mismatch visible
instead.
## What changes are included in this PR?
- A JNI function, `NativeBase.getTzdataVersion`, returns the database
version compiled into libcomet (`chrono_tz::IANA_TZDB_VERSION`). `chrono-tz`
becomes a direct dependency of `datafusion-comet`. It was already in the
dependency tree through arrow's `chrono-tz` feature, so the only `Cargo.lock`
change is that one dependency line.
- When the native library loads, `NativeBase` compares that version with the
JVM's (`ZoneRulesProvider.getVersions("UTC").lastKey()`) and logs a warning if
they differ. The warning names both versions.
- A new "Timezone Database Versions" section in the datetime compatibility
guide explains the difference and which operations it affects.
- A test in `CometTemporalExpressionSuite` checks that the version is
reported and that the warning is built.
## How are these changes tested?
- The new test and all 36 tests in `CometTemporalExpressionSuite` pass on
Spark 4.1, including the `spotless` and `scalastyle` checks.
- On this machine the suite's log shows the warning with the real versions:
`Comet's native library uses timezone database 2025b, but the JVM uses 2023c.`
- `cargo clippy --all-targets -- -D warnings` is clean for
`datafusion-comet`.
- `prettier --check` passes on the edited docs page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
