peterxcli opened a new pull request, #5292:
URL: https://github.com/apache/datafusion-comet/pull/5292
## Which issue does this PR close?
Closes #5279.
Closes #5131.
## Rationale for this change
Comet represented Spark `CalendarIntervalType` as Arrow
`IntervalMonthDayNano`. Converting Spark's microseconds to nanoseconds reduced
the valid elapsed-time range by 1,000x, while the native `datafusion-spark`
kernel also coerced `Decimal(18,6)` seconds to `Float64`, losing microsecond
precision.
Spark represents calendar intervals losslessly as separate months, days, and
microseconds. Comet needs the same representation across JVM/native boundaries
and exact microsecond arithmetic in the native kernel.
## What changes are included in this PR?
- Represent `CalendarIntervalType` as a Spark-tagged Arrow struct containing
`months: Int32`, `days: Int32`, and `microseconds: Int64`.
- Preserve that logical type through Arrow conversion, protobuf serde, FFI,
JVM readers/writers, codegen input/output, Scala UDF codegen, and native
execution.
- Replace the `datafusion-spark` `make_interval` wrapper with an exact
`Decimal(18,6)` microsecond kernel with Spark-compatible NULL and ANSI/TRY
overflow behavior.
- Remove the obsolete incompatibility gate, ignored regressions, and unused
`datafusion-spark` dependency.
- Port Spark 4.2.0 `make_interval` boundary and arity cases with source
permalinks.
- Remove the obsolete duplicate codegen-dispatch benchmark case now that
`make_interval` is fully native.
## How are these changes tested?
- `cargo test --manifest-path native/Cargo.toml -p
datafusion-comet-spark-expr preserves_spark_microsecond_range_and_overflow`
- `cargo check --manifest-path native/Cargo.toml -p
datafusion-comet-spark-expr`
- `cargo check --manifest-path native/Cargo.toml -p datafusion-comet`
- `cargo fmt --manifest-path native/Cargo.toml --all -- --check`
- `make core`
- Focused `CometArrowStreamSuite` CalendarInterval round-trip test
- Focused `CometCodegenSuite` CalendarInterval codegen test
- `CometSqlFileTestSuite make_interval`: 6/6 passed, 0 ignored
- `CometSqlFileTestSuite calendar_interval`: 1/1 passed
- Spotless, Scalastyle, and `git diff --check`
- `CometDatetimeExpressionBenchmark` with 1,048,576 rows on Apple M4/JDK 17:
- Spark: 68 ms, 15.5 M rows/s
- Comet: 27 ms, 38.6 M rows/s (2.5x)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
