sunchao commented on code in PR #4338:
URL: https://github.com/apache/datafusion-comet/pull/4338#discussion_r4104648968
##########
native/core/src/execution/jni_api.rs:
##########
@@ -580,6 +581,7 @@ fn register_datafusion_spark_function(session_ctx:
&SessionContext) {
session_ctx.register_udf(ScalarUDF::new_from_impl(SparkDateSub::default()));
session_ctx.register_udf(ScalarUDF::new_from_impl(SparkFromUtcTimestamp::default()));
session_ctx.register_udf(ScalarUDF::new_from_impl(SparkLastDay::default()));
+
session_ctx.register_udf(ScalarUDF::new_from_impl(SparkMakeDtInterval::default()));
Review Comment:
[P2] Preserve decimal seconds when wiring this UDF. With
`spark.comet.expression.MakeDTInterval.allowIncompatible=true`, a non-folded
`SELECT make_dt_interval(0, 0, 0, s) FROM t`, where `s` is `DECIMAL(18,6)`
containing `99999999999.999999`, should produce `99999999999999999`
microseconds. The registered kernel requires `Float64`, so the planner inserts
a lossy decimal conversion and produces `100000000000000000` microseconds
instead. Neither the input nor the result overflows, so this is separate from
the documented overflow exception/NULL difference. Valid interval values
silently change. Use a decimal-preserving kernel or retain fallback for
affected inputs, and add a column-backed precision regression.
Evidence: Verified the datafusion-spark 53.1.0 crate against the exact-head
Cargo.lock checksum. Compiled its unchanged `make_interval_dt_nano` function
with rustc and applied Arrow 58.3.0's Decimal128-to-Float64 conversion:
unscaled input 99999999999999999 at scale 6 returned Some(100000000000000000).
The negative input showed the symmetric mismatch. Spark 4.1.3 returned exactly
99999999999999999 microseconds. Spark sources across all five supported
profiles use Decimal seconds and `toUnscaledLong`. Exact-head
planner.rs:2837-2904 performs the coercion and inserts the cast. This was a
component reproduction, not an end-to-end Comet run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
