sunchao commented on code in PR #5867:
URL: https://github.com/apache/datafusion-comet/pull/5867#discussion_r4185029069
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -925,10 +989,13 @@ object CometArrayPosition extends
CometExpressionSerde[ArrayPosition] with Array
}
}
-object CometArraysZip extends CometExpressionSerde[ArraysZip] {
+object CometArraysZip extends CometExpressionSerde[ArraysZip] with
CodegenDispatchFallback {
Review Comment:
[P2] Preserve Spark fallback for calendar-interval outputs. Adding
`CodegenDispatchFallback` also dispatches previously unsupported interval
element types. For a Parquet table `t(s DECIMAL(18,6))` containing
`10000000000.000000`, `SELECT arrays_zip(array(make_interval(0,0,0,0,0,0,s)))
FROM t` produces a valid interval in Spark. The dispatched kernel instead
throws `ArithmeticException: long overflow`: `CometBatchKernelCodegenOutput`
multiplies the interval's `10000000000000000` microseconds by 1000 to write
Arrow nanoseconds. Before this change, `CometArraysZip` declined this type and
the projection ran successfully in Spark. Please retain genuine Spark fallback
for these outputs, including nested calendar intervals, until the bridge can
preserve their full range.
Evidence: A disposable Spark 4.1.3 probe compiled the current serializers
and dispatcher sources and evaluated `ArraysZip(CreateArray(MakeInterval(...,
BoundReference(0, DecimalType(18,6), false))))`. The expression resolved
successfully, `getSupportLevel` returned `Unsupported`, and dispatcher
`canHandle` returned `None` (accepted). With 10000000000.000000 seconds, Spark
UnsafeProjection returned an interval with 10000000000000000 microseconds; the
generated Comet kernel threw `java.lang.ArithmeticException: long overflow` at
`Math.multiplyExact`. The same probe with 1.000000 second succeeded.
Reproduction artifacts:
`/tmp/review5867-verify-l1dg8f_6/IntervalSecondsProbe.scala` and
`interval-seconds.log`. End-to-end SQL execution was not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
