andygrove opened a new issue, #6425: URL: https://github.com/apache/datafusion-comet/issues/6425
### Describe the bug Since #5692, Comet dispatches `Invoke` and `StaticInvoke` calls through the JVM codegen dispatcher. That includes calls to DSv2 catalog functions, meaning a `ScalarFunction` with a magic method. Such a function can return a `Decimal` whose scale differs from its declared result type. When it does, the dispatcher writes the unscaled value into the Arrow decimal without rescaling it, so the result is off by a power of ten. The code is in `CometBatchKernelCodegenOutput.scala`, which does `setSafe(idx, d.toUnscaledLong())` with no `changePrecision`. Spark rescales the value with `Decimal.changePrecision` when it writes the row, and returns null if the value doesn't fit. In 1.0.0 these calls fell back to Spark, so this is a regression in 1.1.0. `main` has the same code. ### Steps to reproduce Register a DSv2 function catalog with a scalar function `as_money(int)` whose result type is `DECIMAL(10,2)`, and whose magic method returns `Decimal(i)` (scale 0). Then compare `SELECT i, cat.as_money(i) FROM t` with Comet on and off, for `i` in `(-7), (3), (NULL)`. ### Expected behavior Spark and Comet 1.0.0 return `-7.00, 3.00, NULL`. Comet 1.1.0-rc1 returns `-0.07, 0.03, NULL`. ### Additional context ScalaUDF, the dispatcher's first client, never hit this. `CatalystTypeConverters` applies the declared precision before the value reaches the writer. #5692 already excludes Iceberg's `truncate` on decimals for a related overflow case. The general fix is to apply `changePrecision(precision, scale)` in the dispatcher's decimal output, and write null when it fails, as Spark's `UnsafeRowWriter` does. Found by the 1.1.0 regression audit (#6399) and tracked in #6402. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
