sunchao commented on code in PR #4647:
URL: https://github.com/apache/datafusion-comet/pull/4647#discussion_r4103267682
##########
spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegen.scala:
##########
@@ -83,6 +83,7 @@ object CometBatchKernelCodegen extends Logging with
CometExprTraitShim with Come
* supported when their children are.
*/
def isSupportedDataType(dt: DataType): Boolean = dt match {
+ case NullType => true
Review Comment:
[P2] Please add `NullVector` support to `CometScalaUDFCodegen.specFor`
before admitting `NullType` here. `evaluate` calls that method for every input,
but its scalar match still excludes `NullVector`. Passing the new test's
`Coalesce(Cast(BoundReference(0, NullType, true), IntegerType), Literal(42))`
through the dispatcher with two null rows throws `unsupported Arrow vector
NullVector` instead of returning two `42` values. Arrays, maps, and structs
containing `NullType` fail recursively too. This changes a planning-time
fallback into a runtime query failure. Extend `specFor` and test through
`evaluate`, since the added direct `compile` tests bypass this step.
Evidence: A bounded harness compiled the unchanged production
`CometScalaUDFCodegen.scala` with Spark 4.1.3 and Arrow 18.3.0. Calling
`evaluate` with valid serialized expressions reproduced
`UnsupportedOperationException: CometScalaUDFCodegen: unsupported Arrow vector
NullVector` for scalar, array, map, and struct inputs. Unused kernel
dependencies were stubbed and never reached. Verbatim base/head admission
checks returned false/true for all four shapes. Reproduction sources and logs:
`/tmp/comet-4647-null-repro/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
