Smallfu666 opened a new issue, #6172: URL: https://github.com/apache/datafusion-comet/issues/6172
## Describe the bug The codegen dispatcher writes a null map key as the key type's default value, so the null is silently lost. This is reachable on the default configuration (`spark.comet.exec.scalaUDF.codegen.enabled=true`) and produces a wrong result rather than an error. ## Steps to reproduce ```sql -- m is MAP<BIGINT, INT> holding the key 9999999999, which does not fit in INT SELECT map_keys(transform_values(try_cast(m AS MAP<INT, INT>), (k, v) -> v + 1)) FROM t ``` Spark returns `[1, NULL]`. Comet returns `[1, 0]`. `map_filter` in place of `transform_values` gives the same result. ## Why The higher-order function is routed through the dispatcher with the `TRY_CAST` inside its tree. Spark's generated code for that cast produces a key array of `[1, NULL]`, because a failing key cast becomes null under TRY and `Cast.castMap` builds the result without rejecting a null key. The dispatcher's map writer in `CometBatchKernelCodegenOutput` then writes each key with no null check, on the assumption that a map key is never null. It reads the null slot through the typed getter and writes `0`. Arrow's map format cannot hold a null key at all, so there is no way to write this result correctly. The dispatcher should refuse such a tree at plan time in `CometBatchKernelCodegen.canHandle`, so the projection falls back to Spark. ## Additional context This blocks the fix for #5995. That issue is the direct `try_cast(m AS MAP<INT, INT>)`, which currently fails natively. Marking it `Unsupported` would route it through the dispatcher and turn that error into this wrong result, so this needs to land first. Spark is not self-consistent here either. Its map holds a null key internally, but `collect`, `element_at` and a cast to string all read it back as `0`, and only `map_keys` shows the null. Falling back to Spark is the only way to match it exactly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
