Smallfu666 opened a new pull request, #6178:
URL: https://github.com/apache/datafusion-comet/pull/6178
## Which issue does this PR close?
Closes #6172.
## Rationale for this change
The codegen dispatcher's map writer writes each key with no null check, so a
null key comes out as
the key type's default value. A TRY cast whose key cast fails produces one:
the failing key
becomes null, and Spark's `Cast.castMap` builds the map without rejecting it.
On `main`, over a `MAP<BIGINT, INT>` column holding the key `9999999999`:
```sql
SELECT map_keys(transform_values(try_cast(m AS map<int, int>), (k, v) -> v +
1)) FROM t
```
Spark returns `[1, NULL]` and Comet returns `[1, 0]`. This is on the default
configuration and
gives a wrong answer rather than an error.
## What changes are included in this PR?
- `CometBatchKernelCodegen.canHandle` refuses any bound tree that holds a
TRY cast whose key cast
can fail, so the projection falls back to Spark. Arrow's map format cannot
hold a null key, and
Spark's own readers disagree about one (`map_keys` shows the null, while
`collect`,
`element_at` and a cast to string read the type's default), so only Spark
evaluating the whole
tree matches Spark for every consumer.
- Whether a key cast can fail is decided by a short list of casts that
cannot throw: the same type,
numeric widening, integral to a wide enough decimal, decimal to a wider
decimal, and a few
atomic types to plain string. It is deliberately narrower than Spark's
`Cast.canUpCast`, which
also admits date and timestamp conversions that overflow for extreme
values.
The writer itself is unchanged. Spark's own `GenerateUnsafeProjection` also
writes a map key with
no null check, so storing the default is what Spark does too, and a null key
only diverges when
Comet splits a plan where Spark does not. A Java UDF returning a map with a
null key is one such
case and is not addressed here. Failing on a null key in the writer was
tried and rejected: it
turned several Java UDF queries that match Spark today into errors.
## How are these changes tested?
- `codegen_dispatch_null_map_key.sql`: `transform_values` and `map_filter`
over a narrowing key
cast, a cast of the same map to string, and a date to timestamp key cast
all fall back. A
widening key cast stays in the dispatcher, asserted with `expect_dispatch`.
- `CometCodegenSuite`: a `canHandle` test over each key category, each eval
mode, and a map nested
in an array, a struct field and a map value.
- Both fail on `main`, the SQL file with `[1, 0]` against Spark's `[1,
NULL]`.
- `CometCodegenSuite`, `CometCodegenHOFSuite` and `CometCodegenSourceSuite`
pass, and
`test-compile` passes on the spark-3.4, 3.5, 4.0, 4.1 and 4.2 profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
