Smallfu666 opened a new issue, #5995:
URL: https://github.com/apache/datafusion-comet/issues/5995
## Describe the bug
`TRY_CAST` between map types whose key cast can produce null fails in Comet,
where Spark returns
the row with a null key. Verified against `main` at `58ab5f618` and Spark
4.1.3.
## Steps to reproduce
```sql
-- m is MAP<BIGINT, INT> holding a key that does not fit in INT
SELECT TRY_CAST(m AS MAP<INT, INT>) FROM t
```
Spark returns a map whose key is null. Comet fails the query.
## Why Spark accepts the plan
`Cast.canTryCast` gates its map arm on `!forceNullable(fromKey, toKey)`, and
`forceNullable(LongType, IntegerType)` matches none of the earlier arms, so
it falls through to
`case _ => false`. `LongType` is an `IntegralType`, so the `(_:
FractionalType, _: IntegralType)`
arm does not apply. A string key is rejected by `case (_: StringType, _) =>
true`, but a narrowing
integral key is not.
Spark then casts keys and values recursively through `castArray` and builds
the result with
`ArrayBasedMapData(keys, values)`. Nothing on that path rejects a null key,
and under TRY the
inner cast turns the failure into a null.
## Why it reaches Comet
`CometCast.isSupported`'s map arm decides support purely by recursing into
the key and value
casts:
```scala
case (from_map: MapType, to_map: MapType) =>
isSupported(from_map.keyType, to_map.keyType, timeZoneId, evalMode) match {
case Compatible(_, _) =>
isSupported(from_map.valueType, to_map.valueType, timeZoneId, evalMode)
case other => other
}
```
There is no equivalent of Spark's `forceNullable` gate, so the plan is
marked `Compatible` and
runs natively.
## What Comet does
`cast_map_to_map` builds the entries with the target field nullability. With
a non-nullable target
key field and nulls among the cast keys, the construction fails. On `main`
that is
`StructArray::new`, which is `try_new(...).unwrap()`, so it panics inside
arrow and aborts the
executor. The branch in #5227 uses `try_new` and returns
```
Found unmasked nulls for non-nullable StructArray field "key"
```
so #5227 turns the panic into an error. Neither matches Spark.
## Expected behavior
`TRY_CAST` should match Spark and produce a map with a null key.
Note this is specific to `TRY_CAST`. Plain `CAST` agrees with Spark in both
modes: legacy wraps,
and ANSI raises `[CAST_OVERFLOW]`.
## Additional context
Found while narrowing #5227, which fixes field metadata and the target
sorted flag in the same
function. Deliberately kept out of that PR, because this is a `TRY_CAST`
semantics question
rather than a metadata preservation one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
