unikdahal commented on PR #5414:
URL:
https://github.com/apache/datafusion-comet/pull/5414#issuecomment-5445817975
@sunchao Thanks again for the detailed review. I went through the remaining
empty-struct paths and pushed another round of fixes.
The latest changes address the outstanding coercion/schema issues:
* `GetArrayStructFields` now derives the output field consistently for both
`data_type()` and `evaluate()`, so parent-null propagation cannot produce
runtime data that disagrees with the declared Arrow type. The regression also
validates the produced array against a `RecordBatch`.
* `from_json` now serializes `expr.dataType` instead of the user-provided
schema, matching Spark's `schema.asNullable` result contract.
* multi-argument `CreateArray` with an empty struct in the element type now
falls back before DataFusion can insert a problematic zero-field struct cast.
* `greatest` / `least` now conservatively fall back for multi-argument
inputs containing an empty struct. This also covers Arrow-only container
metadata differences such as `list<e: struct<>>` vs `list<item: struct<>>`,
which Spark's `DataType` does not expose.
* map lookups (`m[key]` / `element_at(m, key)`) now fall back when the map
key type contains an empty struct, since DataFusion's `map_extract` coerces the
lookup key to the map's exact Arrow key type and can otherwise hit the same
zero-field cast failure.
* added/expanded regressions for the above, including the Arrow field-name
mismatch case and NULL vs `{}` validity through shuffle.
Would appreciate another look when you get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
