kosiew opened a new pull request, #22980:
URL: https://github.com/apache/datafusion/pull/22980
## Which issue does this PR close?
* Part of #20835
## Rationale for this change
DataFusion's nested struct adaptation logic already supports recursive
schema evolution for several container types (e.g. `List`, `LargeList`, and
list views), but `FixedSizeList` containing evolved `Struct` values was not
handled consistently.
This created a gap where planner-time compatibility checks and runtime
casting did not support additive struct evolution inside `FixedSizeList`
containers. This change adds explicit support for these cases and verifies
planner/runtime parity through regression tests.
## What changes are included in this PR?
* Add `FixedSizeList` handling to `cast_column` for recursive adaptation of
nested struct values when source and target list sizes match.
* Introduce `cast_fixed_size_list_column` to perform recursive casting of
`FixedSizeList` child values.
* Extend `validate_data_type_compatibility` to validate compatibility of
`FixedSizeList` child fields when list sizes are equal.
* Extend `requires_nested_struct_cast` to detect nested struct adaptation
requirements within `FixedSizeList`.
* Preserve existing behavior for mismatched `FixedSizeList` sizes.
* Add a small helper macro to simplify array downcasting and error reporting.
* Add regression and parity tests covering:
* `test_cast_fixed_size_list_struct`
* `test_validate_fixed_size_list_struct_compatibility`
*
`test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected`
* `test_validate_fixed_size_list_struct_size_mismatch_rejected`
* `test_cast_fixed_size_list_struct_all_null`
* `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type`
* Additional `requires_nested_struct_cast` coverage for `FixedSizeList`
cases.
## Are these changes tested?
Yes.
The PR adds regression tests for `FixedSizeList<Struct>` schema evolution,
including:
* Additive nullable nested field evolution.
* All-null and partially-null fixed-size list handling.
* Rejection of newly added non-nullable fields.
* Rejection of incompatible nested type changes.
* Planner/runtime parity verification for incompatible casts.
* Detection of nested struct adaptation requirements within `FixedSizeList`.
## Are there any user-facing changes?
No user-facing changes are intended. This update extends internal nested
schema adaptation support and improves consistency between planner-time
validation and runtime casting for `FixedSizeList<Struct>` types.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content
has been manually reviewed and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
