Eliaaazzz opened a new pull request, #58517:
URL: https://github.com/apache/spark/pull/58517
### What changes were proposed in this pull request?
`CatalogV2Util.replace` resolves multi-segment paths through collections
with patterns like `case (Seq("element", names @ _*), array @
ArrayType(elementStruct: StructType, _))`, so the traversal only continues when
the child of a map key/value or array element is a struct. When the child is
another collection, e.g. `ARRAY<ARRAY<STRUCT<...>>>`, the path falls through to
the catch-all and throws `_LEGACY_ERROR_TEMP_3223`.
This PR relaxes the three multi-segment cases to accept any
`MapType`/`ArrayType` child and moves the descent into a new `replaceNested`
helper that recurses through nested maps, arrays and structs. Terminal
`key`/`value`/`element` segments behave exactly as in the existing
single-segment cases, structs delegate back to `replace`, and an unresolvable
segment still raises `_LEGACY_ERROR_TEMP_3223` (or is a no-op with `ifExists`).
### Why are the changes needed?
`ResolveSchemaEvolution.computeSchemaChanges` recurses through collections
unconditionally and emits paths such as `AddColumn(["col", "element",
"element", "y"])` for `col ARRAY<ARRAY<STRUCT<x: INT>>>`, but
`CatalogV2Util.applySchemaChanges` cannot apply them:
```
IllegalArgumentException: Cannot find field: element in array<struct<x:int>>
```
so schema evolution fails on any change nested below two levels of
collections. The change emitter and the change applier now agree on which paths
are valid.
### Does this PR introduce _any_ user-facing change?
Yes: schema changes addressing fields nested in collections of collections
(e.g. adding a column to a struct inside an array of arrays, or changing the
type of a map key inside an array) now apply instead of throwing. Previously
valid paths behave as before.
### How was this patch tested?
New tests in `CatalogV2UtilSuite`: adding a column to a struct in an array
of arrays, updating a column type in a struct in a map of arrays, updating a
map key type nested in an array, and an invalid nested path still failing.
```
build/sbt "catalyst/testOnly
org.apache.spark.sql.connector.catalog.CatalogV2UtilSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
