Anish Mahto created SPARK-59188:
-----------------------------------
Summary: `CatalogV2Util.replace` does not recurse through nested
collections
Key: SPARK-59188
URL: https://issues.apache.org/jira/browse/SPARK-59188
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Anish Mahto
`CatalogV2Util.replace` handles paths like `["col", "element", "x"]` by
matching `Seq("element", names @ _*)` against `ArrayType(StructType, _)`, and
similarly for "key" and "value" with `MapType`.
However, these multi-segment cases only match when the child type is a
`StructType`. When the child is another collection -- e.g.
`ArrayType(ArrayType(...))` or `MapType(_, MapType(...), _)` -- the path falls
through to the catch-all and throws _LEGACY_ERROR_TEMP_3223.
Meanwhile, `ResolveSchemaEvolution.computeSchemaChanges` recurses
unconditionally through collections and emits these paths. The two halves
disagree: the change emitter produces paths the change applier cannot handle.
*Example:*
Schema: `col ARRAY<ARRAY<STRUCT<x: INT>>>`
`TableChange` produced by `ResolveSchemaEvolution.computeSchemaChanges`:
`AddColumn(["col", "element", "element", "y"], DoubleType)`
Expected behavior:
The path is applied, producing the new schema `col ARRAY<ARRAY<STRUCT<x: INT,
y: DOUBLE>>>`.
Actual behavior:
`IllegalArgumentException: Cannot find field: element in array<struct<x:int>>`
is thrown
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
