EamesTrinh opened a new pull request, #58831:
URL: https://github.com/apache/spark/pull/58831
### What changes were proposed in this pull request?
This PR updates column-list insert resolution to rename the common prefix of
nested struct fields positionally when the input and target structs have
different numbers of fields. Additional input fields retain their names so that
schema evolution can add them.
### Why are the changes needed?
Column-list inserts resolve nested fields by position. However,
`renameFieldsInStruct` previously skipped positional renaming entirely when the
input and target structs had different numbers of fields. With schema
evolution, this caused existing fields to be resolved by name instead.
For example:
```sql
CREATE TABLE t (s STRUCT<x: INT, y: INT>) USING <v2-provider>;
INSERT WITH SCHEMA EVOLUTION INTO t (s)
SELECT named_struct('y', 1, 'x', 2, 'z', 3);
```
The first two fields should map positionally to `x` and `y`, while `z`
should be added through schema evolution.
### Does this PR introduce _any_ user-facing change?
Yes. Previously, the example above produced `{x: 2, y: 1, z: 3}` because the
unequal nested structs were resolved by name. It now produces the positional
result `{x: 1, y: 2, z: 3}`.
### How was this patch tested?
Added a regression test to `DataSourceV2SQLSuiteV1Filter`. The test was
verified to fail before the implementation change and pass afterward using:
```bash
build/sbt "sql/testOnly
org.apache.spark.sql.connector.DataSourceV2SQLSuiteV1Filter -- -z \"Column list
resolves evolved nested fields by position\""
```
### Was this patch authored or co-authored using generative AI tooling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
