AnhTtis opened a new pull request, #58084:
URL: https://github.com/apache/spark/pull/58084
### What changes were proposed in this pull request?
This PR fixes an inconsistency in
`ResolveInsertionBase.createProjectForByNameQuery` when resolving nested
structs during `INSERT INTO table (column_list)` queries.
Currently, `ResolveInsertionBase` renames struct fields positionally to
align with the target table schema. However, `createProjectForByNameQuery` and
`renameFieldsInStruct` only handle top-level `StructType` and do not traverse
into `ArrayType` or `MapType`. As a result, structs nested inside collections
retain their original field names and are later resolved by name instead of
position in `TableOutputResolver`.
This patch:
1. Generalizes `renameFieldsInStruct` into a recursive method
`renameFieldsInDataType` in
`ResolveInsertionBase.scala` that e`, `ArrayType`, and `MapType`.
2. Updates `createProjectForByNameQuery` to apply `renameFieldsInDataType`,
ensuring positional
rename projection is propagated ac, `ARRAY<STRUCT>`, `MAP<KEY,STRUCT>`).
### Why are the changes needed?
For `INSERT INTO table (column_lis adhere to SQL positional resolution
semantics. The current behavior causes a silent data corruption risk where
values inside nested structs in arrays/maps are assigned to incorrect fields if
the source and target field order differs.
### Does this PR introduce _any_ user-facing change?
Yes. Fixes a bug where structs nested inside arrays and maps were resolved
by name instead of by
position during `INSERT INTO table
### How was this patch tested?
Verified and reproduced directly using PySpark on Spark 4.0.3:
```python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appNa).getOrCreate()
# 1. Create a target table with di, and map of structs
spark.sql("DROP TABLE IF EXISTS target_nested_test")
spark.sql("""
CREATE TABLE target_nested_test (
s STRUCT<x: INT, y: INT>,
arr ARRAY<STRUCT<x: INT, y: INT>>,
m MAP<STRING, STRUCT<x: INT, y: INT>>
) USING parquet
""")
# 2. Insert data where the source struct fields are ordered ('y', 'x')
instead of target's ('x',
'y')
spark.sql("""
INSERT INTO target_nested_test (s,
SELECT
named_struct('y', 20, 'x', 10),
array(named_struct('y', 20, 'x', 10)),
map('k', named_struct('y', 20, 'x', 10))
""")
# 3. Query JSON representation of each field
spark.sql("""
SELECT
to_json(s) AS direct_struct,
to_json(arr[0]) AS array_struct,
to_json(m['k']) AS map_struct
FROM target_nested_test
""").show(truncate=False)
- Before this fix (Actual Output o
+---------------+---------------+---------------+
|direct_struct |array_struct |map_struct |
+---------------+---------------+---------------+
|{"x":20,"y":10}|{"x":10,"y":20}|{
+---------------+---------------+---------------+
(Only direct struct was resolved pructs were incorrectly resolved byname).
- Expected & Result with this fix:
All three struct fields consistently resolve positionally to {"x":20,"y":10}.
Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
