EamesTrinh commented on code in PR #58831:
URL: https://github.com/apache/spark/pull/58831#discussion_r4394884025
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInsertionBase.scala:
##########
@@ -64,25 +65,27 @@ abstract class ResolveInsertionBase extends
Rule[LogicalPlan] {
}
private def renameFieldsInStruct(input: StructType, expected: StructType):
StructType = {
- if (input.length == expected.length) {
+ if (input.length == expected.length || resolveNestedFieldsByPosition) {
val newFields = input.zip(expected).map { case (f1, f2) =>
f1.copy(name = f2.name, dataType = renameFieldsInType(f1.dataType,
f2.dataType))
}
- StructType(newFields)
+ StructType(newFields ++ input.drop(expected.length))
Review Comment:
Thanks for flagging this. I traced both affected paths. For
`ACCEPT_ANY_SCHEMA`, Spark intentionally leaves schema alignment and validation
to the connector, so I don’t think `ResolveInsertionBase` should reject it.
The schema-evolution case is a real issue: the positional rewrite can
produce {x,x,z}, after which `ResolveSchemaEvolution` may add z before
`ResolveOutputRelation` rejects the ambiguous x. Fixing that belongs in
schema-evolution validation/order. Adding a blanket check here would also
change existing V1 error precedence.
I’d prefer to address that separately and keep this PR focused on positional
field alignment. I expanded this PR’s coverage for direct structs, arrays/maps,
wider and narrower inputs, and both configuration states.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
