HyukjinKwon commented on code in PR #59299:
URL: https://github.com/apache/spark/pull/59299#discussion_r4257754739
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala:
##########
@@ -113,7 +119,12 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
targetTable.schema,
sourceSchema,
isByName)
- filterSupportedChanges(targetTable, candidateChanges)
+ val supportedChanges = filterSupportedChanges(targetTable,
candidateChanges)
+ if (isByName && conf.schemaEvolutionPreserveColumnOrder) {
Review Comment:
**Nit (P3):** Nothing currently fails if the `isByName &&` here is dropped.
All six conf-on tests in `InsertIntoSchemaEvolutionTests` pass `byName = true`,
and the existing by-position evolution tests never enable the conf.
`computeSchemaChangesByPosition` names trailing extras after their source
fields. For target `(id, data)` and a by-position source `(a, b, c)`,
`newFieldPosition` would find no anchor for `c` and return `FIRST`, giving `(c,
id, data)` with the values shifted. A conf-on by-position insert whose source
names differ from the target's would cover this. With matching names, the
regression degrades to `AFTER <last>` and stays invisible.
**Verification:**
- **Regression:** A conf-on by-position insert with mismatched source names
keeps the target columns first, appends the extra, and keeps row values aligned.
- **Compatibility:** Existing by-name preserve-order and by-position
evolution INSERT cases keep their expected results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
