AnishMahto opened a new pull request, #58465: URL: https://github.com/apache/spark/pull/58465
### What changes were proposed in this pull request? `SchemaInferenceUtils.diffSchemas` is used to deduce the `TableChanges` that need to be sent to the catalog in order to reflect the latest schema evolved state of datasets declared by a pipeline, during dataset materialization. Today however, the diff does not support nested column level evolution. If some nested column (ex. struct type) has one of its sub-fields changed between runs, but the column's top level name stays the same, it does not emit the correct (or any) column add or delete changes to the catalog. The fix is to recursively diff the schema, traversing every nested column as necessary. This is similar to the existing `ResolveSchemaEvolution.computeSchemaChanges`, except it is not just additive schema evolution - it supports dropping columns in the latest dataset's schema declaration. Two things intentionally not handled by this change: 1. Diff'ing changes to a nested columns `metadata`. That was not respected before, and continues to not be respected after these changes, because DSv2 does not yet support `TableChanges` types for changes to column metadata 2. Respecting order that columns are defined in between the existing and new schemas. Ex. if the exact same column was previously declared as the first column in the schema, but now is declared as the second column in the schema. Instead, existing behavior is preserved; column equality is position invariant. ### Why are the changes needed? Support nested schema evolution in SDP. ### Does this PR introduce _any_ user-facing change? Yes. When a user changes the declared schema for a nested column in their pipeline datasets, the schema changes will actually be propagated to the catalog as a set of `TableChanges`. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Yes, using Claude Opus 4.6. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
