Anish Mahto created SPARK-59164:
-----------------------------------
Summary: Support nested column schema evolution
Key: SPARK-59164
URL: https://issues.apache.org/jira/browse/SPARK-59164
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
`SchemaInferenceUtils.diffSchemas` is used to deduce the `TableChanges` that
need to be sent to the catalog in order to reflect the latest schema evolved
state of datasets declared by a pipeline, during dataset materialization.
Today however, the diff does not support nested column level evolution. If some
nested column (ex. struct type) has one of its sub-fields changed between runs,
but the column's top level name stays the same, it does not emit the correct
(or any) column add or delete changes to the catalog.
The fix is to recursively diff the schema, traversing every nested column as
necessary. This is similar to the existing
`ResolveSchemaEvolution.computeSchemaChanges`, except it is not just additive
schema evolution - it supports dropping columns in the latest dataset's schema
declaration.
Two things intentionally not handled by this change:
1. Diff'ing changes to a nested columns `metadata`. That was not respected
before, and continues to not be respected after these changes, because DSv2
does not yet support `TableChanges` types for changes to column metadata
2. Respecting order that columns are defined in between the existing and new
schemas. Ex. if the exact same column was previously declared as the first
column in the schema, but now is declared as the second column in the schema.
Instead, existing behavior is preserved; column equality is position invariant.
This is a general improvement to SDP's schema evolution infrastructure, but is
being tracked as part of the AutoCDC ignore-null implementation, which will
start requiring AutoCDC's `_cdc_metadata` column to support schema evolution
for a nested version map field.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
