Andreas Neumann created SPARK-58452:
---------------------------------------
Summary: Support schema evolution for SCD Type 2
Key: SPARK-58452
URL: https://issues.apache.org/jira/browse/SPARK-58452
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Andreas Neumann
Currently, SCD Type 2 supports only one type of schema evolution: adding a
non-key, non-tracking column. All other types of evolution fail:
# removing a column from the input schema
# adding a new column as a tracking column
# removing a tracking column from the column selection
# removing a column from the tracked column set
This is troublesome, especially because all of these can be triggered
implicitly by a change in the input schema, if column selection or tracking
column selection not configured (that is, defaults to all columns), or
configured as `* EXCEPT ...`.
This is not a problem in SCD Type 1, and SCD Type 2 should match this
capability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
