Anish Mahto created SPARK-59165:
-----------------------------------
Summary: SCD2 Ignore-null support; Introduce version map in
`_cdc_metadata`
Key: SPARK-59165
URL: https://issues.apache.org/jira/browse/SPARK-59165
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
In order to support ignore-null in SCD2, we need to persist additional metadata
per row that tells us which column values were actually authored by the upsert
event that created the row, and which columns were unauthored by the same
upsert event and instead need to inherit a value from a previous row.
This ticket is concerned with simply defining the version map schema/concept,
and adding it to the `_cdc_metadata` per row. The version map will not be used
yet, but will start materializing in both old and new AutoCDC tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
