Anish Mahto created SPARK-59787:
-----------------------------------
Summary: SCD1 Ignore-null support; Coalesce incoming microbatch
rows per key
Key: SPARK-59787
URL: https://issues.apache.org/jira/browse/SPARK-59787
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
A microbatch can contain multiple CDC events targeting the same key. With SCD1
ignore-null handling, different events may author different leaves, so
selecting a single winning row is insufficient.
Instead, collapse all microbatch rows for each key into one synthetic row. For
every user-data leaf, select the value from the latest event that authors that
leaf. Upserts determine authorship from their version maps, while deletes
author null for every leaf.
The resulting row contains the winning value and authorship sequence for every
leaf, a reconstructed version map, and the greatest row-wide upsert and delete
sequences from the microbatch.
This ticket only reconciles events within one microbatch. It does not reconcile
against persisted auxiliary-table tombstones or existing target-table state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
