[
https://issues.apache.org/jira/browse/SPARK-56890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Mahto resolved SPARK-56890.
---------------------------------
Fix Version/s: 4.2.0
5.0.0
Resolution: Done
> Implement SCD1 Batch Processor; Merge Tombstones onto Microbatch
> ----------------------------------------------------------------
>
> Key: SPARK-56890
> URL: https://issues.apache.org/jira/browse/SPARK-56890
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Anish Mahto
> Priority: Major
> Fix For: 4.2.0, 5.0.0
>
>
> {*}Preamble{*}:
> The SCD type 1 flow is a foreachBatch streaming query on an input
> change-data-feed, and is responsible for reconciling the incoming change data
> onto some target table that follows SCD1 replication semantics.
> SCD1 flows also maintain an "auxiliary" table to keep track of early-arriving
> out-of-order received events state. Each microbatch will need to reconcile
> against this auxiliary table as well, and update the auxiliary table's state
> appropriately for future microbatches.
>
> {*}Merge Tombstones onto Microbatch{*}:
> The auxiliary table produced by an SCD1 flow will [strictly] store tombstones
> accumulated from the flow's change data feed source thus far.
> In SCD1, a tombstone is defined as a delete event that has not been overtaken
> by any upsert event so far (an upsert event whose sequence is geq to the
> delete event's sequence).
> These events/rows are called tombstones because they represent delete events
> that could still be relevant in closing a late-arriving upsert received in
> future microbatches. But we cannot store this type of row in the target
> table, as it would break the contract of what rows an SCD1 compliant replica
> table contains - hence these tombstones are stored in the auxiliary table.
> When a new microbatch is processed, its possible it contains said
> late-arriving upsert events that should be swallowed by some known
> tombstone(s). We need to left anti join the incoming microbatch with the
> auxiliary table on tombstones that do indeed match to the microbatch's
> late-arriving upserts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
