[
https://issues.apache.org/jira/browse/SPARK-60252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60252:
-----------------------------------
Labels: pull-request-available (was: )
> SCD1 Ignore-null support; Field Reconciliation Data Classes
> -----------------------------------------------------------
>
> Key: SPARK-60252
> URL: https://issues.apache.org/jira/browse/SPARK-60252
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
> Labels: pull-request-available
>
> Introduce case classes and helpers that represent user-data fields during
> SCD1 ignore-null reconciliation.
> SCD1 reconciliation combines a key's CDC events into one row holding the
> key's latest state. It first collapses the incoming microbatch to at most one
> row per key, then reconciles that row against the key's persisted tombstone
> and, on merge, against its target row. Without ignore-null, reconciliation is
> row-wide: one row wins every field. With ignore-null, an upsert authors a
> selected leaf only if it provides a non-null value, so different rows can win
> different fields.
> `Scd1FieldToReconcile` describes one field that is reconciled as a unit. That
> is either a leaf of a column selected for ignore-null, or a whole column
> outside the selection, since every upsert authors all of that column's
> leaves. Reconciling it produces an `Scd1ReconciledField`, which holds the
> reconciled value and the sequence that authored it; the version map is built
> from those sequences.
> Both reconciliations, within the microbatch and against the target row, use
> these same two classes, so they define fields and build the version map the
> same way. The transition is one-way, and neither class changes behavior until
> follow-up tickets wire them in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
