Andreas Neumann created SPARK-58391:
---------------------------------------
Summary: Validate AutoCDC sequencing, track-history, and
delete-condition drift across incremental runs
Key: SPARK-58391
URL: https://issues.apache.org/jira/browse/SPARK-58391
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
AutoCDC persists a flow's identity-defining configuration on its auxiliary
table and rejects illegal changes across incremental pipeline runs (the remedy
being a full refresh). Today only TWO of the identity-defining dimensions are
protected; three others that are equally identity-defining for SCD
reconciliation have NO drift protection.
Current state (audited on master):
- Persisted on the auxiliary table as table properties: scdType
(spark.pipelines.autocdc.scdType) and keyColumnNames
(spark.pipelines.autocdc.keyColumnNames). Key column dataTypes are validated
via the recorded key fields.
- DatasetManager runs AutoCdcAuxiliaryTable.validateNoScdTypeDrift and
validateNoKeyColumnDrift on every incremental (non-full-refresh) aux-table
evolution, before evolveTable. A mismatch throws
AUTOCDC_INVALID_STATE.SCD_TYPE_DRIFT / KEY_SCHEMA_DRIFT.
Protected:
- SCD type (Type1 <-> Type2): yes.
- Key columns (add/drop/rename) and their dataTypes: yes (nullability/metadata
intentionally tolerated).
NOT protected (not persisted anywhere, not checked):
- sequencing / sequence_by expression (and its resolved type)
- trackHistorySelection (SCD2 TRACK HISTORY columns)
- deleteCondition
- columnSelection (COLUMNS / EXCEPT) -- lower risk, see below
Why this matters:
- Sequencing change is the highest risk: the persisted __START_AT / __END_AT /
recordStartAt values were computed under the old sequencing expression, so
silently changing it reorders history and yields incoherent intervals. The
sequencing TYPE also feeds the persisted __START_AT / __END_AT / metadata
column types, so a type change can mismatch the stored schema.
- trackHistorySelection change (SCD2) alters which transitions open a new
historical record; mixing old and new tracked-column sets corrupts run
coalescing.
- deleteCondition change reinterprets which events are tombstones and can
invert delete semantics against already-reconciled state.
- columnSelection change is more of a schema-evolution concern (the additive
evolve may absorb it) and is lower corruption risk, but is still a definitional
change worth considering.
Proposed work:
1. Persist the missing configuration on the auxiliary table (canonicalized to
stable strings/JSON): the sequencing expression and its resolved DataType, the
trackHistorySelection, and the deleteCondition. Decide the canonical form (e.g.
Catalyst expression sql/toString for sequencing and deleteCondition; the
existing ColumnSelection JSON shape for trackHistorySelection).
2. Add validateNo*Drift checks at the same DatasetManager call site that
already runs the scdType/key validators, throwing AUTOCDC_INVALID_STATE.*
errors consistent with the existing ones, with the full-refresh remedy.
3. Consider whether columnSelection drift should also be rejected or is
adequately handled by additive schema evolution.
4. Tests mirroring AutoCdcScd1KeyDriftSuite for each new dimension.
Note: this is distinct from SPARK-58347 (thread conf.resolver through
applyToSchema), which is a resolution refactor, not drift protection. Surfaced
while auditing AutoCDC drift coverage after the SCD2 end-to-end work
(SPARK-58321).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
