[
https://issues.apache.org/jira/browse/SPARK-58391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho reassigned SPARK-58391:
---------------------------------
Assignee: Andreas Neumann
> Validate AutoCDC sequencing, track-history, and delete-condition drift across
> incremental runs
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-58391
> URL: https://issues.apache.org/jira/browse/SPARK-58391
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Assignee: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> AutoCDC persists a flow's identity-defining configuration on its auxiliary
> table and rejects illegal changes across incremental pipeline runs (the
> remedy being a full refresh). Today only TWO of the identity-defining
> dimensions are protected; three others that are equally identity-defining for
> SCD reconciliation have NO drift protection.
> Current state (audited on master):
> - Persisted on the auxiliary table as table properties: scdType
> (spark.pipelines.autocdc.scdType) and keyColumnNames
> (spark.pipelines.autocdc.keyColumnNames). Key column dataTypes are validated
> via the recorded key fields.
> - DatasetManager runs AutoCdcAuxiliaryTable.validateNoScdTypeDrift and
> validateNoKeyColumnDrift on every incremental (non-full-refresh) aux-table
> evolution, before evolveTable. A mismatch throws
> AUTOCDC_INVALID_STATE.SCD_TYPE_DRIFT / KEY_SCHEMA_DRIFT.
> Protected:
> - SCD type (Type1 <-> Type2): yes.
> - Key columns (add/drop/rename) and their dataTypes: yes
> (nullability/metadata intentionally tolerated).
> NOT protected (not persisted anywhere, not checked):
> - sequencing / sequence_by expression (and its resolved type)
> - trackHistorySelection (SCD2 TRACK HISTORY columns)
> - deleteCondition
> - columnSelection (COLUMNS / EXCEPT) -- lower risk, see below
> Why this matters:
> - Sequencing change is the highest risk: the persisted __START_AT / __END_AT
> / recordStartAt values were computed under the old sequencing expression, so
> silently changing it reorders history and yields incoherent intervals. The
> sequencing TYPE also feeds the persisted __START_AT / __END_AT / metadata
> column types, so a type change can mismatch the stored schema.
> - trackHistorySelection change (SCD2) alters which transitions open a new
> historical record; mixing old and new tracked-column sets corrupts run
> coalescing.
> - deleteCondition change reinterprets which events are tombstones and can
> invert delete semantics against already-reconciled state.
> - columnSelection change is more of a schema-evolution concern (the additive
> evolve may absorb it) and is lower corruption risk, but is still a
> definitional change worth considering.
> Proposed work:
> 1. Persist the missing configuration on the auxiliary table (canonicalized to
> stable strings/JSON): the sequencing expression and its resolved DataType,
> the trackHistorySelection, and the deleteCondition. Decide the canonical form
> (e.g. Catalyst expression sql/toString for sequencing and deleteCondition;
> the existing ColumnSelection JSON shape for trackHistorySelection).
> 2. Add validateNo*Drift checks at the same DatasetManager call site that
> already runs the scdType/key validators, throwing AUTOCDC_INVALID_STATE.*
> errors consistent with the existing ones, with the full-refresh remedy.
> 3. Consider whether columnSelection drift should also be rejected or is
> adequately handled by additive schema evolution.
> 4. Tests mirroring AutoCdcScd1KeyDriftSuite for each new dimension.
> Note: this is distinct from SPARK-58347 (thread conf.resolver through
> applyToSchema), which is a resolution refactor, not drift protection.
> Surfaced while auditing AutoCDC drift coverage after the SCD2 end-to-end work
> (SPARK-58321).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
