[
https://issues.apache.org/jira/browse/SPARK-58325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andreas Neumann resolved SPARK-58325.
-------------------------------------
Resolution: Won't Do
> Validate AutoCDC reserved columns post-column-selection to allow chained
> replicas
> ---------------------------------------------------------------------------------
>
> Key: SPARK-58325
> URL: https://issues.apache.org/jira/browse/SPARK-58325
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Priority: Major
>
> AutoCDC currently rejects a source change feed that contains a reserved
> column BEFORE the user's column selection is applied. Two problems follow:
> 1. It blocks a legitimate use case. An SCD1 AutoCDC target physically carries
> the reserved _cdc_metadata column. Feeding that target into another AutoCDC
> flow (e.g. re-replicating an SCD1 replica as SCD2, or vice versa) means the
> source contains _cdc_metadata by construction, not by user error. The user's
> natural recourse -- COLUMNS * EXCEPT '__spark_autocdc_metadata' -- cannot
> work today because requireReservedPrefixAbsentInSourceColumns() runs against
> df.schema (the raw source) before column selection.
> 2. It is inconsistent across the framework columns. The prefixed
> _cdc_metadata is guarded pre-selection (in the shared prefix guard,
> Flow.requireReservedPrefixAbsentInSourceColumns), while the non-prefixed
> __START_AT / __END_AT are (per SPARK-57251) guarded by a separate
> framework-column guard and can already be excluded via the column selection.
> From the user's perspective all three are engine-owned columns.
> Proposal: validate that reserved columns are absent AFTER the column
> selection is applied (i.e. against the user-selected schema), not against the
> raw source. This:
> - Lets a user opt out of any reserved column via EXCEPT, enabling chained
> replicas in both directions.
> - Unifies the rule for all three framework columns (__START_AT, __END_AT,
> _cdc_metadata).
> - Remains safe: you can only fail to opt OUT; you can never opt IN (an
> INCLUDE naming a reserved column, or no selection over a colliding source,
> still fails because the column is still present post-selection). Backward
> compatible with the existing 'reject a source _cdc_metadata column with no
> selection' behavior.
> Coordination: this must move BOTH guards post-selection in lockstep to avoid
> flipping the inconsistency:
> - Flow.requireReservedPrefixAbsentInSourceColumns (prefix guard; shared with
> SCD1) -- validate against userSelectedSchema.
> - The SPARK-57251 framework-column guard for __START_AT / __END_AT -- same
> treatment. SPARK-57251 currently validates pre-selection
> (df.schema.fieldNames); align it here or sequence the two changes together.
> Note this touches shared SCD1 behavior, so it is intentionally NOT folded
> into the SCD2 schema-derivation sub-task (SPARK-58319). Add tests: EXCEPT of
> each reserved column from a colliding source is accepted and the engine
> column is re-appended; INCLUDE naming a reserved column still fails; a
> colliding source with no selection still fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
