Andreas Neumann created SPARK-58325:
---------------------------------------
Summary: Validate AutoCDC reserved columns post-column-selection
to allow chained replicas
Key: SPARK-58325
URL: https://issues.apache.org/jira/browse/SPARK-58325
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
AutoCDC currently rejects a source change feed that contains a reserved column
BEFORE the user's column selection is applied. Two problems follow:
1. It blocks a legitimate use case. An SCD1 AutoCDC target physically carries
the reserved _cdc_metadata column. Feeding that target into another AutoCDC
flow (e.g. re-replicating an SCD1 replica as SCD2, or vice versa) means the
source contains _cdc_metadata by construction, not by user error. The user's
natural recourse -- COLUMNS * EXCEPT '__spark_autocdc_metadata' -- cannot work
today because requireReservedPrefixAbsentInSourceColumns() runs against
df.schema (the raw source) before column selection.
2. It is inconsistent across the framework columns. The prefixed _cdc_metadata
is guarded pre-selection (in the shared prefix guard,
Flow.requireReservedPrefixAbsentInSourceColumns), while the non-prefixed
__START_AT / __END_AT are (per SPARK-57251) guarded by a separate
framework-column guard and can already be excluded via the column selection.
From the user's perspective all three are engine-owned columns.
Proposal: validate that reserved columns are absent AFTER the column selection
is applied (i.e. against the user-selected schema), not against the raw source.
This:
- Lets a user opt out of any reserved column via EXCEPT, enabling chained
replicas in both directions.
- Unifies the rule for all three framework columns (__START_AT, __END_AT,
_cdc_metadata).
- Remains safe: you can only fail to opt OUT; you can never opt IN (an INCLUDE
naming a reserved column, or no selection over a colliding source, still fails
because the column is still present post-selection). Backward compatible with
the existing 'reject a source _cdc_metadata column with no selection' behavior.
Coordination: this must move BOTH guards post-selection in lockstep to avoid
flipping the inconsistency:
- Flow.requireReservedPrefixAbsentInSourceColumns (prefix guard; shared with
SCD1) -- validate against userSelectedSchema.
- The SPARK-57251 framework-column guard for __START_AT / __END_AT -- same
treatment. SPARK-57251 currently validates pre-selection
(df.schema.fieldNames); align it here or sequence the two changes together.
Note this touches shared SCD1 behavior, so it is intentionally NOT folded into
the SCD2 schema-derivation sub-task (SPARK-58319). Add tests: EXCEPT of each
reserved column from a colliding source is accepted and the engine column is
re-appended; INCLUDE naming a reserved column still fails; a colliding source
with no selection still fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
