[
https://issues.apache.org/jira/browse/SPARK-58325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099607#comment-18099607
]
Andreas Neumann commented on SPARK-58325:
-----------------------------------------
This is an interesting one. It has an element of complexity that we should
account for:
- we currently don't require that all columns that contribute to the
sequencing expression are included in the column selection
- during batch reconciliation, we first compute the sequencing and project it
onto the batch, encoded into the `_cdc_{_}c{_}metadata` column
- at that time, if the table already contains the `_cdc_{_}c{_}metadata`
column, we would have a conflict.
- so we would have to apply the column selection first, to remove that column.
But if that column contributes to the sequencing expression, then that
expression is broken.
- we could, in theory, add the metadata column with a different name first,
then apply the column selection, and then rename the metadata column to its
actual name
- but that seems overkill for a use case that is probably not common.
This use case (AUTO CDC on a change feed produced by a table that was itself an
AUTO CDC table) is valid. For example, a user might replicate an RDB table to
the cloud, and then replicate that again to another availability zone for DR
reasons. And in that case, it would even make sense to reuse the exact same
sequencing, extracting it from the `_cdc_{_}c{_}metadata` column. And that is
exactly the use case that this improvement would enable.
But there is another way to do that: As part of the streaming query that
provides the `source`, the user can extract the original sequence from the
`_cdc_{_}c{_}metadata` column into a new column, and drop
the`_cdc_{_}c{_}metadata` column itself from the input. Because this
work-around exists, I suggest that we do not implement this feature, at lest
for now. If there is overwhelming demand from users, we can always enable it
later.
> Validate AutoCDC reserved columns post-column-selection to allow chained
> replicas
> ---------------------------------------------------------------------------------
>
> Key: SPARK-58325
> URL: https://issues.apache.org/jira/browse/SPARK-58325
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 5.0.0
> Reporter: Andreas Neumann
> Priority: Major
>
> AutoCDC currently rejects a source change feed that contains a reserved
> column BEFORE the user's column selection is applied. Two problems follow:
> 1. It blocks a legitimate use case. An SCD1 AutoCDC target physically carries
> the reserved _cdc_metadata column. Feeding that target into another AutoCDC
> flow (e.g. re-replicating an SCD1 replica as SCD2, or vice versa) means the
> source contains _cdc_metadata by construction, not by user error. The user's
> natural recourse -- COLUMNS * EXCEPT '__spark_autocdc_metadata' -- cannot
> work today because requireReservedPrefixAbsentInSourceColumns() runs against
> df.schema (the raw source) before column selection.
> 2. It is inconsistent across the framework columns. The prefixed
> _cdc_metadata is guarded pre-selection (in the shared prefix guard,
> Flow.requireReservedPrefixAbsentInSourceColumns), while the non-prefixed
> __START_AT / __END_AT are (per SPARK-57251) guarded by a separate
> framework-column guard and can already be excluded via the column selection.
> From the user's perspective all three are engine-owned columns.
> Proposal: validate that reserved columns are absent AFTER the column
> selection is applied (i.e. against the user-selected schema), not against the
> raw source. This:
> - Lets a user opt out of any reserved column via EXCEPT, enabling chained
> replicas in both directions.
> - Unifies the rule for all three framework columns (__START_AT, __END_AT,
> _cdc_metadata).
> - Remains safe: you can only fail to opt OUT; you can never opt IN (an
> INCLUDE naming a reserved column, or no selection over a colliding source,
> still fails because the column is still present post-selection). Backward
> compatible with the existing 'reject a source _cdc_metadata column with no
> selection' behavior.
> Coordination: this must move BOTH guards post-selection in lockstep to avoid
> flipping the inconsistency:
> - Flow.requireReservedPrefixAbsentInSourceColumns (prefix guard; shared with
> SCD1) -- validate against userSelectedSchema.
> - The SPARK-57251 framework-column guard for __START_AT / __END_AT -- same
> treatment. SPARK-57251 currently validates pre-selection
> (df.schema.fieldNames); align it here or sequence the two changes together.
> Note this touches shared SCD1 behavior, so it is intentionally NOT folded
> into the SCD2 schema-derivation sub-task (SPARK-58319). Add tests: EXCEPT of
> each reserved column from a colliding source is accepted and the engine
> column is re-appended; INCLUDE naming a reserved column still fails; a
> colliding source with no selection still fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
