[
https://issues.apache.org/jira/browse/SPARK-58118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100303#comment-18100303
]
Andreas Neumann commented on SPARK-58118:
-----------------------------------------
Thanks, I will take a look at the PR.
At this time, i am not sure what the right thing to do is here: in this case,
we have two sources of truth: a table declaration with a schema, and an auto
cdc flow with a column selection. At the least, these two are redundant, but
they could be conflicting. I wonder whether, instead of allowing the table
declaration to specify the schema with or without the metadata column, it would
be wiser to simply not allow a schema for tables that have auto cdc flows.
[~joseph.torres] [~anishmahto] [~szehon] what do you think?
> Allow a user-specified AUTO CDC streaming-table schema to omit the reserved
> metadata column
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-58118
> URL: https://issues.apache.org/jira/browse/SPARK-58118
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> **Description:**
> An AUTO CDC (SCD Type 1) flow appends a reserved metadata column named
> `__spark_autocdc_metadata` (a struct of `deleteSequence`/`upsertSequence`) to
> its output schema(`AutoCdcMergeFlow.schema`). Because
> `validateUserSpecifiedSchemas` requires the user-declared schema to *exactly
> equal* the inferred schema, a user who declares a schema on an AUTO CDC
> streaming table is currently forced to also declare this internal reserved
> column, e.g.:
> ```
> CREATE STREAMING TABLE t (
> id INT NOT NULL, name STRING, version BIGINT NOT NULL,
> __spark_autocdc_metadata
> STRUCT<deleteSequence:BIGINT,upsertSequence:BIGINT> NOT NULL
> ) FLOW AUTO CDC ...
> ```
> That column is an engine implementation detail; users should not need to know
> its name or shape. Declaring only the logical data columns should be
> accepted, with the engine owning the metadata column.
> Note: simply relaxing the validation is insufficient.
> `DatasetManager.materializeTable` uses `table.specifiedSchema` verbatim as
> the created-table schema, so if the user omits the metadata column the target
> table is created without it and the SCD1 MERGE fails at runtime with
> `UNRESOLVED_COLUMN ... __spark_autocdc_metadata` (verified empirically). The
> fix must both:
> 1. Relax `validateUserSpecifiedSchemas` to permit the declared schema to
> differ from the inferred schema *only* by the reserved AUTO CDC metadata
> column(s), and
> 2. Backfill the reserved metadata column into the effective output schema at
> materialization when a user schema is specified for an AUTO CDC target, so
> the created table actually contains it.
> This is a deliberate semantics decision ("the user schema describes the
> logical table; the engine owns reserved columns") and should be designed with
> the SDP schema-model owners. Subsumes the ergonomic wart where the combined
> form forces declaring the reserved column.
> Note: this came up during the review of SPARK-57402
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
