[
https://issues.apache.org/jira/browse/SPARK-58118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100254#comment-18100254
]
Naveen Kumar Puppala commented on SPARK-58118:
----------------------------------------------
Opened a PR for this: https://github.com/apache/spark/pull/57644
It does the two things from my earlier comment - relaxes the schema validation
to allow omitting the reserved column, and backfills it at materialization so
the table gets created with it. Kept it to SCD1 for now. Went ahead and opened
it since the approach seemed straightforward, happy to adjust if you had
something different in mind.
> Allow a user-specified AUTO CDC streaming-table schema to omit the reserved
> metadata column
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-58118
> URL: https://issues.apache.org/jira/browse/SPARK-58118
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> **Description:**
> An AUTO CDC (SCD Type 1) flow appends a reserved metadata column named
> `__spark_autocdc_metadata` (a struct of `deleteSequence`/`upsertSequence`) to
> its output schema(`AutoCdcMergeFlow.schema`). Because
> `validateUserSpecifiedSchemas` requires the user-declared schema to *exactly
> equal* the inferred schema, a user who declares a schema on an AUTO CDC
> streaming table is currently forced to also declare this internal reserved
> column, e.g.:
> ```
> CREATE STREAMING TABLE t (
> id INT NOT NULL, name STRING, version BIGINT NOT NULL,
> __spark_autocdc_metadata
> STRUCT<deleteSequence:BIGINT,upsertSequence:BIGINT> NOT NULL
> ) FLOW AUTO CDC ...
> ```
> That column is an engine implementation detail; users should not need to know
> its name or shape. Declaring only the logical data columns should be
> accepted, with the engine owning the metadata column.
> Note: simply relaxing the validation is insufficient.
> `DatasetManager.materializeTable` uses `table.specifiedSchema` verbatim as
> the created-table schema, so if the user omits the metadata column the target
> table is created without it and the SCD1 MERGE fails at runtime with
> `UNRESOLVED_COLUMN ... __spark_autocdc_metadata` (verified empirically). The
> fix must both:
> 1. Relax `validateUserSpecifiedSchemas` to permit the declared schema to
> differ from the inferred schema *only* by the reserved AUTO CDC metadata
> column(s), and
> 2. Backfill the reserved metadata column into the effective output schema at
> materialization when a user schema is specified for an AUTO CDC target, so
> the created table actually contains it.
> This is a deliberate semantics decision ("the user schema describes the
> logical table; the engine owns reserved columns") and should be designed with
> the SDP schema-model owners. Subsumes the ergonomic wart where the combined
> form forces declaring the reserved column.
> Note: this came up during the review of SPARK-57402
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
