[
https://issues.apache.org/jira/browse/SPARK-58118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100605#comment-18100605
]
Naveen Kumar Puppala commented on SPARK-58118:
----------------------------------------------
Good point on the two sources of truth. I allowed the schema because a declared
schema still covers things the column selection doesn't, like NOT NULL,
comments, column ordering, and type assertions. Forbidding it would make AUTO
CDC tables the one table type where you can't declare any of that, which felt
like a bigger restriction than the redundancy it removes.
That said, you all own the schema model here. Before I rework it: do the
constraint/comment/ordering cases matter enough on CDC tables to keep allowing
a schema, or should the flow be the single source of truth?
> Allow a user-specified AUTO CDC streaming-table schema to omit the reserved
> metadata column
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-58118
> URL: https://issues.apache.org/jira/browse/SPARK-58118
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Andreas Neumann
> Priority: Major
> Labels: pull-request-available
>
> **Description:**
> An AUTO CDC (SCD Type 1) flow appends a reserved metadata column named
> `__spark_autocdc_metadata` (a struct of `deleteSequence`/`upsertSequence`) to
> its output schema(`AutoCdcMergeFlow.schema`). Because
> `validateUserSpecifiedSchemas` requires the user-declared schema to *exactly
> equal* the inferred schema, a user who declares a schema on an AUTO CDC
> streaming table is currently forced to also declare this internal reserved
> column, e.g.:
> ```
> CREATE STREAMING TABLE t (
> id INT NOT NULL, name STRING, version BIGINT NOT NULL,
> __spark_autocdc_metadata
> STRUCT<deleteSequence:BIGINT,upsertSequence:BIGINT> NOT NULL
> ) FLOW AUTO CDC ...
> ```
> That column is an engine implementation detail; users should not need to know
> its name or shape. Declaring only the logical data columns should be
> accepted, with the engine owning the metadata column.
> Note: simply relaxing the validation is insufficient.
> `DatasetManager.materializeTable` uses `table.specifiedSchema` verbatim as
> the created-table schema, so if the user omits the metadata column the target
> table is created without it and the SCD1 MERGE fails at runtime with
> `UNRESOLVED_COLUMN ... __spark_autocdc_metadata` (verified empirically). The
> fix must both:
> 1. Relax `validateUserSpecifiedSchemas` to permit the declared schema to
> differ from the inferred schema *only* by the reserved AUTO CDC metadata
> column(s), and
> 2. Backfill the reserved metadata column into the effective output schema at
> materialization when a user schema is specified for an AUTO CDC target, so
> the created table actually contains it.
> This is a deliberate semantics decision ("the user schema describes the
> logical table; the engine owns reserved columns") and should be designed with
> the SDP schema-model owners. Subsumes the ergonomic wart where the combined
> form forces declaring the reserved column.
> Note: this came up during the review of SPARK-57402
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
