naveenp2708 opened a new pull request, #57644: URL: https://github.com/apache/spark/pull/57644
### What changes were proposed in this pull request? An AUTO CDC (SCD1) flow appends a reserved metadata column (`__spark_autocdc_metadata`) to its output schema. Because `validateUserSpecifiedSchemas` required the declared schema to exactly equal the inferred schema, a user declaring a schema on an AUTO CDC streaming table was forced to also declare this engine-internal column. This makes two changes so the user schema can describe just the logical data columns: 1. `GraphValidations.validateUserSpecifiedSchemas` accepts a declared schema that differs from the inferred schema only by the reserved AUTO CDC metadata column(s). Any other mismatch still fails, and a schema that does declare the reserved column stays accepted. 2. `DatasetManager.materializeTable` appends the reserved metadata column(s) to the created table's schema when a user schema omits them, so the SCD1 MERGE resolves them at runtime. ### Why are the changes needed? The reserved column is an engine implementation detail; users should not need to know its name or shape. Relaxing validation alone is insufficient because `materializeTable` uses the user schema verbatim, so the target table would be created without the column and the MERGE would fail at runtime with `UNRESOLVED_COLUMN`. ### Does this PR introduce _any_ user-facing change? Yes. A user-specified schema on an AUTO CDC streaming table may now omit the reserved `__spark_autocdc_metadata` column. ### How was this patch tested? `UserSpecifiedSchemaValidationSuite`: data-only schema accepted (implicit and named flows), wrong data columns still rejected, declaring the metadata column still accepted. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
