szehon-ho commented on PR #57644: URL: https://github.com/apache/spark/pull/57644#issuecomment-5483130351
One upgrade-path concern: users were previously required to declare `__spark_autocdc_metadata`, and under case-insensitive AUTO CDC they could declare it with different casing or positioning. The new helper canonicalizes that field in `VirtualTableInput`, but incremental `evolveTable` merges the existing catalog schema first, so the existing field spelling and position win. For example, suppose an existing target has: ```text id, name, version, __SPARK_AUTOCDC_METADATA ``` Its AUTO CDC flow is case-insensitive, so the MERGE can write to that uppercase column. The same graph also contains a case-sensitive downstream materialized view: ```sql CREATE MATERIALIZED VIEW copy AS SELECT * FROM target; ``` After this change: ```text Schema used to plan/materialize copy: id, name, version, __spark_autocdc_metadata Schema read from the existing target during execution: id, name, version, __SPARK_AUTOCDC_METADATA ``` The AUTO CDC MERGE itself succeeds, but the downstream by-name write can fail because its case-sensitive resolver cannot match the two metadata names. Could `appendEngineOwnedReservedFields` instead: - Preserve the spelling and position of one resolver-matching declared reserved field. - Replace its type/nullability with the engine-owned definition. - Append the canonical field only when the declaration omits it. - Reject ambiguous multiple matches. A regression test could pre-create the uppercase legacy target, run AUTO CDC case-insensitively, and have a case-sensitive downstream materialized view read it in the same graph. cc @anew if you have any thoughts on this as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
