AnishMahto opened a new pull request, #58505: URL: https://github.com/apache/spark/pull/58505
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-59165-scd2-version-map-schema...SPARK-59208-introduce-ignore-null-change-arg ### What changes were proposed in this pull request? `ChangeArgs` holds all of the user-specified configurations (arguments) that should apply to their AutoCDC flow by execution time. Ignore-null is a new AutoCDC argument that allows users to specify a set of columns that AutoCDC should treat as unauthored by an event when they receive a null-value in that event. We need to represent this selection in `ChangeArgs` so that AutoCDC batch processors can later reference it, and validate that the columns selected indeed exist and are eligible for selection. Ex. a key column or reserved AutoCDC column may never be specified as ignore-null. ### Why are the changes needed? To support the ignore-null feature for AutoCDC SCD2 (as per the [approved SPIP](https://docs.google.com/document/d/10gtGoUOJvY_XoXpJuJ8LGJWPLMkdpgzseiMcKnjDloo/edit?tab=t.0#heading=h.9g6a5f8v6xig)), we need to represent the user-specified ignore null column selection during pipeline analysis and execution. ### Does this PR introduce _any_ user-facing change? No. We introduce internal plumbing and representation for ignore-null selection, but no user-facing API exists yet to specify the ignore-null selection. ### How was this patch tested? Unit tests. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
