AnishMahto opened a new pull request, #59528: URL: https://github.com/apache/spark/pull/59528
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/scd1-field-reconciliation-data-classes...scd1-tombstone-rewriting. See previous PR in stack here: https://github.com/apache/spark/pull/59045 ### What changes were proposed in this pull request? Apply persisted tombstones per field during SCD1 ignore-null reconciliation. Before a microbatch is merged, each key's resolved row is dropped if it predates the key's tombstone. Today this check is row-wide, but under ignore-null, leaf-level reconciliation lets different events author different fields of one row, so a tombstone may supersede only some. ### Why are the changes needed? SCD1 ignore-null support. ### Does this PR introduce _any_ user-facing change? No, SCD1 ignore-null is an unreleased and gated feature. ### How was this patch tested? Unit tests added to `Scd1LeafLevelReconciliationSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
