AnishMahto opened a new pull request, #59537: URL: https://github.com/apache/spark/pull/59537
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/scd1-reconcile-microbatch-under-ignore-null...scd1-target-merge-before-aux-merge. See previous PR in stack here: https://github.com/apache/spark/pull/59531 ### What changes were proposed in this pull request? Each microbatch is validated and reconciled into one net row per key, then merged into both auxiliary and target tables before completion. Today auxiliary merges first, which is safe for row-level reconciliation because tombstones only reject older whole-row events. Under leaf-level reconciliation, a tombstone can null fields authored by some upserts in the microbatch and then be removed due to an overall newer upsert in the same microbatch, so failure before target merge makes retry lose that input. The solution is to reverse the merge order, merging to the target before the auxiliary table. Row-level reconciliation remains correct with the same observed behavior, but now leaf-level reconciliation can safely be retried after a failure between table merges too. ### Why are the changes needed? Correctness of ignore-null reconciliation retry after failure between merges. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
