AnishMahto commented on PR #55993: URL: https://github.com/apache/spark/pull/55993#issuecomment-4513167138
The assumption (or more specifically, these are invariants given how auxiliary table and CDC metadata column projection are computed) is none of these will never happen. If a user manually alters these values in the auxiliary/target table to break the invariants, then correct onus falls on them. That being said the implementation is robust OOTB for all of these cases (ex. merging tombstones onto microbatch via left-anti join doesn't care if there are multiple tombstones per key, behavior is preserved, deleteSequence=null tombstones are ignored because of null safe equality checks, etc.). I'm going to go ahead and lock in the behavior for these fringe cases in tests though, they'll be nice stop catches in case future refactorings change the assumptions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
