anew commented on PR #57192: URL: https://github.com/apache/spark/pull/57192#issuecomment-4987553018
@jose-torres Re: > The behavior in the tests doesn't seem right to me. There's a lot of rows that are duplicates in all the user-facing columns. Perhaps reconcileStartAndEndAt is only a piece of the transformation, and there's further logic in a subsequent PR that will route some of these rows to the aux table? You are right, the next two PRs for Scd2BatchProcessor will do some cleanup and then route the records in this intermediate DF to the target table or the aux table: they will follow once this one is merged. - https://github.com/apache/spark/pull/56456 - https://github.com/apache/spark/pull/56457 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
