AnishMahto commented on PR #56208: URL: https://github.com/apache/spark/pull/56208#issuecomment-4595524013
> but how do we know it's the right list Yep totally get that. My goal here with these PRs will be to declare a clear contract (expected input and expected output) per sub-step, and we should review that sub-step in isolation - i.e does the implementation match the declared contract. Eventually once all the sub-steps are in, I'll put out a PR connecting them within the foreachBatch, at which point we should reason about whether SCD2 in its entirety is correctly implemented, given the contracts of each individual sub-step. Fully agree that we might need to go back and update sub-steps at that point, but I'm okay with that given these sub-steps will only be used once we get to that point. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
