AnishMahto commented on code in PR #56160:
URL: https://github.com/apache/spark/pull/56160#discussion_r3314990945
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala:
##########
@@ -271,6 +275,10 @@ class AutoCdcMergeFlow(
selectedSchema
}
+ // If the auxiliary table corresponding to the target already exists, verify
the user is trigging
+ // an AutoCDC transformation to that target using the same keys.
+ validateNoAutoCdcKeyDriftIfAuxTableExists()
Review Comment:
Thats fair. I went back and forth on whether to validate at
analysis/resolution time vs execution time, but I think you're right that we
should prioritize keeping the former side effect free and cheap (no catalog
lookup) - moved the check to flow execution.
I left a comment in the code about this (and we've discussed in other PRs)
but eventually I'd like to come up with a clean refactor where `DatasetManager`
can actually handle creation/validation of the aux table, which would be before
execution time. Doesn't change functionality, but will improve UX for fast
failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
