viirya commented on PR #58562: URL: https://github.com/apache/spark/pull/58562#issuecomment-5595382431
Regarding @n-young's [comment on EXCEPT DISTINCT](https://github.com/apache/spark/pull/58562#pullrequestreview-5147735614): I agree that EXCEPT DISTINCT deserves a separate explanation. Seeing the same row in two Update-mode batches does not by itself establish an incorrect result table, since those batches need to be interpreted as updates rather than concatenated as append output. The stronger justification seems to be that the aggregation introduced by the optimizer rewrite is invisible to the initial unsupported-operation checks, so the output-mode and stateful-operation restrictions are not fully validated. Conservatively rejecting it seems reasonable, but could we clarify this distinction in the PR description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
