HeartSaVioR commented on code in PR #38503: URL: https://github.com/apache/spark/pull/38503#discussion_r1021160270
########## sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala: ########## @@ -188,17 +194,26 @@ class UnsupportedOperationsSuite extends SparkFunSuite with SQLHelper { expectedMsgs = Seq("Complete")) // FlatMapGroupsWithState(Update) in streaming with aggregation - for (outputMode <- Seq(Append, Update, Complete)) { + for (outputMode <- Seq(Update, Complete)) { assertNotSupportedInStreamingPlan( "flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation " + s"with aggregation in $outputMode mode", TestFlatMapGroupsWithState( null, att, att, Seq(att), Seq(att), att, null, Update, isMapGroupsWithState = false, null, Aggregate(Seq(attributeWithWatermark), aggExprs("c"), streamRelation)), outputMode = outputMode, - expectedMsgs = Seq("flatMapGroupsWithState in update mode", "with aggregation")) + expectedMsgs = Seq("Multiple stateful operators", "Update", "Complete")) } + assertNotSupportedInStreamingPlan( + "flatMapGroupsWithState - flatMapGroupsWithState(Update) on streaming relation " + + s"with aggregation in Append mode", + TestFlatMapGroupsWithState( + null, att, att, Seq(att), Seq(att), att, null, Update, isMapGroupsWithState = false, null, Review Comment: This is one of the reason I want to see the allowed operations be more restrictive rather than saying op A - op B is supported with some modes, op B - op A is supported with some modes, etc. The major reason I blocked multiple stateful operators at all was that we figured out stream-stream outer join followed by another stream-stream outer join is not working, and I realized that is not only the case. Similar risks are everywhere and it's really hard to think hard on all matrix. That said, unfortunately, the case we do not fail in unsupported operation checker does not always mean we guarantee the case to work properly. We probably have checked in some cases, but I don't believe it's backed by theory (otherwise we should have known the limit of global watermark much earlier). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org