HeartSaVioR commented on code in PR #38503:
URL: https://github.com/apache/spark/pull/38503#discussion_r1016181346
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:
##########
@@ -169,16 +179,24 @@ object UnsupportedOperationChecker extends Logging {
"DataFrames/Datasets")(plan)
}
- // Disallow multiple streaming aggregations
- val aggregates = collectStreamingAggregates(plan)
+ val statefulOps = plan.collect {
+ case p: LogicalPlan if isStatefulOperation(p) => p
+ }
- if (aggregates.size > 1 && outputMode != InternalOutputModes.Append) {
+ if (statefulOps.size > 1 &&
+ outputMode != InternalOutputModes.Append &&
+ SQLConf.get.statefulOperatorCorrectnessCheckEnabled) {
Review Comment:
I think this configuration should not just exist now as we unlocked various
use cases (except stream-stream time interval join followed by other stateful
operator), but let's handle it as separate issue. (I'll file a JIRA ticket on
it.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
