jose-torres commented on a change in pull request #23156: [SPARK-24063][SS] Add
maximum epoch queue threshold for ContinuousExecution
URL: https://github.com/apache/spark/pull/23156#discussion_r251135559
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/ContinuousSuite.scala
##########
@@ -343,3 +343,31 @@ class ContinuousMetaSuite extends ContinuousSuiteBase {
}
}
}
+
+class ContinuousEpochBacklogSuite extends ContinuousSuiteBase {
+ import testImplicits._
+
+ // We need to specify spark.sql.streaming.continuous.epochBacklogQueueSize.
Review comment:
We should be able to set the queue size conf using withSQLConf; see examples
in e.g. ContinuousAggregationSuite.
I'd suggest a comment explaining that this test forces the backlog to
overflow by not standing up enough executors for the query to make progress.
(Probably we should make that an immediate error at some point, which means
we'd need to come up with another way to test this, but I think the current
strategy is good for now.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
