Bo Zhang created SPARK-36533: -------------------------------- Summary: Allow streaming queries with Trigger.Once run in multiple batches Key: SPARK-36533 URL: https://issues.apache.org/jira/browse/SPARK-36533 Project: Spark Issue Type: New Feature Components: Structured Streaming Affects Versions: 3.2.0 Reporter: Bo Zhang
Currently streaming queries with Trigger.Once will always load all of the available data in a single batch. Because of this, the amount of data the queries can process is limited, or Spark driver will be out of memory. We should allow streaming queries with Trigger.Once run in multiple batches. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org