[ https://issues.apache.org/jira/browse/SPARK-32794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190867#comment-17190867 ]
Apache Spark commented on SPARK-32794: -------------------------------------- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/29651 > Rare corner case error in micro-batch engine with some stateful queries + > no-data-batches + V1 streaming sources > ----------------------------------------------------------------------------------------------------------------- > > Key: SPARK-32794 > URL: https://issues.apache.org/jira/browse/SPARK-32794 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.3.4, 2.4.6, 3.0.0, 3.0.1 > Reporter: Tathagata Das > Assignee: Tathagata Das > Priority: Major > > Structured Streaming micro-batch engine has the contract with V1 data sources > that, after a restart, it will call `source.getBatch()` on the last batch > attempted before the restart. However, a very rare combination of sequences > violates this contract. It occurs only when > - The streaming query has specific types of stateful operations with > watermarks (e.g., aggregation in append, mapGroupsWithState with timeouts). > - These queries can execute a batch even without new data when the > previous updates the watermark and the stateful ops are such that the new > watermark can cause new output/cleanup. Such batches are called > no-data-batches. > - The last batch before termination was an incomplete no-data-batch. Upon > restart, the micro-batch engine fails to call `source.getBatch` when > attempting to re-execute the incomplete no-data-batch. > This occurs because no-data-batches has the same and end offsets, and when a > batch is executed, if the start and end offset is same then calling > `source.getBatch` is skipped as it is assumed the generated plan will be > empty. This only affects V1 data sources which rely on this invariant to > initialize differently when the query is being started from scratch or > restarted. How will a source misbehave is very source-specific. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org