Livia Zhu created SPARK-52008:
---------------------------------
Summary: Add TaskCompletionListener for state stores that abort
updating store and throw error
Key: SPARK-52008
URL: https://issues.apache.org/jira/browse/SPARK-52008
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Livia Zhu
When a user defines a function with foreachBatch that does not completely
consume the passed in iterator, state stores will be opened but not committed
when the batch finishes and no error will be thrown. This will lead to
"changelog/delta file not found" error for the next batch which confuses users.
Instead, we should explicitly throw an error in the TaskCompletionListener that
will abort any state stores still in the updating state and throw an exception
to fail the task (if the task is not already failed or interrupted).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
