L. C. Hsieh created SPARK-40826:
-----------------------------------
Summary: Add additional checkpoint rename file check
Key: SPARK-40826
URL: https://issues.apache.org/jira/browse/SPARK-40826
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.0
Reporter: L. C. Hsieh
We encountered an issue recently that one customer's structured streaming job
failed to read delta file.
The temporary file exists but it was not successfully renamed to final delta
file path.
We currently don't check if renamed file exists but assume it successful. As
the result, failing to read delta file assumed to be committed in last batch
makes re-triggering the job impossible.
We should be able to do a check against checkpoint renamed file to prevent such
difficulty in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
