L. C. Hsieh created SPARK-40826: ----------------------------------- Summary: Add additional checkpoint rename file check Key: SPARK-40826 URL: https://issues.apache.org/jira/browse/SPARK-40826 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 3.4.0 Reporter: L. C. Hsieh
We encountered an issue recently that one customer's structured streaming job failed to read delta file. The temporary file exists but it was not successfully renamed to final delta file path. We currently don't check if renamed file exists but assume it successful. As the result, failing to read delta file assumed to be committed in last batch makes re-triggering the job impossible. We should be able to do a check against checkpoint renamed file to prevent such difficulty in advance. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org