B. Micheal Okutubo created SPARK-55892:
------------------------------------------
Summary: Unable to load state store because reused SST file was
deleted by maintenance
Key: SPARK-55892
URL: https://issues.apache.org/jira/browse/SPARK-55892
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: B. Micheal Okutubo
When there is a large delay between when a snapshot was created and when it was
uploaded by maintenance, it is possible for the files that were reused from old
versions, which are now eligible for deletion by another executor's maintenance
to be deleted.
Hence, when trying to load the state store using the uploaded snapshot, it will
fail with FileNotFoundException for the reused SST file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
