Burak Yavuz created SPARK-18342: ----------------------------------- Summary: HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail Key: SPARK-18342 URL: https://issues.apache.org/jira/browse/SPARK-18342 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 2.0.1 Reporter: Burak Yavuz Priority: Critical
The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org