[ https://issues.apache.org/jira/browse/SPARK-38605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513643#comment-17513643 ]
Apache Spark commented on SPARK-38605: -------------------------------------- User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/35994 > Retrying on file manager operation in HDFSMetadataLog > ----------------------------------------------------- > > Key: SPARK-38605 > URL: https://issues.apache.org/jira/browse/SPARK-38605 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.4.0 > Reporter: L. C. Hsieh > Priority: Major > > Currently HDFSMetadataLog uses CheckpointFileManager to do some file > operation like opening metadata file. It is very easy to be affected by > network blips and causes the streaming query failed. Although we can restart > the streaming query, but it takes more time to recover. > Such file operations should be resilient with such situation by retrying. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org