ulysses you created SPARK-28597: ----------------------------------- Summary: Spark streaming terminated when close meta data log error Key: SPARK-28597 URL: https://issues.apache.org/jira/browse/SPARK-28597 Project: Spark Issue Type: Bug Components: DStreams, Structured Streaming Affects Versions: 2.4.3 Reporter: ulysses you
Now, HDFSMetaDataLog is the only implement of MetaDataLog. If HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be terminated. The meta data log is very important for spark, we should retry some times when exception, and add a config to control retry times. For example HDFS has a bug that close error if has no enough replicas more detail HDFS-11486. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org