[
https://issues.apache.org/jira/browse/SPARK-28597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ulysses you updated SPARK-28597:
--------------------------------
Description:
Now, HDFSMetaDataLog is the only implement of MetaDataLog. If
HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be
terminated.
The meta data log is very important for spark, we should retry some times when
exception, and add a config to control retry times.
For example HDFS has a bug that close error if has no enough replicas detail
HDFS-11486.
was:
Now, HDFSMetaDataLog is the only implement of MetaDataLog. If
HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be
terminated.
The meta data log is very important for spark, we should retry some times when
exception, and add a config to control retry times.
For example HDFS has a bug that close error if has no enough replicas. More
detail HDFS-11486.
> Spark streaming terminated when close meta data log error
> ----------------------------------------------------------
>
> Key: SPARK-28597
> URL: https://issues.apache.org/jira/browse/SPARK-28597
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Structured Streaming
> Affects Versions: 2.4.3
> Reporter: ulysses you
> Priority: Major
>
> Now, HDFSMetaDataLog is the only implement of MetaDataLog. If
> HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be
> terminated.
> The meta data log is very important for spark, we should retry some times
> when exception, and add a config to control retry times.
> For example HDFS has a bug that close error if has no enough replicas detail
> HDFS-11486.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
