[ https://issues.apache.org/jira/browse/SPARK-22788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Imran Rashid resolved SPARK-22788. ---------------------------------- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19983 [https://github.com/apache/spark/pull/19983] > HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support" > ----------------------------------------------------------------------------- > > Key: SPARK-22788 > URL: https://issues.apache.org/jira/browse/SPARK-22788 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 1.6.0 > Reporter: Marcelo Vanzin > Priority: Trivial > Fix For: 2.3.0 > > > Code: > {noformat} > if (conf.getBoolean("hdfs.append.support", false) || > dfs.isInstanceOf[RawLocalFileSystem]) { > dfs.append(dfsPath) > } else { > throw new IllegalStateException("File exists and there is no append > support!") > } > {noformat} > This makes the exception to be thrown if you enable > {{writeAheadLog.closeFileAfterWrite}} with HDFS. > The correct config, from {{DFSConfigKeys}}, is {{dfs.support.append}}, and > the default value is true. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org