[ https://issues.apache.org/jira/browse/SPARK-21470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093726#comment-16093726 ]
Marcelo Vanzin commented on SPARK-21470: ---------------------------------------- That still looks like some issue in your configuration. Seems like the HDFS library doesn't know that "hdfs1" is supposed to be an HA namespace and not a host. e.g., in one of our clusters we have: core-site.xml: {code} <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> {code} hdfs-site.xml: {code} <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <property> <name>dfs.ha.namenodes.ns1</name> <value>namenode19,namenode242</value> </property> {code} Along with a lot of other HA-related configs. > [SPARK 2.2 Regression] Spark History server doesn't support HDFS HA > ------------------------------------------------------------------- > > Key: SPARK-21470 > URL: https://issues.apache.org/jira/browse/SPARK-21470 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: Maciej BryĆski > > With Spark version up to 2.1.1 there was possibility to config history server > to read from hdfs without putting namenode: > spark.history.fs.logDirectory hdfs:///apps/spark > And this works with HDFS HA. > Unfortunately there is regression with Spark 2.2.0 when such configuration > gives error: > {code} > Caused by: java.io.IOException: Incomplete HDFS URI, no host: > hdfs:///apps/spark > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) > at > org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:108) > at > org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:78) > ... 6 more > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org