Marco Gaido created SPARK-24149: ----------------------------------- Summary: Automatic namespaces discovery in HDFS federation Key: SPARK-24149 URL: https://issues.apache.org/jira/browse/SPARK-24149 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 2.4.0 Reporter: Marco Gaido
Hadoop 3 introduced HDFS federation. Spark fails to write on different namespaces when Hadoop federation is turned on and the cluster is secure. This happens because Spark looks for the delegation token only for the defaultFS configured and not for all the available namespaces. A workaround is the usage of the property {{spark.yarn.access.hadoopFileSystems}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org