Marco Gaido created SPARK-24149:
-----------------------------------
Summary: Automatic namespaces discovery in HDFS federation
Key: SPARK-24149
URL: https://issues.apache.org/jira/browse/SPARK-24149
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.4.0
Reporter: Marco Gaido
Hadoop 3 introduced HDFS federation.
Spark fails to write on different namespaces when Hadoop federation is turned
on and the cluster is secure. This happens because Spark looks for the
delegation token only for the defaultFS configured and not for all the
available namespaces. A workaround is the usage of the property
{{spark.yarn.access.hadoopFileSystems}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
