[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16848097#comment-16848097 ]
Marco Gaido commented on SPARK-24149: ------------------------------------- [~Dhruve Ashar] the use case for this change, for instance, is when you have a partitioned table, when the partitions are on different namespaces and there is no viewFS configured. In that case, a user running a query on that table, may or may not get an exception when reading it. Please, notice that a user running a query may be different from the user creating it, so he/she may also not be aware of this situation and understanding what is the problem may be pretty hard. > Automatic namespaces discovery in HDFS federation > ------------------------------------------------- > > Key: SPARK-24149 > URL: https://issues.apache.org/jira/browse/SPARK-24149 > Project: Spark > Issue Type: Improvement > Components: YARN > Affects Versions: 2.4.0 > Reporter: Marco Gaido > Assignee: Marco Gaido > Priority: Minor > Fix For: 2.4.0 > > > Hadoop 3 introduced HDFS federation. > Spark fails to write on different namespaces when Hadoop federation is turned > on and the cluster is secure. This happens because Spark looks for the > delegation token only for the defaultFS configured and not for all the > available namespaces. A workaround is the usage of the property > {{spark.yarn.access.hadoopFileSystems}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org