You need to set the spark configuration property: spark.yarn.access.namenodes to your namenode.
e.g. spark.yarn.access.namenodes=hdfs://mynamenode:8020 Similarly, I'm curious if you're also running high availability HDFS with an HA nameservice. I currently have HA HDFS and kerberos and I've noticed that I must set the above property to the currently active namenode's hostname and port. Simply using the HA nameservice to get delegation tokens does NOT seem to work with Spark 1.1.0 (even though I can confirm the token is acquired). I believe this may be a bug. Unfortunately simply adding both the active and standby name nodes does not work as this actually causes an error. This means that when my active name node fails over, my spark configuration becomes invalid. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Strange-behavior-of-spark-shell-while-accessing-hdfs-tp18549p18656.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
