You need to set the spark configuration property: spark.yarn.access.namenodes
to your namenode.
e.g. spark.yarn.access.namenodes=hdfs://mynamenode:8020
Similarly, I'm curious if you're also running high availability HDFS with an
HA nameservice.
I currently have HA HDFS and kerberos and I've
Only YARN mode is supported with kerberos. You can't use a spark:// master
with kerberos.
Tobias Pfeiffer wrote
When you give a spark://* master, Spark will run on a different machine,
where you have not yet authenticated to HDFS, I think. I don't know how to
solve this, though, maybe some
