Thomas Graves created SPARK-1384:
------------------------------------
Summary: spark-shell on yarn doesn't always work with secure hdfs
Key: SPARK-1384
URL: https://issues.apache.org/jira/browse/SPARK-1384
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 0.9.0, 0.9.1
Reporter: Thomas Graves
I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3
release. It doesn't work with secure HDFS unless you
export SPARK_YARN_MODE=true before starting the shell, or if you happen to do
something immediately with HDFS. If you wait for the connection to the
namenode to timeout it will fail.
I think it was actually this way in the 0.9 release also so I thought I would
send this and get peoples feedback to see if you want it fixed?
Another option would be to document that you have to export
SPARK_YARN_MODE=true for the shell. The fix actually went in with the
authentication changes I made in master but I never realized that change needed
to apply to 0.9.
https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0
See the SparkILoop diff.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
