Thomas Graves created SPARK-1384: ------------------------------------ Summary: spark-shell on yarn doesn't always work with secure hdfs Key: SPARK-1384 URL: https://issues.apache.org/jira/browse/SPARK-1384 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 0.9.0, 0.9.1 Reporter: Thomas Graves
I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 rc3 release. It doesn't work with secure HDFS unless you export SPARK_YARN_MODE=true before starting the shell, or if you happen to do something immediately with HDFS. If you wait for the connection to the namenode to timeout it will fail. I think it was actually this way in the 0.9 release also so I thought I would send this and get peoples feedback to see if you want it fixed? Another option would be to document that you have to export SPARK_YARN_MODE=true for the shell. The fix actually went in with the authentication changes I made in master but I never realized that change needed to apply to 0.9. https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0 See the SparkILoop diff. -- This message was sent by Atlassian JIRA (v6.2#6252)