[ https://issues.apache.org/jira/browse/SPARK-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13957650#comment-13957650 ]
Thomas Graves commented on SPARK-1384: -------------------------------------- https://github.com/apache/spark/pull/287 > spark-shell on yarn on spark 0.9 branch doesn't always work with secure hdfs > ---------------------------------------------------------------------------- > > Key: SPARK-1384 > URL: https://issues.apache.org/jira/browse/SPARK-1384 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 0.9.0, 0.9.1 > Reporter: Thomas Graves > Assignee: Thomas Graves > > I've found an issue with the spark-shell in yarn-client mode in the 0.9.1 > rc3 release. It doesn't work with secure HDFS unless you > export SPARK_YARN_MODE=true before starting the shell, or if you happen to do > something immediately with HDFS. If you wait for the connection to the > namenode to timeout it will fail. > > The fix actually went in to master branch with the authentication changes I > made in master but I never realized that change needed to apply to 0.9. > https://github.com/apache/spark/commit/7edbea41b43e0dc11a2de156be220db8b7952d01#diff-0ae5b834ce90ec37c19af35aa7a5e1a0 > See the SparkILoop diff. -- This message was sent by Atlassian JIRA (v6.2#6252)