[ https://issues.apache.org/jira/browse/SPARK-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769625#comment-15769625 ]
chris snow commented on SPARK-16110: ------------------------------------ +1 for backporting this to 1.6 and 2.0 > Can't set Python via spark-submit for YARN cluster mode when PYSPARK_PYTHON & > PYSPARK_DRIVER_PYTHON are set > ----------------------------------------------------------------------------------------------------------- > > Key: SPARK-16110 > URL: https://issues.apache.org/jira/browse/SPARK-16110 > Project: Spark > Issue Type: Bug > Components: PySpark, YARN > Affects Versions: 1.6.1 > Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-38-generic x86_64), > Spark 1.6.1, Azure HDInsight 3.4) > Reporter: Kevin Grealish > Assignee: Kevin Grealish > Labels: easyfix > Fix For: 2.1.0 > > Original Estimate: 2h > Remaining Estimate: 2h > > When a cluster has PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment > variables set (needed for using non-system Python e.g. > /usr/bin/anaconda/bin/python), then you are unable to override this per > submission in YARN cluster mode. > When using spark-submit (in this case via LIVY) to submit with an override: > spark-submit --master yarn --deploy-mode cluster --conf > 'spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=python3' --conf' > 'spark.yarn.appMasterEnv.PYSPARK_PYTHON=python3' probe.py > the environment variable values will override the conf settings. A workaround > for some can be to unset the env vars but that is not always possible (e.g. > submitting batch via LIVY where you can only pass through the parameters to > spark-submit). > Expectation is that the conf values above override the environment variables. > Fix is to change the order of application of conf and env vars in the yarn > client. > Related discussion:https://issues.cloudera.org/browse/LIVY-159 > Backporting this to 1.6 would be great and unblocking for me. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org