[
https://issues.apache.org/jira/browse/SPARK-26404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740011#comment-16740011
]
Nihar Sheth commented on SPARK-26404:
-------------------------------------
Yup, ran into the same issue then. What I've learned is that PYSPARK_PYTHON
isn't used by the executors, it looks like the driver propagates the value to
the executors, so it needs to be set in the driver's environment. As for
spark.pyspark.python, I'm not entirely sure why it doesn't work. @squito
suggested that it might be a path only covered by spark-submit. I might look
into it in a couple weeks when I have some spare cycles, but if anyone else
wants to give it a go, feel free to ping me with any questions and I can try to
help.
In the meanwhile, setting PYSPARK_PYTHON=<python executable> in the driver was
a valid workaround for me. Either os.environ['PYSPARK_PYTHON']=<python exec>
before SparkSession.builder, or exporting the environment variable in
spark-env.sh
> set spark.pyspark.python or PYSPARK_PYTHON doesn't work in k8s client-cluster
> mode.
> -----------------------------------------------------------------------------------
>
> Key: SPARK-26404
> URL: https://issues.apache.org/jira/browse/SPARK-26404
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Dongqing Liu
> Priority: Major
>
> Neither
> conf.set("spark.executorEnv.PYSPARK_PYTHON", "/opt/pythonenvs/bin/python")
> nor
> conf.set("spark.pyspark.python", "/opt/pythonenvs/bin/python")
> works.
> Looks like the executor always picks python from PATH.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
