Kohsuke Kawaguchi created LIVY-781:
--------------------------------------
Summary: PythonInterpreter doesn't honor
spark.pyspark.driver.python
Key: LIVY-781
URL: https://issues.apache.org/jira/browse/LIVY-781
Project: Livy
Issue Type: Bug
Components: Interpreter
Affects Versions: 0.7.0
Reporter: Kohsuke Kawaguchi
[LIVY-418|https://issues.apache.org/jira/browse/LIVY-418] introduced a
downstream change from
[SPARK-13081|https://issues.apache.org/jira/browse/SPARK-13081] where
{{PythonInterpreter}} honors the {{spark.pyspark.python}} configuration. But it
fails to honor the {{spark.pyspark.driver.python}} that SPARK-13081 also
introduces.
As a result, it's not possible to use different path on the driver and the
executors with Livy, even though it's possible to do so with Spark.
My understanding is that
https://github.com/apache/incubator-livy/commit/64d71aa757a93518427f315e908dcf54373948b2
should start with:
{code}
val pythonExec = conf.getOption("spark.pyspark.driver.python")
.orElse(conf.getOption("spark.pyspark.python"))
.orElse(sys.env.get(...
{code}
... just like
https://github.com/apache/spark/commit/7a9e25c38380e6c62080d62ad38a4830e44fe753
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
