[
https://issues.apache.org/jira/browse/SPARK-28652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16904729#comment-16904729
]
Liang-Chi Hsieh commented on SPARK-28652:
-----------------------------------------
This looks interesting to me. I tried to look into existing tests. I think it
is true that {{spark.kubernetes.pyspark.pythonVersion}} doesn't not pass into
executors. But it looks correct and I think we don't need to pass it.
The python version used by executors is come from Python side at driver, when
wrapping a python function. PythonRunner will later serialize this variable
when it is going to invoke python workers. PythonWorkerFactory also uses this
variable to determine which python executable to run. So in executors, to run
which python executable is not determined by PYSPARK_PYTHON. It means that we
don't need to pass spark.kubernetes.pyspark.pythonVersion to executors, as this
config is only used to choose PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON.
cc [~hyukjin.kwon] too, in case if I miss something.
> spark.kubernetes.pyspark.pythonVersion is never passed to executors
> -------------------------------------------------------------------
>
> Key: SPARK-28652
> URL: https://issues.apache.org/jira/browse/SPARK-28652
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.3
> Reporter: nanav yorbiz
> Priority: Major
>
> I suppose this may not be a priority with Python2 on its way out, but given
> that this setting is only ever sent to the driver and not the executors, no
> actual work can be performed when the versions don't match, which will tend
> to be *always* with the default setting for the driver being changed from 2
> to 3, and the executors using `python`, which defaults to v2, by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
