[
https://issues.apache.org/jira/browse/SPARK-41510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ohad Raviv updated SPARK-41510:
-------------------------------
Description:
When working interactively with Spark through notebooks in various envs -
Databricks/YARN I often encounter a very frustrating process of trying to add
new python modules and even change their code without starting a new spark
session/cluster.
In the driver side it is easy to add things like `sys.path.append()` but if for
example, if a UDF code is importing a function from a local module, then the
pickle boundaries will assume that the module exists in the workers, and fail
on "python module does not exist..".
To update the code "online" I can add NFS volume to the workers' PYTHONPATH.
However, setting the PYTHONPATH in the workers is not easy as it gets
overridden by someone (databricks/spark) along the way. a few ugly workarounds
are suggested like running a "dummy" UDF on the workers to add the folder to
the sys.path.
I think all of that could easily be solved if we just add a dedicated
`spark.conf` the will get merged into the worker's PYTHONPATH, just here:
[https://github.com/apache/spark/blob/0e2d604fd33c8236cfa8ae243eeaec42d3176a06/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala#L94]
please tell me what you think, and I will make the PR.
thanks.
was:
When working interactively with Spark through notebooks in various envs -
Databricks/YARN I often encounter a very frustrating process of trying to add
new python modules and even change their code without starting a new spark
session/cluster.
in the driver side it is easy to add things like `sys.path.append()` but if for
example UDF code is importing function from a local module, then the pickle
boundaries will assume that the module exists in the workers. and then I fail
on "python module does not exist..".
adding NFS volumes to the workers PYTHONPATH could solve it, but it requires
restarting the session/cluster and worse doesn't work in all envs as the
PYTHONPATH gets overridden by someone (databricks/spark) along the way. a few
ugly work around are suggested like running a "dummy" udf on workers to add the
folder to the sys.path.
I think all of that could easily be solved if we add a spark.conf to add to the
worker PYTHONPATH. here:
[https://github.com/apache/spark/blob/0e2d604fd33c8236cfa8ae243eeaec42d3176a06/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala#L94]
please tell me what you think, and I will make the PR.
thanks.
> Support easy way for user defined PYTHONPATH in workers
> -------------------------------------------------------
>
> Key: SPARK-41510
> URL: https://issues.apache.org/jira/browse/SPARK-41510
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.3.1
> Reporter: Ohad Raviv
> Priority: Minor
>
> When working interactively with Spark through notebooks in various envs -
> Databricks/YARN I often encounter a very frustrating process of trying to add
> new python modules and even change their code without starting a new spark
> session/cluster.
> In the driver side it is easy to add things like `sys.path.append()` but if
> for example, if a UDF code is importing a function from a local module, then
> the pickle boundaries will assume that the module exists in the workers, and
> fail on "python module does not exist..".
> To update the code "online" I can add NFS volume to the workers' PYTHONPATH.
> However, setting the PYTHONPATH in the workers is not easy as it gets
> overridden by someone (databricks/spark) along the way. a few ugly
> workarounds are suggested like running a "dummy" UDF on the workers to add
> the folder to the sys.path.
> I think all of that could easily be solved if we just add a dedicated
> `spark.conf` the will get merged into the worker's PYTHONPATH, just here:
> [https://github.com/apache/spark/blob/0e2d604fd33c8236cfa8ae243eeaec42d3176a06/core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala#L94]
>
> please tell me what you think, and I will make the PR.
> thanks.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
