[
https://issues.apache.org/jira/browse/SPARK-13587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174996#comment-15174996
]
Mike Sukmanowsky commented on SPARK-13587:
------------------------------------------
Thanks for letting me know about this [~jeffzhang].
I think in general, I'm +1 on the proposal.
virtualenvs are the way to go to install requirements and ensure isolation of
dependencies between multiple driver scripts. As you noted though, installing
hefty requirements like pandas or numpy (assuming you aren't using Conda),
would add a pretty significant overhead to startup which could be amortized if
the driver was assumed to run for a long enough period of time. Conda of course
would pretty well eliminate that problem as it provides pre-compiled binaries
for most OSs.
I'd like to offer [PEX|https://pex.readthedocs.org/en/stable/] as an
alternative, where spark-submit would build a self-contained virtualenv in a
.pex file on the Spark master node and then distribute to all other nodes.
However, it turns out PEX doesn't support editable requirements and introduces
an assumption that all nodes in a cluster are homogenous so that a Python
package with C extensions compiled on the master node would run on worker nodes
without issue. The latter assumption may be a leap too far for all Spark users.
One thing I'm not entirely sure of is the need for the
spark.pyspark.virtualenv.path property. If the virtualenv is temporary, why
would this path ever be specified? Wouldn't a temporary path be used and
subsequently removed after the Python worker completes?
> Support virtualenv in PySpark
> -----------------------------
>
> Key: SPARK-13587
> URL: https://issues.apache.org/jira/browse/SPARK-13587
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: Jeff Zhang
>
> Currently, it's not easy for user to add third party python packages in
> pyspark.
> * One way is to using --py-files (suitable for simple dependency, but not
> suitable for complicated dependency, especially with transitive dependency)
> * Another way is install packages manually on each node (time wasting, and
> not easy to switch to different environment)
> Python has now 2 different virtualenv implementation. One is native
> virtualenv another is through conda. This jira is trying to migrate these 2
> tools to distributed environment
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
