This is an interesting question. I don't have a solution for you, but you may be interested in taking a look at Anaconda Cluster <http://continuum.io/anaconda-cluster>.
It's made by the same people behind Conda (an alternative to pip focused on data science pacakges) and may offer a better way of doing this. Haven't used it though. On Thu, May 7, 2015 at 5:20 PM alemagnani <ale.magn...@gmail.com> wrote: > I am currently using pyspark with a virtualenv. > Unfortunately I don't have access to the nodes file system and therefore I > cannot manually copy the virtual env over there. > > I have been using this technique: > > I first add a tar ball with the venv > sc.addFile(virtual_env_tarball_file) > > Then in the code used on the node to do the computation I activate the venv > like this: > venv_location = SparkFiles.get(venv_name) > activate_env="%s/bin/activate_this.py" % venv_location > execfile(activate_env, dict(__file__=activate_env)) > > Is there a better way to do this? > One of the problem with this approach is that in > spark/python/pyspark/statcounter.py numpy is imported > before the venv is activated and this can cause conflicts with the venv > numpy. > > Moreover this requires the venv to be sent around in the cluster all the > time. > Any suggestions? > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Virtualenv-pyspark-tp22803.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
