Python Packages in Spark w/Mesos

2015-09-21 Thread John Omernik
Hey all - Curious at the best way to include python packages in my Spark installation. (Such as NLTK). Basically I am running on Mesos, and would like to find a way to include the package in the binary distribution in that I don't want to install packages on all nodes. We should be able to

Re: Python Packages in Spark w/Mesos

2015-09-21 Thread Tim Chen
Hi John, Sorry haven't get time to respond to your questions over the weekend. If you're running client mode, to use the Docker/Mesos integration minimally you just need to set the image configuration 'spark.mesos.executor.docker.image' as stated in the documentation, which Spark will use this