Hey, I just tried to submit a task to my spark cluster using the following command
./spark/bin/spark-submit --py-files file:///root/abc.zip --master spark://xxx.xxx.xxx.xxx:7077 test.py It seems like the dependency I’ve added gets loaded: 14/08/05 23:07:00 INFO spark.SparkContext: Added file file:///root/abc.zip at http://xxx.xxx.xxx.xxx:40346/files/abc.zip with timestamp 1407280020217 However, my python script can’t find the module in this zip file. I already verified if this zip file is not corrupt by install it with “pip install abc.zip” (works fine). Any ideas how to get the content of the archive to the PYTHONPATH on my master and slaves? Traceback (most recent call last): File "/root/test.py", line 7, in <module> import abc ImportError: No module named abc Maybe, it’s just the master complaining since and it only transfers the archive to the slaves (and adds it to the PYTHONPATH)? --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
