Hi there, I am running spark 0.9.0 standalone on a cluster. The documentation http://spark.incubator.apache.org/docs/latest/python-programming-guide.htmlstates that code-dependencies can be deployed through the pyFiles argument to the SparkContext.
But in my case, the relevant code, lets call it myLib is already available in PYTHONPATH on the worker-nodes. However, when trying to access this code through a regular 'import myLib' in the script sent to pyspark, the spark-workers seem to hang in the middle of the script without any specific errors. If I start a regular python-shell on the workers, there is no problem importing myLib and accessing it. Why is this? /Anders
