I had the same problem. One forum post elsewhere suggested that too much
network communication might be using up available ports. I reduced the
partition size via repartition(int) and it solved the problem.
--
View this message in context:
Also, this is the command I use to submit the Spark application:
**
where *recommendation_engine-0.1-py2.7.egg* is a Python egg of my own
library I've written for this application, and *'file'* and
*'/home/spark/enigma_analytics/tests/msg-epims0730_small.json'* are input
arguments for the
Slight update I suppose?
For some reason, sometimes it will connect and continue and the job will be
completed. But most of the time I still run into this error and the job is
killed and the application doesn't finish.
Still have no idea why this is happening. I could really use some help here.
I am having trouble with my standalone Spark cluster and I can't seem to find
a solution anywhere. I hope that maybe someone can figure out what is going
wrong so this issue might be resolved and I can continue with my work.
I am currently attempting to use Python and the pyspark library to do
