Re: Not all workers seem to run in a standalone cluster setup by spark-ec2 script

Whats in your SparkIsAwesome class? Just make sure that you are giving enough partition to spark to evenly distribute the job throughout the cluster. Try submitting the job this way: ~/spark/bin/spark-submit --executor-cores 10 --executor-memory 5G --driver-memory 5G --class

Re: Not all workers seem to run in a standalone cluster setup by spark-ec2 script

Andy, Thank you for replying. I am specifying exactly like it to --master. I just had missed it when writing that email. 2015年12月5日(土) 9:27 Andy Davidson : > Hi Kyohey > > I think you need to pass the argument --master $MASTER_URL \ > > > master_URL is

Re: Not all workers seem to run in a standalone cluster setup by spark-ec2 script

Hi Kyohey I think you need to pass the argument --master $MASTER_URL \ master_URL is something like spark://ec2-54-215-112-121.us-west-1.compute.amazonaws.com:7077 Its the public url to your master Andy From: Kyohey Hamaguchi Date: Friday, December 4, 2015

Re: Not all workers seem to run in a standalone cluster setup by spark-ec2 script

Quick question: Are you processing gzipped files by any chance? It's a common stumbling block people hit. See: http://stackoverflow.com/q/27531816/877069 Nick On Fri, Dec 4, 2015 at 2:28 PM Kyohey Hamaguchi wrote: > Hi, > > I have setup a Spark standalone-cluster, which

Not all workers seem to run in a standalone cluster setup by spark-ec2 script

Hi, I have setup a Spark standalone-cluster, which involves 5 workers, using spark-ec2 script. After submitting my Spark application, I had noticed that just one worker seemed to run the application and other 4 workers were doing nothing. I had confirmed this by checking CPU and memory usage on