Quick question: Are you processing gzipped files by any chance? It's a common stumbling block people hit.
See: http://stackoverflow.com/q/27531816/877069 Nick On Fri, Dec 4, 2015 at 2:28 PM Kyohey Hamaguchi <tnzk.ma...@gmail.com> wrote: > Hi, > > I have setup a Spark standalone-cluster, which involves 5 workers, > using spark-ec2 script. > > After submitting my Spark application, I had noticed that just one > worker seemed to run the application and other 4 workers were doing > nothing. I had confirmed this by checking CPU and memory usage on the > Spark Web UI (CPU usage indicates zero and memory is almost fully > availabile.) > > This is the command used to launch: > > $ ~/spark/ec2/spark-ec2 -k awesome-keypair-name -i > /path/to/.ssh/awesome-private-key.pem --region ap-northeast-1 > --zone=ap-northeast-1a --slaves 5 --instance-type m1.large > --hadoop-major-version yarn launch awesome-spark-cluster > > And the command to run application: > > $ ssh -i ~/path/to/awesome-private-key.pem root@ec2-master-host-name > "mkdir ~/awesome" > $ scp -i ~/path/to/awesome-private-key.pem spark.jar > root@ec2-master-host-name:~/awesome && ssh -i > ~/path/to/awesome-private-key.pem root@ec2-master-host-name > "~/spark-ec2/copy-dir ~/awesome" > $ ssh -i ~/path/to/awesome-private-key.pem root@ec2-master-host-name > "~/spark/bin/spark-submit --num-executors 5 --executor-cores 2 > --executor-memory 5G --total-executor-cores 10 --driver-cores 2 > --driver-memory 5G --class com.example.SparkIsAwesome > awesome/spark.jar" > > How do I let the all of the workers execute the app? > > Or do I have wrong understanding on what workers, slaves and executors are? > > My understanding is: Spark driver(or maybe master?) sends a part of > jobs to each worker (== executor == slave), so a Spark cluster > automatically exploits all resources available in the cluster. Is this > some sort of misconception? > > Thanks, > > -- > Kyohey Hamaguchi > TEL: 080-6918-1708 > Mail: tnzk.ma...@gmail.com > Blog: http://blog.tnzk.org/ > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
