*oops, I mean are you setting --executor-cores to 8 On Mon, Dec 29, 2014 at 10:15 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote:
> Are you setting --num-executors to 8? > > On Mon, Dec 29, 2014 at 10:13 AM, Mukesh Jha <me.mukesh....@gmail.com> > wrote: > >> Sorry Sandy, The command is just for reference but I can confirm that >> there are 4 executors and a driver as shown in the spark UI page. >> >> Each of these machines is a 8 core box with ~15G of ram. >> >> On Mon, Dec 29, 2014 at 11:23 PM, Sandy Ryza <sandy.r...@cloudera.com> >> wrote: >> >>> Hi Mukesh, >>> >>> Based on your spark-submit command, it looks like you're only running >>> with 2 executors on YARN. Also, how many cores does each machine have? >>> >>> -Sandy >>> >>> On Mon, Dec 29, 2014 at 4:36 AM, Mukesh Jha <me.mukesh....@gmail.com> >>> wrote: >>> >>>> Hello Experts, >>>> I'm bench-marking Spark on YARN ( >>>> https://spark.apache.org/docs/latest/running-on-yarn.html) vs a >>>> standalone spark cluster ( >>>> https://spark.apache.org/docs/latest/spark-standalone.html). >>>> I have a standalone cluster with 3 executors, and a spark app running >>>> on yarn with 4 executors as shown below. >>>> >>>> The spark job running inside yarn is 10x slower than the one running on >>>> the standalone cluster (even though the yarn has more number of workers), >>>> also in both the case all the executors are in the same datacenter so there >>>> shouldn't be any latency. On YARN each 5sec batch is reading data from >>>> kafka and processing it in 5sec & on the standalone cluster each 5sec batch >>>> is getting processed in 0.4sec. >>>> Also, In YARN mode all the executors are not getting used up evenly as >>>> vm-13 & vm-14 are running most of the tasks whereas in the standalone mode >>>> all the executors are running the tasks. >>>> >>>> Do I need to set up some configuration to evenly distribute the tasks? >>>> Also do you have any pointers on the reasons the yarn job is 10x slower >>>> than the standalone job? >>>> Any suggestion is greatly appreciated, Thanks in advance. >>>> >>>> YARN(5 workers + driver) >>>> ======================== >>>> Executor ID Address RDD Blocks Memory Used DU AT FT CT TT TT Input >>>> ShuffleRead >>>> ShuffleWrite Thread Dump >>>> 1 vm-18.cloud.com:51796 0 0.0B/530.3MB 0.0 B 1 0 16 17 634 ms 0.0 B 2047.0 >>>> B 1710.0 B Thread Dump >>>> 2 vm-13.cloud.com:57264 0 0.0B/530.3MB 0.0 B 0 0 1427 1427 5.5 m 0.0 B 0.0 >>>> B 0.0 B Thread Dump >>>> 3 vm-14.cloud.com:54570 0 0.0B/530.3MB 0.0 B 0 0 1379 1379 5.2 m 0.0 B >>>> 1368.0 >>>> B 2.8 KB Thread Dump >>>> 4 vm-11.cloud.com:56201 0 0.0B/530.3MB 0.0 B 0 0 10 10 625 ms 0.0 B 1368.0 >>>> B 1026.0 B Thread Dump >>>> 5 vm-5.cloud.com:42958 0 0.0B/530.3MB 0.0 B 0 0 22 22 632 ms 0.0 B 1881.0 >>>> B 2.8 KB Thread Dump >>>> <driver> vm.cloud.com:51847 0 0.0B/530.0MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 >>>> B 0.0 B Thread Dump >>>> >>>> /homext/spark/bin/spark-submit >>>> --master yarn-cluster --num-executors 2 --driver-memory 512m >>>> --executor-memory 512m --executor-cores 2 >>>> --class com.oracle.ci.CmsgK2H /homext/lib/MJ-ci-k2h.jar >>>> vm.cloud.com:2181/kafka spark-yarn avro 1 5000 >>>> >>>> STANDALONE(3 workers + driver) >>>> ============================== >>>> Executor ID Address RDD Blocks Memory Used DU AT FT CT TT TT Input >>>> ShuffleRead >>>> ShuffleWrite Thread Dump >>>> 0 vm-71.cloud.com:55912 0 0.0B/265.0MB 0.0 B 0 0 1069 1069 6.0 m 0.0 B >>>> 1534.0 >>>> B 3.0 KB Thread Dump >>>> 1 vm-72.cloud.com:40897 0 0.0B/265.0MB 0.0 B 0 0 1057 1057 5.9 m 0.0 B >>>> 1368.0 >>>> B 4.0 KB Thread Dump >>>> 2 vm-73.cloud.com:37621 0 0.0B/265.0MB 0.0 B 1 0 1059 1060 5.9 m 0.0 B 2.0 >>>> KB 1368.0 B Thread Dump >>>> <driver> vm.cloud.com:58299 0 0.0B/265.0MB 0.0 B 0 0 0 0 0 ms 0.0 B 0.0 >>>> B 0.0 B Thread Dump >>>> >>>> /homext/spark/bin/spark-submit >>>> --master spark://chsnmvproc71vm3.usdc2.oraclecloud.com:7077 >>>> --class com.oracle.ci.CmsgK2H /homext/lib/MJ-ci-k2h.jar >>>> vm.cloud.com:2181/kafka spark-standalone avro 1 5000 >>>> >>>> PS: I did go through the spark website and >>>> http://www.virdata.com/tuning-spark/, but was out of any luck. >>>> >>>> -- >>>> Cheers, >>>> Mukesh Jha >>>> >>> >>> >> >> >> -- >> >> >> Thanks & Regards, >> >> *Mukesh Jha <me.mukesh....@gmail.com>* >> > >
