The scheduler configurations are helpful as well, but not useful without the information outlined above.
-Sandy On Fri, Jun 26, 2015 at 10:34 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > These are my YARN queue configurations > > Queue State:RUNNINGUsed Capacity:206.7%Absolute Used Capacity:3.1%Absolute > Capacity:1.5%Absolute Max Capacity:10.0%Used Resources:<memory:5578496, > vCores:390>Num Schedulable Applications:7Num Non-Schedulable Applications: > 0Num Containers:390Max Applications:45Max Applications Per User:27Max > Schedulable Applications:1278Max Schedulable Applications Per > User:116Configured > Capacity:1.5%Configured Max Capacity:10.0%Configured Minimum User Limit > Percent:30%Configured User Limit Factor:2.0 > Executors: > ./bin/spark-submit -v --master yarn-cluster --driver-class-path > /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar > --jars > /apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar,/home/dvasthimal/spark1.3/1.3.1.lib/spark_reporting_dep_only-1.0-SNAPSHOT-jar-with-dependencies.jar > * --num-executors 9973 --driver-memory 14g --driver-java-options > "-XX:MaxPermSize=512M -Xmx4096M -Xms4096M -verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps" --executor-memory 14g --executor-cores 1 *--queue > hdmi-others --class com.ebay.ep.poc.spark.reporting.SparkApp > /home/dvasthimal/spark1.3/1.3.1.lib/spark_reporting-1.0-SNAPSHOT.jar > startDate=2015-06-20 endDate=2015-06-21 > input=/apps/hdmi-prod/b_um/epdatasets/exptsession subcommand=viewItem > output=/user/dvasthimal/epdatasets/viewItem buffersize=128 > maxbuffersize=1068 maxResultSize=200G > > > > > On Thu, Jun 25, 2015 at 4:52 PM, Sandy Ryza <sandy.r...@cloudera.com> > wrote: > >> How many nodes do you have, how much space is allocated to each node for >> YARN, how big are the executors you're requesting, and what else is running >> on the cluster? >> >> On Thu, Jun 25, 2015 at 3:57 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> >> wrote: >> >>> I run Spark App on Spark 1.3.1 over YARN. >>> >>> When i request --num-executors 9973 and when i see Executors from >>> Environment tab from SPARK UI its between 200 to 300. >>> >>> What is incorrect here ? >>> >>> -- >>> Deepak >>> >>> >> > > > -- > Deepak > >
