Can a container have multiple JVMs running in YARN? I am comparing Hadoop Mapreduce running on yarn vs spark running on yarn here :
1.Is the difference is in Hadoop Mapreduce job - say I specify 20 reducers and my job uses 10 map tasks then, it need total 30 containers or 30 vcores ? I guess 30 vcores and runs multiple max 4 jvms (set using max vcores in a container) in my case ?So in taotal 8 containers - 7 with 4 vcores and 1 with 2 ? 2.for each map/reduce tasks requested is equal to vcore requested and it launch a new JVM inside same container or per container 1 JVM ? 3.In spark jobs it launches task in same jvm running in a container and 1 container has max 1 JVM and max tasks = max vcore per container limit (4 in my case). On Wed, Jul 15, 2015 at 12:17 AM, Jong Wook Kim <jongw...@nyu.edu> wrote: > it's probably because your YARN cluster has only 40 vCores available. > > Go to your resource manager and check if "VCores Total" and "Memory Total" > exceeds what you have set. (40 cores and 5120 MB) > > If that looks fine, go to "Scheduler" page and find the queue on which > your jobs run, and check the resources allocated for that queue. > > Hope this helps. > > Jong Wook > > > > On Jul 15, 2015, at 01:57, Shushant Arora <shushantaror...@gmail.com> > wrote: > > > > I am running spark application on yarn managed cluster. > > > > When I specify --executor-cores > 4 it fails to start the application. > > I am starting the app as > > > > spark-submit --class classname --num-executors 10 --executor-cores 5 > --master masteradd jarname > > > > Exception in thread "main" org.apache.spark.SparkException: Yarn > application has already ended! It might have been killed or unable to > launch application master. > > > > When I give --executor-cores as 4 , it works fine. > > > > My Cluster has 10 nodes . > > Why am I not able to specify more than 4 concurrent tasks. Is there any > max limit yarn side or spark side which I can override to make use of more > tasks ? > >
