@yohann sorry I am assuming you meant application master if so I believe spark is the one that provides application master. Is there anyway to look for how much resources are being requested and how much yarn is allowed to provide? I would assume this is a common case if so I am not sure why these numbers are not part of resource manager logs?
On Sun, Jul 8, 2018 at 8:09 AM, kant kodali <kanth...@gmail.com> wrote: > yarn.scheduler.capacity.maximum-am-resource-percent by default is set to > 0.1 and I tried changing it to 1.0 and still no luck. same problem > persists. The master here is yarn and I just trying to spawn spark-shell > --master yarn --deploy-mode client and run a simple world count so I am not > sure why it would request for more resources? > > On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjar...@hotmail.com> > wrote: > >> Following the logs from the resource manager: >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue, it is likely set too low. skipping enforcement to >> allow at least one application to start >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue for user, it is likely set too low. skipping >> enforcement to allow at least one application to start >> >> I’d say it has nothing to do with spark. Your master is just asking more >> resources than the default Yarn queue is allowed to provide. >> You might take a look at https://hadoop.apache.org/docs >> /r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search >> for maximum-am-resource-percent. >> >> Regards, >> >> *Yohann Jardin* >> Le 7/8/2018 à 4:40 PM, kant kodali a écrit : >> >> Hi, >> >> It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 >> vCpu! I am not running any code since I can't even spawn spark-shell with >> yarn as master as described in my previous email. I just want to run simple >> word count using yarn as master. >> >> Thanks! >> >> Below is the resource manager log once again if that helps >> >> >> 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.ParentQueue: Application added - >> appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root >> #applications: >> 1 >> >> 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.CapacityScheduler: Accepted >> application application_1531059242261_0001 from user: xxx, in queue: >> default >> >> 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State >> change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED >> >> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. >> resourcemanager.ApplicationMasterService: Registering app attempt : >> appattempt_1531059242261_0001_000001 >> >> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.attempt.RMAppAttemptImpl: >> appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue, it is likely set too low. skipping enforcement to >> allow at least one application to start >> >> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: >> maximum-am-resource-percent is insufficient to start a single >> application in queue for user, it is likely set too low. skipping >> enforcement to allow at least one application to start >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: Application >> application_1531059242261_0001 from user: xxx activated in queue: default >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: >> application_1531059242261_0001 user: org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: >> default #user-pending-applications: 0 #user-active-applications: 1 >> #queue-pending-applications: 0 #queue-active-applications: 1 >> >> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. >> resourcemanager.scheduler.capacity.CapacityScheduler: Added Application >> Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx >> in queue default >> >> 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server. >> resourcemanager.rmapp.attempt.RMAppAttemptImpl: >> appattempt_1531059242261_0001_000001 State change from SUBMITTED to >> SCHEDULED >> >> >> >> >
