yarn.scheduler.capacity.maximum-am-resource-percent by default is set to 0.1 and I tried changing it to 1.0 and still no luck. same problem persists. The master here is yarn and I just trying to spawn spark-shell --master yarn --deploy-mode client and run a simple world count so I am not sure why it would request for more resources?
On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjar...@hotmail.com> wrote: > Following the logs from the resource manager: > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue, it is likely set > too low. skipping enforcement to allow at least one application to start > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue for user, it is > likely set too low. skipping enforcement to allow at least one > application to start > > I’d say it has nothing to do with spark. Your master is just asking more > resources than the default Yarn queue is allowed to provide. > You might take a look at https://hadoop.apache.org/ > docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and > search for maximum-am-resource-percent. > > Regards, > > *Yohann Jardin* > Le 7/8/2018 à 4:40 PM, kant kodali a écrit : > > Hi, > > It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 > vCpu! I am not running any code since I can't even spawn spark-shell with > yarn as master as described in my previous email. I just want to run simple > word count using yarn as master. > > Thanks! > > Below is the resource manager log once again if that helps > > > 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.ParentQueue: Application added - > appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root > #applications: > 1 > > 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.CapacityScheduler: Accepted > application application_1531059242261_0001 from user: xxx, in queue: > default > > 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State > change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED > > 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. > resourcemanager.ApplicationMasterService: Registering app attempt : > appattempt_1531059242261_0001_000001 > > 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue, it is likely set > too low. skipping enforcement to allow at least one application to start > > 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent > is insufficient to start a single application in queue for user, it is > likely set too low. skipping enforcement to allow at least one > application to start > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: Application > application_1531059242261_0001 from user: xxx activated in queue: default > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: > application_1531059242261_0001 user: org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue: > default #user-pending-applications: 0 #user-active-applications: 1 > #queue-pending-applications: 0 #queue-active-applications: 1 > > 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server. > resourcemanager.scheduler.capacity.CapacityScheduler: Added Application > Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx in > queue default > > 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server. > resourcemanager.rmapp.attempt.RMAppAttemptImpl: > appattempt_1531059242261_0001_000001 State change from SUBMITTED to > SCHEDULED > > > >
