[ https://issues.apache.org/jira/browse/YARN-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280357#comment-15280357 ]
Miles Crawford commented on YARN-5039: -------------------------------------- Okay, wow that is wonderful to know. We have a highly variable workload and make almost no use of HDFS, so we use a tiny number of CORE nodes and remove TASK nodes when things are idle to save on costs. We do not use spot instances at all (because of https://issues.apache.org/jira/browse/SPARK-14209) I cannot seem to find any mention of this behavior in the EMR documentation, so it's a bit of a blindside. Additionally, the Node Labels page of the Hadoop UI does not distinguish between spot and task, so I wasn't even aware labeling was going on. I guess things are working as designed, so I'm sorry to take up all your time. Thanks very much for helping. I think I'll follow up with AWS and request a documentation fix. > Applications ACCEPTED but not starting > -------------------------------------- > > Key: YARN-5039 > URL: https://issues.apache.org/jira/browse/YARN-5039 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.7.2 > Reporter: Miles Crawford > Attachments: Screen Shot 2016-05-04 at 1.57.19 PM.png, Screen Shot > 2016-05-04 at 2.41.22 PM.png, capacity-scheduler-at-debug.log.gz, > queue-config.log, resource-manager-application-starts.log.gz, > whole-scheduler-at-debug.log.gz, > yarn-yarn-resourcemanager-ip-10-12-47-144.log.gz > > > Often when we submit applications to an incompletely utilized cluster, they > sit, unable to start for no apparent reason. > There are multiple nodes in the cluster with available resources, but the > resourcemanger logs show that scheduling is being skipped. The scheduling is > skipped because the application itself has reserved the node? I'm not sure > how to interpret this log output: > {code} > 2016-05-04 20:19:21,315 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Trying to fulfill reservation for > application application_1462291866507_0025 on node: > ip-10-12-43-54.us-west-2.compute.internal:8041 > 2016-05-04 20:19:21,316 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Reserved container > application=application_1462291866507_0025 resource=<memory:50688, vCores:1> > queue=default: capacity=1.0, absoluteCapacity=1.0, > usedResources=<memory:1894464, vCores:33>, usedCapacity=0.7126589, > absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 > usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used=<memory:1894464, > vCores:33> cluster=<memory:2658304, vCores:704> > 2016-05-04 20:19:21,316 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Skipping scheduling since node > ip-10-12-43-54.us-west-2.compute.internal:8041 is reserved by application > appattempt_1462291866507_0025_000001 > 2016-05-04 20:19:22,232 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Trying to fulfill reservation for > application application_1462291866507_0025 on node: > ip-10-12-43-53.us-west-2.compute.internal:8041 > 2016-05-04 20:19:22,232 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Reserved container > application=application_1462291866507_0025 resource=<memory:50688, vCores:1> > queue=default: capacity=1.0, absoluteCapacity=1.0, > usedResources=<memory:1894464, vCores:33>, usedCapacity=0.7126589, > absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 > usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used=<memory:1894464, > vCores:33> cluster=<memory:2658304, vCores:704> > 2016-05-04 20:19:22,232 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Skipping scheduling since node > ip-10-12-43-53.us-west-2.compute.internal:8041 is reserved by application > appattempt_1462291866507_0025_000001 > 2016-05-04 20:19:22,316 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Trying to fulfill reservation for > application application_1462291866507_0025 on node: > ip-10-12-43-54.us-west-2.compute.internal:8041 > 2016-05-04 20:19:22,316 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Reserved container > application=application_1462291866507_0025 resource=<memory:50688, vCores:1> > queue=default: capacity=1.0, absoluteCapacity=1.0, > usedResources=<memory:1894464, vCores:33>, usedCapacity=0.7126589, > absoluteUsedCapacity=0.7126589, numApps=2, numContainers=33 > usedCapacity=0.7126589 absoluteUsedCapacity=0.7126589 used=<memory:1894464, > vCores:33> cluster=<memory:2658304, vCores:704> > 2016-05-04 20:19:22,316 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler > (ResourceManager Event Processor): Skipping scheduling since node > ip-10-12-43-54.us-west-2.compute.internal:8041 is reserved by application > appattempt_1462291866507_0025_000001 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org