Jonathan Hung created YARN-10192: ------------------------------------ Summary: CapacityScheduler stuck in loop rejecting allocation proposals Key: YARN-10192 URL: https://issues.apache.org/jira/browse/YARN-10192 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Hung
On a 2.10.0 cluster, we observed containers were being scheduled very slowly. Based on logs, it seems to reject a bunch of allocation proposals, then accept a bunch of reserved containers, but very few containers are actually getting allocated: {noformat} 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,981 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,988 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,988 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,989 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,989 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8957.grid.linkedin.com:8041 2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,095 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15460 resource=<memory:2048, vCores:1> queue=misc_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu 2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal 2020-03-10 06:31:49,132 INFO com.linkedin.hadoop.yarn.ams.NodeLabelRoutingAMSProcessor: Applying node label cpu to container request from application_1582403122262_15460 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000283 Container Transitioned from ACQUIRED to RELEASED 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1582403122262_15460 CONTAINERID=container_e43_1582403122262_15460_01_000283 RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000284 Container Transitioned from ACQUIRED to RELEASED 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1582403122262_15460 CONTAINERID=container_e43_1582403122262_15460_01_000284 RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000285 Container Transitioned from ACQUIRED to RELEASED 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1582403122262_15460 CONTAINERID=container_e43_1582403122262_15460_01_000285 RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu 2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate of application :application_1582403122262_15460 2020-03-10 06:31:49,197 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8950.grid.linkedin.com:8041 2020-03-10 06:31:49,198 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,198 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8650.grid.linkedin.com:8041 2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8696.grid.linkedin.com:8041 2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8689.grid.linkedin.com:8041 2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9327.grid.linkedin.com:8041 2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8617.grid.linkedin.com:8041 2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9342.grid.linkedin.com:8041 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8941.grid.linkedin.com:8041 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3167.grid.linkedin.com:8041 2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,357 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8679.grid.linkedin.com:8041 2020-03-10 06:31:49,358 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,358 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3161.grid.linkedin.com:8041 2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3197.grid.linkedin.com:8041 2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9364.grid.linkedin.com:8041 2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9367.grid.linkedin.com:8041 2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8668.grid.linkedin.com:8041 2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8952.grid.linkedin.com:8041 2020-03-10 06:31:49,383 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,383 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9329.grid.linkedin.com:8041 2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8682.grid.linkedin.com:8041 2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,403 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8708.grid.linkedin.com:8041 2020-03-10 06:31:49,404 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,404 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8932.grid.linkedin.com:8041 2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9340.grid.linkedin.com:8041 2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9378.grid.linkedin.com:8041 2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3171.grid.linkedin.com:8041 2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9365.grid.linkedin.com:8041 2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8919.grid.linkedin.com:8041 2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8644.grid.linkedin.com:8041 2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8614.grid.linkedin.com:8041 2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3176.grid.linkedin.com:8041 2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted 2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8615.grid.linkedin.com:8041 2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> 2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org