[
https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3641:
-------------------------------------
Attachment: MAPREDUCE-3641.patch
Update patch with some bug-fixes and some extra debug info (not for commit).
I'm still hunting a NPE down:
{noformat}
12/01/12 23:47:38 ERROR resourcemanager.ResourceManager: Error in handling
event type NODE_UPDATE to the schedulerjava.lang.NullPointerException at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.unreserve(SchedulerApp.java:392)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.unreserve(LeafQueue.java:1217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1158)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignRackLocalContainers(LeafQueue.java:1027)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:821)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:721)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:568)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:615)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:295)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I'm also seeing negative container counts for the queue etc.
Need to dig some more.
> CapacityScheduler should be more conservative assigning off-switch requests
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-3641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, scheduler
> Affects Versions: 0.23.0
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch
>
>
> In hadoop-1, the CS is very conservative handing out off-switch assignments,
> we need to do the same in YARN.
> We noticed performance regressions due to this, particularly for reduces.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
