[ https://issues.apache.org/jira/browse/YARN-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611086#comment-13611086 ]
Lohit Vijayarenu commented on YARN-502: --------------------------------------- Looks like node might already be removed from nodes map and cleanup was happening on removed node {code} FSSchedulerNode node = nodes.get(rmNode.getNodeID()); Resources.subtractFrom(clusterCapacity, rmNode.getTotalCapability()); // Remove running containers List<RMContainer> runningContainers = node.getRunningContainers(); for (RMContainer container : runningContainers) { completedContainer(container, SchedulerUtils.createAbnormalContainerStatus( container.getContainerId(), SchedulerUtils.LOST_CONTAINER), RMContainerEventType.KILL); } {code} I am trying to find out case where node might have been removed. This was not called twice because I do not see the corresponding log line {code} LOG.info("Removed node " + rmNode.getNodeAddress() + " cluster capacity: " + clusterCapacity); {code} Either node was removed somewhere else or we tried to remove node which was not in map. Either way, having check for NPE makes sense in this. > RM crash with NPE on NODE_REMOVED event > --------------------------------------- > > Key: YARN-502 > URL: https://issues.apache.org/jira/browse/YARN-502 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.0.3-alpha > Reporter: Lohit Vijayarenu > > While running some test and adding/removing nodes, we see RM crashed with the > below exception. We are testing with fair scheduler and running > hadoop-2.0.3-alpha > {noformat} > 2013-03-22 18:54:27,015 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating > Node YYYY:55680 as it is now LOST > 2013-03-22 18:54:27,015 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: YYYY:55680 > Node Transitioned from UNHEALTHY to LOST > 2013-03-22 18:54:27,015 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_REMOVED to the scheduler > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.removeNode(FairScheduler.java:619) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:856) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:98) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:375) > at java.lang.Thread.run(Thread.java:662) > 2013-03-22 18:54:27,016 INFO > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye.. > 2013-03-22 18:54:27,020 INFO org.mortbay.log: Stopped > SelectChannelConnector@XXXX:50030 > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira