Hi Krishna,
4) What's the "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" in
your configuration?
50
I think this config is problematic, too small heartbeat-interval will cause
NM contact RM too often. I would suggest you can set this value larger like
1000.
Thanks,
Wangda
On Wed, Aug 13, 2014 at 4:42 PM, Krishna Kishore Bonagiri <
write2kish...@gmail.com> wrote:
> Hi Wangda,
> Thanks for the reply, here are the details, please see if you could
> suggest anything.
>
> 1) Number of nodes and running app in the cluster
> 2 nodes, and I am running my own application that keeps asking for
> containers,
> a) running something on the containers,
> b) releasing the containers,
> c) ask for more containers with incremented priority value, and repeat the
> same process
>
> 2) What's the version of your Hadoop?
> apache hadoop-2.4.0
>
> 3) Have you set
> "yarn.scheduler.capacity.schedule-asynchronously.enable"=true?
> No
>
> 4) What's the "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms" in
> your configuration?
> 50
>
>
>
>
> On Tue, Aug 12, 2014 at 12:44 PM, Wangda Tan <wheele...@gmail.com> wrote:
>
>> Hi Krishna,
>> To get more understanding about the problem, could you please share
>> following information:
>> 1) Number of nodes and running app in the cluster
>> 2) What's the version of your Hadoop?
>> 3) Have you set
>> "yarn.scheduler.capacity.schedule-asynchronously.enable"=true?
>> 4) What's the "yarn.resourcemanager.nodemanagers.heartbeat-interval-ms"
>> in your configuration?
>>
>> Thanks,
>> Wangda Tan
>>
>>
>>
>> On Sun, Aug 10, 2014 at 11:29 PM, Krishna Kishore Bonagiri <
>> write2kish...@gmail.com> wrote:
>>
>>> Hi,
>>> My YARN resource manager is consuming 100% CPU when I am running an
>>> application that is running for about 10 hours, requesting as many as 27000
>>> containers. The CPU consumption was very low at the starting of my
>>> application, and it gradually went high to over 100%. Is this a known issue
>>> or are we doing something wrong?
>>>
>>> Every dump of the EVent Processor thread is running
>>> LeafQueue::assignContainers() specifically the for loop below from
>>> LeafQueue.java and seems to be looping through some priority list.
>>>
>>> // Try to assign containers to applications in order
>>> for (FiCaSchedulerApp application : activeApplications) {
>>> ...
>>> // Schedule in priority order
>>> for (Priority priority : application.getPriorities()) {
>>>
>>> 3XMTHREADINFO "ResourceManager Event Processor"
>>> J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
>>> java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
>>> 3XMJAVALTHREAD (java/lang/Thread getId:0x1E, isDaemon:false)
>>> 3XMTHREADINFO1 (native thread ID:0x4B64, native priority:0x5,
>>> native policy:UNKNOWN)
>>> 3XMTHREADINFO2 (native stack address range
>>> from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
>>> 3XMCPUTIME *CPU usage total: 42334.614623696 secs*
>>> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=20456
>>> (0x4FE8)
>>> 3XMTHREADINFO3 Java callstack:
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:850(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
>>> entry count: 1)
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 2)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
>>> 4XESTACKTRACE at java/lang/Thread.run(Thread.java:853)
>>>
>>> 3XMTHREADINFO "ResourceManager Event Processor"
>>> J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
>>> java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
>>> 3XMJAVALTHREAD (java/lang/Thread getId:0x1E, isDaemon:false)
>>> 3XMTHREADINFO1 (native thread ID:0x4B64, native priority:0x5,
>>> native policy:UNKNOWN)
>>> 3XMTHREADINFO2 (native stack address range
>>> from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
>>> 3XMCPUTIME CPU usage total: 42379.604203548 secs
>>> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=57280
>>> (0xDFC0)
>>> 3XMTHREADINFO3 Java callstack:
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:841(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
>>> entry count: 1)
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 2)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
>>> 4XESTACKTRACE at java/lang/Thread.run(Thread.java:853)
>>>
>>> 3XMTHREADINFO "ResourceManager Event Processor"
>>> J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
>>> java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
>>> 3XMJAVALTHREAD (java/lang/Thread getId:0x1E, isDaemon:false)
>>> 3XMTHREADINFO1 (native thread ID:0x4B64, native priority:0x5,
>>> native policy:UNKNOWN)
>>> 3XMTHREADINFO2 (native stack address range
>>> from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
>>> 3XMCPUTIME CPU usage total: 42996.394528764 secs
>>> 3XMHEAPALLOC Heap bytes allocated since last GC cycle=475576
>>> (0x741B8)
>>> 3XMTHREADINFO3 Java callstack:
>>> 4XESTACKTRACE at
>>> java/util/TreeMap.successor(TreeMap.java:2001(Compiled Code))
>>> 4XESTACKTRACE at
>>> java/util/TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1127(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> java/util/TreeMap$KeyIterator.next(TreeMap.java:1180(Compiled Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:838(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
>>> entry count: 1)
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 2)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
>>> Code))
>>> 5XESTACKTRACE (entered lock:
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
>>> entry count: 1)
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
>>> Code))
>>> 4XESTACKTRACE at
>>> org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
>>> 4XESTACKTRACE at java/lang/Thread.run(Thread.java:853)
>>>
>>> Thanks,
>>> Kishore
>>>
>>
>>
>
