Hey Stephen,
I was on vacation last week, I'm looking over the logs this week. I've got
a few ideas for a first but may take me a while as I get back into work.
Darin
On Fri, Jul 1, 2016 at 2:43 AM, Stephen Gran <stephen.g...@piksel.com>
wrote:
> Hi,
>
> It's not a problem at all. Anything I can do to help.
>
> I've attached the log file for the relevant time period. This is hadoop
> 2.7.2 - you have a good memory :)
>
> Cheers,
>
> On 30/06/16 22:56, Darin Johnson wrote:
> > Hey Steven,
> >
> > Looks like this might be slightly different than what I was originally
> > expecting. Sorry to keep asking for more info but it will help me
> recreate
> > the issue. Could you possibly get me more of the ResourceManager logs?
> In
> > particular, I'm trying to figure out where upgradeNodeCapacity is getting
> > called from and any transitions of slave2. Also, what version of hadoop
> > are you running, I think I recall it being 2.72 but should verify.
> >
> > Thanks for taken the time to work with me on this.
> >
> > Darin
> >
> > On Thu, Jun 30, 2016 at 5:10 PM, Stephen Gran <stephen.g...@piksel.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Yes - the imaginatively named slave2 was a zero-sized nm at that point -
> >> I am looking at how small a pool of reserved resource I can get away
> >> with, and use FGS for burst activity.
> >>
> >>
> >> Here are all the logs related to that host:port combination around that
> >> time:
> >>
> >> 2016-06-30 19:47:43,756 INFO
> >> org.apache.hadoop.yarn.util.AbstractLivelinessMonitor:
> >> Expired:slave2:24679 Timed out after 2 secs
> >> 2016-06-30 19:47:43,771 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> Deactivating Node slave2:24679 as it is now LOST
> >> 2016-06-30 19:47:43,771 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> slave2:24679 Node Transitioned from RUNNING to LOST
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> >> yarn_Container: [ContainerId: container_1467314892573_0009_01_000005,
> >> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> >> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> >> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> >> cpu and 1 mem.
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> >> Released container container_1467314892573_0009_01_000005 of capacity
> >> <memory:2048, vCores:1> on host slave2:24679, which currently has 1
> >> containers, <memory:2048, vCores:1> used and <memory:2048, vCores:1>
> >> available, release resources=true
> >> 2016-06-30 19:47:43,909 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Application attempt appattempt_1467314892573_0009_000001 released
> >> container container_1467314892573_0009_01_000005 on node: host:
> >> slave2:24679 #containers=1 available=<memory:2048, vCores:1>
> >> used=<memory:2048, vCores:1> with event: KILL
> >> 2016-06-30 19:47:43,909 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> >> Node not found resyncing slave2:24679
> >> 2016-06-30 19:47:43,952 INFO
> >> org.apache.myriad.scheduler.fgs.YarnNodeCapacityManager: Removed task
> >> yarn_Container: [ContainerId: container_1467314892573_0009_01_000006,
> >> NodeId: slave2:24679, NodeHttpAddress: slave2:23177, Resource:
> >> <memory:2048, vCores:1>, Priority: 20, Token: Token { kind:
> >> ContainerToken, service: 10.0.5.5:24679 }, ] with exit status freeing 0
> >> cpu and 1 mem.
> >> 2016-06-30 19:47:43,952 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode:
> >> Released container container_1467314892573_0009_01_000006 of capacity
> >> <memory:2048, vCores:1> on host slave2:24679, which currently has 0
> >> containers, <memory:0, vCores:0> used and <memory:4096, vCores:2>
> >> available, release resources=true
> >> 2016-06-30 19:47:43,952 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Application attempt appattempt_1467314892573_0009_000001 released
> >> container container_1467314892573_0009_01_000006 on node: host:
> >> slave2:24679 #containers=0 available=<memory:4096, vCores:2>
> >> used=<memory:0, vCores:0> with event: KILL
> >> 2016-06-30 19:47:43,952 INFO
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >> Removed node slave2:24679 cluster capacity: <memory:4096, vCores:4>
> >> 2016-06-30 19:47:47,573 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl:
> >> slave2:24679 Node Transitioned from NEW to RUNNING
> >> 2016-06-30 19:47:47,936 INFO
> >> org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService:
> >> NodeManager from node slave2(cmPort: 24679 httpPort: 23177) registered
> >> with capability: <memory:0, vCores:0>, assigned nodeId slave2:24679
> >>
> >>
> >> Looks like it did go into LOST for a bit.
> >>
> >> Cheers,
> >>
> >> On 30/06/16 21:36, Darin Johnson wrote:
> >>> Steven, thanks. I thought I had fixed that but perhaps a regression
> was
> >>> made in another merge. I'll look into it, can you answer a few
> >> questions?
> >>> Was the node (slave2) a zero sided nodemanager (for fgs)? In the node
> >>> manager logs had it recently become unhealthy? I'm pretty concerned
> >> about
> >>> this and will try to get a patch soon.
> >>>
> >>> Thanks,
> >>>
> >>> Darin
> >>> On Jun 30, 2016 3:53 PM, "Stephen Gran" <stephen.g...@piksel.com>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Just playing with the 0.2.0 release (congratulations, by the way!)
> >>>>
> >>>> I have seen this twice now, although it is by no means consistent - I
> >>>> will have a dozen successful runs, and then one of these. This exits
> >>>> the RM, which makes it rather noticable.
> >>>>
> >>>> 2016-06-30 19:47:43,952 INFO
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler:
> >>>> Removed node slave2:24679 cluster capacity: <memory:4096, vCore
> >>>> s:4>
> >>>> 2016-06-30 19:47:43,953 FATAL
> >>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> in
> >>>> handling event type NODE_RESOURCE_UPDATE to the scheduler
> >>>> java.lang.NullPointerException
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNodeResource(AbstractYarnScheduler.java:563)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.updateNodeResource(FairScheduler.java:1652)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1222)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:102)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.myriad.scheduler.yarn.MyriadFairScheduler.handle(MyriadFairScheduler.java:42)
> >>>> at
> >>>>
> >>>>
> >>
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:671)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-06-30 19:47:43,972 INFO
> >>>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
> Exiting,
> >>>> bbye..
> >>>>
> >>>> --
> >>>> Stephen Gran
> >>>> Senior Technical Architect
> >>>>
> >>>> picture the possibilities | piksel.com
> >>>> This message is private and confidential. If you have received this
> >>>> message in error, please notify the sender or serviced...@piksel.com
> >> and
> >>>> remove it from your system.
> >>>>
> >>>> Piksel Inc is a company registered in the United States New York City,
> >>>> 1250 Broadway, Suite 1902, New York, NY 10001. F No. = 2931986
> >>>>
> >>>
> >>
> >> --
> >> Stephen Gran
> >> Senior Technical Architect
> >>
> >> picture the possibilities | piksel.com
> >>
> >
>
> --
> Stephen Gran
> Senior Technical Architect
>
> picture the possibilities | piksel.com
>
