Re: Challenges after MapR 5.1 Upgrade.
Hey John, I noticed these lines in your yarn-site.xml: yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.minimum-allocation-vcores 1 If your attempting to launch a zero resource nodemanager for fgs that will result in the first stack trace. Both should be explicitly 0 for that feature to work (defaults are 1024 and 1 resp, which will fail). You do have them set below to 0, however I'm in certain which would take precedence. On Apr 4, 2016 5:19 PM, "John Omernik" wrote: > This was a Upgrade from 5.0. I will post here, note: I have removed the > mapr_shuffle to get node managers to work, however, I am seeing other odd > things, so any help would be appreciated. > > > > > > > yarn.nodemanager.aux-services > mapreduce_shuffle,myriad_executor > > > > yarn.resourcemanager.hostname > myriadprod.marathonprod.mesos > > > yarn.nodemanager.aux-services.mapreduce_shuffle.class > org.apache.hadoop.mapred.ShuffleHandler > > > yarn.nodemanager.aux-services.myriad_executor.class > org.apache.myriad.executor.MyriadExecutorAuxService > > > yarn.nm.liveness-monitor.expiry-interval-ms > 2000 > > > yarn.am.liveness-monitor.expiry-interval-ms > 1 > > > yarn.resourcemanager.nm.liveness-monitor.interval-ms > 1000 > > > > yarn.nodemanager.resource.cpu-vcores > ${nodemanager.resource.cpu-vcores} > > > yarn.nodemanager.resource.memory-mb > ${nodemanager.resource.memory-mb} > > > > > yarn.scheduler.minimum-allocation-mb > 512 > > > > yarn.scheduler.minimum-allocation-vcores > 1 > > > > > > > yarn.nodemanager.address > ${myriad.yarn.nodemanager.address} > > > yarn.nodemanager.webapp.address > ${myriad.yarn.nodemanager.webapp.address} > > > yarn.nodemanager.webapp.https.address > ${myriad.yarn.nodemanager.webapp.address} > > > yarn.nodemanager.localizer.address > ${myriad.yarn.nodemanager.localizer.address} > > > > > yarn.resourcemanager.scheduler.class > org.apache.myriad.scheduler.yarn.MyriadFairScheduler > > > > > yarn.scheduler.minimum-allocation-vcores > 0 > > > yarn.scheduler.minimum-allocation-vcores > 0 > > > > Who will execute(launch) the containers. > yarn.nodemanager.container-executor.class > ${yarn.nodemanager.container-executor.class} > > > The class which should help the LCE handle > resources. > > > yarn.nodemanager.linux-container-executor.resources-handler.class > > > ${yarn.nodemanager.linux-container-executor.resources-handler.class} > > > > yarn.nodemanager.linux-container-executor.cgroups.hierarchy > > > ${yarn.nodemanager.linux-container-executor.cgroups.hierarchy} > > > > yarn.nodemanager.linux-container-executor.cgroups.mount > > ${yarn.nodemanager.linux-container-executor.cgroups.mount} > > > > yarn.nodemanager.linux-container-executor.cgroups.mount-path > > > ${yarn.nodemanager.linux-container-executor.cgroups.mount-path} > > > yarn.nodemanager.linux-container-executor.group > ${yarn.nodemanager.linux-container-executor.group} > > > yarn.nodemanager.linux-container-executor.path > ${yarn.home}/bin/container-executor > > > yarn.http.policy > HTTP_ONLY > > > > On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman > > wrote: > > > YarnDefaultProperties.java that defines class for mapr_direct_shuffle > > should be there even in 5.0, so nothing new there even if maprfs jar is > > outdated - could you also check that? > > Also could you paste content of your yarn-site.xml here? > > Thanks,Yuliya > > > > From: yuliya Feldman > > To: "dev@myriad.incubator.apache.org" > > Sent: Monday, April 4, 2016 1:43 PM > > Subject: Re: Challenges after MapR 5.1 Upgrade. > > > > Hello John, > > Did you upgrade to 5.1 or installed new one? > > Feels like MapR default properties were not loaded - I need to poke > around > > and then I will ask you for additional info > > Thanks,Yuliya > > > > From: John Omernik > > To: dev@myriad.incubator.apache.org > > Sent: Monday, April 4, 2016 12:29 PM > > Subject: Challenges after MapR 5.1 Upgrade. > > > > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, > and > > repackaged my hadoop tgz for remote distribution and now I have two > > problems occurring. > > > > 1. At first when I had the mapr direct shuffle enabled per the > > yarn-site.xml on the myriad documentaion, node managers would not start, > > and would fail with the error below. > > > > 2. Once I removed the mapr
Re: Challenges after MapR 5.1 Upgrade.
Hmmm... You did not have to specify yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler either Can you check if maprfs*.jar is on the classpath (should be - otherwise you would probably get more weird errors) and it has following class: YarnDefaultProperties Thanks,Yuliya From: John Omernik To: dev@myriad.incubator.apache.org; yuliya Feldman Sent: Monday, April 4, 2016 2:18 PM Subject: Re: Challenges after MapR 5.1 Upgrade. This was a Upgrade from 5.0. I will post here, note: I have removed the mapr_shuffle to get node managers to work, however, I am seeing other odd things, so any help would be appreciated. yarn.nodemanager.aux-services mapreduce_shuffle,myriad_executor yarn.resourcemanager.hostname myriadprod.marathonprod.mesos yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.aux-services.myriad_executor.class org.apache.myriad.executor.MyriadExecutorAuxService yarn.nm.liveness-monitor.expiry-interval-ms 2000 yarn.am.liveness-monitor.expiry-interval-ms 1 yarn.resourcemanager.nm.liveness-monitor.interval-ms 1000 yarn.nodemanager.resource.cpu-vcores ${nodemanager.resource.cpu-vcores} yarn.nodemanager.resource.memory-mb ${nodemanager.resource.memory-mb} yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.minimum-allocation-vcores 1 yarn.nodemanager.address ${myriad.yarn.nodemanager.address} yarn.nodemanager.webapp.address ${myriad.yarn.nodemanager.webapp.address} yarn.nodemanager.webapp.https.address ${myriad.yarn.nodemanager.webapp.address} yarn.nodemanager.localizer.address ${myriad.yarn.nodemanager.localizer.address} yarn.resourcemanager.scheduler.class org.apache.myriad.scheduler.yarn.MyriadFairScheduler yarn.scheduler.minimum-allocation-vcores 0 yarn.scheduler.minimum-allocation-vcores 0 Who will execute(launch) the containers. yarn.nodemanager.container-executor.class ${yarn.nodemanager.container-executor.class} The class which should help the LCE handle resources. yarn.nodemanager.linux-container-executor.resources-handler.class ${yarn.nodemanager.linux-container-executor.resources-handler.class} yarn.nodemanager.linux-container-executor.cgroups.hierarchy ${yarn.nodemanager.linux-container-executor.cgroups.hierarchy} yarn.nodemanager.linux-container-executor.cgroups.mount ${yarn.nodemanager.linux-container-executor.cgroups.mount} yarn.nodemanager.linux-container-executor.cgroups.mount-path ${yarn.nodemanager.linux-container-executor.cgroups.mount-path} yarn.nodemanager.linux-container-executor.group ${yarn.nodemanager.linux-container-executor.group} yarn.nodemanager.linux-container-executor.path ${yarn.home}/bin/container-executor yarn.http.policy HTTP_ONLY On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman wrote: > YarnDefaultProperties.java that defines class for mapr_direct_shuffle > should be there even in 5.0, so nothing new there even if maprfs jar is > outdated - could you also check that? > Also could you paste content of your yarn-site.xml here? > Thanks,Yuliya > > From: yuliya Feldman > To: "dev@myriad.incubator.apache.org" > Sent: Monday, April 4, 2016 1:43 PM > Subject: Re: Challenges after MapR 5.1 Upgrade. > > Hello John, > Did you upgrade to 5.1 or installed new one? > Feels like MapR default properties were not loaded - I need to poke around > and then I will ask you for additional info > Thanks,Yuliya > > From: John Omernik > To: dev@myriad.incubator.apache.org > Sent: Monday, April 4, 2016 12:29 PM > Subject: Challenges after MapR 5.1 Upgrade. > > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and > repackaged my hadoop tgz for remote distribution and now I have two > problems occurring. > > 1. At first when I had the mapr direct shuffle enabled per the > yarn-site.xml on the myriad documentaion, node managers would not start, > and would fail with the error below. > > 2. Once I removed the mapr shuffle from the yarn-site, I got node managers > started however, when I tried to launch a size 0, I got the other error > below. Not sure what's happening here. > > Any thoughts would be appreciated. Like I said, this was working with 5.0, > and now doesn't work in 5.1. > > Thanks! > > John > > SHuffle Error > > 16/04/04 13:46:34 INFO service.AbstractService:
Re: Challenges after MapR 5.1 Upgrade.
This was a Upgrade from 5.0. I will post here, note: I have removed the mapr_shuffle to get node managers to work, however, I am seeing other odd things, so any help would be appreciated. yarn.nodemanager.aux-services mapreduce_shuffle,myriad_executor yarn.resourcemanager.hostname myriadprod.marathonprod.mesos yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.aux-services.myriad_executor.class org.apache.myriad.executor.MyriadExecutorAuxService yarn.nm.liveness-monitor.expiry-interval-ms 2000 yarn.am.liveness-monitor.expiry-interval-ms 1 yarn.resourcemanager.nm.liveness-monitor.interval-ms 1000 yarn.nodemanager.resource.cpu-vcores ${nodemanager.resource.cpu-vcores} yarn.nodemanager.resource.memory-mb ${nodemanager.resource.memory-mb} yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.minimum-allocation-vcores 1 yarn.nodemanager.address ${myriad.yarn.nodemanager.address} yarn.nodemanager.webapp.address ${myriad.yarn.nodemanager.webapp.address} yarn.nodemanager.webapp.https.address ${myriad.yarn.nodemanager.webapp.address} yarn.nodemanager.localizer.address ${myriad.yarn.nodemanager.localizer.address} yarn.resourcemanager.scheduler.class org.apache.myriad.scheduler.yarn.MyriadFairScheduler yarn.scheduler.minimum-allocation-vcores 0 yarn.scheduler.minimum-allocation-vcores 0 Who will execute(launch) the containers. yarn.nodemanager.container-executor.class ${yarn.nodemanager.container-executor.class} The class which should help the LCE handle resources. yarn.nodemanager.linux-container-executor.resources-handler.class ${yarn.nodemanager.linux-container-executor.resources-handler.class} yarn.nodemanager.linux-container-executor.cgroups.hierarchy ${yarn.nodemanager.linux-container-executor.cgroups.hierarchy} yarn.nodemanager.linux-container-executor.cgroups.mount ${yarn.nodemanager.linux-container-executor.cgroups.mount} yarn.nodemanager.linux-container-executor.cgroups.mount-path ${yarn.nodemanager.linux-container-executor.cgroups.mount-path} yarn.nodemanager.linux-container-executor.group ${yarn.nodemanager.linux-container-executor.group} yarn.nodemanager.linux-container-executor.path ${yarn.home}/bin/container-executor yarn.http.policy HTTP_ONLY On Mon, Apr 4, 2016 at 3:53 PM, yuliya Feldman wrote: > YarnDefaultProperties.java that defines class for mapr_direct_shuffle > should be there even in 5.0, so nothing new there even if maprfs jar is > outdated - could you also check that? > Also could you paste content of your yarn-site.xml here? > Thanks,Yuliya > > From: yuliya Feldman > To: "dev@myriad.incubator.apache.org" > Sent: Monday, April 4, 2016 1:43 PM > Subject: Re: Challenges after MapR 5.1 Upgrade. > > Hello John, > Did you upgrade to 5.1 or installed new one? > Feels like MapR default properties were not loaded - I need to poke around > and then I will ask you for additional info > Thanks,Yuliya > > From: John Omernik > To: dev@myriad.incubator.apache.org > Sent: Monday, April 4, 2016 12:29 PM > Subject: Challenges after MapR 5.1 Upgrade. > > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and > repackaged my hadoop tgz for remote distribution and now I have two > problems occurring. > > 1. At first when I had the mapr direct shuffle enabled per the > yarn-site.xml on the myriad documentaion, node managers would not start, > and would fail with the error below. > > 2. Once I removed the mapr shuffle from the yarn-site, I got node managers > started however, when I tried to launch a size 0, I got the other error > below. Not sure what's happening here. > > Any thoughts would be appreciated. Like I said, this was working with 5.0, > and now doesn't work in 5.1. > > Thanks! > > John > > SHuffle Error > > 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed > in state INITED; cause: java.lang.RuntimeException: No class defined for > mapr_direct_shuffle > java.lang.RuntimeException: No class defined for mapr_direct_shuffle > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > > org.apache.hadoop.yarn.s
Re: Challenges after MapR 5.1 Upgrade.
YarnDefaultProperties.java that defines class for mapr_direct_shuffle should be there even in 5.0, so nothing new there even if maprfs jar is outdated - could you also check that? Also could you paste content of your yarn-site.xml here? Thanks,Yuliya From: yuliya Feldman To: "dev@myriad.incubator.apache.org" Sent: Monday, April 4, 2016 1:43 PM Subject: Re: Challenges after MapR 5.1 Upgrade. Hello John, Did you upgrade to 5.1 or installed new one? Feels like MapR default properties were not loaded - I need to poke around and then I will ask you for additional info Thanks,Yuliya From: John Omernik To: dev@myriad.incubator.apache.org Sent: Monday, April 4, 2016 12:29 PM Subject: Challenges after MapR 5.1 Upgrade. I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and repackaged my hadoop tgz for remote distribution and now I have two problems occurring. 1. At first when I had the mapr direct shuffle enabled per the yarn-site.xml on the myriad documentaion, node managers would not start, and would fail with the error below. 2. Once I removed the mapr shuffle from the yarn-site, I got node managers started however, when I tried to launch a size 0, I got the other error below. Not sure what's happening here. Any thoughts would be appreciated. Like I said, this was working with 5.0, and now doesn't work in 5.1. Thanks! John SHuffle Error 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.RuntimeException: No class defined for mapr_direct_shuffle java.lang.RuntimeException: No class defined for mapr_direct_shuffle at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics system... Zero Sized Node Manager Error: 16/04/04 14:22:49 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224) ... 6 more
Re: Challenges after MapR 5.1 Upgrade.
Hello John, Did you upgrade to 5.1 or installed new one? Feels like MapR default properties were not loaded - I need to poke around and then I will ask you for additional info Thanks,Yuliya From: John Omernik To: dev@myriad.incubator.apache.org Sent: Monday, April 4, 2016 12:29 PM Subject: Challenges after MapR 5.1 Upgrade. I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and repackaged my hadoop tgz for remote distribution and now I have two problems occurring. 1. At first when I had the mapr direct shuffle enabled per the yarn-site.xml on the myriad documentaion, node managers would not start, and would fail with the error below. 2. Once I removed the mapr shuffle from the yarn-site, I got node managers started however, when I tried to launch a size 0, I got the other error below. Not sure what's happening here. Any thoughts would be appreciated. Like I said, this was working with 5.0, and now doesn't work in 5.1. Thanks! John SHuffle Error 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.RuntimeException: No class defined for mapr_direct_shuffle java.lang.RuntimeException: No class defined for mapr_direct_shuffle at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics system... Zero Sized Node Manager Error: 16/04/04 14:22:49 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224) ... 6 more
Re: Challenges after MapR 5.1 Upgrade.
So prior to the upgrade, I had this code block commented out, however, it doesn't work without the comments either: yarn.scheduler.minimum-allocation-vcores 0 yarn.scheduler.minimum-allocation-vcores 0 On Mon, Apr 4, 2016 at 3:04 PM, John Omernik wrote: > > > yarn.resourcemanager.scheduler.class > > org.apache.myriad.scheduler.yarn.MyriadFairScheduler > > > > > On Mon, Apr 4, 2016 at 2:52 PM, Darin Johnson > wrote: > >> Hey John, I think the MapR guys will have some answers for you on the >> first >> stack trace, sounds like a missing jar on you class path. On the second >> I'm interested in knowing what MyriadScheduler your attempting to run, >> FAIR, CAPACITY, etc (Will be in your yarn-site.xml). >> >> Darin >> >> On Mon, Apr 4, 2016 at 3:29 PM, John Omernik wrote: >> >> > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, >> and >> > repackaged my hadoop tgz for remote distribution and now I have two >> > problems occurring. >> > >> > 1. At first when I had the mapr direct shuffle enabled per the >> > yarn-site.xml on the myriad documentaion, node managers would not start, >> > and would fail with the error below. >> > >> > 2. Once I removed the mapr shuffle from the yarn-site, I got node >> managers >> > started however, when I tried to launch a size 0, I got the other error >> > below. Not sure what's happening here. >> > >> > Any thoughts would be appreciated. Like I said, this was working with >> 5.0, >> > and now doesn't work in 5.1. >> > >> > Thanks! >> > >> > John >> > >> > SHuffle Error >> > >> > 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager >> failed >> > in state INITED; cause: java.lang.RuntimeException: No class defined for >> > mapr_direct_shuffle >> > java.lang.RuntimeException: No class defined for mapr_direct_shuffle >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) >> > at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> > at >> > >> > >> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) >> > at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> > at >> > >> > >> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) >> > at >> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) >> > 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager >> metrics >> > system... >> > >> > >> > Zero Sized Node Manager Error: >> > >> > 16/04/04 14:22:49 INFO service.AbstractService: Service >> > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed >> in >> > state STARTED; cause: >> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: >> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved >> SHUTDOWN >> > signal from Resourcemanager ,Registration of NodeManager failed, Message >> > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com >> > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the >> > NodeManager. >> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: >> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved >> SHUTDOWN >> > signal from Resourcemanager ,Registration of NodeManager failed, Message >> > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com >> > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the >> > NodeManager. >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) >> > at >> > >> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) >> > at >> > >> > >> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) >> > at >> > >> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) >> > at >> > >> > >> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) >> > Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: >> Recieved >> > SHUTDOWN signal from Resourcemanager ,Registration of NodeManager >> failed, >> > Message from ResourceManager: Nod
Re: Challenges after MapR 5.1 Upgrade.
yarn.resourcemanager.scheduler.class org.apache.myriad.scheduler.yarn.MyriadFairScheduler On Mon, Apr 4, 2016 at 2:52 PM, Darin Johnson wrote: > Hey John, I think the MapR guys will have some answers for you on the first > stack trace, sounds like a missing jar on you class path. On the second > I'm interested in knowing what MyriadScheduler your attempting to run, > FAIR, CAPACITY, etc (Will be in your yarn-site.xml). > > Darin > > On Mon, Apr 4, 2016 at 3:29 PM, John Omernik wrote: > > > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, > and > > repackaged my hadoop tgz for remote distribution and now I have two > > problems occurring. > > > > 1. At first when I had the mapr direct shuffle enabled per the > > yarn-site.xml on the myriad documentaion, node managers would not start, > > and would fail with the error below. > > > > 2. Once I removed the mapr shuffle from the yarn-site, I got node > managers > > started however, when I tried to launch a size 0, I got the other error > > below. Not sure what's happening here. > > > > Any thoughts would be appreciated. Like I said, this was working with > 5.0, > > and now doesn't work in 5.1. > > > > Thanks! > > > > John > > > > SHuffle Error > > > > 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager > failed > > in state INITED; cause: java.lang.RuntimeException: No class defined for > > mapr_direct_shuffle > > java.lang.RuntimeException: No class defined for mapr_direct_shuffle > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) > > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > > at > > > > > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) > > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > > at > > > > > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) > > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) > > 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager > metrics > > system... > > > > > > Zero Sized Node Manager Error: > > > > 16/04/04 14:22:49 INFO service.AbstractService: Service > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in > > state STARTED; cause: > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN > > signal from Resourcemanager ,Registration of NodeManager failed, Message > > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > > NodeManager. > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN > > signal from Resourcemanager ,Registration of NodeManager failed, Message > > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > > NodeManager. > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) > > at > > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > > at > > > > > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) > > at > > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) > > Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > Recieved > > SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, > > Message from ResourceManager: NodeManager from > > hadoopmapr4.brewingintel.com > > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > > NodeManager. > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298) > > at > > > > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224) > > ... 6 more > > >
Re: Challenges after MapR 5.1 Upgrade.
Hey John, I think the MapR guys will have some answers for you on the first stack trace, sounds like a missing jar on you class path. On the second I'm interested in knowing what MyriadScheduler your attempting to run, FAIR, CAPACITY, etc (Will be in your yarn-site.xml). Darin On Mon, Apr 4, 2016 at 3:29 PM, John Omernik wrote: > I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and > repackaged my hadoop tgz for remote distribution and now I have two > problems occurring. > > 1. At first when I had the mapr direct shuffle enabled per the > yarn-site.xml on the myriad documentaion, node managers would not start, > and would fail with the error below. > > 2. Once I removed the mapr shuffle from the yarn-site, I got node managers > started however, when I tried to launch a size 0, I got the other error > below. Not sure what's happening here. > > Any thoughts would be appreciated. Like I said, this was working with 5.0, > and now doesn't work in 5.1. > > Thanks! > > John > > SHuffle Error > > 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed > in state INITED; cause: java.lang.RuntimeException: No class defined for > mapr_direct_shuffle > java.lang.RuntimeException: No class defined for mapr_direct_shuffle > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) > 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics > system... > > > Zero Sized Node Manager Error: > > 16/04/04 14:22:49 INFO service.AbstractService: Service > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in > state STARTED; cause: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN > signal from Resourcemanager ,Registration of NodeManager failed, Message > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > NodeManager. > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN > signal from Resourcemanager ,Registration of NodeManager failed, Message > from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > NodeManager. > at > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) > Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved > SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, > Message from ResourceManager: NodeManager from > hadoopmapr4.brewingintel.com > doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the > NodeManager. > at > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298) > at > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224) > ... 6 more >
Challenges after MapR 5.1 Upgrade.
I had at one point Myriad working fine in MapR 5.0. I updated to 5.1, and repackaged my hadoop tgz for remote distribution and now I have two problems occurring. 1. At first when I had the mapr direct shuffle enabled per the yarn-site.xml on the myriad documentaion, node managers would not start, and would fail with the error below. 2. Once I removed the mapr shuffle from the yarn-site, I got node managers started however, when I tried to launch a size 0, I got the other error below. Not sure what's happening here. Any thoughts would be appreciated. Like I said, this was working with 5.0, and now doesn't work in 5.1. Thanks! John SHuffle Error 16/04/04 13:46:34 INFO service.AbstractService: Service NodeManager failed in state INITED; cause: java.lang.RuntimeException: No class defined for mapr_direct_shuffle java.lang.RuntimeException: No class defined for mapr_direct_shuffle at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:139) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:250) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:256) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:476) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) 16/04/04 13:46:34 INFO impl.MetricsSystemImpl: Stopping NodeManager metrics system... Zero Sized Node Manager Error: 16/04/04 14:22:49 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:230) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:267) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:477) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:524) Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoopmapr4.brewingintel.com doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:298) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:224) ... 6 more