Re: Question about YARN HA and myriad/Mesos
Start rm1 Start rm2 - rm2 registers the myriad framework When you started rm2, how did it register the myriad framework? Was rm2 active the moment you started it? Only the active rm should initialize myriad. Can you also paste the yarn-site.xml from both rm1 and rm2 here ? Thanks, Santosh On Mon, Jun 22, 2015 at 1:14 PM, Bill Sparks jspa...@cray.com wrote: A single RM per-node. It seems the second RM to start is the one that registers the Myriad-Alpha framework with Mesos. But both RM's are still in standby state - when I activate one RM, then I get the error, but the Mesos framework still show the other as the location of the framework. It goes something like Š Start rm1 Start rm2 - rm2 registers the myriad framework Both RM's are in standby rm1 standby rm2 standby rm1 - to active - Get the errorŠ yarn rmadmin -transitionToActive rm1 Operation failed: Error when transitioning to Active mode at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:304) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.tra nsitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService $2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Prot obufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) Caused by: org.apache.hadoop.ha.ServiceFailedException: Failed to initialize myriad at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(Admin Service.java:626) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:297) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 6/22/15 3:05 PM, Santosh Marella smare...@maprtech.com wrote: The cause for the exception shows Address already in use. And looks like the 'address in question is the webserver for Myriad. Can you please check if you are spinning up two ResourceManagers on the same node? That's currently not supported. Santosh On Mon, Jun 22, 2015 at 12:48 PM, Bill Sparks jspa...@cray.com wrote: I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule r failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp ositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair Scheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j ava:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer vices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn itActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit (ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour ceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector. java:216) at
Re: Question about YARN HA and myriad/Mesos
The cause for the exception shows Address already in use. And looks like the 'address in question is the webserver for Myriad. Can you please check if you are spinning up two ResourceManagers on the same node? That's currently not supported. Santosh On Mon, Jun 22, 2015 at 12:48 PM, Bill Sparks jspa...@cray.com wrote: I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:31) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc.
Re: Question about YARN HA and myriad/Mesos
A single RM per-node. It seems the second RM to start is the one that registers the Myriad-Alpha framework with Mesos. But both RM's are still in standby state - when I activate one RM, then I get the error, but the Mesos framework still show the other as the location of the framework. It goes something like Š Start rm1 Start rm2 - rm2 registers the myriad framework Both RM's are in standby rm1 standby rm2 standby rm1 - to active - Get the errorŠ yarn rmadmin -transitionToActive rm1 Operation failed: Error when transitioning to Active mode at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:304) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.tra nsitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService $2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Prot obufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) Caused by: org.apache.hadoop.ha.ServiceFailedException: Failed to initialize myriad at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(Admin Service.java:626) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:297) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 6/22/15 3:05 PM, Santosh Marella smare...@maprtech.com wrote: The cause for the exception shows Address already in use. And looks like the 'address in question is the webserver for Myriad. Can you please check if you are spinning up two ResourceManagers on the same node? That's currently not supported. Santosh On Mon, Jun 22, 2015 at 12:48 PM, Bill Sparks jspa...@cray.com wrote: I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule r failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp ositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair Scheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j ava:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer vices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn itActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit (ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour ceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector. java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnect or.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68) at
Question about YARN HA and myriad/Mesos
I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:31) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc.
