Re: myriad scheduler startup with HDP2.7
Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha d oop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar
Re: myriad scheduler startup with HDP2.7
Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, Adam Bordelon a...@mesosphere.io wrote: Myriad should be using jackson 2.5.1 https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 743/myriad-scheduler/build.gradle#L13 Double-check your build.gradle, and make sure you don't have a jackson 2.2.3 preinstalled somewhere else on your system On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks jspa...@cray.com wrote: Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, Bill Sparks jspa...@cray.com wrote: Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, yuliya Feldman yufeld...@yahoo.com.INVALID wrote: This method is part of JsonFactory class which is part of jackson-core jar See if you have some other jars on the classpath (different versions) that precede jackson-core-2.5.1.jar From: Bill Sparks jspa...@cray.com To: dev@myriad.incubator.apache.org dev@myriad.incubator.apache.org Sent: Wednesday, August 19, 2015 7:08 AM Subject: myriad scheduler startup with HDP2.7 I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In pu t Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact or y .java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java :2 0 11) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep to r .init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co mp o siteInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa ir S cheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService .j a va:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS er v ices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAnd In i tActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceIn it ( ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 ) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Reso ur c eManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.j ar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/us r/ h dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp /2 . 3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2. 3. 0 .0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/h dp / 2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/ us r /hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-m ap r educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ ja v a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/ us r /hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0. 0- 2 557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ ha d oop-yarn
Re: Question about YARN HA and myriad/Mesos
A single RM per-node. It seems the second RM to start is the one that registers the Myriad-Alpha framework with Mesos. But both RM's are still in standby state - when I activate one RM, then I get the error, but the Mesos framework still show the other as the location of the framework. It goes something like Š Start rm1 Start rm2 - rm2 registers the myriad framework Both RM's are in standby rm1 standby rm2 standby rm1 - to active - Get the errorŠ yarn rmadmin -transitionToActive rm1 Operation failed: Error when transitioning to Active mode at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:304) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.tra nsitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService $2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Prot obufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) Caused by: org.apache.hadoop.ha.ServiceFailedException: Failed to initialize myriad at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(Admin Service.java:626) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:297) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 6/22/15 3:05 PM, Santosh Marella smare...@maprtech.com wrote: The cause for the exception shows Address already in use. And looks like the 'address in question is the webserver for Myriad. Can you please check if you are spinning up two ResourceManagers on the same node? That's currently not supported. Santosh On Mon, Jun 22, 2015 at 12:48 PM, Bill Sparks jspa...@cray.com wrote: I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule r failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto r.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp ositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair Scheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j ava:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer vices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn itActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit (ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour ceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector. java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnect or.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68
Question about YARN HA and myriad/Mesos
I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:31) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc.
