Re: myriad scheduler startup with HDP2.7
Well thats the point, I do have 2.2.3 installed as that's the version shipped with HDP 2.3 and that gets loaded first in the classpath for YARN resourcemanager. I guess I have three alternatives. 1) build myriad using 2.2.3, thus matching the HDP installed jar's 2) replace the HDP version with 2.5.1, not sure what's that going to do for HDP compatibility 3) prepend a new classpath for yarn resourcemanager to pick up myriad versioned jars first. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 12:36 PM, "Adam Bordelon" wrote: >Myriad should be using jackson 2.5.1 >https://github.com/mesos/myriad/blob/d6d765736ba1c8f59aa967457527331e1dab6 >743/myriad-scheduler/build.gradle#L13 >Double-check your build.gradle, and make sure you don't have a jackson >2.2.3 preinstalled somewhere else on your system > >On Wed, Aug 19, 2015 at 8:20 AM, Bill Sparks wrote: > >> Odd the class path reported in the yarn log contains jackson-core-2.2.3 >> and not 2.5.1. Is there a way to build myriad to match the version >> supported by HDP - that being 2.2.3 ? >> >> >> -- >> Jonathan (Bill) Sparks >> Software Architecture >> Cray Inc. >> >> >> >> >> >> On 8/19/15 10:11 AM, "Bill Sparks" wrote: >> >> >Thanks I'll check.. >> > >> >-- >> >Jonathan (Bill) Sparks >> >Software Architecture >> >Cray Inc. >> > >> > >> > >> > >> > >> >On 8/19/15 10:09 AM, "yuliya Feldman" >> wrote: >> > >> >>This method is part of JsonFactory class which is part of jackson-core >> >>jar >> >>See if you have some other jars on the classpath (different versions) >> >>that precede jackson-core-2.5.1.jar >> >> From: Bill Sparks >> >> To: "dev@myriad.incubator.apache.org" >> >> >> Sent: Wednesday, August 19, 2015 7:08 AM >> >> Subject: myriad scheduler startup with HDP2.7 >> >> >> >>I'm sure this is been resolved, but I've been triaging why I'm getting >> >>the following error on resourcemanager startup. Everything on the >> >>configuration side looks correct, but I must have missed something. >> >> >> >> >> >> >> >>2015-08-19 08:53:04,718 FATAL >> >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error >> >>starting ResourceManager >> >> >> >>java.lang.NoSuchMethodError: >> >>>>com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/In >>>>pu >> >>t >> >>Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; >> >> >> >>at >> >>>>com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact >>>>or >> >>y >> >>.java:299) >> >> >> >>at >> >>>>com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFact >>>>or >> >>y >> >>.java:14) >> >> >> >>at >> >>>>com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java >>>>:2 >> >>0 >> >>11) >> >> >> >>at com.ebay.myriad.Main.initialize(Main.java:70) >> >> >> >>at >> >>>>com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercep >>>>to >> >>r >> >>.init(MyriadInitializationInterceptor.java:32) >> >> >> >>at >> >>>>com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Co >>>>mp >> >>o >> >>siteInterceptor.java:76) >> >> >> >>at >> >>>>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFa >>>>ir >> >>S >> >>cheduler.java:50) >> >> >> >>at >> >>>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163 >>>>) >> >> >> >>at >> >>>>org.apache.hadoop.service.CompositeService.serviceInit(CompositeService >>>>.j >> >>a >> >>va:107) >> >> >> >>at >> >>>>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveS >>>>er >> >>v >> >>ices.serviceInit(ResourceManager.java:572) >> >> >> >>at >> >>>&g
Re: myriad scheduler startup with HDP2.7
Odd the class path reported in the yarn log contains jackson-core-2.2.3 and not 2.5.1. Is there a way to build myriad to match the version supported by HDP - that being 2.2.3 ? -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:11 AM, "Bill Sparks" wrote: >Thanks I'll check.. > >-- >Jonathan (Bill) Sparks >Software Architecture >Cray Inc. > > > > > >On 8/19/15 10:09 AM, "yuliya Feldman" wrote: > >>This method is part of JsonFactory class which is part of jackson-core >>jar >>See if you have some other jars on the classpath (different versions) >>that precede jackson-core-2.5.1.jar >> From: Bill Sparks >> To: "dev@myriad.incubator.apache.org" >> Sent: Wednesday, August 19, 2015 7:08 AM >> Subject: myriad scheduler startup with HDP2.7 >> >>I'm sure this is been resolved, but I've been triaging why I'm getting >>the following error on resourcemanager startup. Everything on the >>configuration side looks correct, but I must have missed something. >> >> >> >>2015-08-19 08:53:04,718 FATAL >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error >>starting ResourceManager >> >>java.lang.NoSuchMethodError: >>com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Inpu >>t >>Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; >> >>at >>com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor >>y >>.java:299) >> >>at >>com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactor >>y >>.java:14) >> >>at >>com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2 >>0 >>11) >> >>at com.ebay.myriad.Main.initialize(Main.java:70) >> >>at >>com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto >>r >>.init(MyriadInitializationInterceptor.java:32) >> >>at >>com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp >>o >>siteInterceptor.java:76) >> >>at >>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair >>S >>cheduler.java:50) >> >>at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> >>at >>org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j >>a >>va:107) >> >>at >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer >>v >>ices.serviceInit(ResourceManager.java:572) >> >>at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> >>at >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn >>i >>tActiveServices(ResourceManager.java:972) >> >>at >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit >>( >>ResourceManager.java:259) >> >>at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> >>at >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour >>c >>eManager.java:1202) >> >> >>I have placed all the myriad jar in the hadoop-yarn/lib directory and the >>classpath reflect that. >> >> >>cp /tmp/myriad/myriad-scheduler/build/libs/* >>/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib >> >>cp >>/tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar >>/usr/libexec/mesos/ >> >> >>[root@nid00037 myriad]# su - yarn >> >>-bash-4.1$ yarn classpath >> >>/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/ >>h >>dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2 >>. >>3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3. >>0 >>.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp >>/ >>2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/us >>r >>/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-map >>r >>educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/ja >>v >>a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/us >>r >>/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0- >>2 >>557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/ha >>d >>oop-yarn-client/lib/* >> >> >> >>ls /usr/hdp
Re: myriad scheduler startup with HDP2.7
Thanks I'll check.. -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 8/19/15 10:09 AM, "yuliya Feldman" wrote: >This method is part of JsonFactory class which is part of jackson-core jar >See if you have some other jars on the classpath (different versions) >that precede jackson-core-2.5.1.jar > From: Bill Sparks > To: "dev@myriad.incubator.apache.org" > Sent: Wednesday, August 19, 2015 7:08 AM > Subject: myriad scheduler startup with HDP2.7 > >I'm sure this is been resolved, but I've been triaging why I'm getting >the following error on resourcemanager startup. Everything on the >configuration side looks correct, but I must have missed something. > > > >2015-08-19 08:53:04,718 FATAL >org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error >starting ResourceManager > >java.lang.NoSuchMethodError: >com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/Input >Stream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; > >at >com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory >.java:299) > >at >com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory >.java:14) > >at >com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:20 >11) > >at com.ebay.myriad.Main.initialize(Main.java:70) > >at >com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor >.init(MyriadInitializationInterceptor.java:32) > >at >com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Compo >siteInterceptor.java:76) > >at >com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairS >cheduler.java:50) > >at >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > >at >org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.ja >va:107) > >at >org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServ >ices.serviceInit(ResourceManager.java:572) > >at >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > >at >org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIni >tActiveServices(ResourceManager.java:972) > >at >org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit( >ResourceManager.java:259) > >at >org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > >at >org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resourc >eManager.java:1202) > > >I have placed all the myriad jar in the hadoop-yarn/lib directory and the >classpath reflect that. > > >cp /tmp/myriad/myriad-scheduler/build/libs/* >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib > >cp >/tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar >/usr/libexec/mesos/ > > >[root@nid00037 myriad]# su - yarn > >-bash-4.1$ yarn classpath > >/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/h >dp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2. >3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0 >.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/ >2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr >/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapr >educe/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/jav >a/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr >/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0-2 >557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/had >oop-yarn-client/lib/* > > > >ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries > > >-bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* > >-rw-r--r-- 1 root root 3456 Aug 19 08:50 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar > >-rw-r--r-- 1 root root 950687 Aug 19 08:50 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar > >and > > >-bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* > >-rw-r--r-- 1 root root 39817 Aug 17 18:32 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar > >-rw-r--r-- 1 root root 192699 Jul 14 08:22 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar > >-rw-r--r-- 1 root root 229860 Aug 17 18:32 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar > >-rw-r--r-- 1 root root 232248 Jul 14 08:22 >/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar > >-rw-r--r-- 1 root root 1138921
myriad scheduler startup with HDP2.7
I'm sure this is been resolved, but I've been triaging why I'm getting the following error on resourcemanager startup. Everything on the configuration side looks correct, but I must have missed something. 2015-08-19 08:53:04,718 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.yaml.YAMLFactory._decorate(Ljava/io/InputStream;Lcom/fasterxml/jackson/core/io/IOContext;)Ljava/io/InputStream; at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:299) at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2011) at com.ebay.myriad.Main.initialize(Main.java:70) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:32) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:76) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:50) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:572) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:972) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:259) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1202) I have placed all the myriad jar in the hadoop-yarn/lib directory and the classpath reflect that. cp /tmp/myriad/myriad-scheduler/build/libs/* /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib cp /tmp/myriad/myriad-executor/build/libs/myriad-executor-runnable-0.0.1.jar /usr/libexec/mesos/ [root@nid00037 myriad]# su - yarn -bash-4.1$ yarn classpath /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/*:/usr/hdp/2.3.0.0-2557/hadoop/.//*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/./:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-hdfs/.//*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-yarn/.//*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/lib/*:/usr/hdp/2.3.0.0-2557/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/2.3.0.0-2557/tez/*:/usr/hdp/2.3.0.0-2557/tez/lib/*:/usr/hdp/2.3.0.0-2557/tez/conf:/usr/hdp/current/hadoop-yarn-client/.//*:/usr/hdp/current/hadoop-yarn-client/lib/* ls /usr/hdp/current/hadoop-yarn-client/lib/* has all the libraries -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad* -rw-r--r-- 1 root root 3456 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-commons-0.0.1.jar -rw-r--r-- 1 root root 950687 Aug 19 08:50 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib//myriad-scheduler-0.0.1.jar and -bash-4.1$ ls -l /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson* -rw-r--r-- 1 root root 39817 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-annotations-2.5.1.jar -rw-r--r-- 1 root root 192699 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.2.3.jar -rw-r--r-- 1 root root 229860 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-2.5.1.jar -rw-r--r-- 1 root root 232248 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 1138921 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-databind-2.5.1.jar -rw-r--r-- 1 root root 321751 Aug 17 18:32 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-dataformat-yaml-2.5.1.jar -rw-r--r-- 1 root root 18336 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar -rw-r--r-- 1 root root 780664 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar -rw-r--r-- 1 root root 27084 Jul 14 08:22 /usr/hdp/2.3.0.0-2557/hadoop-yarn/lib/jackson-xc-1.9.13.jar -bash-4.1$ jar tf /usr/hdp/current/hadoop-yarn-client/lib/jackson-dataformat-yaml-2.5.1.jar | grep YAMLFactory com/fasterxml/jackson/dataformat/yaml/YAMLFactory.class
Re: Question about YARN HA and myriad/Mesos
A single RM per-node. It seems the second RM to start is the one that registers the Myriad-Alpha framework with Mesos. But both RM's are still in standby state - when I activate one RM, then I get the error, but the Mesos framework still show the other as the location of the framework. It goes something like Š Start rm1 Start rm2 -> rm2 registers the myriad framework Both RM's are in standby rm1 standby rm2 standby rm1 - to active - Get the errorŠ yarn rmadmin -transitionToActive rm1 Operation failed: Error when transitioning to Active mode at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:304) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.tra nsitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService $2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(Prot obufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) Caused by: org.apache.hadoop.ha.ServiceFailedException: Failed to initialize myriad at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(Admin Service.java:626) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActi ve(AdminService.java:297) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc. On 6/22/15 3:05 PM, "Santosh Marella" wrote: >The cause for the exception shows "Address already in use". And looks like >the 'address" in question is the webserver for Myriad. > >Can you please check if you are spinning up two ResourceManagers on the >same node? That's currently not supported. > >Santosh > >On Mon, Jun 22, 2015 at 12:48 PM, Bill Sparks wrote: > >> I thought I'd ask before diving into it more, but is YARN HA >> ResourceManager and myriad/mesos supported? >> >> The reason why I ask, is that I just brought up YARN HA >>ResourceManager(s) >> and the resourcemanager fails to start and the stack trace has myriad >>on it. >> >> 15/06/22 13:17:47 INFO service.AbstractService: Service >> >>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedule >>r >> failed in state INITED; cause: java.lang.RuntimeException: Failed to >> initialize myriad >> java.lang.RuntimeException: Failed to initialize myriad >> at >> >>com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationIntercepto >>r.init(MyriadInitializationInterceptor.java:34) >> at >> >>com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(Comp >>ositeInterceptor.java:63) >> at >> >>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFair >>Scheduler.java:37) >> at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> >>org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.j >>ava:107) >> at >> >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveSer >>vices.serviceInit(ResourceManager.java:573) >> at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndIn >>itActiveServices(ResourceManager.java:1004) >> at >> >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit >>(ResourceManager.java:270) >> at >>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) >> at >> >>org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(Resour >>ceManager.java:1225) >> Caused by: java.net.BindException: Address already in use >> at sun.nio.ch.Net.bind0(Native Method) >> at sun.nio.ch.Net.bind(Net.java:344) >> at sun.nio.ch.Net.bind(Net.java:336) >> at >> >>sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) >> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) >> at >> >>org.mortbay.jetty.nio
Question about YARN HA and myriad/Mesos
I thought I'd ask before diving into it more, but is YARN HA ResourceManager and myriad/mesos supported? The reason why I ask, is that I just brought up YARN HA ResourceManager(s) and the resourcemanager fails to start and the stack trace has myriad on it. 15/06/22 13:17:47 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED; cause: java.lang.RuntimeException: Failed to initialize myriad java.lang.RuntimeException: Failed to initialize myriad at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:34) at com.ebay.myriad.scheduler.yarn.interceptor.CompositeInterceptor.init(CompositeInterceptor.java:63) at com.ebay.myriad.scheduler.yarn.MyriadFairScheduler.serviceInit(MyriadFairScheduler.java:37) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:573) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1004) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:270) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1225) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:344) at sun.nio.ch.Net.bind(Net.java:336) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:199) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) at org.mortbay.jetty.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:315) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.ebay.myriad.webapp.MyriadWebServer.start(MyriadWebServer.java:50) at com.ebay.myriad.Main.initWebApp(Main.java:109) at com.ebay.myriad.Main.run(Main.java:86) at com.ebay.myriad.Main.initialize(Main.java:68) at com.ebay.myriad.scheduler.yarn.interceptor.MyriadInitializationInterceptor.init(MyriadInitializationInterceptor.java:31) ... 10 more -- Jonathan (Bill) Sparks Software Architecture Cray Inc.
Re: Myriad/YARN configuration
Ok, so if we/I have 48 nodes, each with 64 HT-cores and 128GB memory and I want
YARN to FULLY utilize the whole system. My profile would/should be
cpu: 52 # 80% to core max
mem: 104448 # 80% of node mem
Or basically set it to yarn-site.xml yarn.nodemanager.resource.cpu-vcores and
yarn.scheduler.maximum-allocation-mb
--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.
From: Darin Johnson mailto:dbjohnson1...@gmail.com>>
Date: Friday, June 12, 2015 9:06 AM
To: Bill Sparks mailto:jspa...@cray.com>>
Cc: "dev@myriad.incubator.apache.org<mailto:dev@myriad.incubator.apache.org>"
mailto:dev@myriad.incubator.apache.org>>
Subject: Re: Myriad/YARN configuration
Bill,
I'm at a computer now so I'll take a little more time to explain. The profiles
are used by the REST API so that you can tell a the resource manager to add a
new node manager with cpu and memory defined by the profile. If you set a
profile large as:
large:
cpu: 16
mem: 32768
then you could flexup using {profile: large, instances: 10} to get 10
nodemanagers running with 32 GB memory and 16 cpu core capacity. You could
flexdown {instance: 10} to remove the capacity. How to flexup/flexdown is
documented in API.md. You can also flexup/flexdown using the webui that runs
in port 8192.
In the near future we plan to have fine grain scaling, then when you launch a
resource manager you'll also launch instance of node manager with 0 cpu and 0
mem capacity. As you run yarn applications the node managers will add cpu and
mem capacity to meet your jobs container needs.
Let me know if you have more questions.
Darin
On Thu, Jun 11, 2015 at 6:43 PM, Bill Sparks
mailto:jspa...@cray.com>> wrote:
I've changed them with no ill effect. Just wondering what the purpose is. My
yarn configurations will allow much more that the profiles set by mesos/myriad.
So is myriad scheduler going to bounce the request based on profiles or be
passive and let yarn containers do the work. All this is early days for me.
--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.
From: Darin Johnson mailto:dbjohnson1...@gmail.com>>
Date: Thursday, June 11, 2015 5:36 PM
To: "dev@myriad.incubator.apache.org<mailto:dev@myriad.incubator.apache.org>"
mailto:dev@myriad.incubator.apache.org>>
Cc: Bill Sparks mailto:jspa...@cray.com>>
Subject: Re: Myriad/YARN configuration
I've changed the profiles before with no ill effect, you should be fine.
On Thu, Jun 11, 2015 at 6:28 PM, Adam Bordelon
mailto:a...@mesosphere.io>> wrote:
+dev@myriad, -user@mesos
On Thu, Jun 11, 2015 at 8:51 AM, Bill Sparks
mailto:jspa...@cray.com>> wrote:
> I have a question about the profiles configured
> in myriad-config-default.yml, do they have to match parameters specified in
> yarn-site.xml.
>
> E.g.
> myriad-config-default.yml:
>large:
> cpu: 4
> mem: 4096
>
> yarn-site.xml:
>
> yarn.nodemanager.resource.cpu-vcores
> ${nodemanager.resource.cpu-vcores}
>
>
>
> yarn.nodemanager.resource.memory-mb
> ${nodemanager.resource.memory-mb}
>
>
> I ask because we have large memory/high core count nodes 48 cores/64GB
> memory.
>
> --
> Jonathan (Bill) Sparks
> Software Architecture
> Cray Inc.
>
Re: Myriad/YARN configuration
I've changed them with no ill effect. Just wondering what the purpose is. My
yarn configurations will allow much more that the profiles set by mesos/myriad.
So is myriad scheduler going to bounce the request based on profiles or be
passive and let yarn containers do the work. All this is early days for me.
--
Jonathan (Bill) Sparks
Software Architecture
Cray Inc.
From: Darin Johnson mailto:dbjohnson1...@gmail.com>>
Date: Thursday, June 11, 2015 5:36 PM
To: "dev@myriad.incubator.apache.org<mailto:dev@myriad.incubator.apache.org>"
mailto:dev@myriad.incubator.apache.org>>
Cc: Bill Sparks mailto:jspa...@cray.com>>
Subject: Re: Myriad/YARN configuration
I've changed the profiles before with no ill effect, you should be fine.
On Thu, Jun 11, 2015 at 6:28 PM, Adam Bordelon
mailto:a...@mesosphere.io>> wrote:
+dev@myriad, -user@mesos
On Thu, Jun 11, 2015 at 8:51 AM, Bill Sparks
mailto:jspa...@cray.com>> wrote:
> I have a question about the profiles configured
> in myriad-config-default.yml, do they have to match parameters specified in
> yarn-site.xml.
>
> E.g.
> myriad-config-default.yml:
>large:
> cpu: 4
> mem: 4096
>
> yarn-site.xml:
>
> yarn.nodemanager.resource.cpu-vcores
> ${nodemanager.resource.cpu-vcores}
>
>
>
> yarn.nodemanager.resource.memory-mb
> ${nodemanager.resource.memory-mb}
>
>
> I ask because we have large memory/high core count nodes 48 cores/64GB
> memory.
>
> --
> Jonathan (Bill) Sparks
> Software Architecture
> Cray Inc.
>
executing YARN node manager as "yarn"
I have a question about to to configure myriad executor to start the YARN node manager as user "yarn". I though I'd put the correct value in HADOOP_CONF_DIR/myriad-config-default.yml frameworkUser: yarn Also I updated both the myriad-executor-0.0.1.jar and myriad-0.0.1.jar to contain the same configuration, just in case they didn't get the correct file due to classpath issues. But when the executor runs node manager it is as hduser "Myriad User" and not as yarn. E.g. 0 0 5714 1 20 0 4000252 50136 futex_ Ssl ? 45:07 /usr/sbin/mesos-slave --master=zk://newt-0001-n3:2181,newt-0002-n3:2181,newt-0003-n3:2181/mesos --log_dir=/var/log/m 0 0 5720 5714 20 0 100892 596 pipe_w S? 0:00 \_ logger -p user.info -t mesos-slave[5714] 0 0 5721 5714 20 0 100892 708 pipe_w S? 0:00 \_ logger -p user.err -t mesos-slave[5714] 4 40037 6659 5714 20 0 108200 1420 wait Ss ? 0:00 \_ sh -c echo "export CAPSULE_CACHE_DIR=`pwd` ;echo $CAPSULE_CACHE_DIR; java -Dcapsule.log=verbose -jar myriad-exec 0 40037 6659 20 0 20538068 140472 futex_ Sl ? 0:03 \_ java -Dcapsule.log=verbose -jar myriad-executor-0.0.1.jar 0 40037 6709 20 0 24559132 431196 futex_ Sl ? 0:12 \_ /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre/bin/java -Dcapsule.app=com.ebay.myriad.executor.Myria 0 40037 42106 6709 20 0 1990188 352052 futex_ Sl ? 0:05 \_ /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Dproc_nodemanager -Xmx1024m -Dhdp.version=2.2.4.2-2 -bash-4.1# grep 40037 /etc/passwd hduser:x:40037:30550:Myriad user:/var/lib/mesos:/bin/bash -- Jonathan (Bill) Sparks Software Architecture Cray Inc.
