Hi Santosh, Thanks for looking into it.
I got it working and now I am able to run Node Managers through Myriad. There are couple of issues which i am still facing , I am sure you guys would have faced those and already have the solutions for those. 1. I am not seeing Nodemanager logs, I see YARN LOG DIR is set however logs are not going there 2. Log aggregation is also not working in YARN , don't know why , as I don't have logs 3. Other issue is when i do first time flex up then it launches 2 node managers instead of one. Please let me know if u know about those issues. Thanks, Mayank On Mon, Feb 22, 2016 at 4:21 PM, Santosh Marella <smare...@maprtech.com> wrote: > Hey guys.. the flow of things is as below: > > 1. Myriad registers itself as a Mesos framework. (Frameworks view on Mesos > UI should show this) > 2. Myriad by default attempts to launch 1 medium profile NM as soon as RM > process is up. > This is queued up until Myriad receives an offer from Mesos that is big > enough to launch a medium profle NM. > (Mesos' UI should show a task such as "nm.medium.xxxx" if Myriad > attempted to launch a NM). > > Mayank - in your cluster, does Mesos UI show Myriad as a registered > framework? Does it show if Myriad launched NM tasks? Perhaps a good > starting point is to check what are the sizes of the offers Myriad is > receiving (Mesos' master log should show this). You can also enable DEBUG > logging in RM for myriad by setting "org.apache.myriad" to DEBUG in RM's > log4j.properties. > > > > Santosh > > On Mon, Feb 22, 2016 at 10:39 AM, Björn Hagemeier < > b.hageme...@fz-juelich.de > > wrote: > > > Hi Mayank, > > > > it seems you and I are having similar issues. Let's see whether they can > > be traced to the same causes. > > > > > > Best regards, > > Björn > > > > Am 22.02.2016 um 19:33 schrieb Mayank Bansal: > > > Hi, > > > > > > I am trying to use Hadoop 2.6.3 with Myriad origin/0.1.x > > > > > > > > > I have followed all the instructions in the wiki > > > > > > So the resource manager is up and i am able to launch the web UI of > > myriad > > > as well however whenever I try to use flex up it launches some tasks > and > > > put them in to pending queue and after that nothing launches. > > > > > > I am attaching below the resourcemanager.out and log snippets. > > > > > > Please let me know if I am missing something. > > > > > > Thanks, > > > > > > > > > *I see some exceptions in resourcemanager.out files those are as > follows* > > > > > > > > > Feb 22, 2016 6:21:42 PM > > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > > > resolve > > > > > > SEVERE: null > > > > > > java.lang.IllegalAccessException: Class > > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > > > can not access a member of class javax.ws.rs.core.Response with > modifiers > > > "protected" > > > > > > at > sun.reflect.Reflection.ensureMemberAccess(Reflection.java:102) > > > > > > at java.lang.Class.newInstance(Class.java:436) > > > > > > at > > > > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467) > > > > > > at > > > > > > com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181) > > > > > > at > > > > > > com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81) > > > > > > at > > > > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518) > > > > > > at > > > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124) > > > > > > at > > > > > > com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104) > > > > > > at > > > > > > com.sun.jersey.server.impl.wadl.WadlResource.getWadl(WadlResource.java:89) > > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > > > at > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > > > > at > > > > > > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > > > > > > at > > > > > > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) > > > > > > at > > > > > > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > > > > > > at > > > > > > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > > > > > > at > > > > > > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > > > > > > at > > > > > > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > > > > > > at > > > > > > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > > > > > > at > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > > > > > > at > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > > > > > > at > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > > > > > > at > > > > > > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > > > > > > at > > > > > > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > > > > > > at > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > > > > > > at > > > > > > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) > > > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > > > > > at > > > > > > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) > > > > > > at > > > > > > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) > > > > > > at > > > > > > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > > > > > > at > > > > > > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) > > > > > > at > > > > > > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > > > > > > at > > > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > > > > > > at > > > > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > > > > > > at > > > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > > > > > at > > > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > > > > > > at > > > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > > > > > at org.mortbay.jetty.Server.handle(Server.java:326) > > > > > > at > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > > > > > at > > > > > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > > > > > > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > > > > > > at > > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > > > > > > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > > > > > > at > > > > > > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > > > > > > at > > > > > > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > > > > > > > > > Feb 22, 2016 6:21:42 PM > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > > > attachTypes > > > > > > INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response > > > > > > Feb 22, 2016 6:21:42 PM > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > > > attachTypes > > > > > > INFO: Couldn't find JAX-B element for class > > > org.apache.myriad.configuration.MyriadConfiguration > > > > > > Feb 22, 2016 6:21:42 PM > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > > > attachTypes > > > > > > INFO: Couldn't find JAX-B element for class > > > org.apache.myriad.api.model.GetSchedulerStateResponse > > > > > > Feb 22, 2016 6:21:42 PM > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator > > > attachTypes > > > > > > INFO: Couldn't find JAX-B element for class > > > org.apache.myriad.api.model.FlexUpServiceRequest > > > > > > Feb 22, 2016 6:21:42 PM > > > > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 > > > resolve > > > > > > SEVERE: null > > > > > > E0222 18:22:30.863994 47309 socket.hpp:174] Shutdown failed on fd=231: > > > Transport endpoint is not connected [107] > > > > > > E0222 18:25:19.055994 47309 socket.hpp:174] Shutdown failed on fd=231: > > > Transport endpoint is not connected [107] > > > > > > E0222 18:27:44.208027 47309 socket.hpp:174] Shutdown failed on fd=231: > > > Transport endpoint is not connected [107] > > > > > > *This is the resource manager logs* > > > > > > 2016-02-22 17:37:27,747 INFO org.apache.hadoop.ipc.Server: IPC Server > > > listener on 8033: starting > > > > > > 2016-02-22 17:39:31,266 INFO org.apache.myriad.api.ClustersResource: > > > Received flexup request. Profile: medium, Instances: 1, Constraints: > null > > > > > > 2016-02-22 17:39:31,266 INFO > > org.apache.myriad.scheduler.MyriadOperations: > > > Adding 1 NM instances to cluster > > > > > > 2016-02-22 17:39:31,267 INFO org.apache.myriad.state.SchedulerState: > > Marked > > > taskId nm.medium.efd5f3b4-fe5f-4eed-bd19-6906696a22bc pending, size of > > > pending queue for nm is: 1 > > > > > > 2016-02-22 17:47:25,450 INFO > > > > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler: > > > Release request cache is cleaned up > > > > > > 2016-02-22 18:22:22,093 INFO org.apache.myriad.api.ClustersResource: > > > Received flexup request. Profile: medium, Instances: 1, Constraints: > null > > > > > > 2016-02-22 18:22:22,094 INFO > > org.apache.myriad.scheduler.MyriadOperations: > > > Adding 1 NM instances to cluster > > > > > > 2016-02-22 18:22:22,094 INFO org.apache.myriad.state.SchedulerState: > > Marked > > > taskId nm.medium.607ebf49-045b-44bb-8940-8e3bd8c115a8 pending, size of > > > pending queue for nm is: 2 > > > > > > > > > > > > Thanks, > > > > > > > > > -- > > Dipl.-Inform. Björn Hagemeier > > Federated Systems and Data > > Juelich Supercomputing Centre > > Institute for Advanced Simulation > > > > Phone: +49 2461 61 1584 > > Fax : +49 2461 61 6656 > > Email: b.hageme...@fz-juelich.de > > Skype: bhagemeier > > WWW : http://www.fz-juelich.de/jsc > > > > JSC is the coordinator of the > > John von Neumann Institute for Computing > > and member of the > > Gauss Centre for Supercomputing > > > > > > > ------------------------------------------------------------------------------------- > > > > > ------------------------------------------------------------------------------------- > > Forschungszentrum Juelich GmbH > > 52425 Juelich > > Sitz der Gesellschaft: Juelich > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > > Prof. Dr. Sebastian M. Schmidt > > > > > ------------------------------------------------------------------------------------- > > > > > ------------------------------------------------------------------------------------- > > > > > -- - Mayank
