[ https://issues.apache.org/jira/browse/MYRIAD-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santosh Marella updated MYRIAD-7: --------------------------------- Issue Type: Improvement (was: Bug) > Run MyriadMesosScheduler inside YARN's resource manager JVM. > ------------------------------------------------------------ > > Key: MYRIAD-7 > URL: https://issues.apache.org/jira/browse/MYRIAD-7 > Project: Myriad > Issue Type: Improvement > Reporter: Santosh Marella > Assignee: Santosh Marella > > The objective of this change is to run MyriadMesosScheduler inside YARN's > resource manager JVM. > 1. Added Myriad{Fair,Capacity,Fifo}Scheduler classes that extend from Yarn's > {Fair,Capacity,Fifo}Scheduler classes respectively. > 2. Added build dependencies on Hadoop. > 3. Encountered slf4j's "multiple binding" conflicts because drop wizard uses > logback implementation while YARN uses slf4j-log4j12 implementation for > slf4j API. > After discussions with Mohit, we've decided to remove dependencies on drop > wizard. > 4. Refactored the rest of the code to that effect. Retained non-conflicting > deps like > codahale metrics, jackson's dataformat/databind/annotations/yaml and > hibernate validator. > 5. Added build rules to package the myriad jar and the dependencies to > build/libs dir. > These DO NOT INCLUDE hadoop jars and their dependencies as the intent is to > deploy myriad jar and it's deps into an existing YARN installation. > Build and Deployment guidelines: > 1. From myriad dir in the local git repo, run "./gradlew jar". > 2. This should produce myriad-0.0.1.jar and other deps under build/libs dir. > 3. Copy build/libs/*.jar to HADOOP_HOME/share/hadoop/yarn. > 4. Modify HADOOP_HOME/etc/hadoop/yarn-site.xml to have the following entry: > {code:xml} > <property> > <name>yarn.resourcemanager.scheduler.class</name> > <value>com.ebay.myriad.scheduler.yarn.MyriadFairScheduler</value> > </property> > {code} > 5. Restart Resource Manager process. > CAVEATS: > - The REST API to myriad is currently broken as we eliminated the > dependencies on dropwizard. > We need to bring up a web app for myriad and expose the > REST API through that. > - Mesos also requires a native library (libmesos.so). Our build process > currently > does not add that to myriad jar. We need to manually add that to YARN's > native lib dir > $HADOOP_HOME/lib/native. > I'll open tasks for both the above and track them separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)