Tongqing Qiu created TWILL-117:
----------------------------------
Summary: MapR extension
Key: TWILL-117
URL: https://issues.apache.org/jira/browse/TWILL-117
Project: Apache Twill
Issue Type: New Feature
Components: yarn
Affects Versions: 0.5.0-incubating
Reporter: Tongqing Qiu
Priority: Minor
I made some extensions to make twill working for mapr. Besides of what I
included in github, there are some other issues. Currently, I have to include
maprfs library in yarn-example pom file, then related jars will be bundled and
deployed into yarn. These libraries, however, are included in the HADOOP_CLASS,
I don't know why it is not been loading when twill/yarn application runs. So if
we don't are not using bundle jar application while let twill to scan dependent
classes (like CDAP startup services did), the jars are missing. The way to make
it work right now is a hacking:
1. add new MapRFileSystem() in the code of TwillContainerMain and
Hadoop21YarnAppClient class, so that the dependency will be included, even
though it does nothing. BTW, I think it works for other hadoop distribution,
because there two classes already depends the hdfs related stuff and jars are
packaged into container.jar and application jar (which I think is not necessary
since very yarn node should already contain these jars)
2. Add twillRunner.setJVMOptions("-Djava.library.path=/opt/mapr/lib") when run
the twill application.
I think it is really not the right way to do, but not sure how to avoid that,
just let yarn/twill know to include HADOOP_CLASSPATH when run the application?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
