[
https://issues.apache.org/jira/browse/YARN-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260609#comment-15260609
]
Xuan Gong commented on YARN-4577:
---------------------------------
[~sjlee0] Thanks for the review.
Attached a new patch to address the comments.
Unfortunately, I am not able to create a unit test for this. But I did test it
manually.
Here is how I test it:
1. Create a customized TestAuxService which extends AuxiliaryService.
2. Create two jar file which have the same jar file name: TestAuxSerivce.jar
and have the same class name: TestAuxService.java
3. Each TestAuxService.java has different log message. something like
"TestAuxService in NM ClassPath" and "TestAuxService in Customer ClassPath"
4. Put one TestAuxService.jar into NM ClassPath, and put another
TestAuxService.jar into customer class path, such as
"/Users/xuan/dep/TestAuxService.jar"
5. modify several configuration in YARN-SITE.XML
{code}
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,TestAuxService</value>
<description>shuffle service that needs to be set for Map Reduce to run
</description>
</property>
<property>
<name>yarn.nodemanager.aux-services.TestAuxService.class</name>
<value>org.aux.TestAuxService</value>
</property>
{code}
6. start NM, and verified the log message in NM logs, we can see
{code}
Test My AuxService in NM ClassPath in Service Init stage
Test My AuxService in NM ClassPath in Service Start stage
{code}
And we can verify that we load the TestAuxService class from NM Class Path
7. add one more configuration into yarn-site.xml
{code}
<property>
<name>yarn.nodemanager.aux-services.TestAuxService.class.classpath</name>
<value>/Users/xuan/dep/TestAuxService.jar</value>
</property>
{code}
8. Start NM, and check log message in NM log, we can find
{code}
Test My AuxService in Customer ClassPath in Service Init stage
Test My AuxService in Customer ClassPath in Service Start stage
{code}
we can verify that if we set the customer class path, we would load
TestAuxService from customer class path instead of NM classpath.
> Enable aux services to have their own custom classpath/jar file
> ---------------------------------------------------------------
>
> Key: YARN-4577
> URL: https://issues.apache.org/jira/browse/YARN-4577
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.8.0
> Reporter: Xuan Gong
> Assignee: Xuan Gong
> Attachments: YARN-4577.1.patch, YARN-4577.2.patch,
> YARN-4577.20160119.1.patch, YARN-4577.20160204.patch, YARN-4577.3.patch,
> YARN-4577.3.rebase.patch, YARN-4577.4.patch, YARN-4577.5.patch,
> YARN-4577.poc.patch
>
>
> Right now, users have to add their jars to the NM classpath directly, thus
> put them on the system classloader. But if multiple versions of the plugin
> are present on the classpath, there is no control over which version actually
> gets loaded. Or if there are any conflicts between the dependencies
> introduced by the auxiliary service and the NM itself, they can break the NM,
> the auxiliary service, or both.
> The solution could be: to instantiate aux services using a classloader that
> is different from the system classloader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
