Prabhu Joseph created YARN-9967:
-----------------------------------
Summary: Fix NodeManager failing to start when Hdfs Auxillary Jar
is set
Key: YARN-9967
URL: https://issues.apache.org/jira/browse/YARN-9967
Project: Hadoop YARN
Issue Type: Bug
Components: auxservices, nodemanager
Affects Versions: 3.3.0
Reporter: Prabhu Joseph
Loading an auxiliary jar from a Hdfs location on a node manager fails with
ClassNotFound Exception
{code:java}
2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader:
classpath: []
2019-11-08 03:59:49,256 INFO org.apache.hadoop.util.ApplicationClassLoader:
system classes: [java., javax.accessibility., javax.activation.,
javax.activity., javax.annotation., javax.annotation.processing.,
javax.crypto., javax.imageio., javax.jws., javax.lang.model.,
-javax.management.j2ee., javax.management., javax.naming., javax.net.,
javax.print., javax.rmi., javax.script., -javax.security.auth.message.,
javax.security.auth., javax.security.cert., javax.security.sasl., javax.sound.,
javax.sql., javax.swing., javax.tools., javax.transaction.,
-javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax.,
org.apache.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase.,
org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml,
yarn-default.xml]
2019-11-08 03:59:49,257 INFO org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed
in state INITED
java.lang.ClassNotFoundException: org.apache.auxtest.AuxServiceFromHDFS
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:270)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:321)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:478)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)
{code}
*Repro:*
{code:java}
1. Prepare a custom auxiliary service jar and place it on hdfs
[hdfs@yarndocker-1 yarn]$ cat TestShuffleHandler2.java
package org;
import org.apache.hadoop.yarn.server.api.AuxiliaryService;
import org.apache.hadoop.yarn.server.api.ApplicationInitializationContext;
import org.apache.hadoop.yarn.server.api.ApplicationTerminationContext;
import java.nio.ByteBuffer;
public class TestShuffleHandler2 extends AuxiliaryService {
public static final String MAPREDUCE_TEST_SHUFFLE_SERVICEID =
"test_shuffle2";
public TestShuffleHandler2() {
super("testshuffle2");
}
@Override
public void initializeApplication(ApplicationInitializationContext context)
{
}
@Override
public void stopApplication(ApplicationTerminationContext context) {
}
@Override
public synchronized ByteBuffer getMetaData() {
return ByteBuffer.allocate(0);
}
}
[hdfs@yarndocker-1 yarn]$ javac -d . -cp `hadoop classpath`
TestShuffleHandler2.java
[hdfs@yarndocker-1 yarn]$ jar cvf auxhdfs.jar org/
[hdfs@yarndocker-1 mapreduce]$ hadoop fs -mkdir /AUX
[hdfs@yarndocker-1 mapreduce]$ hadoop fs -put /tmp/auxhdfs.jar /AUX
[hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 777 /AUX
[hdfs@yarndocker-1 mapreduce]$ hadoop fs -chmod 600 /AUX/auxhdfs.jar
[hdfs@yarndocker-1 mapreduce]$ hadoop fs -chown -R yarn:hadoop /AUX
2. Configure YARN NodeManager (yarn-site.xml) to pick from Hdfs
<property>
<name>yarn.nodemanager.aux-services</name>
<value>auxhdfs</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.auxhdfs.class</name>
<value>org.TestShuffleHandler2</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.auxhdfs.remote-classpath</name>
<value>/AUX/auxhdfs.jar</value>
</property>
{code}
YARN-9965 issue is applicable for branch-3.2 and trunk (with
yarn.nodemanager.aux-services.auxhdfs.system-classes configured) where NM runs
fine on first time and not on subsequent restart.
This issue is applicable for trunk where NM fails to start any attempt without
yarn.nodemanager.aux-services.auxhdfs.system-classes. This was an optional
config in branch-3.2. YARN-9075 has set the configured custom class as a system
class by default which caused the ClassNotFoundException as this class won't be
in system class loader.
AuxServices#getSystemClasses has to return Null if not configured
{code:java}
private static String[] getSystemClasses(AuxServiceRecord service, String
className) {
AuxServiceConfiguration serviceConf =
service.getConfiguration();
if (serviceConf != null && serviceConf.getProperty(SYSTEM_CLASSES) != null)
{
return StringUtils.split(serviceConf.getProperty(SYSTEM_CLASSES));
}
return null;
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
