Hiroyuki Adachi created YARN-11030: -------------------------------------- Summary: ClassNotFoundException when aux service class is loaded from customized classpath Key: YARN-11030 URL: https://issues.apache.org/jira/browse/YARN-11030 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.3.1, 3.3.0 Reporter: Hiroyuki Adachi
NodeManager failed to load the aux service with ClassNotFoundException while loading the class from the customized classpath. {noformat} <configuration> <property name="yarn.nodemanager.aux-services" value="spark_shuffle"/> <property name="yarn.nodemanager.aux-services.spark_shuffle.class" value="org.apache.spark.network.yarn.YarnShuffleService"/> <property name="yarn.nodemanager.aux-services.spark_shuffle.classpath" value="/tmp/spark-3.1.2-yarn-shuffle.jar"/> <!-- Note that the spark_shuffle classpath is not included in HADOOP_CLASSPATH and yarn.application.classpath. --> </configuration> {noformat} {noformat} 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader: classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar] 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader: system classes: [org.apache.spark.network.yarn.YarnShuffleService] 2021-12-06 15:32:09,169 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:482) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:761) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:327) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:494) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:962) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1042) Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.ja va:165) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxServiceFromLocalClasspath(AuxServices.java:242) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxService(AuxServices.java:271) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:452) ... 10 more 2021-12-06 15:32:09,172 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED{noformat} YARN-9075 may cause this problem. The default system classes were changed by this patch. Before YARN-9075: isSystemClass() returns false since the system classes does not contain the aux service class itself, and the class will be loaded from the customized classpath. [https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java#L176] {noformat} 2021-12-06 15:50:21,332 INFO org.apache.hadoop.util.ApplicationClassLoader: classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar] 2021-12-06 15:50:21,332 INFO org.apache.hadoop.util.ApplicationClassLoader: system classes: [java., javax.accessibility., -javax.activation., javax.activity., j avax.annotation., javax.annotation.processing., javax.crypto., javax.imageio., javax.jws., javax.lang.model., -javax.management.j2ee., javax.management., javax. naming., javax.net., javax.print., javax.rmi., javax.script., -javax.security.auth.message., javax.security.auth., javax.security.cert., javax.security.sasl., j avax.sound., javax.sql., javax.swing., javax.tools., javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml., org.w3c.dom., org.xml.sax., org.apac he.commons.logging., org.apache.log4j., -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml] {noformat} After YARN-9075: isSystemClass() returns true and the parent classloader will be used. It does not know the customized classpath and throws the ClassNotFoundException. [https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ApplicationClassLoader.java#L189] {noformat} 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader: classpath: [file:/tmp/spark-3.1.2-yarn-shuffle.jar] 2021-12-06 15:32:09,168 INFO org.apache.hadoop.util.ApplicationClassLoader: system classes: [org.apache.spark.network.yarn.YarnShuffleService]{noformat} workaround: set dummy system-classes so that isSystemClass() returns false. {noformat} <configuration> <property name="yarn.nodemanager.aux-services.spark_shuffle.system-classes" value="dummy"/> </configuration>{noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org