[
https://issues.apache.org/jira/browse/YARN-8223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-8223:
-----------------------------
Fix Version/s: 3.1.1
3.2.0
> ClassNotFoundException when auxiliary service is loaded from HDFS
> -----------------------------------------------------------------
>
> Key: YARN-8223
> URL: https://issues.apache.org/jira/browse/YARN-8223
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Charan Hebri
> Assignee: Zian Chen
> Priority: Blocker
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8223.001.patch, YARN-8223.002.patch
>
>
> Loading an auxiliary jar from a local location on a node manager works as
> expected,
> {noformat}
> 2018-04-26 15:09:26,179 INFO util.ApplicationClassLoader
> (ApplicationClassLoader.java:<init>(98)) - classpath:
> [file:/grid/0/hadoop/yarn/local/aux-service-local.jar]
> 2018-04-26 15:09:26,179 INFO util.ApplicationClassLoader
> (ApplicationClassLoader.java:<init>(99)) - system classes: [java.,
> javax.accessibility., javax.activation., javax.activity., javax.annotation.,
> javax.annotation.processing., javax.crypto., javax.imageio., javax.jws.,
> javax.lang.model., -javax.management.j2ee., javax.management., javax.naming.,
> javax.net., javax.print., javax.rmi., javax.script.,
> -javax.security.auth.message., javax.security.auth., javax.security.cert.,
> javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools.,
> javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml.,
> org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j.,
> -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml,
> hdfs-default.xml, mapred-default.xml, yarn-default.xml]
> 2018-04-26 15:09:26,181 INFO containermanager.AuxServices
> (AuxServices.java:serviceInit(252)) - The aux service:test_aux_local are
> using the custom classloader
> 2018-04-26 15:09:26,182 WARN containermanager.AuxServices
> (AuxServices.java:serviceInit(268)) - The Auxiliary Service named
> 'test_aux_local' in the configuration is for class
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader
> which has a name of 'org.apache.auxtest.AuxServiceFromLocal with custom
> class loader'. Because these are not the same tools trying to send
> ServiceData and read Service Meta Data may have issues unless the refer to
> the name in the config.
> 2018-04-26 15:09:26,182 INFO containermanager.AuxServices
> (AuxServices.java:addService(103)) - Adding auxiliary service
> org.apache.auxtest.AuxServiceFromLocal with custom class loader,
> "test_aux_local"{noformat}
> But loading the same jar from a location on HDFS fails with a
> ClassNotFoundException.
> {noformat}
> 018-04-26 15:14:39,683 INFO util.ApplicationClassLoader
> (ApplicationClassLoader.java:<init>(98)) - classpath: []
> 2018-04-26 15:14:39,683 INFO util.ApplicationClassLoader
> (ApplicationClassLoader.java:<init>(99)) - system classes: [java.,
> javax.accessibility., javax.activation., javax.activity., javax.annotation.,
> javax.annotation.processing., javax.crypto., javax.imageio., javax.jws.,
> javax.lang.model., -javax.management.j2ee., javax.management., javax.naming.,
> javax.net., javax.print., javax.rmi., javax.script.,
> -javax.security.auth.message., javax.security.auth., javax.security.cert.,
> javax.security.sasl., javax.sound., javax.sql., javax.swing., javax.tools.,
> javax.transaction., -javax.xml.registry., -javax.xml.rpc., javax.xml.,
> org.w3c.dom., org.xml.sax., org.apache.commons.logging., org.apache.log4j.,
> -org.apache.hadoop.hbase., org.apache.hadoop., core-default.xml,
> hdfs-default.xml, mapred-default.xml, yarn-default.xml]
> 2018-04-26 15:14:39,687 INFO service.AbstractService
> (AbstractService.java:noteFailure(267)) - Service
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed
> in state INITED
> java.lang.ClassNotFoundException: org.apache.auxtest.AuxServiceFromLocal
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:249)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:316)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:472)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:918)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:979)
> {noformat}
> The difference between the 2 logs is the classpath variable in the 1st line
> of the log is empty in the HDFS case. It doesn't have the URL/filename of the
> jar file specified in the config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
