[ https://issues.apache.org/jira/browse/TEZ-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
László Bodor updated TEZ-4376: ------------------------------ Description: When I try to start yarn nodemanager with tez shufflehandler configured, I get this: {code:java} java.lang.NoClassDefFoundError: org/apache/tez/shaded/javax/security/cert/X509Certificate at org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38) at org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601) at org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054) Caused by: java.lang.ClassNotFoundException: org.apache.tez.shaded.javax.security.cert.X509Certificate at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157) ... 15 more {code} this is because in netty 4, EmptyArrays class - which is originally referred to by DefaultChannelId, here the stacktrace shows that I simply copied it to ShuffleHandler code to insert some logging - refers to X509Certificate I tried to add javax.security exclude as: {code:java} <relocation> <pattern>javax</pattern> <shadedPattern>org.apache.tez.shaded.$0</shadedPattern> <excludes> <exclude>javax.crypto.*</exclude> <exclude>javax.security.*</exclude> <!-- HERE --> </excludes> </relocation> {code} but it didn't work, not sure why *when I completely removed javax relocation, it started to work* I'm tempted to make javax.security exclusion work, but first I need to understand why javax relocation is needed at all is there a chance that we want to package a custom javax to be used instead of what is provided on nodemanager's classpath? > Remove javax.security from the tez-auxservices shaded jar > --------------------------------------------------------- > > Key: TEZ-4376 > URL: https://issues.apache.org/jira/browse/TEZ-4376 > Project: Apache Tez > Issue Type: Sub-task > Affects Versions: 0.10.1 > Reporter: László Bodor > Assignee: László Bodor > Priority: Major > Labels: ShuffleHandler > Fix For: 0.10.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When I try to start yarn nodemanager with tez shufflehandler configured, I > get this: > {code:java} > java.lang.NoClassDefFoundError: > org/apache/tez/shaded/javax/security/cert/X509Certificate > at > org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38) > at > org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601) > at > org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054) > Caused by: java.lang.ClassNotFoundException: > org.apache.tez.shaded.javax.security.cert.X509Certificate > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > at > org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189) > at > org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157) > ... 15 more > {code} > this is because in netty 4, EmptyArrays class - which is originally referred > to by DefaultChannelId, here the stacktrace shows that I simply copied it to > ShuffleHandler code to insert some logging - refers to X509Certificate > I tried to add javax.security exclude as: > {code:java} > <relocation> > <pattern>javax</pattern> > <shadedPattern>org.apache.tez.shaded.$0</shadedPattern> > <excludes> > <exclude>javax.crypto.*</exclude> > <exclude>javax.security.*</exclude> <!-- HERE --> > </excludes> > </relocation> > {code} > but it didn't work, not sure why > *when I completely removed javax relocation, it started to work* > I'm tempted to make javax.security exclusion work, but first I need to > understand why javax relocation is needed at all > is there a chance that we want to package a custom javax to be used instead > of what is provided on nodemanager's classpath? -- This message was sent by Atlassian Jira (v8.20.10#820010)