[ https://issues.apache.org/jira/browse/HDDS-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Soumitra Sulav updated HDDS-781: -------------------------------- Description: HDP integrated with Ambari has a _/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the libraries needed for a MR job to run and is copied in the yarn containers at time of execution. As introducing ozone filesystem, relevant jars need to be packaged as part of the tar, also the tar is placed as part of _yum install hadoop_ components done by Ambari during cluster setup. During an MR Job run, I faced below java.lang.*NoClassDefFoundError* exceptions : {code:java} org/apache/hadoop/fs/ozone/OzoneFileSystem org/apache/ratis/proto/RaftProtos$ReplicationLevel org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum {code} Adding the relevant jar in the mentioned tar file resolves the exception. Complete stacktrace for one of the *NoClassDefFoundError* exception : {code:java} 2018-10-31 10:03:05,191 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64) at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221) at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116) at org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421) at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214) at org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116) at org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691) Caused by: java.lang.ClassNotFoundException: org.apache.ratis.proto.RaftProtos$ReplicationLevel at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 29 more 2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.NoClassDefFoundError: org/apache/ratis/proto/RaftProtos$ReplicationLevel{code} was: HDP integrated with Ambari has a _/usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz_ file containing all the libraries needed for a MR job to run and is copied in the yarn containers at time of execution. As introducing ozone filesystem, relevant jars need to be packaged as part of the tar, also the tar is placed as part of _yum install hadoop_ components done by Ambari during cluster setup. During an MR Job run, I faced below java.lang.NoClassDefFoundError exceptions : org/apache/hadoop/fs/ozone/OzoneFileSystem org/apache/ratis/proto/RaftProtos$ReplicationLevel org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum Adding the relevant jar in the mentioned tar file resolves the exception. > Ambari HDP NoClassDefFoundError for MR jobs > ------------------------------------------- > > Key: HDDS-781 > URL: https://issues.apache.org/jira/browse/HDDS-781 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem > Affects Versions: 0.3.0 > Reporter: Soumitra Sulav > Priority: Major > > HDP integrated with Ambari has a > _/*usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz*_ file containing all the > libraries needed for a MR job to run and is copied in the yarn containers at > time of execution. > As introducing ozone filesystem, relevant jars need to be packaged as part of > the tar, also the tar is placed as part of _yum install hadoop_ components > done by Ambari during cluster setup. > During an MR Job run, I faced below java.lang.*NoClassDefFoundError* > exceptions : > > {code:java} > org/apache/hadoop/fs/ozone/OzoneFileSystem > org/apache/ratis/proto/RaftProtos$ReplicationLevel > org/apache/ratis/thirdparty/com/google/protobuf/ProtocolMessageEnum > {code} > > Adding the relevant jar in the mentioned tar file resolves the exception. > > Complete stacktrace for one of the *NoClassDefFoundError* exception : > {code:java} > 2018-10-31 10:03:05,191 ERROR [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster > java.lang.NoClassDefFoundError: > org/apache/ratis/proto/RaftProtos$ReplicationLevel > at org.apache.hadoop.hdds.scm.ScmConfigKeys.<clinit>(ScmConfigKeys.java:64) > at org.apache.hadoop.ozone.OzoneConfigKeys.<clinit>(OzoneConfigKeys.java:221) > at org.apache.hadoop.ozone.client.OzoneBucket.<init>(OzoneBucket.java:116) > at > org.apache.hadoop.ozone.client.rpc.RpcClient.getBucketDetails(RpcClient.java:421) > at org.apache.hadoop.ozone.client.OzoneVolume.getBucket(OzoneVolume.java:214) > at > org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:127) > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:160) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.<init>(FileOutputCommitter.java:116) > at > org.apache.hadoop.mapreduce.lib.output.PathOutputCommitterFactory.createFileOutputCommitter(PathOutputCommitterFactory.java:134) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitterFactory.createOutputCommitter(FileOutputCommitterFactory.java:35) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:338) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:552) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$3.call(MRAppMaster.java:534) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1802) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:534) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:311) > at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1760) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757) > at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691) > Caused by: java.lang.ClassNotFoundException: > org.apache.ratis.proto.RaftProtos$ReplicationLevel > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 29 more > 2018-10-31 10:03:05,203 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting > with status 1: java.lang.NoClassDefFoundError: > org/apache/ratis/proto/RaftProtos$ReplicationLevel{code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org