[
https://issues.apache.org/jira/browse/TEZ-4567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850257#comment-17850257
]
Bilwa S T commented on TEZ-4567:
--------------------------------
cc [~ayushsaxena] [~abstractdog]
> Failed to load Lz4 codec
> ------------------------
>
> Key: TEZ-4567
> URL: https://issues.apache.org/jira/browse/TEZ-4567
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.10.3
> Reporter: Bilwa S T
> Assignee: Bilwa S T
> Priority: Major
>
> Currently we use hadoop 3.3.6 version. As part of this Jira HADOOP-17292, Lz4
> is a provided dependency in Hadoop Common 3.3.1 for Lz4Codec, so we need to
> add the dependency in tez as well. Otherwise we get the below exception when
> we run hive job on tez
> {code:java}
> Caused by: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
> at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> at
> org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)
> at
> org.apache.hadoop.io.compress.Lz4Codec.createOutputStream(Lz4Codec.java:70)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.createCompressedStream(Utilities.java:949)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116) at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
> org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:556)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> ... 19 more Caused by: java.lang.ClassNotFoundException:
> net.jpountz.lz4.LZ4Factory at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 53
> more{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
