Bilwa S T created TEZ-4567:
------------------------------
Summary: Failed to load Lz4 codec
Key: TEZ-4567
URL: https://issues.apache.org/jira/browse/TEZ-4567
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.10.3
Reporter: Bilwa S T
Assignee: Bilwa S T
Currently we use hadoop 3.3.6 version of hadoop. As part of this Jira
HADOOP-17292, Lz4 is a provided dependency in Hadoop Common 3.3.1 for Lz4Codec,
so we need to add the dependency in tez as well. Otherwise we get the below
exception when we run hive job on tez
{code:java}
Caused by: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory at
org.apache.hadoop.io.compress.lz4.Lz4Compressor.<init>(Lz4Compressor.java:66)
at org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152) at
org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131)
at org.apache.hadoop.io.compress.Lz4Codec.createOutputStream(Lz4Codec.java:70)
at
org.apache.hadoop.hive.ql.exec.Utilities.createCompressedStream(Utilities.java:949)
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926) at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993) at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939) at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:556) at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 19 more Caused by: java.lang.ClassNotFoundException:
net.jpountz.lz4.LZ4Factory at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 53
more{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
