Dmitriy Fingerman created HIVE-27105: ----------------------------------------
Summary: Querying parquet table with zstd encryption is failing Key: HIVE-27105 URL: https://issues.apache.org/jira/browse/HIVE-27105 Project: Hive Issue Type: Bug Reporter: Dmitriy Fingerman Assignee: Dmitriy Fingerman Steps to reproduce on the local with upstream or downstream code: * Start HiveServer2 * Start beeline * Run {code:java} set hive.execution.engine=tez;{code} * Run {code:java} CREATE TABLE emp(id int, name string, department string, salary float) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS PARQUET TBLPROPERTIES ("parquet.compression"="zstd"); {code} * Run {code:java} Insert into emp VALUES (1, 'some name', 'some dept', 1.15); {code} * Getting java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool Java call stack for the error: {code:java} java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:90) E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83) E at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:111) E at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236) E at org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.<init>(BaseVectorizedColumnReader.java:137) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.<init>(VectorizedPrimitiveColumnReader.java:58) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:515) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:446) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:406) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:347) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:95) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:376) E at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59) E at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151) E at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) E at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) E at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297) E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280) E at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82) E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69) E at java.security.AccessController.doPrivileged(Native Method) E at javax.security.auth.Subject.doAs(Subject.java:422) E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69) E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39) E at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) E at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) E at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) E at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) E at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) E at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) E at java.lang.Thread.run(Thread.java:748) E Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.RecyclingBufferPool E at java.net.URLClassLoader.findClass(URLClassLoader.java:382) E at java.lang.ClassLoader.loadClass(ClassLoader.java:418) E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) E at java.lang.ClassLoader.loadClass(ClassLoader.java:351) E ... 36 more E , errorMessage=Cannot recover from this error:java.lang.NoClassDefFoundError: com/github/luben/zstd/RecyclingBufferPool E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:90) E at org.apache.parquet.hadoop.codec.ZstandardCodec.createInputStream(ZstandardCodec.java:83) E at org.apache.parquet.hadoop.CodecFactory$HeapBytesDecompressor.decompress(CodecFactory.java:111) E at org.apache.parquet.hadoop.ColumnChunkPageReadStore$ColumnChunkPageReader.readDictionaryPage(ColumnChunkPageReadStore.java:236) E at org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.<init>(BaseVectorizedColumnReader.java:137) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedPrimitiveColumnReader.<init>(VectorizedPrimitiveColumnReader.java:58) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:515) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:446) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:406) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:347) E at org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:95) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:376) E at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119) E at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59) E at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151) E at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116) E at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) E at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437) E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297) E at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280) E at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82) E at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69) E at java.security.AccessController.doPrivileged(Native Method) E at javax.security.auth.Subject.doAs(Subject.java:422) E at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69) E at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39) E at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) E at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) E at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69) E at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) E at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) E at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) E at java.lang.Thread.run(Thread.java:748) E Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.RecyclingBufferPool E at java.net.URLClassLoader.findClass(URLClassLoader.java:382) E at java.lang.ClassLoader.loadClass(ClassLoader.java:418) E at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) E at java.lang.ClassLoader.loadClass(ClassLoader.java:351) E ... 36 more{code} This is what happens during the execution of this insert statement: # At some point Hive calls Tez execution on YARN cluster # Tez calls the code in Hive-Exec jar. Tez classpath does not have zstd-jni.jar. Tez classpath include Tez jars, Tez lib jars and hive-exec. # Hive code calls Parquet shaded in hive-exec. # Parquet is failing because it cannot find zstd-jni classes at runtime. -- This message was sent by Atlassian Jira (v8.20.10#820010)