[
https://issues.apache.org/jira/browse/HIVE-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781745#comment-17781745
]
Basapuram Kumar edited comment on HIVE-27835 at 11/1/23 2:17 PM:
-----------------------------------------------------------------
Adding the {color:#00875a}lz4-java-1.7.1.jar{color} into the
{color:#0747a6}HIVE_AUX_JARS_PATH{color} from {color:#0747a6}hive-env{color}
would help to fix this issue/
{code:java}
export
HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}:/usr/odp/3.2.2.0-1/hive/lib/lz4-java-1.7.1.jar
{code}
Followed by restart the hive.
And from beeline, we see that this change got reflected.
{code:java}
0: jdbc:hive2://odp01.ubuntu.ad.ce:2181,odp02> set hive.aux.jars.path;
+----------------------------------------------------+
| set |
+----------------------------------------------------+
|
hive.aux.jars.path=file:///usr/odp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,file:///usr/odp/3.2.2.0-1/hive/lib/lz4-java-1.7.1.jar
| {code}
was (Author: JIRAUSER295887):
Adding the {color:#00875a}lz4-java-1.7.1.jar{color} into the
{color:#0747a6}HIVE_AUX_JARS_PATH{color} from {color:#0747a6}hive-env{color}
would help to fix this issue/
{code:java}
export
HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH}:/usr/odp/3.2.2.0-1/hive/lib/lz4-java-1.7.1.jar
{code}
> java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for
> DruidStorageHandler
> ----------------------------------------------------------------------------------
>
> Key: HIVE-27835
> URL: https://issues.apache.org/jira/browse/HIVE-27835
> Project: Hive
> Issue Type: Task
> Components: Hive
> Affects Versions: 4.0.0-alpha-1
> Environment: hive-4-alpha-1
> Druid-27.0.0
> Reporter: Basapuram Kumar
> Assignee: Basapuram Kumar
> Priority: Major
>
> Hello Team,
> I am trying to use Druid storage handler to create a table.
> Steps to repro
> Create table
>
> {code:java}
> CREATE TABLE emp_zlib_3110_2038 (
> `__time` TIMESTAMP,
> id int,
> name string,
> age int,
> gender string)
> STORED AS ORC
> TBLPROPERTIES ('transactional'='true',"orc.compress"="ZLIB");
> {code}
> insert data
> {code:java}
> insert into emp_zlib_3110_2038 values(CURRENT_TIMESTAMP(),10,'basa',30,'M');
> {code}
> Create a DruidStorageHandler table
> {code:java}
> CREATE TABLE druid_hive_table_3110_2038_02
> STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
> TBLPROPERTIES ("druid.datasource" = "druid_hive_table_3110_2038_02")
> AS
> select `__time`, id,name,age,gender from emp_zlib_3110_2038 ; {code}
> Error:
> {code:java}
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1698753624381_0021)
> ----------------------------------------------------------------------------------------------
>
> VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED
> KILLED
> ----------------------------------------------------------------------------------------------
> Map 1 .......... container SUCCEEDED 1 1 0 0
> 0 0
> Reducer 2 ... container RUNNING 2 1 0 1
> 4 0
> ----------------------------------------------------------------------------------------------
> VERTICES: 01/02 [=================>>---------] 66% ELAPSED TIME: 37.91 s
> ----------------------------------------------------------------------------------------------
> ERROR : Status: FailedERROR : Vertex failed, vertexName=Reducer 2,
> vertexId=vertex_1698753624381_0021_3_01, diagnostics=[Task failed,
> taskId=task_1698753624381_0021_3_01_000000, diagnostics=[TaskAttempt 0
> failed, info=[Error: Error while running task ( failure ) :
> attempt_1698753624381_0021_3_01_000000_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:348)
>
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
>
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
>
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
>
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
>
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at java.lang.Thread.run(Thread.java:750)Caused by:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:311)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
>
> ... 16 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
> Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:385)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:301)
>
> ... 18 more
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
> net/jpountz/lz4/LZ4Factory
> at
> org.apache.hive.druid.com.google.common.base.Throwables.propagate(Throwables.java:241)
>
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:209)
>
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.getSegmentIdentifierAndMaybePush(DruidRecordWriter.java:164)
>
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.write(DruidRecordWriter.java:258)
>
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1160)
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:94)
>
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
>
> ... 19 moreCaused
> by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError:
> net/jpountz/lz4/LZ4Factory at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
>
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
>
> at
> org.apache.hive.druid.com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
>
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:173)
>
> ... 25 more {code}
> I see that lz4-java jar is available from hive/lib
> {code:java}
> $ grep -irn "net.jpountz.lz4.LZ4Factory" hive/lib/
> Binary file hive/lib/lz4-java-1.7.1.jar matches {code}
> Can u help us how to fix this.?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
