[jira] [Commented] (HIVE-11651) Index creation is failing from beeline when execution engine is set to Tez

[Venkata Srinivasa Rao Kolla (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712891#comment-14712891
 ] 

Venkata Srinivasa Rao Kolla  commented on HIVE-11651:
-

I have tried this on Hive 0.13 version and Tez 0.4

> Index creation is failing from beeline when execution engine is set to Tez
> --
>
> Key: HIVE-11651
> URL: https://issues.apache.org/jira/browse/HIVE-11651
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline, Hive, Tez
>Reporter: Venkata Srinivasa Rao Kolla 
>
> In Hive, with execution engine is set to Tez and when we tried to rebuild 
> index on a table (which is not empty) using beeline, index creation is not 
> happening. It is not throwing any error too on to console. But in the 
> corresponding tez job logs we found below error:
> 2015-08-24 18:08:43,999 WARN [AMShutdownThread] 
> org.apache.tez.dag.history.recovery.RecoveryService: Error when closing 
> summary stream 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>  No lease on 
> /tmp/hive-alti-test-01/_tez_session_dir/573df26e-4a22-4bf5-ad5c-a0d72cdacec6/application_1439396922313_1732/recovery/1/application_1439396922313_1732.summary:
>  File does not exist. Holder DFSClient_NONMAPREDUCE_-117450461_1 does not 
> have any open files. at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2956)
>  at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3027)
>  at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3007)
>  at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:641)
>  at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:484)
>  at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>  at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at 
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at 
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at 
> java.security.AccessController.doPrivileged(Native Method) at 
> javax.security.auth.Subject.doAs(Subject.java:415) at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> Below are the set of commands we have used: CREATE TABLE table02(column1 
> String, column2 string, column3 int, column4 string) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY ','; LOAD DATA LOCAL INPATH 'posts_us' OVERWRITE INTO 
> TABLE table02; CREATE INDEX table02_index ON TABLE table02 (column3) AS 
> 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED 
> REBUILD; ALTER INDEX table02_index ON table02 REBUILD;
> Does any one seen this problem?
> Regards, Srinivas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HIVE-11651) Index creation is failing from beeline when execution engine is set to Tez

[Venkata Srinivasa Rao Kolla (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712892#comment-14712892
 ] 

Venkata Srinivasa Rao Kolla  commented on HIVE-11651:
-

Also tried to run the same set of scripts on Hive 1.2 + Tez 0.7, i saw some 
errors in the job log. 

2015-08-26 09:48:31,178 INFO [Dispatcher thread: Central] 
history.HistoryEventHandler: 
[HISTORY][DAG:dag_1440569395691_0214_1][Event:DAG_FINISHED]: 
dagId=dag_1440569395691_0214_1, startTime=1440582480900, 
finishTime=1440582511086, timeTaken=30186, status=FAILED, diagnostics=Vertex 
failed, vertexName=Map 1, vertexId=vertex_1440569395691_0214_1_00, 
diagnostics=[Task failed, taskId=task_1440569395691_0214_1_00_00, 
diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.RuntimeException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:192)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.(TezGroupedSplitsInputFormat.java:131)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:97)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80)
at 
org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:614)
at 
org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:593)
at 
org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:141)
at 
org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:370)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:127)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
... 14 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:270)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:234)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:189)
... 25 more
], TaskAttempt 1 failed, info=[Error: Failure while running 
task:java.lang.RuntimeException: java.lang.RuntimeException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
at 
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor