[jira] [Created] (HIVE-14596) Canceling hive query takes very long time

2016-08-22 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-14596:
---

 Summary: Canceling hive query takes very long time
 Key: HIVE-14596
 URL: https://issues.apache.org/jira/browse/HIVE-14596
 Project: Hive
  Issue Type: Bug
Reporter: Yongzhi Chen


when the Hue user clicks cancel, the Hive query does not stop immediately, it 
can take very long time. And in the yarn job history you will see exceptions 
like following:
{noformat}
org.apache.hadoop.hive.ql.metadata.HiveException: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 No lease on 
/tmp/hive/hive/80a5cfdb-9f98-44d2-ae53-332c8dae62a3/hive_2016-08-20_07-06-12_819_8780093905859269639-3/-mr-1/.hive-staging_hive_2016-08-20_07-06-12_819_8780093905859269639-3/_task_tmp.-ext-10001/_tmp.00_0
 (inode 28224): File does not exist. Holder 
DFSClient_attempt_1471630445417_0034_m_00_0_-50732711_1 does not have any 
open files.
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3624)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3427)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3283)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:677)
at 
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:213)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:485)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.abortWriters(FileSinkOperator.java:246)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1007)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:206)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51287: The ConsoleReader.readLine method never echoes back the executed line. Instead, Beeline.info is used, which is redirected to stderr with other messages. On interactive mode b

2016-08-22 Thread Miklos Csanady 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51287/
---

(Updated Aug. 22, 2016, 2:55 p.m.)


Review request for hive, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.


Bugs: HIVE-14525
https://issues.apache.org/jira/browse/HIVE-14525


Repository: hive-git


Description
---

HIVE-14525: beeline still writing log data to stdout as of version 2.1.0


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java e0fa032 

Diff: https://reviews.apache.org/r/51287/diff/


Testing
---

Beeline with redirected input and output with script file and without one.


Thanks,

Miklos Csanady

Re: Review Request 51287: The ConsoleReader.readLine method never echoes back the executed line. Instead, Beeline.info is used, which is redirected to stderr with other messages. On interactive mode b

2016-08-22 Thread Miklos Csanady 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51287/
---

(Updated Aug. 22, 2016, 3:15 p.m.)


Review request for hive, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.


Changes
---

Fixed whitespaces


Bugs: HIVE-14525
https://issues.apache.org/jira/browse/HIVE-14525


Repository: hive-git


Description
---

HIVE-14525: beeline still writing log data to stdout as of version 2.1.0


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java e0fa032 

Diff: https://reviews.apache.org/r/51287/diff/


Testing
---

Beeline with redirected input and output with script file and without one.


Thanks,

Miklos Csanady

Re: Review Request 51287: The ConsoleReader.readLine method never echoes back the executed line. Instead, Beeline.info is used, which is redirected to stderr with other messages. On interactive mode b

2016-08-22 Thread Peter Vary 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51287/#review146358
---



Hi Miklos,

Thanks for the patch Miklos!

I am not absoultely sure about the "business" requirements, but if we accept 
that then there only a few nits in the code.

Could this change mean, than the following lines could be separated in the 
beeline command line output?
Like this:
0: jdbc:hive2://localhost:1> show create table escape_comments_tbl1;
++
show create table escape_comments_tbl1;
|   createtab_stmt   |

Have not tried it, just curious :)

Thanks,
Peter


beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 992)


nit space between ){



beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 993)


nit: spaces after ','


- Peter Vary


On Aug. 22, 2016, 3:15 p.m., Miklos Csanady wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51287/
> ---
> 
> (Updated Aug. 22, 2016, 3:15 p.m.)
> 
> 
> Review request for hive, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14525
> https://issues.apache.org/jira/browse/HIVE-14525
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14525: beeline still writing log data to stdout as of version 2.1.0
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java e0fa032 
> 
> Diff: https://reviews.apache.org/r/51287/diff/
> 
> 
> Testing
> ---
> 
> Beeline with redirected input and output with script file and without one.
> 
> 
> Thanks,
> 
> Miklos Csanady
> 
>

[GitHub] hive pull request #98: HIVE-14217: Druid integration

2016-08-22 Thread jcamachor 
GitHub user jcamachor opened a pull request:

https://github.com/apache/hive/pull/98

HIVE-14217: Druid integration



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jcamachor/hive HIVE-druid

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/98.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #98


commit e633db81bb1e3064b6a638b897098d585da93659
Author: Jesus Camacho Rodriguez 
Date:   2016-08-12T11:55:46Z

HIVE-14217: Druid integration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-14597) Support for Druid custom granularities

2016-08-22 Thread Jesus Camacho Rodriguez (JIRA) 
Jesus Camacho Rodriguez created HIVE-14597:
--

 Summary: Support for Druid custom granularities
 Key: HIVE-14597
 URL: https://issues.apache.org/jira/browse/HIVE-14597
 Project: Hive
  Issue Type: Sub-task
  Components: Druid integration
Affects Versions: 2.2.0
Reporter: Jesus Camacho Rodriguez


In addition to Druid native granularities YEAR, QUARTER, MONTH, WEEK, DAY, 
HOUR, MINUTE, SECOND, NONE, and ALL, we support and translate custom 
granularities into Druid _duration_ and _period_ granularities.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51287: The ConsoleReader.readLine method never echoes back the executed line. Instead, Beeline.info is used, which is redirected to stderr with other messages. On interactive mode b

2016-08-22 Thread Miklos Csanady 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51287/
---

(Updated Aug. 22, 2016, 5:40 p.m.)


Review request for hive, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.


Changes
---

missing spaces


Bugs: HIVE-14525
https://issues.apache.org/jira/browse/HIVE-14525


Repository: hive-git


Description
---

HIVE-14525: beeline still writing log data to stdout as of version 2.1.0


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java e0fa032 

Diff: https://reviews.apache.org/r/51287/diff/


Testing
---

Beeline with redirected input and output with script file and without one.


Thanks,

Miklos Csanady

[jira] [Created] (HIVE-14598) Disallow newlines in query name

2016-08-22 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-14598:
-

 Summary: Disallow newlines in query name
 Key: HIVE-14598
 URL: https://issues.apache.org/jira/browse/HIVE-14598
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth


Hive sets up the query name based on the query text. This is truncated to a 
certain number of characters, but newlines are allowed.
This makes logs really ugly - spanning across lines for the same log line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14599) Log hive query id to llap activity/history log

2016-08-22 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-14599:
-

 Summary: Log hive query id to llap activity/history log
 Key: HIVE-14599
 URL: https://issues.apache.org/jira/browse/HIVE-14599
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14600) LLAP zookeeper registry failures do not fail the daemon

2016-08-22 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-14600:
---

 Summary: LLAP zookeeper registry failures do not fail the daemon
 Key: HIVE-14600
 URL: https://issues.apache.org/jira/browse/HIVE-14600
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14601) Altering table/partition file format with preexisting data should not be allowed

2016-08-22 Thread Barna Zsombor Klara (JIRA) 
Barna Zsombor Klara created HIVE-14601:
--

 Summary: Altering table/partition file format with preexisting 
data should not be allowed
 Key: HIVE-14601
 URL: https://issues.apache.org/jira/browse/HIVE-14601
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Barna Zsombor Klara
Assignee: Barna Zsombor Klara
Priority: Minor


The file format of a table or a partition can be changed using an alter 
statement. However this only affects the metadata, the data in hdfs is not 
changed, leading to a table from which you cannot select anymore. 
Changing the file format back fixes the issue, but a better approach would be 
to prevent the alter to the file format if we have data in the tables.

The issue is reproducible by executing the following commands:
{code}
create table test (id int);
insert into test values (1);
alter table test set fileformat parquet;
insert into test values (2);
select * from test;
{code}
Will result in:
{code}
java.lang.RuntimeException: .../00_0 is not a Parquet file (too small) 
(state=,code=0)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14602) NPE when running delete_orig_table.q under MiniLlap

2016-08-22 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-14602:


 Summary: NPE when running delete_orig_table.q under MiniLlap
 Key: HIVE-14602
 URL: https://issues.apache.org/jira/browse/HIVE-14602
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


When delete_orig_table.q is run under TestMiniLlapCliDriver, NPE is thrown

{code}
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : 
attempt_1471902873303_0001_31_01_00_1:java.lang.RuntimeException: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row (tag=0) 
{"key":{"reducesinkkey0":null},"value":null}
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row (tag=0) {"key":{"reducesinkkey0":null},"value":null}
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:289)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row (tag=0) {"key":{"reducesinkkey0":null},"value":null}
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
... 17 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:764)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
... 18 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14603) AssertionError when running explainuser_3.q under MiniLlap

2016-08-22 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-14603:


 Summary: AssertionError when running explainuser_3.q under MiniLlap
 Key: HIVE-14603
 URL: https://issues.apache.org/jira/browse/HIVE-14603
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


When running explainuser_3.q under TestMiniLlapCliDriver, following error is 
thrown

{code}
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
attempt_1471903600528_0001_32_01_00_3:java.lang.RuntimeException: 
java.lang.RuntimeException: java.lang.AssertionError
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.AssertionError
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:395)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
... 15 more
Caused by: java.lang.AssertionError
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setBatchSize(VectorizedBatchUtil.java:125)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:459)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:386)
... 18 more
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14604) ClassCastException when running orc_vectorization_ppd.q under MiniLlap

2016-08-22 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-14604:


 Summary: ClassCastException when running orc_vectorization_ppd.q 
under MiniLlap
 Key: HIVE-14604
 URL: https://issues.apache.org/jira/browse/HIVE-14604
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


When orc_vectorization_ppd.q is run under TestMiniLlapCliDriver, following 
exception is thrown

{code}
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) : 
attempt_1471904180593_0001_44_00_00_3:java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:391)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
... 15 more
Caused by: java.io.IOException: java.io.IOException: 
java.lang.ClassCastException: org.apache.hadoop.hive.common.io.DiskRangeList 
cannot be cast to org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
... 17 more
Caused by: java.io.IOException: java.lang.ClassCastException: 
org.apache.hadoop.hive.common.io.DiskRangeList cannot be cast to 
org.apache.orc.impl.BufferChunk
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.rethrowErrorIfAny(LlapInputFormat.java:383)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.nextCvb(LlapInputFormat.java:338)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:278)
at 
org.apache.hadoop.hive.llap.io.api.impl.LlapInputFormat$LlapRecordReader.next(LlapInputFormat.java:167)
at 
org.apache.hadoop.hive.ql.io.BatchToRowReader.ensureBatch(BatchToRowReader.java:167)
at 
org.apache.hadoop.hive.ql.io.BatchToRowReader.next(BatchToRowReader.java:140)
at 
org.apache.hadoop.hive.ql.io.BatchToRowReader.next(BatchToR

[jira] [Created] (HIVE-14605) MetaException thrown when running partition_column_names_with_leading_and_trailing_spaces.q under MiniLlap

2016-08-22 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-14605:


 Summary: MetaException thrown when running 
partition_column_names_with_leading_and_trailing_spaces.q under MiniLlap
 Key: HIVE-14605
 URL: https://issues.apache.org/jira/browse/HIVE-14605
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran


Following exception is thrown when running 
partition_column_names_with_leading_and_trailing_spaces.q under 
TestMiniLlapCliDriver

{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
MetaException(message:Expecting a partition with name p=a, but metastore is 
returning a partition with name p=a .)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2120)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2052)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$TableSpec.(BaseSemanticAnalyzer.java:1094)
... 58 more
Caused by: MetaException(message:Expecting a partition with name p=a, but 
metastore is returning a partition with name p=a .)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMPartition(ObjectStore.java:1892)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionWithAuth(ObjectStore.java:2149)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
at com.sun.proxy.$Proxy116.getPartitionWithAuth(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partition_with_auth(HiveMetaStore.java:3165)
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99)
at com.sun.proxy.$Proxy117.get_partition_with_auth(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartitionWithAuthInfo(HiveMetaStoreClient.java:1315)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:162)
at com.sun.proxy.$Proxy118.getPartitionWithAuthInfo(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:2111)
... 60 more

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14606) Beeline fails if quoted string begins with \\

2016-08-22 Thread Sahil Takiar (JIRA) 
Sahil Takiar created HIVE-14606:
---

 Summary: Beeline fails if quoted string begins with \\
 Key: HIVE-14606
 URL: https://issues.apache.org/jira/browse/HIVE-14606
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Sahil Takiar
Assignee: Sahil Takiar


The following query fails {{select '\\' as literal;}} in Beeline with the 
following exception:

{code}
FAILED: ParseException line 1:22 extraneous input ';' expecting EOF near ''
16/08/22 15:46:15 [023ddb3b-1f3c-4db6-bd4e-bba392d6e4bb main]: ERROR ql.Driver: 
FAILED: ParseException line 1:22 extraneous input ';' expecting EOF near ''
org.apache.hadoop.hive.ql.parse.ParseException: line 1:22 extraneous input ';' 
expecting EOF near ''
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:215)
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:414)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1226)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1195)
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:282)
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:324)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:497)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:485)
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:294)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:505)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1412)
at com.sun.proxy.$Proxy30.ExecuteStatement(Unknown Source)
at 
org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:976)
at org.apache.hive.beeline.Commands.execute(Commands.java:1132)
at org.apache.hive.beeline.Commands.sql(Commands.java:1062)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1168)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:999)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:909)
at 
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:511)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:494)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}

This bug is a regression introduced by HIVE-12646



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14607) ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 1

2016-08-22 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-14607:
-

 Summary:  ORC split generation failed with exception: 
java.lang.ArrayIndexOutOfBoundsException: 1
 Key: HIVE-14607
 URL: https://issues.apache.org/jira/browse/HIVE-14607
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 2.2.0
Reporter: Eugene Koifman


Steps to repro: 
in TestTxnCommands2WithSplitUpdate remove the overridden method testOrcPPD().
Then run:
mvn test -Dtest=TestTxnCommands2WithSplitUpdate#testOrcPPD

it will fail with ArrayIndexOutOfBounds.  HIVE-14448 was supposed to have fixed 
it


{noformat}
2016-08-22T15:54:17,654  INFO [main] mapreduce.JobSubmitter: Cleaning up the 
staging area 
file:/Users/ekoifman/dev/hiverwgit/ql/target/tmp/hadoop-tmp/mapred/staging/ekoifman99742506\
0/.staging/job_local997425060_0002
2016-08-22T15:54:17,663 ERROR [main] exec.Task: Job Submission failed with 
exception 'java.lang.RuntimeException(ORC split generation failed with 
exception: java.lang.ArrayIndexOutO\
fBoundsException: 1)'
java.lang.RuntimeException: ORC split generation failed with exception: 
java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1670)
at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1756)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:370)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:488)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:329)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:321)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1297)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1294)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at 
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:417)
at 
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:141)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1983)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1674)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1410)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1134)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1122)
at 
org.apache.hadoop.hive.ql.TestTxnCommands2.runStatementOnDriver(TestTxnCommands2.java:1392)
at 
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD(TestTxnCommands2.java:195)
at 
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcPPD(TestTxnCommands2.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.

Re: Review Request 51287: The ConsoleReader.readLine method never echoes back the executed line. Instead, Beeline.info is used, which is redirected to stderr with other messages. On interactive mode b

2016-08-22 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51287/#review146463
---



Could you paste the test you made for this patch? like before & after the patch 
output?

- Sergio Pena


On Aug. 22, 2016, 5:40 p.m., Miklos Csanady wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51287/
> ---
> 
> (Updated Aug. 22, 2016, 5:40 p.m.)
> 
> 
> Review request for hive, Peter Vary, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14525
> https://issues.apache.org/jira/browse/HIVE-14525
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14525: beeline still writing log data to stdout as of version 2.1.0
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java e0fa032 
> 
> Diff: https://reviews.apache.org/r/51287/diff/
> 
> 
> Testing
> ---
> 
> Beeline with redirected input and output with script file and without one.
> 
> 
> Thanks,
> 
> Miklos Csanady
> 
>

Question about Hive on Spark

2016-08-22 Thread Tao Li 
Hi,

Looks like the Spark client (SparkClientImpl class) submits Sparks jobs to the 
YARN cluster by forking a process and kicking off spark-submit script. Are we 
provisioning new containers every time we submit a job? There could be a perf 
hit by doing that.

Thanks.

[jira] [Created] (HIVE-14608) LLAP: ZK registry doesn't remove nodes on kill

2016-08-22 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-14608:
---

 Summary: LLAP: ZK registry doesn't remove nodes on kill 
 Key: HIVE-14608
 URL: https://issues.apache.org/jira/browse/HIVE-14608
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Prasanth Jayachandran


...and presumably doesn't disable them for scheduling. I haven't looked in 
details though, I just see some harmless killed tasks in queries after I kill 
some LLAP nodes manually between queries
{noformat}
  public void workerNodeRemoved(ServiceInstance serviceInstance) {
 // FIXME: disabling this for now
instanceToNodeMap.remove(serviceInstance.getWorkerIdentity());
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-08-22 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51312/
---

Review request for hive and Prasanth_J.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/ServiceInstanceSet.java
 99ead9b 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java
 e9456f2 
  
llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java
 64d2617 
  llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/SlotZnode.java 
PRE-CREATION 
  
llap-client/src/java/org/apache/hadoop/hive/llap/security/LlapTokenClient.java 
921e050 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java
 17ce69b 
  
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
 efd774d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HostAffinitySplitLocationProvider.java
 c06499e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/Utils.java 8a4fc08 
  
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestHostAffinitySplitLocationProvider.java
 d98a5ff 

Diff: https://reviews.apache.org/r/51312/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed

2016-08-22 Thread Yibing Shi (JIRA) 
Yibing Shi created HIVE-14609:
-

 Summary: HS2 cannot drop a function whose associated jar file has 
been removed
 Key: HIVE-14609
 URL: https://issues.apache.org/jira/browse/HIVE-14609
 Project: Hive
  Issue Type: Bug
Reporter: Yibing Shi


Create a permanent function with below command:
{code:sql}
create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar 
'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
{code}

After that, delete the HDFS file 
{{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and 
*restart HS2 to remove the loaded class*.

Now the function cannot be dropped:
{noformat}
0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
INFO  : Compiling 
command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded): show 
functions yshi.dummy
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, 
type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded); Time 
taken: 1.259 seconds
INFO  : Executing 
command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded): show 
functions yshi.dummy
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
INFO  : Completed executing 
command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded); Time 
taken: 0.024 seconds
INFO  : OK
+-+--+
|  tab_name   |
+-+--+
| yshi.dummy  |
+-+--+
1 row selected (3.877 seconds)
0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
INFO  : Compiling 
command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc): drop 
function yshi.dummy
INFO  : converting to local 
hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
ERROR : Failed to read external resource 
hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
java.lang.RuntimeException: Failed to read external resource 
hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
at 
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
at 
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
at 
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
at 
org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
at 
org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
at 
org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
at 
org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
at 
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
at 
org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:718)
at 
org.apache.thrift.server.TThreadPoolServer$Worke

Re: Question about Hive on Spark

2016-08-22 Thread Xuefu Zhang 
That happens per session or if certain configs are changed in the session.

On Mon, Aug 22, 2016 at 5:53 PM, Tao Li  wrote:

> Hi,
>
> Looks like the Spark client (SparkClientImpl class) submits Sparks jobs to
> the YARN cluster by forking a process and kicking off spark-submit script.
> Are we provisioning new containers every time we submit a job? There could
> be a perf hit by doing that.
>
> Thanks.
>

Re: Review Request 50888: Reduce number of partition check calls in add_partitions

2016-08-22 Thread Rajesh Balamohan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50888/
---

(Updated Aug. 23, 2016, 5:04 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14462
https://issues.apache.org/jira/browse/HIVE-14462


Repository: hive-git


Description
---

Metastore already does all the validations. Lots of MS calls are made just 
before add_partitions to double check if the partitions exists.  This impacts 
perf when large number of partitions are present.


Diffs (updated)
-

  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
3f85ca6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a59b781 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/CheckResult.java ec9deeb 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreChecker.java 
a164b12 
  ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveMetaStoreChecker.java 
5b8ec60 

Diff: https://reviews.apache.org/r/50888/diff/


Testing
---


Thanks,

Rajesh Balamohan