[jira] [Commented] (HIVE-7232) VectorReduceSink is emitting incorrect JOIN keys

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044397#comment-14044397
 ] 

Hive QA commented on HIVE-7232:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652546/HIVE-7232.2.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5654 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/594/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/594/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-594/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652546

> VectorReduceSink is emitting incorrect JOIN keys
> 
>
> Key: HIVE-7232
> URL: https://issues.apache.org/jira/browse/HIVE-7232
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-7232-extra-logging.patch, HIVE-7232.1.patch.txt, 
> HIVE-7232.2.patch.txt, q5.explain.txt, q5.sql
>
>
> After HIVE-7121, tpc-h query5 has resulted in incorrect results.
> Thanks to [~navis], it has been tracked down to the auto-parallel settings 
> which were initialized for ReduceSinkOperator, but not for 
> VectorReduceSinkOperator. The vector version inherits, but doesn't call 
> super.initializeOp() or set up the variable correctly from ReduceSinkDesc.
> The query is tpc-h query5, with extra NULL checks just to be sure.
> {code}
> ELECT n_name,
>sum(l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer,
>  orders,
>  lineitem,
>  supplier,
>  nation,
>  region
> WHERE c_custkey = o_custkey
>   AND l_orderkey = o_orderkey
>   AND l_suppkey = s_suppkey
>   AND c_nationkey = s_nationkey
>   AND s_nationkey = n_nationkey
>   AND n_regionkey = r_regionkey
>   AND r_name = 'ASIA'
>   AND o_orderdate >= '1994-01-01'
>   AND o_orderdate < '1995-01-01'
>   and l_orderkey is not null
>   and c_custkey is not null
>   and l_suppkey is not null
>   and c_nationkey is not null
>   and s_nationkey is not null
>   and n_regionkey is not null
> GROUP BY n_name
> ORDER BY revenue DESC;
> {code}
> The reducer which has the issue has the following plan
> {code}
> Reducer 3
> Reduce Operator Tree:
>   Join Operator
> condition map:
>  Inner Join 0 to 1
> condition expressions:
>   0 {KEY.reducesinkkey0} {VALUE._col2}
>   1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
> outputColumnNames: _col0, _col3, _col10, _col11, _col14
> Statistics: Num rows: 18344 Data size: 95229140992 Basic 
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
>   key expressions: _col10 (type: int)
>   sort order: +
>   Map-reduce partition columns: _col10 (type: int)
>   Statistics: Num rows: 18344 Data size: 95229140992 
> Basic stats: COMPLETE Column stats: NONE
>   value expressions: _col0 (type: int), _col3 (type: int), 
> _col11 (type: int), _col14 (type: string)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7127) Handover more details on exception in hiveserver2

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044386#comment-14044386
 ] 

Szehon Ho commented on HIVE-7127:
-

Thanks, +1 pending test

> Handover more details on exception in hiveserver2
> -
>
> Key: HIVE-7127
> URL: https://issues.apache.org/jira/browse/HIVE-7127
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7127.1.patch.txt, HIVE-7127.2.patch.txt, 
> HIVE-7127.4.patch.txt, HIVE-7127.5.patch.txt
>
>
> NO PRECOMMIT TESTS
> Currently, JDBC hands over exception message and error codes. But it's not 
> helpful for debugging.
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 1:0 cannot recognize input near 
> 'createa' 'asd' ''
>   at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
>   at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
>   at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
>   at org.apache.hive.beeline.Commands.execute(Commands.java:736)
>   at org.apache.hive.beeline.Commands.sql(Commands.java:657)
>   at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:889)
>   at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:744)
>   at 
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:459)
>   at org.apache.hive.beeline.BeeLine.main(BeeLine.java:442)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> {noformat}
> With this patch, JDBC client can get more details on hiveserver2. 
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while 
> compiling statement: FAILED: ParseException line 1:0 cannot recognize input 
> near 'createa' 'asd' ''
>   at org.apache.hive.service.cli.operation.SQLOperation.prepare(Unknown 
> Source)
>   at org.apache.hive.service.cli.operation.SQLOperation.run(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(Unknown
>  Source)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(Unknown
>  Source)
>   at org.apache.hive.service.cli.CLIService.executeStatementAsync(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
>  Source)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
>  Source)
>   at org.apache.thrift.ProcessFunction.process(Unknown Source)
>   at org.apache.thrift.TBaseProcessor.process(Unknown Source)
>   at org.apache.hive.service.auth.TSetIpAddressProcessor.process(Unknown 
> Source)
>   at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown 
> Source)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>   at java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7211) Throws exception if the name of conf var starts with "hive." does not exists in HiveConf

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7211:


Labels:   (was: TODOC14)

> Throws exception if the name of conf var starts with "hive." does not exists 
> in HiveConf
> 
>
> Key: HIVE-7211
> URL: https://issues.apache.org/jira/browse/HIVE-7211
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Fix For: 0.14.0
>
> Attachments: HIVE-7211.1.patch.txt, HIVE-7211.2.patch.txt, 
> HIVE-7211.3.patch.txt, HIVE-7211.4.patch.txt
>
>
> Some typos in configurations are very hard to find.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044384#comment-14044384
 ] 

Szehon Ho commented on HIVE-7220:
-

Forgot to rebase.  Thank you [~hagleitn] for that.

> Empty dir in external table causes issue (root_dir_external_table.q failure)
> 
>
> Key: HIVE-7220
> URL: https://issues.apache.org/jira/browse/HIVE-7220
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-7220.2.patch, HIVE-7220.3.patch, HIVE-7220.4.patch, 
> HIVE-7220.patch
>
>
> While looking at root_dir_external_table.q failure, which is doing a query on 
> an external table located at root ('/'), I noticed that latest Hadoop2 
> CombineFileInputFormat returns split representing empty directories (like 
> '/Users'), which leads to failure in Hive's CombineFileRecordReader as it 
> tries to open the directory for processing.
> Tried with an external table in a normal HDFS directory, and it also returns 
> the same error.  Looks like a real bug.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044381#comment-14044381
 ] 

Szehon Ho commented on HIVE-6394:
-

Ah got it, thanks.  Looks good, just one (unrelated) note, as HIVE-6375 is 
committed in 0.13, should we qualify the CTAS limitation?

> Implement Timestmap in ParquetSerde
> ---
>
> Key: HIVE-6394
> URL: https://issues.apache.org/jira/browse/HIVE-6394
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Reporter: Jarek Jarcec Cecho
>Assignee: Szehon Ho
>  Labels: Parquet, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, 
> HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, 
> HIVE-6394.patch
>
>
> This JIRA is to implement timestamp support in Parquet SerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-2379) Hive/HBase integration could be improved

[Gautam Gopalakrishnan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gautam Gopalakrishnan updated HIVE-2379:


Description: 
  For now any Hive/HBase queries would require the following jars to be 
explicitly added via hive's add jar command:

add jar /usr/lib/hive/lib/hbase-0.90.1-cdh3u0.jar;
add jar /usr/lib/hive/lib/hive-hbase-handler-0.7.0-cdh3u0.jar;
add jar /usr/lib/hive/lib/zookeeper-3.3.1.jar;
add jar /usr/lib/hive/lib/guava-r06.jar;

the longer term solution, perhaps, should be to have the code at submit time 
call hbase's 
TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it 
in distributedcache.

  was:
For now any Hive/HBase queries would require the following jars to be 
explicitly added via hive's add jar command:

add jar /usr/lib/hive/lib/hbase-0.90.1-cdh3u0.jar;
add jar /usr/lib/hive/lib/hive-hbase-handler-0.7.0-cdh3u0.jar;
add jar /usr/lib/hive/lib/zookeeper-3.3.1.jar;
add jar /usr/lib/hive/lib/guava-r06.jar;

the longer term solution, perhaps, should be to have the code at submit time 
call hbase's 
TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship it 
in distributedcache.


> Hive/HBase integration could be improved
> 
>
> Key: HIVE-2379
> URL: https://issues.apache.org/jira/browse/HIVE-2379
> Project: Hive
>  Issue Type: Bug
>  Components: CLI, Clients, HBase Handler
>Affects Versions: 0.7.1, 0.8.0, 0.9.0
>Reporter: Roman Shaposhnik
>Assignee: Navis
>Priority: Critical
> Fix For: 0.12.0
>
> Attachments: HIVE-2379-0.11.patch.txt, HIVE-2379.D7347.1.patch, 
> HIVE-2379.D7347.2.patch, HIVE-2379.D7347.3.patch
>
>
>   For now any Hive/HBase queries would require the following jars to be 
> explicitly added via hive's add jar command:
> add jar /usr/lib/hive/lib/hbase-0.90.1-cdh3u0.jar;
> add jar /usr/lib/hive/lib/hive-hbase-handler-0.7.0-cdh3u0.jar;
> add jar /usr/lib/hive/lib/zookeeper-3.3.1.jar;
> add jar /usr/lib/hive/lib/guava-r06.jar;
> the longer term solution, perhaps, should be to have the code at submit time 
> call hbase's 
> TableMapREduceUtil.addDependencyJar(job, HBaseStorageHandler.class) to ship 
> it in distributedcache.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044376#comment-14044376
 ] 

Szehon Ho commented on HIVE-6637:
-

Udf_infile test is failing, which looks related..  Can you make sure it passes 
first before commit?

> UDF in_file() doesn't take CHAR or VARCHAR as input
> ---
>
> Key: HIVE-6637
> URL: https://issues.apache.org/jira/browse/HIVE-6637
> Project: Hive
>  Issue Type: Bug
>  Components: Types, UDF
>Affects Versions: 0.14.0
>Reporter: Xuefu Zhang
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
>
>
> {code}
> hive> desc alter_varchar_1;
> key   string  None
> value varchar(3)  None
> key2  int None
> value2varchar(10) None
> hive> select in_file(value, value2) from alter_varchar_1;
> FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch 
> 'value': The 1st argument of function IN_FILE must be a string but 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
>  was given.
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HIVE-7211) Throws exception if the name of conf var starts with "hive." does not exists in HiveConf

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043127#comment-14043127
 ] 

Navis edited comment on HIVE-7211 at 6/26/14 6:04 AM:
--

This adds several configuration parameters to HiveConf.java: 

* hive.test.dummystats.aggregator (internal)
* hive.test.dummystats.publisher (internal)
* hive.io.rcfile.record.interval
* hive.io.rcfile.column.number.conf
* hive.io.rcfile.tolerate.corruptions
* hive.io.rcfile.record.buffer.size
* hive.hbase.generatehfiles
* hive.index.compact.file (internal)
* hive.index.blockfilter.file (internal)

Except for the internal parameters, they need definitions in 
hive-default.xml.template.  Then they should be documented in the wiki 
(https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties).


was (Author: le...@hortonworks.com):
This adds several configuration parameters to HiveConf.java: 

* hive.test.dummystats.aggregator
* hive.test.dummystats.publisher
* hive.io.rcfile.record.interval
* hive.io.rcfile.column.number.conf
* hive.io.rcfile.tolerate.corruptions
* hive.io.rcfile.record.buffer.size
* hive.hbase.generatehfiles
* hive.index.compact.file (internal)
* hive.index.blockfilter.file (internal)

Except for the internal parameters, they need definitions in 
hive-default.xml.template.  Then they should be documented in the wiki 
(https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties).

> Throws exception if the name of conf var starts with "hive." does not exists 
> in HiveConf
> 
>
> Key: HIVE-7211
> URL: https://issues.apache.org/jira/browse/HIVE-7211
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7211.1.patch.txt, HIVE-7211.2.patch.txt, 
> HIVE-7211.3.patch.txt, HIVE-7211.4.patch.txt
>
>
> Some typos in configurations are very hard to find.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6468) HS2 out of memory error when curl sends a get request

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-6468:


Attachment: HIVE-6468.2.patch.txt

Good to know that someone is interested in this. Rebased to trunk.

> HS2 out of memory error when curl sends a get request
> -
>
> Key: HIVE-6468
> URL: https://issues.apache.org/jira/browse/HIVE-6468
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
> Environment: Centos 6.3, hive 12, hadoop-2.2
>Reporter: Abin Shahab
>Assignee: Navis
> Attachments: HIVE-6468.1.patch.txt, HIVE-6468.2.patch.txt
>
>
> We see an out of memory error when we run simple beeline calls.
> (The hive.server2.transport.mode is binary)
> curl localhost:1
> Exception in thread "pool-2-thread-8" java.lang.OutOfMemoryError: Java heap 
> space
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>   at 
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7298) desc database extended does not show properties of the database

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044372#comment-14044372
 ] 

Ashutosh Chauhan commented on HIVE-7298:


+1

> desc database extended does not show properties of the database
> ---
>
> Key: HIVE-7298
> URL: https://issues.apache.org/jira/browse/HIVE-7298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7298.1.patch.txt
>
>
> HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7127) Handover more details on exception in hiveserver2

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044370#comment-14044370
 ] 

Navis commented on HIVE-7127:
-

Updated review board entry and kicked test manually.

> Handover more details on exception in hiveserver2
> -
>
> Key: HIVE-7127
> URL: https://issues.apache.org/jira/browse/HIVE-7127
> Project: Hive
>  Issue Type: Improvement
>  Components: JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7127.1.patch.txt, HIVE-7127.2.patch.txt, 
> HIVE-7127.4.patch.txt, HIVE-7127.5.patch.txt
>
>
> NO PRECOMMIT TESTS
> Currently, JDBC hands over exception message and error codes. But it's not 
> helpful for debugging.
> {noformat}
> org.apache.hive.service.cli.HiveSQLException: Error while compiling 
> statement: FAILED: ParseException line 1:0 cannot recognize input near 
> 'createa' 'asd' ''
>   at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
>   at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
>   at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
>   at org.apache.hive.beeline.Commands.execute(Commands.java:736)
>   at org.apache.hive.beeline.Commands.sql(Commands.java:657)
>   at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:889)
>   at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:744)
>   at 
> org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:459)
>   at org.apache.hive.beeline.BeeLine.main(BeeLine.java:442)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> {noformat}
> With this patch, JDBC client can get more details on hiveserver2. 
> {noformat}
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while 
> compiling statement: FAILED: ParseException line 1:0 cannot recognize input 
> near 'createa' 'asd' ''
>   at org.apache.hive.service.cli.operation.SQLOperation.prepare(Unknown 
> Source)
>   at org.apache.hive.service.cli.operation.SQLOperation.run(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(Unknown
>  Source)
>   at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(Unknown
>  Source)
>   at org.apache.hive.service.cli.CLIService.executeStatementAsync(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(Unknown 
> Source)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
>  Source)
>   at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
>  Source)
>   at org.apache.thrift.ProcessFunction.process(Unknown Source)
>   at org.apache.thrift.TBaseProcessor.process(Unknown Source)
>   at org.apache.hive.service.auth.TSetIpAddressProcessor.process(Unknown 
> Source)
>   at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown 
> Source)
>   at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>   at java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 21922: Handover more details on exception in hiveserver2

[Navis Ryu]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21922/
---

(Updated June 26, 2014, 5:49 a.m.)


Review request for hive.


Changes
---

Just rebase


Bugs: HIVE-7127
https://issues.apache.org/jira/browse/HIVE-7127


Repository: hive-git


Description
---

NO_PRECOMMIT_TEST

Currently, JDBC hands over exception message and error codes. But it's not 
helpful for debugging.
{noformat}
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: ParseException line 1:0 cannot recognize input near 'createa' 'asd' 
''
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at org.apache.hive.beeline.Commands.execute(Commands.java:736)
at org.apache.hive.beeline.Commands.sql(Commands.java:657)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:889)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:744)
at 
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:459)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:442)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{noformat}

With this patch, JDBC client can get more details on hiveserver2. 

{noformat}
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling 
statement: FAILED: ParseException line 1:0 cannot recognize input near 
'createa' 'asd' ''
at org.apache.hive.service.cli.operation.SQLOperation.prepare(Unknown 
Source)
at org.apache.hive.service.cli.operation.SQLOperation.run(Unknown 
Source)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(Unknown
 Source)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(Unknown
 Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(Unknown 
Source)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(Unknown 
Source)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
 Source)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(Unknown
 Source)
at org.apache.thrift.ProcessFunction.process(Unknown Source)
at org.apache.thrift.TBaseProcessor.process(Unknown Source)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(Unknown 
Source)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{noformat}


Diffs (updated)
-

  jdbc/src/java/org/apache/hive/jdbc/Utils.java f0834bd 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 401e639 
  
ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorResponse.java 
2eb4438 
  service/src/java/org/apache/hive/service/cli/HiveSQLException.java 8c9496e 
  
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
 f708650 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d6651ba 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
5e7ee93 
  service/src/test/org/apache/hive/service/cli/TestHiveSQLException.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/21922/diff/


Testing
---


Thanks,

Navis Ryu

[jira] [Updated] (HIVE-7298) desc database extended does not show properties of the database

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7298:


Attachment: HIVE-7298.1.patch.txt

> desc database extended does not show properties of the database
> ---
>
> Key: HIVE-7298
> URL: https://issues.apache.org/jira/browse/HIVE-7298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7298.1.patch.txt
>
>
> HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7298) desc database extended does not show properties of the database

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7298:


Status: Patch Available  (was: Open)

> desc database extended does not show properties of the database
> ---
>
> Key: HIVE-7298
> URL: https://issues.apache.org/jira/browse/HIVE-7298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: HIVE-7298.1.patch.txt
>
>
> HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7298) desc database extended does not show properties of the database

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7298:


Attachment: HIVE-7298.1.patch.txt

> desc database extended does not show properties of the database
> ---
>
> Key: HIVE-7298
> URL: https://issues.apache.org/jira/browse/HIVE-7298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
>
> HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7298) desc database extended does not show properties of the database

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7298:


Attachment: (was: HIVE-7298.1.patch.txt)

> desc database extended does not show properties of the database
> ---
>
> Key: HIVE-7298
> URL: https://issues.apache.org/jira/browse/HIVE-7298
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
>
> HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7298) desc database extended does not show properties of the database

[Navis (JIRA)]

Navis created HIVE-7298:
---

 Summary: desc database extended does not show properties of the 
database
 Key: HIVE-7298
 URL: https://issues.apache.org/jira/browse/HIVE-7298
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor


HIVE-6386 added owner information to desc, but not updated schema of it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7295) FileStatus.getOwner on Windows returns name of group the user belongs to, instead of user name expected, fails many authorization related unit tests

[Chris Nauroth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044354#comment-14044354
 ] 

Chris Nauroth commented on HIVE-7295:
-

Hi, [~xiaobingo].  Yes, this is something particular to Windows.  If you're 
logged in as a user in the administrators group, then files you create on the 
local file system will be owned by the administrators group, not your 
individual user.  Here is an example running winutils ls on a file I created 
while logged in as a member of the administrators group:

{code}
C:\test>C:\hdc\hadoop-common-project\hadoop-common\target\bin\winutils.exe ls 
C:\test\hello
-rwx-- 1 BUILTIN\Administrators CNAUROTH-VM-1\None 8 Jan 14 2014 
C:\test\hello
{code}

Here is the same thing run on a different VM, where my user is not a member of 
the administrators group:

{code}
C:\test>C:\hdc\hadoop-common-project\hadoop-common\target\bin\winutils.exe ls 
C:\test\hello
-rwx-- 1 WIN-NCDLEQLC13J\cnauroth WIN-NCDLEQLC13J\None 8 Jun 26 2014 
C:\test\hello
{code}

There isn't really any way to change code to work around this.  The only 
workaround is to run unit tests as a user who is not a member of the 
administrators group.

Note that when you set up a non-admin user for yourself, you'll need to make 
sure that user has Create Symbolic Links permission.  The code in Hadoop core 
relies on creating symlinks for a couple of important things, notably 
{{MiniYARNCluster}}.  Admins get this permission automatically, but regular 
users don't.  This page describes how to grant the permission to a regular user:

http://superuser.com/questions/104845/permission-to-make-symbolic-links-in-windows-7

Hope this helps.


> FileStatus.getOwner on Windows returns name of group the user belongs to, 
> instead of user name expected, fails many authorization related unit tests
> 
>
> Key: HIVE-7295
> URL: https://issues.apache.org/jira/browse/HIVE-7295
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HCatalog, Security, Windows
>Affects Versions: 0.13.0
> Environment: Windows Server 2008 R2
>Reporter: Xiaobing Zhou
>Priority: Critical
>
> Unit test in TestHdfsAuthorizationProvider, e.g. 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps. 
> fails to run.
> Running org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.799 sec 
> <<< FAILURE! - in org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> testTableOps(org.apache.hcatalog.security.TestHdfsAuthorizationProvider)  
> Time elapsed: 15.546 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: FAILED: AuthorizationException 
> org.apache.hadoop.security.AccessControlException: action WRITE not permitted 
> on path pfile:/Users/xz
> hou/hworks/workspace/hwx-hive-ws/hive/hcatalog/core/target/warehouse for user 
> xzhou expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.exec(TestHdfsAuthorizationProvider.java:172)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps(TestHdfsAuthorizationProvider.java:307)
> 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7297) org.apache.hadoop.hive.ql.WindowsPathUtil is not easy to be reused across different hive projects

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HIVE-7297:
---

 Summary: org.apache.hadoop.hive.ql.WindowsPathUtil is not easy to 
be reused across different hive projects
 Key: HIVE-7297
 URL: https://issues.apache.org/jira/browse/HIVE-7297
 Project: Hive
  Issue Type: Improvement
  Components: Testing Infrastructure, Tests
Affects Versions: 0.13.0
Reporter: Xiaobing Zhou
Assignee: Xiaobing Zhou
Priority: Minor


WindowsPathUtil converts Windows path notation(c:\Uses\xzhou) to HDFS 
compatible Unix notation(/Users/xzhou). It was used in several places, will be 
reused by other fixes like patches for HIVE-7276 and HIVE-7295. There's need to 
change those influenced projects pom.xml to include dependency. Or could pushed 
WindowsPathUtil down to hive-exec library code.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7220:
-

Attachment: HIVE-7220.4.patch

Oops. Missed to include the new files. .4 has those.

> Empty dir in external table causes issue (root_dir_external_table.q failure)
> 
>
> Key: HIVE-7220
> URL: https://issues.apache.org/jira/browse/HIVE-7220
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-7220.2.patch, HIVE-7220.3.patch, HIVE-7220.4.patch, 
> HIVE-7220.patch
>
>
> While looking at root_dir_external_table.q failure, which is doing a query on 
> an external table located at root ('/'), I noticed that latest Hadoop2 
> CombineFileInputFormat returns split representing empty directories (like 
> '/Users'), which leads to failure in Hive's CombineFileRecordReader as it 
> tries to open the directory for processing.
> Tried with an external table in a normal HDFS directory, and it also returns 
> the same error.  Looks like a real bug.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7220:
-

Attachment: HIVE-7220.3.patch

Rebased. Only change was to move the new q file to the now separate file used 
to configure mini mr files.

> Empty dir in external table causes issue (root_dir_external_table.q failure)
> 
>
> Key: HIVE-7220
> URL: https://issues.apache.org/jira/browse/HIVE-7220
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-7220.2.patch, HIVE-7220.3.patch, HIVE-7220.patch
>
>
> While looking at root_dir_external_table.q failure, which is doing a query on 
> an external table located at root ('/'), I noticed that latest Hadoop2 
> CombineFileInputFormat returns split representing empty directories (like 
> '/Users'), which leads to failure in Hive's CombineFileRecordReader as it 
> tries to open the directory for processing.
> Tried with an external table in a normal HDFS directory, and it also returns 
> the same error.  Looks like a real bug.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7296) big data approximate processing at a very low cost based on hive sql

[wangmeng (JIRA)]

wangmeng created HIVE-7296:
--

 Summary: big data approximate processing  at a very  low cost  
based on hive sql 
 Key: HIVE-7296
 URL: https://issues.apache.org/jira/browse/HIVE-7296
 Project: Hive
  Issue Type: New Feature
Reporter: wangmeng


For big data analysis, we often need to do the following query and statistics：

1.Cardinality Estimation,   count the number of different elements in the 
collection, such as Unique Visitor ,UV)

Now we can use hive-query:
Select distinct(id)  from TestTable ;

2.Frequency Estimation: estimate number of an element is repeated, such as the 
site visits of  a user 。

Hive query: select  count(1)  from TestTable where name=”wangmeng”

3.Heavy Hitters, top-k elements: such as top-100 shops 

Hive query: select count(1), name  from TestTable  group by name ;  need UDF……

4.Range Query: for example, to find out the number of  users between 20 to 30

Hive query : select  count(1) from TestTable where age>20 and age <30

5.Membership Query : for example, whether  the user name is already registered?

According to the implementation mechanism of hive , it  will cost too large 
memory space and a long query time.

However ,in many cases, we do not need very accurate results and a small error 
can be tolerated. In such case  , we can use  approximate processing  to 
greatly improve the time and space efficiency.

Now , based  on some theoretical analysis materials ,I want to  do some for 
these new features so much .

I am familiar with hive and  hadoop , and  I have implemented an efficient  
storage format based on hive.( 
https://github.com/sjtufighter/Data---Storage--).

So, is there anything I can do ?  Many Thanks.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7024) Escape control characters for explain result

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7024:


Attachment: HIVE-7024.3.patch.txt

> Escape control characters for explain result
> 
>
> Key: HIVE-7024
> URL: https://issues.apache.org/jira/browse/HIVE-7024
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7024.1.patch.txt, HIVE-7024.2.patch.txt, 
> HIVE-7024.3.patch.txt
>
>
> Comments for columns are now delimited by 0x00, which is binary and make git 
> refuse to make proper diff file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)

[Navis (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044344#comment-14044344
 ] 

Navis commented on HIVE-7220:
-

[~szehon] Seemed need to be rebased on trunk. Could you do that once more?

> Empty dir in external table causes issue (root_dir_external_table.q failure)
> 
>
> Key: HIVE-7220
> URL: https://issues.apache.org/jira/browse/HIVE-7220
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-7220.2.patch, HIVE-7220.patch
>
>
> While looking at root_dir_external_table.q failure, which is doing a query on 
> an external table located at root ('/'), I noticed that latest Hadoop2 
> CombineFileInputFormat returns split representing empty directories (like 
> '/Users'), which leads to failure in Hive's CombineFileRecordReader as it 
> tries to open the directory for processing.
> Tried with an external table in a normal HDFS directory, and it also returns 
> the same error.  Looks like a real bug.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-1662) Add file pruning into Hive.

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-1662:


Attachment: HIVE-1662.16.patch.txt

Rebased to trunk

> Add file pruning into Hive.
> ---
>
> Key: HIVE-1662
> URL: https://issues.apache.org/jira/browse/HIVE-1662
> Project: Hive
>  Issue Type: New Feature
>Reporter: He Yongqiang
>Assignee: Navis
> Attachments: HIVE-1662.10.patch.txt, HIVE-1662.11.patch.txt, 
> HIVE-1662.12.patch.txt, HIVE-1662.13.patch.txt, HIVE-1662.14.patch.txt, 
> HIVE-1662.15.patch.txt, HIVE-1662.16.patch.txt, HIVE-1662.8.patch.txt, 
> HIVE-1662.9.patch.txt, HIVE-1662.D8391.1.patch, HIVE-1662.D8391.2.patch, 
> HIVE-1662.D8391.3.patch, HIVE-1662.D8391.4.patch, HIVE-1662.D8391.5.patch, 
> HIVE-1662.D8391.6.patch, HIVE-1662.D8391.7.patch
>
>
> now hive support filename virtual column. 
> if a file name filter presents in a query, hive should be able to only add 
> files which passed the filter to input paths.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7289) revert HIVE-6469

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044340#comment-14044340
 ] 

Xuefu Zhang commented on HIVE-7289:
---

Are the test failures related?

> revert HIVE-6469
> 
>
> Key: HIVE-7289
> URL: https://issues.apache.org/jira/browse/HIVE-7289
> Project: Hive
>  Issue Type: Task
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Jayesh
>Assignee: Jayesh
> Attachments: HIVE-7289.patch
>
>
> this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 22996: HIVE-7090 Support session-level temporary tables in Hive

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22996/#review46713
---



ql/src/java/org/apache/hadoop/hive/ql/Context.java


What is the purpose of removing from here? I don't see any other changes?



ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java


if (not) 

else

This should be reversed.






ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java


We shouldn't lose the stack trace here



ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java


Losing stack trace here



ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java


Could we have a better message?



ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java


a bunch of 

if (not) 

else

statements here


- Brock Noland


On June 26, 2014, 2:05 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22996/
> ---
> 
> (Updated June 26, 2014, 2:05 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Navis Ryu, and Harish Butani.
> 
> 
> Bugs: HIVE-7090
> https://issues.apache.org/jira/browse/HIVE-7090
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Temp tables managed in memory by SessionState.
> SessionHiveMetaStoreClient overrides table-related methods in HiveMetaStore 
> to access the temp tables saved in the SessionState when appropriate.
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 9fb7550 
>   itests/qtest/testconfiguration.properties 6731561 
>   metastore/if/hive_metastore.thrift cc802c6 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 9e8d912 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java abc4290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 24f829f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4d35176 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3df2690 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 1270520 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 71471f4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 83d09c0 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 2537b75 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java cb5d64c 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2143d0c 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 43125f7 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager.java 98c3cc3 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> 91de8da 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  20d08b3 
>   ql/src/test/queries/clientnegative/temp_table_authorize_create_tbl.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_column_stats.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_create_like_partitions.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_index.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_partitions.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_rename.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/show_create_table_temp_table.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_external.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_gb1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_join1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_names.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_options1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_precedence.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_subquery1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_windowing_expressions.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_authorize_create_tbl.q.out 

[jira] [Commented] (HIVE-7024) Escape control characters for explain result

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044336#comment-14044336
 ] 

Hive QA commented on HIVE-7024:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652541/HIVE-7024.2.patch.txt

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5669 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_ppr2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/593/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/593/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-593/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652541

> Escape control characters for explain result
> 
>
> Key: HIVE-7024
> URL: https://issues.apache.org/jira/browse/HIVE-7024
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7024.1.patch.txt, HIVE-7024.2.patch.txt
>
>
> Comments for columns are now delimited by 0x00, which is binary and make git 
> refuse to make proper diff file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 22996: HIVE-7090 Support session-level temporary tables in Hive

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22996/#review46712
---



itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java


We should be checking for something more specific than a sql exception. 
error code, message, etc.


- Brock Noland


On June 26, 2014, 2:05 a.m., Jason Dere wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22996/
> ---
> 
> (Updated June 26, 2014, 2:05 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Navis Ryu, and Harish Butani.
> 
> 
> Bugs: HIVE-7090
> https://issues.apache.org/jira/browse/HIVE-7090
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Temp tables managed in memory by SessionState.
> SessionHiveMetaStoreClient overrides table-related methods in HiveMetaStore 
> to access the temp tables saved in the SessionState when appropriate.
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
> 9fb7550 
>   itests/qtest/testconfiguration.properties 6731561 
>   metastore/if/hive_metastore.thrift cc802c6 
>   metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 9e8d912 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java abc4290 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 24f829f 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4d35176 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3df2690 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 1270520 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
> 71471f4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 83d09c0 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 2537b75 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java cb5d64c 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2143d0c 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 43125f7 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager.java 98c3cc3 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
> 91de8da 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
>  20d08b3 
>   ql/src/test/queries/clientnegative/temp_table_authorize_create_tbl.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_column_stats.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_create_like_partitions.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_index.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_partitions.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/temp_table_rename.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/show_create_table_temp_table.q 
> PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_external.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_gb1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_join1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_names.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_options1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_precedence.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_subquery1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/temp_table_windowing_expressions.q 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_authorize_create_tbl.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_column_stats.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_create_like_partitions.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_index.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_partitions.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/temp_table_rename.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/show_create_table_temp_table.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/temp_table.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/temp_table_external.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/temp_table_gb1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/temp_table_join1.q.out PRE-CREATION 
>   ql/src/test/results/cli

[jira] [Commented] (HIVE-7295) FileStatus.getOwner on Windows returns name of group the user belongs to, instead of user name expected, fails many authorization related unit tests

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044330#comment-14044330
 ] 

Xiaobing Zhou commented on HIVE-7295:
-

This is a bug from HDFS, I guess. 
It was reported that permissions check would go through if a dedicated user 
detached from administrators group is used to login windows and run the unit 
tests. I will try that and get back.

> FileStatus.getOwner on Windows returns name of group the user belongs to, 
> instead of user name expected, fails many authorization related unit tests
> 
>
> Key: HIVE-7295
> URL: https://issues.apache.org/jira/browse/HIVE-7295
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HCatalog, Security, Windows
>Affects Versions: 0.13.0
> Environment: Windows Server 2008 R2
>Reporter: Xiaobing Zhou
>Priority: Critical
>
> Unit test in TestHdfsAuthorizationProvider, e.g. 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps. 
> fails to run.
> Running org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.799 sec 
> <<< FAILURE! - in org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> testTableOps(org.apache.hcatalog.security.TestHdfsAuthorizationProvider)  
> Time elapsed: 15.546 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: FAILED: AuthorizationException 
> org.apache.hadoop.security.AccessControlException: action WRITE not permitted 
> on path pfile:/Users/xz
> hou/hworks/workspace/hwx-hive-ws/hive/hcatalog/core/target/warehouse for user 
> xzhou expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.exec(TestHdfsAuthorizationProvider.java:172)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps(TestHdfsAuthorizationProvider.java:307)
> 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-6367) Implement Decimal in ParquetSerde

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-6367:
--

Labels: Parquet  (was: Parquet TODOC14)

> Implement Decimal in ParquetSerde
> -
>
> Key: HIVE-6367
> URL: https://issues.apache.org/jira/browse/HIVE-6367
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Xuefu Zhang
>  Labels: Parquet
> Fix For: 0.14.0
>
> Attachments: HIVE-6367.patch, dec.parq
>
>
> Some code in the Parquet Serde deals with decimal and other does not. For 
> example in ETypeConverter we convert Decimal to double (which is invalid) 
> whereas in DataWritableWriter and other locations we throw an exception if 
> decimal is used.
> This JIRA is to implement decimal support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6367) Implement Decimal in ParquetSerde

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044329#comment-14044329
 ] 

Xuefu Zhang commented on HIVE-6367:
---

Thanks, [~leftylev]. I removed the label.

> Implement Decimal in ParquetSerde
> -
>
> Key: HIVE-6367
> URL: https://issues.apache.org/jira/browse/HIVE-6367
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Xuefu Zhang
>  Labels: Parquet
> Fix For: 0.14.0
>
> Attachments: HIVE-6367.patch, dec.parq
>
>
> Some code in the Parquet Serde deals with decimal and other does not. For 
> example in ETypeConverter we convert Decimal to double (which is invalid) 
> whereas in DataWritableWriter and other locations we throw an exception if 
> decimal is used.
> This JIRA is to implement decimal support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7232) VectorReduceSink is emitting incorrect JOIN keys

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-7232:
--

Status: Patch Available  (was: Open)

> VectorReduceSink is emitting incorrect JOIN keys
> 
>
> Key: HIVE-7232
> URL: https://issues.apache.org/jira/browse/HIVE-7232
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-7232-extra-logging.patch, HIVE-7232.1.patch.txt, 
> HIVE-7232.2.patch.txt, q5.explain.txt, q5.sql
>
>
> After HIVE-7121, tpc-h query5 has resulted in incorrect results.
> Thanks to [~navis], it has been tracked down to the auto-parallel settings 
> which were initialized for ReduceSinkOperator, but not for 
> VectorReduceSinkOperator. The vector version inherits, but doesn't call 
> super.initializeOp() or set up the variable correctly from ReduceSinkDesc.
> The query is tpc-h query5, with extra NULL checks just to be sure.
> {code}
> ELECT n_name,
>sum(l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer,
>  orders,
>  lineitem,
>  supplier,
>  nation,
>  region
> WHERE c_custkey = o_custkey
>   AND l_orderkey = o_orderkey
>   AND l_suppkey = s_suppkey
>   AND c_nationkey = s_nationkey
>   AND s_nationkey = n_nationkey
>   AND n_regionkey = r_regionkey
>   AND r_name = 'ASIA'
>   AND o_orderdate >= '1994-01-01'
>   AND o_orderdate < '1995-01-01'
>   and l_orderkey is not null
>   and c_custkey is not null
>   and l_suppkey is not null
>   and c_nationkey is not null
>   and s_nationkey is not null
>   and n_regionkey is not null
> GROUP BY n_name
> ORDER BY revenue DESC;
> {code}
> The reducer which has the issue has the following plan
> {code}
> Reducer 3
> Reduce Operator Tree:
>   Join Operator
> condition map:
>  Inner Join 0 to 1
> condition expressions:
>   0 {KEY.reducesinkkey0} {VALUE._col2}
>   1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
> outputColumnNames: _col0, _col3, _col10, _col11, _col14
> Statistics: Num rows: 18344 Data size: 95229140992 Basic 
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
>   key expressions: _col10 (type: int)
>   sort order: +
>   Map-reduce partition columns: _col10 (type: int)
>   Statistics: Num rows: 18344 Data size: 95229140992 
> Basic stats: COMPLETE Column stats: NONE
>   value expressions: _col0 (type: int), _col3 (type: int), 
> _col11 (type: int), _col14 (type: string)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7232) VectorReduceSink is emitting incorrect JOIN keys

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-7232:
--

Attachment: HIVE-7232.2.patch.txt

> VectorReduceSink is emitting incorrect JOIN keys
> 
>
> Key: HIVE-7232
> URL: https://issues.apache.org/jira/browse/HIVE-7232
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-7232-extra-logging.patch, HIVE-7232.1.patch.txt, 
> HIVE-7232.2.patch.txt, q5.explain.txt, q5.sql
>
>
> After HIVE-7121, tpc-h query5 has resulted in incorrect results.
> Thanks to [~navis], it has been tracked down to the auto-parallel settings 
> which were initialized for ReduceSinkOperator, but not for 
> VectorReduceSinkOperator. The vector version inherits, but doesn't call 
> super.initializeOp() or set up the variable correctly from ReduceSinkDesc.
> The query is tpc-h query5, with extra NULL checks just to be sure.
> {code}
> ELECT n_name,
>sum(l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer,
>  orders,
>  lineitem,
>  supplier,
>  nation,
>  region
> WHERE c_custkey = o_custkey
>   AND l_orderkey = o_orderkey
>   AND l_suppkey = s_suppkey
>   AND c_nationkey = s_nationkey
>   AND s_nationkey = n_nationkey
>   AND n_regionkey = r_regionkey
>   AND r_name = 'ASIA'
>   AND o_orderdate >= '1994-01-01'
>   AND o_orderdate < '1995-01-01'
>   and l_orderkey is not null
>   and c_custkey is not null
>   and l_suppkey is not null
>   and c_nationkey is not null
>   and s_nationkey is not null
>   and n_regionkey is not null
> GROUP BY n_name
> ORDER BY revenue DESC;
> {code}
> The reducer which has the issue has the following plan
> {code}
> Reducer 3
> Reduce Operator Tree:
>   Join Operator
> condition map:
>  Inner Join 0 to 1
> condition expressions:
>   0 {KEY.reducesinkkey0} {VALUE._col2}
>   1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
> outputColumnNames: _col0, _col3, _col10, _col11, _col14
> Statistics: Num rows: 18344 Data size: 95229140992 Basic 
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
>   key expressions: _col10 (type: int)
>   sort order: +
>   Map-reduce partition columns: _col10 (type: int)
>   Statistics: Num rows: 18344 Data size: 95229140992 
> Basic stats: COMPLETE Column stats: NONE
>   value expressions: _col0 (type: int), _col3 (type: int), 
> _col11 (type: int), _col14 (type: string)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7295) FileStatus.getOwner on Windows returns name of group the user belongs to, instead of user name expected, fails many authorization related unit tests

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044328#comment-14044328
 ] 

Xiaobing Zhou commented on HIVE-7295:
-

After digging, AuthorizationException is thrown inside 
HdfsAuthorizationProvider.checkPermissions(final FileSystem fs...) because 
permissions to operate on DB path are not met. FileStatus.getOwner gives out 
name of group, not user name expected for permission check going through. e.g. 
Administrators, not xzhou. 

FileStatus.getGroup also returns group like machine-name\None, which does not 
match group the user running program belongs to, operation on the path not 
permitted.



> FileStatus.getOwner on Windows returns name of group the user belongs to, 
> instead of user name expected, fails many authorization related unit tests
> 
>
> Key: HIVE-7295
> URL: https://issues.apache.org/jira/browse/HIVE-7295
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, HCatalog, Security, Windows
>Affects Versions: 0.13.0
> Environment: Windows Server 2008 R2
>Reporter: Xiaobing Zhou
>Priority: Critical
>
> Unit test in TestHdfsAuthorizationProvider, e.g. 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps. 
> fails to run.
> Running org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.799 sec 
> <<< FAILURE! - in org.apache.hcatalog.security.TestHdfsAuthorizationProvider
> testTableOps(org.apache.hcatalog.security.TestHdfsAuthorizationProvider)  
> Time elapsed: 15.546 sec  <<< FAILURE!
> junit.framework.AssertionFailedError: FAILED: AuthorizationException 
> org.apache.hadoop.security.AccessControlException: action WRITE not permitted 
> on path pfile:/Users/xz
> hou/hworks/workspace/hwx-hive-ws/hive/hcatalog/core/target/warehouse for user 
> xzhou expected:<0> but was:<4>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.exec(TestHdfsAuthorizationProvider.java:172)
> at 
> org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps(TestHdfsAuthorizationProvider.java:307)
> 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6367) Implement Decimal in ParquetSerde

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044325#comment-14044325
 ] 

Lefty Leverenz commented on HIVE-6367:
--

Documented in the wiki, please review and remove "TODOC14" label if it's okay:

* [Language Manual -- Parquet -- Limitations | 
https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Limitations]

> Implement Decimal in ParquetSerde
> -
>
> Key: HIVE-6367
> URL: https://issues.apache.org/jira/browse/HIVE-6367
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Affects Versions: 0.13.0
>Reporter: Brock Noland
>Assignee: Xuefu Zhang
>  Labels: Parquet, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6367.patch, dec.parq
>
>
> Some code in the Parquet Serde deals with decimal and other does not. For 
> example in ETypeConverter we convert Decimal to double (which is invalid) 
> whereas in DataWritableWriter and other locations we throw an exception if 
> decimal is used.
> This JIRA is to implement decimal support.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044322#comment-14044322
 ] 

Lefty Leverenz commented on HIVE-6394:
--

How's this?  I added decimal too (HIVE-6367).

* [Language Manual -- Parquet -- Limitations | 
https://cwiki.apache.org/confluence/display/Hive/Parquet#Parquet-Limitations]

> Implement Timestmap in ParquetSerde
> ---
>
> Key: HIVE-6394
> URL: https://issues.apache.org/jira/browse/HIVE-6394
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Reporter: Jarek Jarcec Cecho
>Assignee: Szehon Ho
>  Labels: Parquet, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, 
> HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, 
> HIVE-6394.patch
>
>
> This JIRA is to implement timestamp support in Parquet SerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7295) FileStatus.getOwner on Windows returns name of group the user belongs to, instead of user name expected, fails many authorization related unit tests

[Xiaobing Zhou (JIRA)]

Xiaobing Zhou created HIVE-7295:
---

 Summary: FileStatus.getOwner on Windows returns name of group the 
user belongs to, instead of user name expected, fails many authorization 
related unit tests
 Key: HIVE-7295
 URL: https://issues.apache.org/jira/browse/HIVE-7295
 Project: Hive
  Issue Type: Bug
  Components: Authorization, HCatalog, Security, Windows
Affects Versions: 0.13.0
 Environment: Windows Server 2008 R2
Reporter: Xiaobing Zhou
Priority: Critical


Unit test in TestHdfsAuthorizationProvider, e.g. 
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps. fails 
to run.

Running org.apache.hcatalog.security.TestHdfsAuthorizationProvider
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.799 sec <<< 
FAILURE! - in org.apache.hcatalog.security.TestHdfsAuthorizationProvider
testTableOps(org.apache.hcatalog.security.TestHdfsAuthorizationProvider)  Time 
elapsed: 15.546 sec  <<< FAILURE!
junit.framework.AssertionFailedError: FAILED: AuthorizationException 
org.apache.hadoop.security.AccessControlException: action WRITE not permitted 
on path pfile:/Users/xz
hou/hworks/workspace/hwx-hive-ws/hive/hcatalog/core/target/warehouse for user 
xzhou expected:<0> but was:<4>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at 
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.exec(TestHdfsAuthorizationProvider.java:172)
at 
org.apache.hcatalog.security.TestHdfsAuthorizationProvider.testTableOps(TestHdfsAuthorizationProvider.java:307)




--
This message was sent by Atlassian JIRA
(v6.2#6252)

"desc database extended " doesn't print dbproperties?

[Sumit Kumar]

Hey guys,

I just discovered that this syntax doesn't print the dbproperties any more. 
I've two hive versions that i'm testing following query on:

  create database test2 with dbproperties ('key1' = 'value1', 'key2' = 
'value2');
  desc database extended test2;


The output on hive 11 is:

hive>   desc database extended test2;   
 
OK
test2 hdfs://:9000/warehouse/test2.db   {key2=value2, 
key1=value1}
Time taken: 0.021 seconds, Fetched: 1 row(s)


The output on hive 13 is:
hive> desc database extended test2; 
 
OK
test2 hdfs://:9000/warehouse/test2.db    hadoop
Time taken: 0.023 seconds, Fetched: 1 row(s)


If you look closely, you would notice that no key value information from 
dbproperties was printed in hive13 case and somehow magically "hadoop" (i guess 
it's my userid) appeared.

Any idea if this functionality changed since hive 11? Do we have a reference 
jira? I searched on the wikis and JIRAs but couldn't find a reference; 
surprised that the language manual wiki 
(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL) doesn't 
even talk about this functionality any more. Would appreciate input on this.


Thanks,
-Sumit

[jira] [Updated] (HIVE-7232) VectorReduceSink is emitting incorrect JOIN keys

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-7232:
--

Status: Open  (was: Patch Available)

Need to rebase patch to match recent qtest changes made HIVE-7258

> VectorReduceSink is emitting incorrect JOIN keys
> 
>
> Key: HIVE-7232
> URL: https://issues.apache.org/jira/browse/HIVE-7232
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-7232-extra-logging.patch, HIVE-7232.1.patch.txt, 
> q5.explain.txt, q5.sql
>
>
> After HIVE-7121, tpc-h query5 has resulted in incorrect results.
> Thanks to [~navis], it has been tracked down to the auto-parallel settings 
> which were initialized for ReduceSinkOperator, but not for 
> VectorReduceSinkOperator. The vector version inherits, but doesn't call 
> super.initializeOp() or set up the variable correctly from ReduceSinkDesc.
> The query is tpc-h query5, with extra NULL checks just to be sure.
> {code}
> ELECT n_name,
>sum(l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer,
>  orders,
>  lineitem,
>  supplier,
>  nation,
>  region
> WHERE c_custkey = o_custkey
>   AND l_orderkey = o_orderkey
>   AND l_suppkey = s_suppkey
>   AND c_nationkey = s_nationkey
>   AND s_nationkey = n_nationkey
>   AND n_regionkey = r_regionkey
>   AND r_name = 'ASIA'
>   AND o_orderdate >= '1994-01-01'
>   AND o_orderdate < '1995-01-01'
>   and l_orderkey is not null
>   and c_custkey is not null
>   and l_suppkey is not null
>   and c_nationkey is not null
>   and s_nationkey is not null
>   and n_regionkey is not null
> GROUP BY n_name
> ORDER BY revenue DESC;
> {code}
> The reducer which has the issue has the following plan
> {code}
> Reducer 3
> Reduce Operator Tree:
>   Join Operator
> condition map:
>  Inner Join 0 to 1
> condition expressions:
>   0 {KEY.reducesinkkey0} {VALUE._col2}
>   1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
> outputColumnNames: _col0, _col3, _col10, _col11, _col14
> Statistics: Num rows: 18344 Data size: 95229140992 Basic 
> stats: COMPLETE Column stats: NONE
> Reduce Output Operator
>   key expressions: _col10 (type: int)
>   sort order: +
>   Map-reduce partition columns: _col10 (type: int)
>   Statistics: Num rows: 18344 Data size: 95229140992 
> Basic stats: COMPLETE Column stats: NONE
>   value expressions: _col0 (type: int), _col3 (type: int), 
> _col11 (type: int), _col14 (type: string)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7282) HCatLoader fail to load Orc map with null key

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-7282:
-

Attachment: HIVE-7282-2.patch

Add test case.

> HCatLoader fail to load Orc map with null key
> -
>
> Key: HIVE-7282
> URL: https://issues.apache.org/jira/browse/HIVE-7282
> Project: Hive
>  Issue Type: Bug
>  Components: HCatalog
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 0.14.0
>
> Attachments: HIVE-7282-1.patch, HIVE-7282-2.patch
>
>
> Here is the stack:
> Get exception:
> AttemptID:attempt_1403634189382_0011_m_00_0 Info:Error: 
> org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error 
> converting read value to tuple
> at org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
> at org.apache.hive.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:58)
> at 
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
> at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToPigMap(PigHCatUtil.java:469)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.extractPigObject(PigHCatUtil.java:404)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:456)
> at 
> org.apache.hive.hcatalog.pig.PigHCatUtil.transformToTuple(PigHCatUtil.java:374)
> at org.apache.hive.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:64)
> ... 13 more



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044315#comment-14044315
 ] 

Lefty Leverenz commented on HIVE-6394:
--

Not quite, because 'timestamp' is still a limitation for releases prior to 0.14.

I'll make a change and you can review it.  (That'll be quicker than writing my 
suggestion here.)

> Implement Timestmap in ParquetSerde
> ---
>
> Key: HIVE-6394
> URL: https://issues.apache.org/jira/browse/HIVE-6394
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Reporter: Jarek Jarcec Cecho
>Assignee: Szehon Ho
>  Labels: Parquet, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, 
> HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, 
> HIVE-6394.patch
>
>
> This JIRA is to implement timestamp support in Parquet SerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7090) Support session-level temporary tables in Hive

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044309#comment-14044309
 ] 

Hive QA commented on HIVE-7090:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652537/HIVE-7090.4.patch

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 5685 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformat
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nullformatCTAS
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_alter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_db_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_delimited
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats19
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/592/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-592/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652537

> Support session-level temporary tables in Hive
> --
>
> Key: HIVE-7090
> URL: https://issues.apache.org/jira/browse/HIVE-7090
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Reporter: Gunther Hagleitner
>Assignee: Jason Dere
> Attachments: HIVE-7090.1.patch, HIVE-7090.2.patch, HIVE-7090.3.patch, 
> HIVE-7090.4.patch
>
>
> It's common to see sql scripts that create some temporary table as an 
> intermediate result, run some additional queries against it and then clean up 
> at the end.
> We should support temporary tables properly, meaning automatically manage the 
> life cycle and make sure the visibility is restricted to the creating 
> connection/session. Without these it's common to see left over tables in 
> meta-store or weird errors with clashing tmp table names.
> Proposed syntax:
> CREATE TEMPORARY TABLE 
> CTAS, CTL, INSERT INTO, should all be supported as usual.
> Knowing that a user wants a temp table can enable us to further optimize 
> access to it. E.g.: temp tables should be kept in memory where possible, 
> compactions and merging table files aren't required, ...



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7024) Escape control characters for explain result

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7024:


Attachment: HIVE-7024.2.patch.txt

> Escape control characters for explain result
> 
>
> Key: HIVE-7024
> URL: https://issues.apache.org/jira/browse/HIVE-7024
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7024.1.patch.txt, HIVE-7024.2.patch.txt
>
>
> Comments for columns are now delimited by 0x00, which is binary and make git 
> refuse to make proper diff file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7024) Escape control characters for explain result

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-7024:


Status: Patch Available  (was: Open)

> Escape control characters for explain result
> 
>
> Key: HIVE-7024
> URL: https://issues.apache.org/jira/browse/HIVE-7024
> Project: Hive
>  Issue Type: Bug
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-7024.1.patch.txt, HIVE-7024.2.patch.txt
>
>
> Comments for columns are now delimited by 0x00, which is binary and make git 
> refuse to make proper diff file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6637) UDF in_file() doesn't take CHAR or VARCHAR as input

[Ashish Kumar Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044274#comment-14044274
 ] 

Ashish Kumar Singh commented on HIVE-6637:
--

Thanks [~xuefuz] and [~wilbur.yang] for reviewing.

> UDF in_file() doesn't take CHAR or VARCHAR as input
> ---
>
> Key: HIVE-6637
> URL: https://issues.apache.org/jira/browse/HIVE-6637
> Project: Hive
>  Issue Type: Bug
>  Components: Types, UDF
>Affects Versions: 0.14.0
>Reporter: Xuefu Zhang
>Assignee: Ashish Kumar Singh
> Attachments: HIVE-6637.1.patch, HIVE-6637.2.patch
>
>
> {code}
> hive> desc alter_varchar_1;
> key   string  None
> value varchar(3)  None
> key2  int None
> value2varchar(10) None
> hive> select in_file(value, value2) from alter_varchar_1;
> FAILED: SemanticException [Error 10016]: Line 1:15 Argument type mismatch 
> 'value': The 1st argument of function IN_FILE must be a string but 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableHiveVarcharObjectInspector@10f1f34a
>  was given.
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 22996: HIVE-7090 Support session-level temporary tables in Hive

[Jason Dere]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22996/
---

Review request for hive, Gunther Hagleitner, Navis Ryu, and Harish Butani.


Bugs: HIVE-7090
https://issues.apache.org/jira/browse/HIVE-7090


Repository: hive-git


Description
---

Temp tables managed in memory by SessionState.
SessionHiveMetaStoreClient overrides table-related methods in HiveMetaStore to 
access the temp tables saved in the SessionState when appropriate.


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 
9fb7550 
  itests/qtest/testconfiguration.properties 6731561 
  metastore/if/hive_metastore.thrift cc802c6 
  metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java 9e8d912 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java abc4290 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 24f829f 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4d35176 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 3df2690 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
1270520 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g f934ac4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
71471f4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 83d09c0 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 2537b75 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java cb5d64c 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 2143d0c 
  ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java 43125f7 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager.java 98c3cc3 
  ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java 
91de8da 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/authorization/TestHiveAuthorizationTaskFactory.java
 20d08b3 
  ql/src/test/queries/clientnegative/temp_table_authorize_create_tbl.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_column_stats.q PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_create_like_partitions.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_index.q PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_partitions.q PRE-CREATION 
  ql/src/test/queries/clientnegative/temp_table_rename.q PRE-CREATION 
  ql/src/test/queries/clientpositive/show_create_table_temp_table.q 
PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_external.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_gb1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_join1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_names.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_options1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_precedence.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_subquery1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/temp_table_windowing_expressions.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_authorize_create_tbl.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_column_stats.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_create_like_partitions.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_index.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_partitions.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/temp_table_rename.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_create_table_temp_table.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_external.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_gb1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_join1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_names.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_options1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_precedence.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_subquery1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/temp_table_windowing_expressions.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/tez/temp_table.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/22996/diff/


Testing
---


Thanks,

Jason Dere

[jira] [Updated] (HIVE-7090) Support session-level temporary tables in Hive

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-7090:
-

Attachment: HIVE-7090.4.patch

Attaching patch v4.
Most of the precommit test failures were due to the scratch directory changes.  
As a result ql.Context must now be be created after SessionState.startSession() 
is called. The failing tests have been updated.
Also added more tests and some fixes based on the tests.

> Support session-level temporary tables in Hive
> --
>
> Key: HIVE-7090
> URL: https://issues.apache.org/jira/browse/HIVE-7090
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Reporter: Gunther Hagleitner
>Assignee: Jason Dere
> Attachments: HIVE-7090.1.patch, HIVE-7090.2.patch, HIVE-7090.3.patch, 
> HIVE-7090.4.patch
>
>
> It's common to see sql scripts that create some temporary table as an 
> intermediate result, run some additional queries against it and then clean up 
> at the end.
> We should support temporary tables properly, meaning automatically manage the 
> life cycle and make sure the visibility is restricted to the creating 
> connection/session. Without these it's common to see left over tables in 
> meta-store or weird errors with clashing tmp table names.
> Proposed syntax:
> CREATE TEMPORARY TABLE 
> CTAS, CTL, INSERT INTO, should all be supported as usual.
> Knowing that a user wants a temp table can enable us to further optimize 
> access to it. E.g.: temp tables should be kept in memory where possible, 
> compactions and merging table files aren't required, ...



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7090) Support session-level temporary tables in Hive

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-7090:
-

Status: Patch Available  (was: Open)

> Support session-level temporary tables in Hive
> --
>
> Key: HIVE-7090
> URL: https://issues.apache.org/jira/browse/HIVE-7090
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Reporter: Gunther Hagleitner
>Assignee: Jason Dere
> Attachments: HIVE-7090.1.patch, HIVE-7090.2.patch, HIVE-7090.3.patch, 
> HIVE-7090.4.patch
>
>
> It's common to see sql scripts that create some temporary table as an 
> intermediate result, run some additional queries against it and then clean up 
> at the end.
> We should support temporary tables properly, meaning automatically manage the 
> life cycle and make sure the visibility is restricted to the creating 
> connection/session. Without these it's common to see left over tables in 
> meta-store or weird errors with clashing tmp table names.
> Proposed syntax:
> CREATE TEMPORARY TABLE 
> CTAS, CTL, INSERT INTO, should all be supported as usual.
> Knowing that a user wants a temp table can enable us to further optimize 
> access to it. E.g.: temp tables should be kept in memory where possible, 
> compactions and merging table files aren't required, ...



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Hive-branch-0.12-hadoop2 - Build # 31 - Still Failing

[Apache Jenkins Server]

Changes for Build #26

Changes for Build #27

Changes for Build #28

Changes for Build #29

Changes for Build #30

Changes for Build #31



No tests ran.

The Apache Jenkins build system has built Hive-branch-0.12-hadoop2 (build #31)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Hive-branch-0.12-hadoop2/31/ to view the results.

[jira] [Updated] (HIVE-7292) Hive on Spark

[Xuefu Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuefu Zhang updated HIVE-7292:
--

Attachment: Hive-on-Spark.pdf

> Hive on Spark
> -
>
> Key: HIVE-7292
> URL: https://issues.apache.org/jira/browse/HIVE-7292
> Project: Hive
>  Issue Type: Improvement
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: Hive-on-Spark.pdf
>
>
> Spark as an open-source data analytics cluster computing framework has gained 
> significant momentum recently. Many Hive users already have Spark installed 
> as their computing backbone. To take advantages of Hive, they still need to 
> have either MapReduce or Tez on their cluster. This initiative will provide 
> user a new alternative so that those user can consolidate their backend. 
> Secondly, providing such an alternative further increases Hive's adoption as 
> it exposes Spark users  to a viable, feature-rich de facto standard SQL tools 
> on Hadoop.
> Finally, allowing Hive to run on Spark also has performance benefits. Hive 
> queries, especially those involving multiple reducer stages, will run faster, 
> thus improving user experience as Tez does.
> This is an umber JIRA which will cover many coming subtask. Design doc will 
> be attached here shortly, and will be on the wiki as well. Feedback from the 
> community is greatly appreciated!



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7220) Empty dir in external table causes issue (root_dir_external_table.q failure)

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044123#comment-14044123
 ] 

Szehon Ho commented on HIVE-7220:
-

ping, do people still want to proceed with this, to fix the last test failure?

> Empty dir in external table causes issue (root_dir_external_table.q failure)
> 
>
> Key: HIVE-7220
> URL: https://issues.apache.org/jira/browse/HIVE-7220
> Project: Hive
>  Issue Type: Bug
>Reporter: Szehon Ho
>Assignee: Szehon Ho
> Attachments: HIVE-7220.2.patch, HIVE-7220.patch
>
>
> While looking at root_dir_external_table.q failure, which is doing a query on 
> an external table located at root ('/'), I noticed that latest Hadoop2 
> CombineFileInputFormat returns split representing empty directories (like 
> '/Users'), which leads to failure in Hive's CombineFileRecordReader as it 
> tries to open the directory for processing.
> Tried with an external table in a normal HDFS directory, and it also returns 
> the same error.  Looks like a real bug.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7294) sql std auth - authorize show grant statements

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-7294:


Description: 
A non admin user should not be allowed to run show grant commands only for 
themselves or a role they belong to.


  was:
A non admin user should be allowed to run show grant commands only for 
themselves or a role they belong to.



> sql std auth - authorize show grant statements
> --
>
> Key: HIVE-7294
> URL: https://issues.apache.org/jira/browse/HIVE-7294
> Project: Hive
>  Issue Type: Bug
>  Components: Authorization, SQLStandardAuthorization
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>
> A non admin user should not be allowed to run show grant commands only for 
> themselves or a role they belong to.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7294) sql std auth - authorize show grant statements

[Thejas M Nair (JIRA)]

Thejas M Nair created HIVE-7294:
---

 Summary: sql std auth - authorize show grant statements
 Key: HIVE-7294
 URL: https://issues.apache.org/jira/browse/HIVE-7294
 Project: Hive
  Issue Type: Bug
  Components: Authorization, SQLStandardAuthorization
Reporter: Thejas M Nair
Assignee: Thejas M Nair


A non admin user should be allowed to run show grant commands only for 
themselves or a role they belong to.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7293) Hive-trunk does not build against JDK8 with generic class checks

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-7293:
--

Labels: Vectorization  (was: )

> Hive-trunk does not build against JDK8 with generic class checks
> 
>
> Key: HIVE-7293
> URL: https://issues.apache.org/jira/browse/HIVE-7293
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: java version "1.8.0"
> Java(TM) SE Runtime Environment (build 1.8.0-b132)
> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
>  Labels: Vectorization
>
> The current build and tests on my laptop are failing due to generic argument 
> mismatch errors.
> {code}
> hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPGreaterThan.java:[46,82]
>  incompatible types
> found   : 
> java.lang.Class
> required: java.lang.Class org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression>
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7293) Hive-trunk does not build against JDK8 with generic class checks

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-7293:
--

Component/s: Query Processor

> Hive-trunk does not build against JDK8 with generic class checks
> 
>
> Key: HIVE-7293
> URL: https://issues.apache.org/jira/browse/HIVE-7293
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Affects Versions: 0.14.0
> Environment: java version "1.8.0"
> Java(TM) SE Runtime Environment (build 1.8.0-b132)
> Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Minor
>  Labels: Vectorization
>
> The current build and tests on my laptop are failing due to generic argument 
> mismatch errors.
> {code}
> hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPGreaterThan.java:[46,82]
>  incompatible types
> found   : 
> java.lang.Class
> required: java.lang.Class org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression>
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7293) Hive-trunk does not build against JDK8 with generic class checks

[Gopal V (JIRA)]

Gopal V created HIVE-7293:
-

 Summary: Hive-trunk does not build against JDK8 with generic class 
checks
 Key: HIVE-7293
 URL: https://issues.apache.org/jira/browse/HIVE-7293
 Project: Hive
  Issue Type: Bug
Affects Versions: 0.14.0
 Environment: java version "1.8.0"
Java(TM) SE Runtime Environment (build 1.8.0-b132)
Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor


The current build and tests on my laptop are failing due to generic argument 
mismatch errors.

{code}
hive-trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPGreaterThan.java:[46,82]
 incompatible types
found   : 
java.lang.Class
required: java.lang.Class
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7289) revert HIVE-6469

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043984#comment-14043984
 ] 

Hive QA commented on HIVE-7289:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652480/HIVE-7289.patch

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5669 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/591/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/591/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-591/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652480

> revert HIVE-6469
> 
>
> Key: HIVE-7289
> URL: https://issues.apache.org/jira/browse/HIVE-7289
> Project: Hive
>  Issue Type: Task
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Jayesh
>Assignee: Jayesh
> Attachments: HIVE-7289.patch
>
>
> this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7292) Hive on Spark

[Xuefu Zhang (JIRA)]

Xuefu Zhang created HIVE-7292:
-

 Summary: Hive on Spark
 Key: HIVE-7292
 URL: https://issues.apache.org/jira/browse/HIVE-7292
 Project: Hive
  Issue Type: Improvement
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


Spark as an open-source data analytics cluster computing framework has gained 
significant momentum recently. Many Hive users already have Spark installed as 
their computing backbone. To take advantages of Hive, they still need to have 
either MapReduce or Tez on their cluster. This initiative will provide user a 
new alternative so that those user can consolidate their backend. 

Secondly, providing such an alternative further increases Hive's adoption as it 
exposes Spark users  to a viable, feature-rich de facto standard SQL tools on 
Hadoop.

Finally, allowing Hive to run on Spark also has performance benefits. Hive 
queries, especially those involving multiple reducer stages, will run faster, 
thus improving user experience as Tez does.

This is an umber JIRA which will cover many coming subtask. Design doc will be 
attached here shortly, and will be on the wiki as well. Feedback from the 
community is greatly appreciated!



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-5976) Decouple input formats from STORED as keywords

[David Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043966#comment-14043966
 ] 

David Chen commented on HIVE-5976:
--

Hi Brock,

Thank you for working on this! This will make a lot of things easier, including 
some of the work I am doing to help improve HCatalog's test coverage.

If you would like, I'd be glad to help out with finishing up this patch.

Thanks,
David

> Decouple input formats from STORED as keywords
> --
>
> Key: HIVE-5976
> URL: https://issues.apache.org/jira/browse/HIVE-5976
> Project: Hive
>  Issue Type: Task
>Reporter: Brock Noland
>Assignee: Brock Noland
> Attachments: HIVE-5976.patch, HIVE-5976.patch, HIVE-5976.patch, 
> HIVE-5976.patch
>
>
> As noted in HIVE-5783, we hard code the input formats mapped to keywords. 
> It'd be nice if there was a registration system so we didn't need to do that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7286) Parameterize HCatMapReduceTest for testing against all Hive storage formats

[David Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043955#comment-14043955
 ] 

David Chen commented on HIVE-7286:
--

Thanks for your feedback, Szehon.

I also think that this patch will be much cleaner once HIVE-5976 is done. Do we 
expect that it will be committed soon?

For SERDEUSINGMETASTOREFORSCHEMA, do you mean that we can use it to have the 
AvroSerDe use the metastore schema and not have to specify an Avro schema?

> Parameterize HCatMapReduceTest for testing against all Hive storage formats
> ---
>
> Key: HIVE-7286
> URL: https://issues.apache.org/jira/browse/HIVE-7286
> Project: Hive
>  Issue Type: Test
>  Components: HCatalog
>Reporter: David Chen
>Assignee: David Chen
> Attachments: HIVE-7286.1.patch
>
>
> Currently, HCatMapReduceTest, which is extended by the following test suites:
>  * TestHCatDynamicPartitioned
>  * TestHCatNonPartitioned
>  * TestHCatPartitioned
>  * TestHCatExternalDynamicPartitioned
>  * TestHCatExternalNonPartitioned
>  * TestHCatExternalPartitioned
>  * TestHCatMutableDynamicPartitioned
>  * TestHCatMutableNonPartitioned
>  * TestHCatMutablePartitioned
> These tests run against RCFile. Currently, only TestHCatDynamicPartitioned is 
> run against any other storage format (ORC).
> Ideally, HCatalog should be tested against all storage formats supported by 
> Hive. The easiest way to accomplish this is to turn HCatMapReduceTest into a 
> parameterized test fixture that enumerates all Hive storage formats. Until 
> HIVE-5976 is implemented, we would need to manually create the mapping of 
> SerDe to InputFormat and OutputFormat. This way, we can explicitly keep track 
> of which storage formats currently work with HCatalog or which ones are 
> untested or have test failures. The test fixture should also use Reflection 
> to find all classes in the classpath that implements the SerDe interface and 
> raise a failure if any of them are not enumerated.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6394) Implement Timestmap in ParquetSerde

[Szehon Ho (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043942#comment-14043942
 ] 

Szehon Ho commented on HIVE-6394:
-

[~leftylev] Do we just need to remove 'timestamp' from the following sentence?

{noformat}
Binary, timestamp, date, char, varchar or decimal support are pending 
(HIVE-6384)
{noformat}

> Implement Timestmap in ParquetSerde
> ---
>
> Key: HIVE-6394
> URL: https://issues.apache.org/jira/browse/HIVE-6394
> Project: Hive
>  Issue Type: Sub-task
>  Components: Serializers/Deserializers
>Reporter: Jarek Jarcec Cecho
>Assignee: Szehon Ho
>  Labels: Parquet, TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6394.2.patch, HIVE-6394.3.patch, HIVE-6394.4.patch, 
> HIVE-6394.5.patch, HIVE-6394.6.patch, HIVE-6394.6.patch, HIVE-6394.7.patch, 
> HIVE-6394.patch
>
>
> This JIRA is to implement timestamp support in Parquet SerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.

[Jayesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043889#comment-14043889
 ] 

Jayesh commented on HIVE-7100:
--

Sure, created HIVE-7289 for reverting patch.
I ll updated this patch accordingly once we have HIVE-7289 committed. 
Thanks!

> Users of hive should be able to specify skipTrash when dropping tables.
> ---
>
> Key: HIVE-7100
> URL: https://issues.apache.org/jira/browse/HIVE-7100
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Ravi Prakash
>Assignee: Jayesh
> Attachments: HIVE-7100.1.patch, HIVE-7100.patch
>
>
> Users of our clusters are often running up against their quota limits because 
> of Hive tables. When they drop tables, they have to then manually delete the 
> files from HDFS using skipTrash. This is cumbersome and unnecessary. We 
> should enable users to skipTrash directly when dropping tables.
> We should also be able to provide this functionality without polluting SQL 
> syntax.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7291) Refactor TestParser to understand test-property file

[Szehon Ho (JIRA)]

Szehon Ho created HIVE-7291:
---

 Summary: Refactor TestParser to understand test-property file
 Key: HIVE-7291
 URL: https://issues.apache.org/jira/browse/HIVE-7291
 Project: Hive
  Issue Type: Sub-task
  Components: Testing Infrastructure
Reporter: Szehon Ho
Assignee: Szehon Ho






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7290) BoneCP Connection Pooling throwing JDODataStoreException with Hive 0.13.1

[Bala Krishna (JIRA)]

Bala Krishna created HIVE-7290:
--

 Summary: BoneCP Connection Pooling throwing JDODataStoreException 
with Hive 0.13.1 
 Key: HIVE-7290
 URL: https://issues.apache.org/jira/browse/HIVE-7290
 Project: Hive
  Issue Type: Bug
  Components: Database/Schema, Metastore
Affects Versions: 0.13.1
 Environment: Hadoop 2.2.0
Hive 0.13.1
BoneCP 0.8.0.RELEASE
Reporter: Bala Krishna


Hi folks,

we're seeing an intermittent issue between our Hive 0.13 metastore and mysql 
instance. After being idle for about 5 minutes or so, any transactions 
involving the metastore and mysql causes the following error to appear in the 
metastore log:

2014-06-09 05:13:52,066 ERROR bonecp.ConnectionHandle 
(ConnectionHandle.java:markPossiblyBroken(388)) - Database access problem. 
Killing off this connection and all remaining connectio
ns in the connection pool. SQL State = 08S01
2014-06-09 05:13:52,068 ERROR metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invoke(157)) - Retrying HMSHandler after 1000 ms 
(attempt 1 of 1) with error: javax.jdo.JDODataStore
Exception: Communications link failure

The last packet successfully received from the server was 502,751 milliseconds 
ago.  The last packet sent successfully to the server was 1 milliseconds ago.
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:275)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:900)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:832)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)

[full stack trace is below]

Subsequent transactions go through fine, however. This only occurs if the 
metastore has has been idle for a while.

We've been looking at the code and the problem seems to lie in the channel 
between the driver and metastore connection pool. We looked at the wait time 
configuration settings both mysql and BoneCP. Both of them are set for max 
value. 

Our hive 0.12 install uses DBCP for datanucleus.connectionPoolingType, not 
BoneCP and we don't see these issues there. We are not running Tez.

This seems like such a basic issue that we'd thought to check and see if anyone 
else has encountered it. Any insights would be greatly appreciated.

=

Full stack trace:

2014-06-09 05:13:52,066 ERROR bonecp.ConnectionHandle 
(ConnectionHandle.java:markPossiblyBroken(388)) - Database access problem. 
Killing off this connection and all remaining connectio
ns in the connection pool. SQL State = 08S01
2014-06-09 05:13:52,068 ERROR metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invoke(157)) - Retrying HMSHandler after 1000 ms 
(attempt 1 of 1) with error: javax.jdo.JDODataStore
Exception: Communications link failure

The last packet successfully received from the server was 502,751 milliseconds 
ago.  The last packet sent successfully to the server was 1 milliseconds ago.
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:275)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:900)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:832)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
at com.sun.proxy.$Proxy0.getTable(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1559)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
at com.sun.proxy.$Proxy11.get_table(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table.getResult(ThriftHiveMetastore.java:8146)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table.getResult(ThriftHiveMetastore.java:8130)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at 
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at 

[jira] [Assigned] (HIVE-7288) Enable support for -libjars and -archives in WebHcat for Streaming MapReduce jobs

[shanyu zhao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shanyu zhao reassigned HIVE-7288:
-

Assignee: shanyu zhao

> Enable support for -libjars and -archives in WebHcat for Streaming MapReduce 
> jobs
> -
>
> Key: HIVE-7288
> URL: https://issues.apache.org/jira/browse/HIVE-7288
> Project: Hive
>  Issue Type: New Feature
>  Components: WebHCat
>Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1
> Environment: HDInsight deploying HDP 2.1;  Also HDP 2.1 on Windows 
>Reporter: Azim Uddin
>Assignee: shanyu zhao
>
> Issue:
> ==
> Due to lack of parameters (or support for) equivalent of '-libjars' and 
> '-archives' in WebHcat REST API, we cannot use an external Java Jars or 
> Archive files with a Streaming MapReduce job, when the job is submitted via 
> WebHcat/templeton. 
> I am citing a few use cases here, but there can be plenty of scenarios like 
> this-
> #1 
> (for -archives):In order to use R with a hadoop distribution like HDInsight 
> or HDP on Windows, we could package the R directory up in a zip file and 
> rename it to r.jar and put it into HDFS or WASB. We can then do 
> something like this from hadoop command line (ignore the wasb syntax, same 
> command can be run with hdfs) - 
> hadoop jar %HADOOP_HOME%\lib\hadoop-streaming.jar -archives 
> wasb:///example/jars/r.jar -files 
> "wasb:///example/apps/mapper.r,wasb:///example/apps/reducer.r" -mapper 
> "./r.jar/bin/Rscript.exe mapper.r" -reducer "./r.jar/bin/Rscript.exe 
> reducer.r" -input /example/data/gutenberg -output /probe/r/wordcount
> This works from hadoop command line, but due to lack of support for 
> '-archives' parameter in WebHcat, we can't submit the same Streaming MR job 
> via WebHcat.
> #2 (for -libjars):
> Consider a scenario where a user would like to use a custom inputFormat with 
> a Streaming MapReduce job and wrote his own custom InputFormat JAR. From a 
> hadoop command line we can do something like this - 
> hadoop jar /path/to/hadoop-streaming.jar \
> -libjars /path/to/custom-formats.jar \
> -D map.output.key.field.separator=, \
> -D mapred.text.key.partitioner.options=-k1,1 \
> -input my_data/ \
> -output my_output/ \
> -outputformat test.example.outputformat.DateFieldMultipleOutputFormat 
> \
> -mapper my_mapper.py \
> -reducer my_reducer.py \
> But due to lack of support for '-libjars' parameter for streaming MapReduce 
> job in WebHcat, we can't submit the above streaming MR job (that uses a 
> custom Java JAR) via WebHcat.
> Impact:
> 
> We think, being able to submit jobs remotely is a vital feature for hadoop to 
> be enterprise-ready and WebHcat plays an important role there. Streaming 
> MapReduce job is also very important for interoperability. So, it would be 
> very useful to keep WebHcat on par with hadoop command line in terms of 
> streaming MR job submission capability.
> Ask:
> 
> Enable parameter support for 'libjars' and 'archives' in WebHcat for Hadoop 
> streaming jobs in WebHcat.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7289) revert HIVE-6469

[Jayesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh updated HIVE-7289:
-

Status: Patch Available  (was: Open)

> revert HIVE-6469
> 
>
> Key: HIVE-7289
> URL: https://issues.apache.org/jira/browse/HIVE-7289
> Project: Hive
>  Issue Type: Task
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Jayesh
>Assignee: Jayesh
> Attachments: HIVE-7289.patch
>
>
> this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7289) revert HIVE-6469

[Jayesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh updated HIVE-7289:
-

Attachment: HIVE-7289.patch

> revert HIVE-6469
> 
>
> Key: HIVE-7289
> URL: https://issues.apache.org/jira/browse/HIVE-7289
> Project: Hive
>  Issue Type: Task
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Jayesh
>Assignee: Jayesh
> Attachments: HIVE-7289.patch
>
>
> this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HIVE-7289) revert HIVE-6469

[Jayesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jayesh reassigned HIVE-7289:


Assignee: Jayesh

> revert HIVE-6469
> 
>
> Key: HIVE-7289
> URL: https://issues.apache.org/jira/browse/HIVE-7289
> Project: Hive
>  Issue Type: Task
>  Components: CLI
>Affects Versions: 0.14.0
>Reporter: Jayesh
>Assignee: Jayesh
>
> this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HIVE-7289) revert HIVE-6469

[Jayesh (JIRA)]

Jayesh created HIVE-7289:


 Summary: revert HIVE-6469
 Key: HIVE-7289
 URL: https://issues.apache.org/jira/browse/HIVE-7289
 Project: Hive
  Issue Type: Task
  Components: CLI
Affects Versions: 0.14.0
Reporter: Jayesh


this task is to revert HIVE-6469



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7100) Users of hive should be able to specify skipTrash when dropping tables.

[Xuefu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043703#comment-14043703
 ] 

Xuefu Zhang commented on HIVE-7100:
---

I think it might be better if we revert HIVE-6469 separately and mark it as not 
valid. Then, we can commit a patch here to add the purge option at syntax level.

> Users of hive should be able to specify skipTrash when dropping tables.
> ---
>
> Key: HIVE-7100
> URL: https://issues.apache.org/jira/browse/HIVE-7100
> Project: Hive
>  Issue Type: Improvement
>Affects Versions: 0.13.0
>Reporter: Ravi Prakash
>Assignee: Jayesh
> Attachments: HIVE-7100.1.patch, HIVE-7100.patch
>
>
> Users of our clusters are often running up against their quota limits because 
> of Hive tables. When they drop tables, they have to then manually delete the 
> files from HDFS using skipTrash. This is cumbersome and unnecessary. We 
> should enable users to skipTrash directly when dropping tables.
> We should also be able to provide this functionality without polluting SQL 
> syntax.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Purge as an option to drop table.

[Ravi Prakash]

Hi folks!

We have an outstanding JIRA https://issues.apache.org/jira/browse/HIVE-7100 
that we would like to get review comments on. A lot of our hive users see their 
HDFS quotas fill up because their dropped tables went to trash.

The patch adds a new keyword called PURGE just like 
http://docs.oracle.com/cd/B28359_01/server.111/b28286/statements_9003.htm#SQLRF01806

 
Could you please voice your support / opposition for this addition?

Thanks
Ravi

[jira] [Resolved] (HIVE-7283) CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based planning

[Harish Butani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harish Butani resolved HIVE-7283.
-

Resolution: Fixed

John thanks for reviewing

> CBO: plumb in HepPlanner and FieldTrimmer(ColumnPruner) into Optiq based 
> planning
> -
>
> Key: HIVE-7283
> URL: https://issues.apache.org/jira/browse/HIVE-7283
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Processor
>Reporter: Harish Butani
>Assignee: Harish Butani
> Attachments: HIVE-7283.1.patch
>
>
> 1.
> HepPlanner initially used for:
> - Predicate Pushdown
> - Transitive Predicate inference
> - Partition Pruning
> 2. Use Optiq's FieldTrimmer for ColumnPruner
> To begin with the rules are copies of Optiq base rules. Once Optiq is 
> refactored to work on Base RelNode classes, the copied rules will be removed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6468) HS2 out of memory error when curl sends a get request

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043642#comment-14043642
 ] 

Hive QA commented on HIVE-6468:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12633065/HIVE-6468.1.patch.txt

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/590/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/590/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-590/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-590/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1605453.

At revision 1605453.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12633065

> HS2 out of memory error when curl sends a get request
> -
>
> Key: HIVE-6468
> URL: https://issues.apache.org/jira/browse/HIVE-6468
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
> Environment: Centos 6.3, hive 12, hadoop-2.2
>Reporter: Abin Shahab
>Assignee: Navis
> Attachments: HIVE-6468.1.patch.txt
>
>
> We see an out of memory error when we run simple beeline calls.
> (The hive.server2.transport.mode is binary)
> curl localhost:1
> Exception in thread "pool-2-thread-8" java.lang.OutOfMemoryError: Java heap 
> space
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>   at 
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-6468) HS2 out of memory error when curl sends a get request

[Ravi Prakash (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043615#comment-14043615
 ] 

Ravi Prakash commented on HIVE-6468:


This is a pretty egregious bug! Any reviews for the patch Navis posted? Can we 
please check in some guard? 

> HS2 out of memory error when curl sends a get request
> -
>
> Key: HIVE-6468
> URL: https://issues.apache.org/jira/browse/HIVE-6468
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 0.12.0
> Environment: Centos 6.3, hive 12, hadoop-2.2
>Reporter: Abin Shahab
>Assignee: Navis
> Attachments: HIVE-6468.1.patch.txt
>
>
> We see an out of memory error when we run simple beeline calls.
> (The hive.server2.transport.mode is binary)
> curl localhost:1
> Exception in thread "pool-2-thread-8" java.lang.OutOfMemoryError: Java heap 
> space
>   at 
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>   at 
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>   at 
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>   at 
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7271) Speed up unit tests

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043278#comment-14043278
 ] 

Lefty Leverenz commented on HIVE-7271:
--

Perhaps this should be documented here too:

* [Unit Test Parallel Execution | 
https://cwiki.apache.org/confluence/display/Hive/Unit+Test+Parallel+Execution]

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7271) Speed up unit tests

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043273#comment-14043273
 ] 

Lefty Leverenz commented on HIVE-7271:
--

[~hagleitn], the wiki has a section for test parameters (hive.test.mode, etc.) 
so this can be documented with them.  I recommend that the release note & 
description explicitly advise people not to set this parameter to true except 
when running unit tests.  Of course the wiki will link to this Jira so they'll 
be able to find the information, but why make them hunt for it?

When HIVE-6037 gets committed, all parameter descriptions will be in 
HiveConf.java and then hive-default.xml.template will be generated from 
HiveConf.java.  You can have an empty description if you prefer, but the 
parameter will still appear in hive-default.xml.template at that point.

* [Configuration Properties -- Test Properties | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-TestProperties]

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7271) Speed up unit tests

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-7271:
-

Release Note: 
hive.exec.submit.local.task.via.child

Determines whether local tasks (typically mapjoin hashtable generation phase) 
runs in separate JVM ("true", recommended) or not. Avoids the overhead of 
spawning new JVM, but can lead to out-of-memory issues.

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7271) Speed up unit tests

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043251#comment-14043251
 ] 

Gunther Hagleitner commented on HIVE-7271:
--

[~leftylev] Just a thought: This parameter is useful to run unit tests quicker, 
but in general not something i would recommend anyone to set to true outside of 
that. It basically runs the hashtable generation for mapjoins in the CLI 
process - something that could give you memory issues. I'll put that in the 
release notes, but in general I think parameters like this needn't be exposed 
to users. In these cases should we just leave it out of the xml?

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-1662) Add file pruning into Hive.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043241#comment-14043241
 ] 

Hive QA commented on HIVE-1662:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652372/HIVE-1662.15.patch.txt

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/589/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/589/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-589/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-589/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'conf/hive-default.xml.template'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/TServerSocketKeepAlive.java'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 'service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target 
itests/custom-serde/target itests/util/target hcatalog/target 
hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/webhcat/svr/target 
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target 
hwi/target common/target common/src/gen contrib/target service/target 
serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1605303.

At revision 1605302.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652372

> Add file pruning into Hive.
> ---
>
> Key: HIVE-1662
> URL: https://issues.apache.org/jira/browse/HIVE-1662
> Project: Hive
>  Issue Type: New Feature
>Reporter: He Yongqiang
>Assignee: Navis
> Attachments: HIVE-1662.10.patch.txt, HIVE-1662.11.patch.txt, 
> HIVE-1662.12.patch.txt, HIVE-1662.13.patch.txt, HIVE-1662.14.patch.txt, 
> HIVE-1662.15.patch.txt, HIVE-1662.8.patch.txt, HIVE-1662.9.patch.txt, 
> HIVE-1662.D8391.1.patch, HIVE-1662.D8391.2.patch, HIVE-1662.D8391.3.patch, 
> HIVE-1662.D8391.4.patch, HIVE-1662.D8391.5.patch, HIVE-1662.D8391.6.patch, 
> HIVE-1662.D8391.7.patch
>
>
> now hive support filename virtual column. 
>

[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout for all transports types

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043240#comment-14043240
 ] 

Hive QA commented on HIVE-6679:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652339/HIVE-6679.1.patch.txt

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 5669 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/587/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/587/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-587/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652339

> HiveServer2 should support configurable the server side socket timeout for 
> all transports types
> ---
>
> Key: HIVE-6679
> URL: https://issues.apache.org/jira/browse/HIVE-6679
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 0.13.0
>Reporter: Prasad Mujumdar
>Assignee: Navis
> Attachments: HIVE-6679.1.patch.txt
>
>
>  HiveServer2 should support configurable the server side socket read timeout 
> and TCP keep-alive option. Metastore server already support this (and the so 
> is the old hive server). 
> We now have multiple client connectivity options like Kerberos, Delegation 
> Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The 
> configuration should be applicable to all types (if possible).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-1662) Add file pruning into Hive.

[Navis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navis updated HIVE-1662:


Attachment: HIVE-1662.15.patch.txt

> Add file pruning into Hive.
> ---
>
> Key: HIVE-1662
> URL: https://issues.apache.org/jira/browse/HIVE-1662
> Project: Hive
>  Issue Type: New Feature
>Reporter: He Yongqiang
>Assignee: Navis
> Attachments: HIVE-1662.10.patch.txt, HIVE-1662.11.patch.txt, 
> HIVE-1662.12.patch.txt, HIVE-1662.13.patch.txt, HIVE-1662.14.patch.txt, 
> HIVE-1662.15.patch.txt, HIVE-1662.8.patch.txt, HIVE-1662.9.patch.txt, 
> HIVE-1662.D8391.1.patch, HIVE-1662.D8391.2.patch, HIVE-1662.D8391.3.patch, 
> HIVE-1662.D8391.4.patch, HIVE-1662.D8391.5.patch, HIVE-1662.D8391.6.patch, 
> HIVE-1662.D8391.7.patch
>
>
> now hive support filename virtual column. 
> if a file name filter presents in a query, hive should be able to only add 
> files which passed the filter to input paths.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7241) Wrong lock acquired for alter table rename partition

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043204#comment-14043204
 ] 

Lefty Leverenz commented on HIVE-7241:
--

The wiki's Locking doc doesn't include "alter table T1 partition P1 rename to 
partition P2" so it ought to be added to the table of Hive commands and locks 
acquired in the Use Cases section:

* [Hive Concurrency Model -- Use Cases | 
https://cwiki.apache.org/confluence/display/Hive/Locking?from=_YKhAQ#Locking-UseCases]

> Wrong lock acquired for alter table rename partition
> 
>
> Key: HIVE-7241
> URL: https://issues.apache.org/jira/browse/HIVE-7241
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7241.patch, HIVE-7241.patch
>
>
> Doing an "alter table foo partition (bar='x') rename to partition (bar='y')" 
> acquires a read lock on table foo.  It should instead acquire an exclusive 
> lock on partition bar=x.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7241) Wrong lock acquired for alter table rename partition

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-7241:
-

Labels: TODOC14  (was: )

> Wrong lock acquired for alter table rename partition
> 
>
> Key: HIVE-7241
> URL: https://issues.apache.org/jira/browse/HIVE-7241
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7241.patch, HIVE-7241.patch
>
>
> Doing an "alter table foo partition (bar='x') rename to partition (bar='y')" 
> acquires a read lock on table foo.  It should instead acquire an exclusive 
> lock on partition bar=x.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-4924) Support query timeout for jdbc2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043187#comment-14043187
 ] 

Hive QA commented on HIVE-4924:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12652329/HIVE-4924.3.patch.txt

{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5655 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/586/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/586/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-586/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12652329

> Support query timeout for jdbc2
> ---
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2, JDBC
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.D11787.1.patch, 
> HIVE-4924.D11787.2.patch
>
>
> support 
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7242) alter table drop partition is acquiring the wrong type of lock

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043185#comment-14043185
 ] 

Lefty Leverenz commented on HIVE-7242:
--

The wiki already documents the correct behavior:

{quote}
*Hive Command* alter table T1 drop partition P1
*Locks Acquired* S on T1, X on T1.P1
{quote}

Should this bug fix & version be noted in the wiki?

* [Hive Concurrency Model -- Use Cases (see table) | 
https://cwiki.apache.org/confluence/display/Hive/Locking?from=_YKhAQ#Locking-UseCases]

> alter table drop partition is acquiring the wrong type of lock
> --
>
> Key: HIVE-7242
> URL: https://issues.apache.org/jira/browse/HIVE-7242
> Project: Hive
>  Issue Type: Bug
>  Components: Locking
>Affects Versions: 0.13.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 0.14.0
>
> Attachments: HIVE-7242.patch
>
>
> Doing an "alter table foo drop partition ('bar=x')" acquired a shared-write 
> lock on partition bar=x.  It should be acquiring an exclusive lock in that 
> case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7271) Speed up unit tests

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-7271:
-

Fix Version/s: 0.14.0

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7271) Speed up unit tests

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-7271:
-

Labels: TODOC14  (was: )

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7271) Speed up unit tests

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043139#comment-14043139
 ] 

Lefty Leverenz commented on HIVE-7271:
--

This adds configuration parameter *hive.exec.submit.local.task.via.child* to 
HiveConf.java, so it needs a description in hive-default.xml.template.

Or you can put the description in a release note, and I'll make sure it gets 
into the new version of HiveConf.java (which will include descriptions) after 
HIVE-6037 gets committed.

> Speed up unit tests
> ---
>
> Key: HIVE-7271
> URL: https://issues.apache.org/jira/browse/HIVE-7271
> Project: Hive
>  Issue Type: Bug
>Reporter: Gunther Hagleitner
>Assignee: Gunther Hagleitner
>  Labels: TODOC14
> Attachments: HIVE-7271.1.patch, HIVE-7271.2.patch, HIVE-7271.3.patch, 
> HIVE-7271.4.patch, HIVE-7271.5.patch, HIVE-7271.6.patch, HIVE-7271.7.patch
>
>
> Did some experiments to see if there's a way to speed up unit tests. 
> TestCliDriver seemed to take a lot of time just spinning up/tearing down 
> JVMs. I was also curious to see if running everything on a ram disk would 
> help.
> Results (I ran tests up to authorization_2):
> - Current setup: 40 minutes
> - Single JVM (not using child JVM to run all queries): 8 minutes
> - Single JVM + ram disk: 7 minutes
> So the ram disk didn't help that much. But running tests in single JVM seems 
> worthwhile doing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HIVE-7211) Throws exception if the name of conf var starts with "hive." does not exists in HiveConf

[Lefty Leverenz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lefty Leverenz updated HIVE-7211:
-

Labels: TODOC14  (was: )

> Throws exception if the name of conf var starts with "hive." does not exists 
> in HiveConf
> 
>
> Key: HIVE-7211
> URL: https://issues.apache.org/jira/browse/HIVE-7211
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
>  Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-7211.1.patch.txt, HIVE-7211.2.patch.txt, 
> HIVE-7211.3.patch.txt, HIVE-7211.4.patch.txt
>
>
> Some typos in configurations are very hard to find.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HIVE-7211) Throws exception if the name of conf var starts with "hive." does not exists in HiveConf

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043127#comment-14043127
 ] 

Lefty Leverenz commented on HIVE-7211:
--

This adds several configuration parameters to HiveConf.java: 

* hive.test.dummystats.aggregator
* hive.test.dummystats.publisher
* hive.io.rcfile.record.interval
* hive.io.rcfile.column.number.conf
* hive.io.rcfile.tolerate.corruptions
* hive.io.rcfile.record.buffer.size
* hive.hbase.generatehfiles
* hive.index.compact.file (internal)
* hive.index.blockfilter.file (internal)

Except for the internal parameters, they need definitions in 
hive-default.xml.template.  Then they should be documented in the wiki 
(https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties).

> Throws exception if the name of conf var starts with "hive." does not exists 
> in HiveConf
> 
>
> Key: HIVE-7211
> URL: https://issues.apache.org/jira/browse/HIVE-7211
> Project: Hive
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Navis
>Assignee: Navis
>Priority: Trivial
> Fix For: 0.14.0
>
> Attachments: HIVE-7211.1.patch.txt, HIVE-7211.2.patch.txt, 
> HIVE-7211.3.patch.txt, HIVE-7211.4.patch.txt
>
>
> Some typos in configurations are very hard to find.



--
This message was sent by Atlassian JIRA
(v6.2#6252)