Re: Review Request 34877: HIVE-11201: HCatalog is ignoring user specified avro schema in the table definition

2015-09-11 Thread Bing Li 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34877/
---

(Updated Sept. 11, 2015, 7:21 a.m.)


Review request for hive.


Bugs: HIVE-11201
https://issues.apache.org/jira/browse/HIVE-11201


Repository: hive-git


Description
---

This patch is created based on the latest code in master branch.
With this patch, Hive would get the correct avro schema.


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/SpecialCases.java
 f38d53b 

Diff: https://reviews.apache.org/r/34877/diff/


Testing
---

I have tested it in a real cluster. And also ran Hive UTs in local successfully.


Thanks,

Bing Li

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

2015-09-11 Thread Yongzhi Chen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/
---

(Updated Sept. 11, 2015, 1:02 p.m.)


Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-11745
https://issues.apache.org/jira/browse/HIVE-11745


Repository: hive-git


Description
---

Alter table Exchange partition with multiple partition_spec does not work in 
cluster mode because in rename, the parent folder for destination path does not 
physically exist. Some files system(hdfs for instance) does not support(or 
allow) this. Fix by create parent folder first.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 
bed621d3eb74f01e54110552f68538afd228018d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
1840e76cc567e95e1942d912b8ab0db516d63a3b 
  ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
  ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/38216/diff/


Testing
---

Add minimr unit test.


Thanks,

Yongzhi Chen

Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks memory on long running Hive Server2 Instances

2015-09-11 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review98616
---



common/src/java/org/apache/hive/common/util/ShutdownHookManager.java (line 59)


Side question: I'm wondering why we need two methods which are the same 
thing. Could we get rid of one?


- Xuefu Zhang


On Sept. 11, 2015, 5:54 a.m., Navis Ryu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> ---
> 
> (Updated Sept. 11, 2015, 5:54 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our 
> long running HiveServer2 instances,taken up more than 100MB on heap.
>   Most of the paths contains a suffix of ".pipeout".
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hive/common/util/ShutdownHookManager.java 
> fd2f20a 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f 
>   
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
>  bcc66cf 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> cc9df76 
> 
> Diff: https://reviews.apache.org/r/38292/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Navis Ryu
> 
>

[jira] [Created] (HIVE-11799) The output of explain query for multiple lateral views is huge

2015-09-11 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-11799:
---

 Summary: The output of explain query for multiple lateral views is 
huge
 Key: HIVE-11799
 URL: https://issues.apache.org/jira/browse/HIVE-11799
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 2.0.0
Reporter: Aihua Xu
Assignee: Aihua Xu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11800) GenMapRedWalker in MapReduceCompiler.java seems not efficient

2015-09-11 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-11800:
---

 Summary: GenMapRedWalker in MapReduceCompiler.java seems not 
efficient
 Key: HIVE-11800
 URL: https://issues.apache.org/jira/browse/HIVE-11800
 Project: Hive
  Issue Type: Sub-task
  Components: Logical Optimizer
Reporter: Aihua Xu


Investigate the implementation of GenMapRedWalker in MapReduceCompiler.java to 
see if the performance can be improved. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11801) In HMS HA env, "show databases" fails when"current" HMS is stopped.

2015-09-11 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-11801:
---

 Summary: In HMS HA env, "show databases" fails when"current" HMS 
is stopped.
 Key: HIVE-11801
 URL: https://issues.apache.org/jira/browse/HIVE-11801
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 1.1.0, 1.2.0, 0.14.0, 2.0.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


Reproduce steps:
# Enable HMS HA on a cluster
# Use beeline to connect to HS2 and execute command {{show databases}}. Don't 
quit beeline after command has finished
# Stop the first HMS in configuration {{hive.metastore.uri}}
# Execute {{show databases}} in beeline again. Will get below error:
{noformat}
MetaException(message:Got exception: 
org.apache.thrift.transport.TTransportException java.net.SocketException: 
Broken pipe)
{noformat}

The error message in HS2 is as below:
{noformat}
2015-09-08 12:06:53,236 ERROR hive.log: Got exception: 
org.apache.thrift.transport.TTransportException java.net.SocketException: 
Broken pipe
org.apache.thrift.transport.TTransportException: java.net.SocketException: 
Broken pipe
at 
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at 
org.apache.thrift.transport.TSaslTransport.flush(TSaslTransport.java:501)
at 
org.apache.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:37)
at 
org.apache.hadoop.hive.thrift.TFilterTransport.flush(TFilterTransport.java:77)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_databases(ThriftHiveMetastore.java:692)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:684)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:964)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:91)
at com.sun.proxy.$Proxy6.getDatabases(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:1909)
at com.sun.proxy.$Proxy6.getDatabases(Unknown Source)
at 
org.apache.hive.service.cli.operation.GetSchemasOperation.runInternal(GetSchemasOperation.java:59)
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at 
org.apache.hive.service.cli.session.HiveSessionImpl.getSchemas(HiveSessionImpl.java:462)
at 
org.apache.hive.service.cli.CLIService.getSchemas(CLIService.java:296)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.GetSchemas(ThriftCLIService.java:534)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1373)
at 
org.apache.hive.service.cli.thrift.TCLIService$Processor$GetSchemas.getResult(TCLIService.java:1358)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:692)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at 
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
... 31 more
2015-09-08 12:06:53,238 ERROR hive.log: Converting exception to MetaException
2015-09-08 12:06:53,238 WARN 
org.apache.hive.service.cli.thrift.ThriftCLIService: Error getting schemas:
org.apache.hive.service.cli.HiveSQLException: MetaException(message:Got 
exception: org.apache.thrift.transport.TTransportException

[jira] [Created] (HIVE-11802) Float-point numbers are displayed with different precision in Beeline/JDBC

2015-09-11 Thread JIRA 
Sergio Peña created HIVE-11802:
--

 Summary: Float-point numbers are displayed with different 
precision in Beeline/JDBC
 Key: HIVE-11802
 URL: https://issues.apache.org/jira/browse/HIVE-11802
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Sergio Peña
Assignee: Sergio Peña


When inserting float-point numbers to a table, the values displayed on beeline 
or jdbc are with different precision.

How to reproduce:
{noformat}
0: jdbc:hive2://localhost:1> create table decimals (f float, af 
array, d double, ad array) stored as parquet;
No rows affected (0.294 seconds)

0: jdbc:hive2://localhost:1> insert into table decimals select 1.10058, 
array(cast(1.10058 as float)), 2.0133, array(2.0133) from dummy limit 1;
...
No rows affected (20.089 seconds)

0: jdbc:hive2://localhost:1> select f, af, af[0], d, ad[0] from decimals;
+-++-+-+-+--+
|  f  | af | _c2 |d|   _c4   |
+-++-+-+-+--+
| 1.1005799770355225  | [1.10058]  | 1.1005799770355225  | 2.0133  | 2.0133  |
+-++-+-+-+--+
{noformat}

When displaying arrays, the values are displayed correctly, but if I print a 
specific element, it is then displayed with more decimal positions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11803) LLAP: MiniLlapCluster tests are slow

2015-09-11 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-11803:


 Summary: LLAP: MiniLlapCluster tests are slow
 Key: HIVE-11803
 URL: https://issues.apache.org/jira/browse/HIVE-11803
 Project: Hive
  Issue Type: Sub-task
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Session closing for llap was not handle properly causing llap tests to run slow.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11804) Different describe formatted behavior depending on whether the table name is qualified with database name or not

2015-09-11 Thread Mark Grover (JIRA) 
Mark Grover created HIVE-11804:
--

 Summary: Different describe formatted behavior depending on 
whether the table name is qualified with database name or not
 Key: HIVE-11804
 URL: https://issues.apache.org/jira/browse/HIVE-11804
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Mark Grover


I have a simple text file based managed table on HDFS:
{quote}
show create table src;
+---+--+
|createtab_stmt 
|
+---+--+
| CREATE TABLE `src`(   
|
|   `first` string, 
|
|   `word` string)  
|
| PARTITIONED BY (  
|
|   `length` int)   
|
| ROW FORMAT SERDE  
|
|   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
|
| STORED AS INPUTFORMAT 
|
|   'org.apache.hadoop.mapred.TextInputFormat'  
|
| OUTPUTFORMAT  
|
|   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
|
| LOCATION  
|
|   'hdfs://name-node:8020/user/hive/warehouse/my.db/src'  |
| TBLPROPERTIES (   
|
|   'transient_lastDdlTime'='1441921577')   
|
+---+--+
{quote}

The describe formatted with the database name returns:
{quote}
describe formatted my.src first partition(length=1);
+-+---+---+---++-+--+--++-+--+--+
|col_name |   data_type   |  min  |  
max  | num_nulls  | distinct_count  | avg_col_len  | max_col_len  | num_trues  
| num_falses  | comment  |
+-+---+---+---++-+--+--++-+--+--+
| # col_name  | data_type | comment   | 
  | NULL   | NULL| NULL | NULL | NULL   | 
NULL| NULL |
| | NULL  | NULL  | 
NULL  | NULL   | NULL| NULL | NULL | NULL   
| NULL| NULL |
| first   | string| from deserializer | 
NULL  | NULL   | NULL| NULL | NULL | NULL   
| NULL| NULL |
+-+---+---+---++-+--+--++-+--+--+
{quote}

while without it returns:
{quote}
describe formatted src first partition(length=1);
+---+---+---+--+
|   col_name| data_type 
|comment|
+---+---+---+--+
| # col_name| data_type 
| comment   |
|   | NULL  
| NULL  |
| first | string
|   |
| word  | string
|   |
|   | NULL  
| NULL  |
| # Partition Information   | NULL  
| NULL  |
| # col_name| data_type 
| comment   |
|   | NULL

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

2015-09-11 Thread Yongzhi Chen 


> On Sept. 10, 2015, 6:02 p.m., Szehon Ho wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java, 
> > line 2552
> > 
> >
> > I think this whole method can be moved to FileUtils for organization.  
> > Also please check if there's any method there already.
> 
> Yongzhi Chen wrote:
> I think it may be better as a private method in the HiveMetaStore class 
> for it will using its private variable wh (hdfs warehouse) .
> 
> Szehon Ho wrote:
> Actually looking more into the code, this method should not be necessary. 
>  You can just call wh.mkdirs directly.  The underlying FileSystem.mkdirs has 
> the same semantics as -p, there should be no file system that violates this.  
> If there were, many other partition codes would break..

Thanks Szehon. As you pointed out and the name of the function, the wh.mkdirs 
should be the same as mkdir -p in all the filesystem. I have worried too much. 
The third patch remove the createFullPath method and use mkdirs directly. I 
also add a new test case to cover the case when more than one intermediate dirs 
are missing.


- Yongzhi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/#review98435
---


On Sept. 11, 2015, 1:02 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38216/
> ---
> 
> (Updated Sept. 11, 2015, 1:02 p.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-11745
> https://issues.apache.org/jira/browse/HIVE-11745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Alter table Exchange partition with multiple partition_spec does not work in 
> cluster mode because in rename, the parent folder for destination path does 
> not physically exist. Some files system(hdfs for instance) does not 
> support(or allow) this. Fix by create parent folder first.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> bed621d3eb74f01e54110552f68538afd228018d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 1840e76cc567e95e1942d912b8ab0db516d63a3b 
>   ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
>   ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38216/diff/
> 
> 
> Testing
> ---
> 
> Add minimr unit test.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

[jira] [Created] (HIVE-11805) LLAP: TaskCommunicator should not register MultiMRInput Inputs for state updates

2015-09-11 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-11805:
-

 Summary: LLAP: TaskCommunicator should not register MultiMRInput 
Inputs for state updates
 Key: HIVE-11805
 URL: https://issues.apache.org/jira/browse/HIVE-11805
 Project: Hive
  Issue Type: Sub-task
Affects Versions: llap
Reporter: Siddharth Seth
Assignee: Siddharth Seth
 Fix For: llap






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Fwd: Over writing files

2015-09-11 Thread Raajay 
Hi,

How do I go about renaming the output directory for different runs of the DAG 
generated through hive?

- Raajay


Begin forwarded message:

> From: Hitesh Shah 
> Date: September 11, 2015 at 3:22:25 PM CDT
> To: u...@tez.apache.org
> Subject: Re: Over writing files
> Reply-To: u...@tez.apache.org
> 
> This is probably a question for the Hive dev mailing list on how the 
> staging/output directory name is determined. i.e. 
> ".hive-staging_hive_2015-09-11_00-07-40_043_6365145769624003668-1”. You may 
> need to change this value in the config being used to configure the output of 
> the vertex that is doing the write to HDFS.
> 
> — Hitesh
> 
> 
>> On Sep 11, 2015, at 1:09 PM, Raajay  wrote:
>> 
>> I am running DAGs generated by Hive using my custom Tez Client. So I 
>> serialize a DAG, load it back and submit it later. Everything works great 
>> the first time; however, on second runs the I get a RunTime exception 
>> (snippet below)
>> 
>> My guess, it since the same DAG is run again, the output tables (have same 
>> id) and that prevents overwrite. 
>> 
>> Where should i introduce randomness in the file name ? Should I change some 
>> name field in FileSinkDescriptor every time I re-run the dag ? 
>> 
>> Thanks
>> Raajay
>> 
>> 
>> Vertex failed, vertexName=Reducer 3, 
>> vertexId=vertex_1441949856963_0011_1_04, diagnostics=[Task failed, 
>> taskId=task_1441949856963_0011_1_04_00, diagnostics=[TaskAttempt 0 
>> failed, info=[Error: Failure while running task:java.lang.RuntimeException: 
>> java.lang.RuntimeException: Hive Runtime Error while closing operators: 
>> Unable to rename output from: 
>> hdfs://10.10.1.2:8020/apps/hive/output_tab/.hive-staging_hive_2015-09-11_00-07-40_043_6365145769624003668-1/_task_tmp.-ext-1/_tmp.00_0
>>  to: 
>> hdfs://10.10.1.2:8020/apps/hive/output_tab/.hive-staging_hive_2015-09-11_00-07-40_043_6365145769624003668-1/_tmp.-ext-1/00_0
>> at 
>> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171)
>> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
>> at 
>> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:345)
>> at 
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
>> at 
>> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171
>

Re: Review Request 38247: HIVE-11778 Merge beeline-cli branch to trunk

2015-09-11 Thread Xuefu Zhang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38247/#review98694
---



beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 96)


Could we remove this static import, as it's not common in Hive and it's 
referred only once in the code?



beeline/src/java/org/apache/hive/beeline/BeeLine.java (line 1085)


Again I have trouble understanding this: if cmdMap.size() > 1, then there 
must be at lease one match, right? Then, how cmdMap.get(line) can return null, 
as suggested in line 1088? Further, how can handle == null suggest multiple 
matches? I know this is old code, but let's clean it up if necessary.



beeline/src/java/org/apache/hive/beeline/Commands.java (line 815)


This seems weird as the method isn't new but shown as new here.



ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 393)


Nit: remove leading space/tab.


- Xuefu Zhang


On Sept. 10, 2015, 3:35 a.m., cheng xu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38247/
> ---
> 
> (Updated Sept. 10, 2015, 3:35 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The changes we made in beeline-cli branch
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 3cd2a8b 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 3388391 
>   beeline/src/java/org/apache/hive/beeline/ClientCommandHookFactory.java 
> PRE-CREATION 
>   beeline/src/java/org/apache/hive/beeline/ClientHook.java PRE-CREATION 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 3cdcfb8 
>   beeline/src/java/org/apache/hive/beeline/cli/CliOptionsProcessor.java 
> PRE-CREATION 
>   beeline/src/java/org/apache/hive/beeline/cli/HiveCli.java PRE-CREATION 
>   beeline/src/test/org/apache/hive/beeline/TestClientCommandHookFactory.java 
> PRE-CREATION 
>   beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java PRE-CREATION 
>   beeline/src/test/resources/hive-site.xml PRE-CREATION 
>   bin/ext/cli.sh 914aae3 
>   bin/ext/util/execHiveCmd.sh 167cc40 
>   cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java d62fd5c 
>   common/src/java/org/apache/hadoop/hive/conf/HiveVariableSource.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/VariableSubstitution.java 
> PRE-CREATION 
>   common/src/test/org/apache/hadoop/hive/conf/TestVariableSubstitution.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4030075 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 8b7a2e8 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/VariableSubstitution.java 
> e8b1d96 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/AddResourceProcessor.java 
> 0558c53 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/CompileProcessor.java 
> 25ce168 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/DeleteResourceProcessor.java 
> 9052c82 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/DfsProcessor.java cc0414d 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/SetProcessor.java 2414e12 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f 
>   
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java
>  bcc66cf 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> cc9df76 
> 
> Diff: https://reviews.apache.org/r/38247/diff/
> 
> 
> Testing
> ---
> 
> UT and smoke test passed locally
> 
> 
> Thanks,
> 
> cheng xu
> 
>

[jira] [Created] (HIVE-11806) Create test for HIVE-11174

2015-09-11 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-11806:
-

 Summary: Create test for HIVE-11174
 Key: HIVE-11806
 URL: https://issues.apache.org/jira/browse/HIVE-11806
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.2.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K
Priority: Minor


We are lacking tests for HIVE-11174. Adding one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38136: CBO: Calcite Operator To Hive Operator (Calcite Return Path): ctas after order by has problem

2015-09-11 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38136/#review98718
---

Ship it!


Ship It!

- John Pullokkaran


On Sept. 8, 2015, 8:42 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38136/
> ---
> 
> (Updated Sept. 8, 2015, 8:42 p.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> with return path on, "create table b as select * from src order by key" will 
> fail. Attached two test cases, one (cbo_rp_cross_product_check_2.q) is to 
> test that "create table b as select * from src order by key" will not fail. 
> The other(cbo_rp_auto_join17.q) is to correct current wrong table alias on 
> master.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/PlanModifierForReturnPath.java
>  81cc474 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 73ae7c4 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f6052e3 
>   ql/src/test/queries/clientpositive/cbo_rp_auto_join17.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/cbo_rp_cross_product_check_2.q 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_auto_join17.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/cbo_rp_cross_product_check_2.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38136/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

2015-09-11 Thread Szehon Ho 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/#review98723
---


I dont know if you saw in the earlier comments, please add a test to the file 
'FolderPermissionBase' to verify permission inheritance works with the feature.

- Szehon Ho


On Sept. 11, 2015, 1:02 p.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38216/
> ---
> 
> (Updated Sept. 11, 2015, 1:02 p.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-11745
> https://issues.apache.org/jira/browse/HIVE-11745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Alter table Exchange partition with multiple partition_spec does not work in 
> cluster mode because in rename, the parent folder for destination path does 
> not physically exist. Some files system(hdfs for instance) does not 
> support(or allow) this. Fix by create parent folder first.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 
> bed621d3eb74f01e54110552f68538afd228018d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 1840e76cc567e95e1942d912b8ab0db516d63a3b 
>   ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
>   ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38216/diff/
> 
> 
> Testing
> ---
> 
> Add minimr unit test.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

2015-09-11 Thread Yongzhi Chen 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/
---

(Updated Sept. 12, 2015, 4:07 a.m.)


Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.


Changes
---

add test to test permission inheritance.


Bugs: HIVE-11745
https://issues.apache.org/jira/browse/HIVE-11745


Repository: hive-git


Description
---

Alter table Exchange partition with multiple partition_spec does not work in 
cluster mode because in rename, the parent folder for destination path does not 
physically exist. Some files system(hdfs for instance) does not support(or 
allow) this. Fix by create parent folder first.


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
 f28edc66ea4644c5847ee6abe2e26306f9fbb43e 
  itests/src/test/resources/testconfiguration.properties 
bed621d3eb74f01e54110552f68538afd228018d 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
1840e76cc567e95e1942d912b8ab0db516d63a3b 
  ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
  ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/38216/diff/


Testing
---

Add minimr unit test.


Thanks,

Yongzhi Chen