[jira] [Created] (HIVE-11807) Set ORC buffer size in relation to set stripe size

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-11807:


 Summary: Set ORC buffer size in relation to set stripe size
 Key: HIVE-11807
 URL: https://issues.apache.org/jira/browse/HIVE-11807
 Project: Hive
  Issue Type: Improvement
  Components: File Formats
Reporter: Owen O'Malley
Assignee: Owen O'Malley


A customer produced ORC files with very small stripe sizes (10k rows/stripe) by 
setting a small 64MB stripe size and 256K buffer size for a 54 column table. At 
that size, each of the streams only get a buffer or two before the stripe size 
is reached. The current code uses the available memory instead of the stripe 
size and thus doesn't shrink the buffer size if the JVM has much more memory 
than the stripe size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #112

[hiveqa]

See 


Changes:

[Chao Sun] HIVE-11761: DoubleWritable hashcode for GroupBy is not properly 
generated (Aihua Xu, reviewed by Chao Sun)

[sershe] HIVE-11587 : Fix memory estimates for mapjoin hashtable (Wei Zheng, 
reviewed by Sergey Shelukhin)

[khorgath] HIVE-11510 : Metatool updateLocation warning on views (Wei Zheng via 
Sushanth Sowmyan)

[gunther] HIVE-11727: Hive on Tez through Oozie: Some queries fail with fnf 
exception (Gunther Hagleitner, reviewed by Vikram Dixit K)

[vikram] HIVE-11605: Incorrect results with bucket map join in tez. (Vikram 
Dixit K, reviewed by Sergey Shelukhin)

[vikram] HIVE-11606: Bucket map joins fail at hash table construction time 
(Vikram Dixit K, reviewed by Sergey Shelukhin)

[Ashutosh Chauhan] HIVE-11645 : Add in-place updates for dynamic partitions 
loading (Ashutosh Chauhan via Prasanth J)

[j.prasanth.j] HIVE-11751: hive-exec-log4j2.xml settings causes DEBUG messages 
to be generated and ignored (Prasanth Jayachandran reviewed by Sergey Shelukhin)

[Ashutosh Chauhan] HIVE-11781 : Remove HiveLimit operator (Jesus Camacho 
Rodriguez via Ashutosh Chauhan)

[Jimmy Xiang] HIVE-11771: Parquet timestamp conversion errors (Jimmy, reviewed 
by Szehon)

[Ashutosh Chauhan] HIVE-10980 : Merge of dynamic partitions loads all data to 
default partition (Illya Yalovyy via Ashutosh Chauhan)

[pxiong] HIVE-11614: CBO: Calcite Operator To Hive Operator (Calcite Return 
Path): ctas after order by has problem (Pengcheng Xiong, reviewd by Laljo John 
Pullokkaran)

[Sergio Pena] HIVE-11763: Use * instead of sum(hash(*)) on Parquet predicate 
(PPD) integration tests (Sergio Pena, reviewed by Ferdinand Xu)

--
[...truncated 324 lines...]
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/AbstractSSHCommand.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/AbstractSSHResult.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/SSHExecutionException.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/AbortDroneException.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/LocalCommandFactory.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ExecutionPhase.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/TestsFailedException.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/PrepPhase.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/LocalCommand.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ReportingPhase.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/HostExecutor.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/Templates.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/PTest.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/LogDirectoryCleaner.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/Phase.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/Dirs.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/JUnitReportParser.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/HostExecutorBuilder.java
Aptest2/src/main/java/org/apache/hive/ptest/api
Aptest2/src/main/java/org/apache/hive/ptest/api/request
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestStopRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestListRequest.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestLogRequest.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStatusRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/client
Aptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
Aptest2/src/main/java/org/apache/hive/ptest/api/response
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestListResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/GenericResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStartResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestLogResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatusResponse.java
Aptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatus.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStopResponse.java
Aptest2/src/main/java/org/apache/hive/ptest/api/Status.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server
Aptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server/Test.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server/TestLogger.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
Aptest2/pom.xml
Checked 

[jira] [Created] (HIVE-11808) In ORC removing the dynamic dispatch for StringTreeReader improves read by 10%

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-11808:


 Summary: In ORC removing the dynamic dispatch for StringTreeReader 
improves read by 10%
 Key: HIVE-11808
 URL: https://issues.apache.org/jira/browse/HIVE-11808
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


When we introduced the dictionary/direct encodings for ORC, we made subclasses 
of StringTreeReader named StringDirectTreeReader and StringDictionaryTreeReader 
and introduce an additional dynamic dispatch in the inner loop. For tables with 
a lot of string columns, removing that extra dispatch improves performance 10%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38216: HIVE-11745: Alter table Exchange partition with multiple partition_spec is not working

[Yongzhi Chen]

> On Sept. 12, 2015, 1:23 a.m., Szehon Ho wrote:
> > I dont know if you saw in the earlier comments, please add a test to the 
> > file 'FolderPermissionBase' to verify permission inheritance works with the 
> > feature.

Sorry, overlooked the comments. I added a test to cover this. 
The fix respect original design: destination partition folder inheritance 
original partition folder's permission.
For the intermediate folders between destination partition folder and base 
table folder, if they do not exist, the permission inherit from base table 
folder's(the same behavior as when add a new partition), otherwise keep their 
original permission.


- Yongzhi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38216/#review98723
---


On Sept. 12, 2015, 4:07 a.m., Yongzhi Chen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38216/
> ---
> 
> (Updated Sept. 12, 2015, 4:07 a.m.)
> 
> 
> Review request for hive, Chao Sun, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-11745
> https://issues.apache.org/jira/browse/HIVE-11745
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Alter table Exchange partition with multiple partition_spec does not work in 
> cluster mode because in rename, the parent folder for destination path does 
> not physically exist. Some files system(hdfs for instance) does not 
> support(or allow) this. Fix by create parent folder first.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/FolderPermissionBase.java
>  f28edc66ea4644c5847ee6abe2e26306f9fbb43e 
>   itests/src/test/resources/testconfiguration.properties 
> bed621d3eb74f01e54110552f68538afd228018d 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> 1840e76cc567e95e1942d912b8ab0db516d63a3b 
>   ql/src/test/queries/clientpositive/exchgpartition2lel.q PRE-CREATION 
>   ql/src/test/results/clientpositive/exchgpartition2lel.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38216/diff/
> 
> 
> Testing
> ---
> 
> Add minimr unit test.
> 
> 
> Thanks,
> 
> Yongzhi Chen
> 
>

[jira] [Created] (HIVE-11809) Insert into table partition failed but partition directory created

[niklaus.xiao (JIRA)]

niklaus.xiao created HIVE-11809:
---

 Summary: Insert into table partition failed but partition 
directory created
 Key: HIVE-11809
 URL: https://issues.apache.org/jira/browse/HIVE-11809
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.1, 1.1.0, 1.2.0
Reporter: niklaus.xiao


1. userA creates a partition table: create table w1(id int, name string, addr 
string) partitioned by (dt string) row format delimited fields terminated by 
',' stored as textfile;
2. userB tries to insert into userA's table: insert into table w1 
partition(dt='2') select id, name, addr from t1;
3. userB does not have select privilege of table t1 and insert privilege of w1, 
step 2 throws: Error: Error while compiling statement: FAILED: 
HiveAccessControlException Permission denied: Principal [name=hdfs, type=USER] 
does not have following privileges for operation QUERY [[SELECT] on Object 
[type=TABLE_OR_VIEW, name=default.t1]] (state=42000,code=4)
but the partition directory "/user/hive/warehouse/w1/dt=2" created:
hdfs dfs -ls /user/hive/warehouse/w1/
15/09/14 11:16:42 INFO hdfs.PeerCache: SocketCache disabled.
Found 2 items
drwx--   - userA hive  0 2015-09-14 10:05 
/user/hive/warehouse/w1/dt=1
drwx--   - userB  hive  0 2015-09-14 10:23 
/user/hive/warehouse/w1/dt=2




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11810) LLAP: Exception is ignored if MiniLlap cluster fails to start

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-11810:


 Summary: LLAP: Exception is ignored if MiniLlap cluster fails to 
start
 Key: HIVE-11810
 URL: https://issues.apache.org/jira/browse/HIVE-11810
 Project: Hive
  Issue Type: Sub-task
Reporter: Prasanth Jayachandran






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)