[GitHub] hive pull request #435: HIVE-20627: Concurrent async queries intermittently ...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/435


---

Review Request 68868: HIVE-17043: Remove non unique columns from group by keys if not referenced later

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68868/
---

Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-17043
https://issues.apache.org/jira/browse/HIVE-17043


Repository: hive-git


Description
---

This patch implements/test the following optimizations
* Removal of group by on primary keys
* Reduction of group by keys on primary keys
* is NOT NULL filter removal if NOT NULL constraint is defined


Diffs
-

  itests/src/test/resources/testconfiguration.properties def356176b 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveDefaultRelMetadataProvider.java
 635d27e723 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java 
42e60de6a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateJoinTransposeRule.java
 f43ef01293 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelFieldTrimmer.java
 5857f730a8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/EstimateUniqueKeys.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdColumnUniqueness.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 1ca1937ed9 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdUniqueKeys.java
 3bf62c535c 
  ql/src/test/queries/clientpositive/constraints_optimization.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/constraints_optimization.q.out 
PRE-CREATION 


Diff: https://reviews.apache.org/r/68868/diff/1/


Testing
---


Thanks,

Vineet Garg

[jira] [Created] (HIVE-20650) trunc string type throw FAILED: ArrayIndexOutOfBoundsException 1

[Yuming Wang (JIRA)]

Yuming Wang created HIVE-20650:
--

 Summary: trunc string type throw FAILED: 
ArrayIndexOutOfBoundsException 1
 Key: HIVE-20650
 URL: https://issues.apache.org/jira/browse/HIVE-20650
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.3
Reporter: Yuming Wang


{code:sql}
hive> select trunc('2.5');
FAILED: ArrayIndexOutOfBoundsException 1
hive> SELECT trunc('2009-02-12');
FAILED: ArrayIndexOutOfBoundsException 1
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68683: Add new configuration to set the size of the global compile lock

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68683/#review209081
---



Thanks Denys,
I like this new version.
My last comments are below.
What do you think is it worth to create a new version of the patch?
Thanks,
Peter


ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLock.java
Lines 44 (patched)


Would it be a good idea to remove the public constructor? We are using 
factory to create CompileLock, so we might want to emphasize that



ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLock.java
Lines 64 (patched)


We do not use this anywhere - we might want to consider to remove this 
altogether and keep only the one without parameters?



ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLock.java
Lines 110 (patched)


Can this cause problems if called after a failed tryAcquire? Since this 
method is not used anywhere outside this class, it might be a good idea to 
merge with close.


- Peter Vary


On szept. 26, 2018, 1:08 du, denys kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68683/
> ---
> 
> (Updated szept. 26, 2018, 1:08 du)
> 
> 
> Review request for hive, Zoltan Haindrich, Zoltan Haindrich, Naveen Gangam, 
> and Peter Vary.
> 
> 
> Bugs: HIVE-20535
> https://issues.apache.org/jira/browse/HIVE-20535
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> When removing the compile lock, it is quite risky to remove it entirely.
> 
> It would be good to provide a pool size for the concurrent compilation, so 
> the administrator can limit the load
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8c39de3e77 
>   ql/src/java/org/apache/hadoop/hive/ql/Driver.java 737debd2ad 
>   ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLock.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/lock/CompileLockFactory.java 
> PRE-CREATION 
>   ql/src/test/org/apache/hadoop/hive/ql/CompileLockTest.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68683/diff/7/
> 
> 
> Testing
> ---
> 
> Added CompileLockTest
> 
> 
> File Attachments
> 
> 
> HIVE-20535.1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/09/13/41f5a84a-70e5-4882-99c1-1cf98c4364e4__HIVE-20535.1.patch
> HIVE-20535.14.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/09/25/335b0f4b-ea94-41d4-881a-ec8bb870a376__HIVE-20535.14.patch
> HIVE-20535.14.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/09/25/a92b6da2-eeba-46ee-9409-162653826172__HIVE-20535.14.patch
> HIVE-20535.14.patch
>   
> https://reviews.apache.org/media/uploaded/files/2018/09/25/9db4cf76-9188-48fb-bd3d-5b28e43a791b__HIVE-20535.14.patch
> 
> 
> Thanks,
> 
> denys kuzmenko
> 
>

[jira] [Created] (HIVE-20649) LLAP aware memory manager for Orc writers

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-20649:


 Summary: LLAP aware memory manager for Orc writers
 Key: HIVE-20649
 URL: https://issues.apache.org/jira/browse/HIVE-20649
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


ORC writer has its own memory manager that assumes memory usage or memory 
available based on JVM heap (MemoryMX bean). This works on tez container mode 
execution model but not in LLAP where container sizes (and Xmx) are typically 
high and there are multiple executors per LLAP daemon. This custom memory 
manager should be aware of memory bounds per executor. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20648) LLAP: Vector group by operator should use memory per executor

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-20648:


 Summary: LLAP: Vector group by operator should use memory per 
executor
 Key: HIVE-20648
 URL: https://issues.apache.org/jira/browse/HIVE-20648
 Project: Hive
  Issue Type: Bug
Affects Versions: 4.0.0, 3.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


HIVE-15503 treatment has to be applied for vector group by operator as well. 
Vector group by currently uses MemoryMX bean to get heap usage and heap max 
memory which will not work for LLAP. Instead it should use memory per executor 
as upper bound to make flush decision.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 68828: HIVE-20601 : EnvironmentContext null in ALTER_PARTITION event in DbNotificationListener

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68828/#review209069
---


Ship it!




Ship It!

- Peter Vary


On szept. 24, 2018, 8:42 du, Bharathkrishna Guruvayoor Murali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68828/
> ---
> 
> (Updated szept. 24, 2018, 8:42 du)
> 
> 
> Review request for hive and Alexander Kolbasov.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> It will be useful to have the environmentContext passed to 
> DbNotificationListener in this case, to know if the alter happened due to a 
> stat change.
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
>  f52ff91a8f2e7710801dcadc4a83ce454992a66a 
> 
> 
> Diff: https://reviews.apache.org/r/68828/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bharathkrishna Guruvayoor Murali
> 
>

[GitHub] hive pull request #439: HIVE-20644: Avoid exposing sensitive infomation thro...

[ashutosh-bapat]

GitHub user ashutosh-bapat opened a pull request:

https://github.com/apache/hive/pull/439

HIVE-20644: Avoid exposing sensitive infomation through a Hive Runtime 
exception (Ashutosh Bapat)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ashutosh-bapat/hive hive20644

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/439.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #439


commit f5c4b22ebbfc4893b2aa436d9ea9b4241f04340b
Author: Ashutosh Bapat 
Date:   2018-09-27T05:51:52Z

HIVE-20644: Avoid exposing sensitive infomation through a Hive Runtime 
exception (Ashutosh Bapat)




---

[jira] [Created] (HIVE-20647) HadoopVer was ignored in QTestUtil

[denys kuzmenko (JIRA)]

denys kuzmenko created HIVE-20647:
-

 Summary: HadoopVer was ignored in QTestUtil
 Key: HIVE-20647
 URL: https://issues.apache.org/jira/browse/HIVE-20647
 Project: Hive
  Issue Type: Bug
  Components: Test
Reporter: denys kuzmenko
 Attachments: HIVE-20647.1.patch





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20646) Partition filter condition is not pushed down to metastore query if it has IS NOT NULL.

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-20646:
---

 Summary: Partition filter condition is not pushed down to 
metastore query if it has IS NOT NULL.
 Key: HIVE-20646
 URL: https://issues.apache.org/jira/browse/HIVE-20646
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2, Standalone Metastore
Affects Versions: 4.0.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan


If the partition filter condition has "is not null" then the filter query isn't 
getting pushed to the SQL query in RDMBS. 
This slows down metastore api calls for getting list of partitions with filter 
condition.

This condition gets added by optimizer in many cases so this is affecting many 
queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-20645) Vectorization: Implicit casting causes scratch vector reuse Wrong Results

[Matt McCline (JIRA)]

Matt McCline created HIVE-20645:
---

 Summary: Vectorization: Implicit casting causes scratch vector 
reuse Wrong Results
 Key: HIVE-20645
 URL: https://issues.apache.org/jira/browse/HIVE-20645
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Matt McCline
Assignee: Matt McCline


The bug fix in HIVE-20563 exposes a Wrong Results bug in vectorized_cast.q



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)