[jira] [Created] (HIVE-22289) Regenerate test output for tests broken due to commit race

2019-10-03 Thread John Sherman (Jira) 
John Sherman created HIVE-22289:
---

 Summary: Regenerate test output for tests broken due to commit race
 Key: HIVE-22289
 URL: https://issues.apache.org/jira/browse/HIVE-22289
 Project: Hive
  Issue Type: Task
Reporter: John Sherman
Assignee: John Sherman


HIVE-22042 got committed which changed the plans of a few tests (by enabling 
nonstrict partitioning mode by default) then HIVE-22269 got committed which 
fixes a bug with stats not being correctly calculated on some operators. Each 
patch got green runs individually but together causes test output differences.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22290) ObjectStore.cleanWriteNotificationEvents and ObjectStore.cleanupEvents OutOfMemory on large number of pending events

2019-10-03 Thread Thomas Prelle (Jira) 
Thomas Prelle created HIVE-22290:


 Summary: ObjectStore.cleanWriteNotificationEvents and 
ObjectStore.cleanupEvents OutOfMemory on large number of pending events
 Key: HIVE-22290
 URL: https://issues.apache.org/jira/browse/HIVE-22290
 Project: Hive
  Issue Type: Bug
  Components: HCatalog, repl
Affects Versions: 4.0.0
Reporter: Thomas Prelle


As in [https://jira.apache.org/jira/browse/HIVE-19430] if there are large 
number of events that haven't been cleaned up for some reason, then 
ObjectStore.cleanWriteNotificationEvents() and ObjectStore.cleanupEvents can 
run out of memory while it loads all the events to be deleted.
It should fetch events in batches.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71575: HIVE-22284: Improve LLAP CacheContentsTracker to collect and display correct statistics

2019-10-03 Thread Adam Szita via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71575/
---

(Updated Oct. 3, 2019, 3:20 p.m.)


Review request for hive.


Changes
---

Added test, fixed findbugs and checkstyle errors


Bugs: HIVE-22284
https://issues.apache.org/jira/browse/HIVE-22284


Repository: hive-git


Description
---

When keeping track of which buffers correspond to what Hive objects, 
CacheContentsTracker relies on cache tags.

Currently a tag is a simple String that ideally holds DB and table name, and a 
partition spec concatenated by . and / . The information here is derived from 
the Path of the file that is getting cached. Needless to say sometimes this 
produces a wrong tag especially for external tables.

Also there's a bug when calculating aggregated stats for a 'parent' tag 
(corresponding to the table of the partition) because the overall maxCount and 
maxSize do not add up to the sum of those in the partitions. This happens when 
buffers get removed from the cache.


Diffs (updated)
-

  llap-common/src/java/org/apache/hadoop/hive/llap/LlapUtil.java 
a351a193c6bc558bb420049c54b7657cd7d04b7c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/CacheContentsTracker.java
 64c0125833af100fd7012b9751d075ab536ad1b0 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapCacheableBuffer.java 
f91a5d91a5b739dcbee98a1485ad4c59f6a9057b 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LlapDataBuffer.java 
405fca2d4fae9fe0e3fd6d6d1345d55255d6df78 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCache.java 
4dd3826a67dfff66ce9c90027d61a9012c0a15e8 
  llap-server/src/java/org/apache/hadoop/hive/llap/cache/LowLevelCacheImpl.java 
62d7e5534486b53634de332875c5fd5d336c29b4 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 2a39d2d32807a51346baad28b04d87670381b6d5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SimpleBufferManager.java 
41855e171eaa5bf8da638bc62bce3d0d49dc4bae 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
c63ee5f79b4f9fc356f033960e0af1a7b0058038 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 1378a01f44ef774a15f769460833064c6305b2d6 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 2a0c5ca92f3c7431f3c399f309a538f47eb27597 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 85a42f945624c3ca468790772f52363b4064d8fc 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 d414b1405b7672767196b3eaad02baa516169288 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/MetadataCache.java 
8400fe98411ed07bd525a51a223fc35423136efb 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcFileEstimateErrors.java
 30dc1b9da2002689b8b1917f46ae3ca24194f3be 
  
llap-server/src/test/org/apache/hadoop/hive/llap/cache/TestCacheContentsTracker.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/llap/LlapCacheAwareFs.java 
af04a51b5536550b2d2f7d3e008cf2b2dea607d4 
  ql/src/java/org/apache/hadoop/hive/llap/LlapHiveUtils.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/EncodedReaderImpl.java 
241a3001e6e0002377736d6d0e820fde004b0bac 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/Reader.java 
210c987b7f580dacda5bdb487af9cf234a738b79 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/encoded/ReaderImpl.java 
a9a9f101948a970e0dbf2f77eeb6f688a88d1cbd 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java
 61e2556b08fe4247f35673f24378505ada20a605 
  storage-api/src/java/org/apache/hadoop/hive/common/io/CacheTag.java 
PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/common/io/DataCache.java 
2ac0a18a5026e76e65c3c3a8b81d5a844c472ed2 
  storage-api/src/java/org/apache/hadoop/hive/common/io/FileMetadataCache.java 
d7de3619380d24a1aeea2bac9a66485d7d468517 


Diff: https://reviews.apache.org/r/71575/diff/2/

Changes: https://reviews.apache.org/r/71575/diff/1-2/


Testing
---


Thanks,

Adam Szita

[jira] [Created] (HIVE-22291) HMS Translation: Limit translation to hive default catalog only

2019-10-03 Thread Naveen Gangam (Jira) 
Naveen Gangam created HIVE-22291:


 Summary: HMS Translation: Limit translation to hive default 
catalog only
 Key: HIVE-22291
 URL: https://issues.apache.org/jira/browse/HIVE-22291
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


HMS Translation should only be limited to a single catalog.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions

2019-10-03 Thread Krisztian Kasa (Jira) 
Krisztian Kasa created HIVE-22292:
-

 Summary: Implement Hypothetical-Set Aggregate Functions
 Key: HIVE-22292
 URL: https://issues.apache.org/jira/browse/HIVE-22292
 Project: Hive
  Issue Type: Improvement
  Components: UDF
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa
 Fix For: 4.0.0


{code}
 ::=
   
   
  

 ::=
  RANK
  | DENSE_RANK
  | PERCENT_RANK
  | CUME_DIST
{code}
Example:
{code}
CREATE TABLE table1 (column1 int);
INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL), 
(NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
{code}
{code}
SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
{code}
{code}
2
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)