[jira] [Created] (HIVE-22725) Lazy evaluate HiveMetastore::fireReadTablePreEvent table computation

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-22725:
---

 Summary: Lazy evaluate HiveMetastore::fireReadTablePreEvent table 
computation
 Key: HIVE-22725
 URL: https://issues.apache.org/jira/browse/HIVE-22725
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


"TransactionalValidationListener" gets added in the pre-event listeners of HMS 
by default. 

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L559]

This causes issue in short select queries, as table details are computed for 
any partition lookups.

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L4984]

It would be good to lazy evaluate table lookup in this codepath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22724) ObjectStore: Reduce number of DB calls

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-22724:
---

 Summary: ObjectStore: Reduce number of DB calls
 Key: HIVE-22724
 URL: https://issues.apache.org/jira/browse/HIVE-22724
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22723) SHOW CREATE TABLE omits backtics, results in syntax error

[Alex Yankelevich (Jira)]

Alex Yankelevich created HIVE-22723:
---

 Summary: SHOW CREATE TABLE omits backtics, results in syntax error
 Key: HIVE-22723
 URL: https://issues.apache.org/jira/browse/HIVE-22723
 Project: Hive
  Issue Type: Bug
  Components: Hive
 Environment: Run SHOW CREATE TABLE via AWS Athena
Reporter: Alex Yankelevich


CREATE EXTERNAL TABLE `alerts1`( `_id` string COMMENT 'from deserializer', 
`org` string COMMENT 'from deserializer', `collector` string COMMENT 'from 
deserializer', `source` string COMMENT 'from deserializer', `entitytype` string 
COMMENT 'from deserializer', `entity` string COMMENT 'from deserializer', 
`rule` string COMMENT 'from deserializer', `findingdescid` string COMMENT 'from 
deserializer', `details` 
struct>,findingtype:string,producttype:string,entityname:string,extra:struct,entityparams:struct,version:string,build:string>>
 COMMENT 'from deserializer', `seenat` array> 
COMMENT 'from deserializer', `execution` array> 
COMMENT 'from deserializer', `lastmodified` struct<__numberint:string> COMMENT 
'from deserializer') ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' WITH 
SERDEPROPERTIES ( 
'paths'='_id,collector,details,entity,entityType,execution,findingDescId,lastModified,org,rule,seenAt,source')
 STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 
's3://redshift-poc-ay/alerts' TBLPROPERTIES ( 
'CrawlerSchemaDeserializerVersion'='1.0', 
'CrawlerSchemaSerializerVersion'='1.0', 
'UPDATED_BY_CRAWLER'='redshift-poc-alerts-ay', 'averageRecordSize'='9619', 
'classification'='json', 'compressionType'='none', 'objectCount'='1', 
'recordCount'='21028', 'sizeKey'='202270843', 
'transient_lastDdlTime'='1578946186', 'typeOfData'='file')    __numberint, 
__numberlong columns are not generated enclosed in backticks even though they 
were explicitly created with them.  Therefore, the generated DDL is 
syntactically incorrect. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: HMS Hive database schema+upgrade scripts

[Alan Gates]

The hive directory isn't for installing new instances of HMS, since the HMS
can't use Hive itself as its database.  It's for the information schema
support.  So when I moved the HMS schemas over into standalone-metastore I
didn't move the hive directory, as it served a different purpose.  I agree
it would be better to make that clear in the source tree.

Alan.

On Mon, Jan 13, 2020 at 1:51 AM Zoltan Chovan 
wrote:

> Hi all,
>
> I've recently done some work on the Hive database schema/upgrade files and
> found the following:
>
>- there are two locations where we store the schema/upgrade files:
>   - hive/metastore/scripts/upgrade (old)
>   - hive/standalone-metastore/metastore-server/src/main/sql (new)
>- the old location includes the 'hive' folder and the hive database
>schema (this schema includes the sys and infromation_schema databases)
>- the new location is missing this entirely
>- looking at the packaging target directory the hive schema makes it's
>way in there as the directory
>
>  
> 'hive/packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade'
>includes the hive folder with the schema and upgrade scripts
>
>
> With all this I'm a bit confused why the hive folder is not in the
> standalone-metastore module. Could someone shed some light on this?
>
> Thanks,
> Zoltan
>

Re: Review Request 71888: HIVE-22568: Process compaction candidates in parallel by the Initiator

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71888/#review219239
---


Ship it!




+1 (non-binding)

- Laszlo Pinter


On Dec. 6, 2019, 12:54 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71888/
> ---
> 
> (Updated Dec. 6, 2019, 12:54 p.m.)
> 
> 
> Review request for hive, Laszlo Pinter and Peter Vary.
> 
> 
> Bugs: HIVE-22568
> https://issues.apache.org/jira/browse/HIVE-22568
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> `checkForCompaction` includes many file metadata checks and may be expensive. 
> Therefore, make sense using a thread pool here and running 
> `checkForCompactions` in parallel.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 4393a2825e 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7a0e32463d 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> 564839324f 
> 
> 
> Diff: https://reviews.apache.org/r/71888/diff/1/
> 
> 
> Testing
> ---
> 
> unit test
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 71988: HIVE-22703: Compaction configuration check when starting HMS/HS2

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71988/#review219237
---



LGTM, just minor comments


service/src/java/org/apache/hive/service/server/HiveServer2.java
Lines 437 (patched)


do we have constant for "hs2" literal?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
Lines 10135 (patched)


small nit: you can use static import to reduce log command length (i.e. use 
LOG instead of HMSHandler.LOG)


- Denys Kuzmenko


On Jan. 13, 2020, 2:12 p.m., Laszlo Pinter wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71988/
> ---
> 
> (Updated Jan. 13, 2020, 2:12 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22703: Compaction configuration check when starting HMS/HS2
> 
> 
> Diffs
> -
> 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 
> a93cc1b7e1aadf6e2724d667b6e4c9c9ecc38a75 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  94698e6771890a050b00b374ca0ee926f768aa0e 
> 
> 
> Diff: https://reviews.apache.org/r/71988/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Laszlo Pinter
> 
>

Review Request 71988: HIVE-22703: Compaction configuration check when starting HMS/HS2

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71988/
---

Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary.


Repository: hive-git


Description
---

HIVE-22703: Compaction configuration check when starting HMS/HS2


Diffs
-

  service/src/java/org/apache/hive/service/server/HiveServer2.java 
a93cc1b7e1aadf6e2724d667b6e4c9c9ecc38a75 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 94698e6771890a050b00b374ca0ee926f768aa0e 


Diff: https://reviews.apache.org/r/71988/diff/1/


Testing
---


Thanks,

Laszlo Pinter

Re: Review Request 71761: HIVE-22489

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71761/
---

(Updated Jan. 13, 2020, 1:32 p.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-22489
https://issues.apache.org/jira/browse/HIVE-22489


Repository: hive-git


Description
---

Reduce Sink operator orders nulls first
===
1. Set the default null sort order by hive config when creating Reduce Sink 
Desc.
2. Hash join uses 
`org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite`
 or `BinarySortableDeserializeRead` for selializing keys. For bigtable keys 
always ascending and nulls first ordering was hardcoded. This patch changes 
this behaviour to use the `Operator.getConf().TableDesc.getProperties()` (in 
this case `MapJoinOperator`) to setup ordering in `BinarySortableSerializeWrite`
3. Use null ordering set in ReduceRecordSource at Reduce phase when comparing 
keys in `CommonMergeJoinOperator` (This is the null ordering of the children 
Reduce Sink operators)


Diffs (updated)
-

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out 7c552621f2 
  contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out 
6846720d95 
  hbase-handler/src/test/results/positive/hbase_queries.q.out a32ef81a7b 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 e997fa65cf 
  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 73fc3e514f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 
3974627a24 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
72446afeda 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
 2380d936f2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
 f587517b08 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
 cdee3fd957 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
 e5d9fdae19 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
 29c531bd51 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
 a4cda921a5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
 43f093d906 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
 8dce5b82d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 a35401d9b2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringCommon.java
 1b108a8c14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
 446feb2526 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
 c28ef9be2b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
 17bd5fda93 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
 4ab8902a3f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
 21c355cb42 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java
 de1ee15c3b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java
 42573f0898 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java
 829a03737d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java
 18e1435019 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java
 da0e8365b1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java
 6c4d8a81d1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java
 a6b754c7eb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
 fdcd83dde7 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 5c409e4573 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
a50ad78e8f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 0f95d7788c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
89b55001f0 
  
ql/src/java/org/apache/hadoop/hi

[jira] [Created] (HIVE-22722) timestamptz_2 test failure

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-22722:
---

 Summary: timestamptz_2 test failure
 Key: HIVE-22722
 URL: https://issues.apache.org/jira/browse/HIVE-22722
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


the min/max value seems to be off in some cases; this was highly non 
deterministic; and hard to reproduce - but in the recent QA runs it started 
failing more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22721) Add option for queries to only read from LLAP cache

[Jira]

Ádám Szita created HIVE-22721:
-

 Summary: Add option for queries to only read from LLAP cache
 Key: HIVE-22721
 URL: https://issues.apache.org/jira/browse/HIVE-22721
 Project: Hive
  Issue Type: Test
Reporter: Ádám Szita
Assignee: Ádám Szita


Testing features of LLAP cache sometimes requires to validate if e.g. a 
particular table/partition is cached, or not.

This is to avoid relying on counters that are dependent on the underlying (ORC) 
file format (which may produce different number of bytes among its different 
versions).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

HMS Hive database schema+upgrade scripts

[Zoltan Chovan]

Hi all,

I've recently done some work on the Hive database schema/upgrade files and
found the following:

   - there are two locations where we store the schema/upgrade files:
  - hive/metastore/scripts/upgrade (old)
  - hive/standalone-metastore/metastore-server/src/main/sql (new)
   - the old location includes the 'hive' folder and the hive database
   schema (this schema includes the sys and infromation_schema databases)
   - the new location is missing this entirely
   - looking at the packaging target directory the hive schema makes it's
   way in there as the directory
   
'hive/packaging/target/apache-hive-4.0.0-SNAPSHOT-bin/apache-hive-4.0.0-SNAPSHOT-bin/scripts/metastore/upgrade'
   includes the hive folder with the schema and upgrade scripts


With all this I'm a bit confused why the hive folder is not in the
standalone-metastore module. Could someone shed some light on this?

Thanks,
Zoltan

Review Request 71984: HIVE-22666

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71984/
---

Review request for hive and Jesús Camacho Rodríguez.


Bugs: HIVE-22666
https://issues.apache.org/jira/browse/HIVE-22666


Repository: hive-git


Description
---

implement supporting partitioning in TopNKeyOperator
enable push down of partitioned TopNKeyOperator


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/TopNKeyOperator.java b7c1250220 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/CommonKeyPrefix.java 
049466050b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyProcessor.java 
ce6efa4919 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
 348fbb5faf 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TopNKeyDesc.java a9e2a527df 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java 38f66f978c 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestTopNKeyFilter.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/topnkey/TestCommonKeyPrefix.java
 51d2382278 
  ql/src/test/queries/clientpositive/topnkey_windowing.q PRE-CREATION 
  ql/src/test/queries/clientpositive/topnkey_windowing_order.q PRE-CREATION 
  ql/src/test/queries/clientpositive/windowing_filter.q 2483c18416 
  ql/src/test/results/clientpositive/llap/topnkey.q.out 3150eefc7d 
  ql/src/test/results/clientpositive/llap/topnkey_grouping_sets.q.out 
b1ec8d8847 
  ql/src/test/results/clientpositive/llap/topnkey_grouping_sets_functions.q.out 
0ec6e6de43 
  ql/src/test/results/clientpositive/llap/topnkey_grouping_sets_order.q.out 
d3ff1a6256 
  ql/src/test/results/clientpositive/llap/topnkey_windowing.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/topnkey_windowing_order.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/windowing_filter.q.out 78240b58b3 
  ql/src/test/results/clientpositive/perf/tez/constraints/query49.q.out 
37fc51697b 
  ql/src/test/results/clientpositive/perf/tez/constraints/query67.q.out 
9ed7306c46 
  ql/src/test/results/clientpositive/perf/tez/constraints/query70.q.out 
40dfaa2f13 
  ql/src/test/results/clientpositive/perf/tez/query49.q.out 239592bc1d 
  ql/src/test/results/clientpositive/perf/tez/query67.q.out 54d76e9c1a 
  ql/src/test/results/clientpositive/perf/tez/query70.q.out 23f9166a4f 


Diff: https://reviews.apache.org/r/71984/diff/1/


Testing
---

New q test were added
topnkey_windowing.q,topnkey_windowing_order.q

New UT:
TestTopNKeyFilter.java

existing:
windowing_filter.q,topnkey.q,topnkey_grouping_sets.q,topnkey_grouping_sets_functions.q,topnkey_grouping_sets_order.q

existing UT:
TestCommonKeyPrefix


Thanks,

Krisztian Kasa