Re: Review Request 71932: HIVE-22652

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71932/#review219122
---




ql/src/test/queries/clientpositive/topnkey_grouping_sets.q
Lines 41 (patched)


Can we add more examples to this q file? In particular, I was thinking 
about queries:
- With a lower limit.
- With custom nulls order spec in order by.
- With order by with more than a single column and different order spec, 
e.g., order by a desc, b asc.
- With aggregate functions, e.g., sum, min, max.
- Without including all grouping sets in spec, e.g., ((a), (b)), ((a, b), 
()), rollup, cube.
- With more than a single group by in the query.


- Jesús Camacho Rodríguez


On Jan. 2, 2020, 8:02 a.m., Krisztian Kasa wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71932/
> ---
> 
> (Updated Jan. 2, 2020, 8:02 a.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-22652
> https://issues.apache.org/jira/browse/HIVE-22652
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-22652: TopNKey push through Group by with Grouping sets
> 
> Enable TNK op push down through Group by with Grouping sets by removing the 
> lines which checked whether the GBY operator has GROUPING SETS
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/topnkey/TopNKeyPushdownProcessor.java
>  c79c371a8b 
>   ql/src/test/queries/clientpositive/topnkey_grouping_sets.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/topnkey_grouping_sets.q.out 
> PRE-CREATION 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  c7e837905d 
>   ql/src/test/results/clientpositive/perf/tez/cbo_query14.q.out d1e8c3806e 
>   ql/src/test/results/clientpositive/perf/tez/cbo_query77.q.out aa080603e1 
>   ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query14.q.out 
> 59fcf951fe 
>   ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query77.q.out 
> 39da7ea903 
>   ql/src/test/results/clientpositive/perf/tez/constraints/query14.q.out 
> 65d3faa20f 
>   ql/src/test/results/clientpositive/perf/tez/constraints/query27.q.out 
> e1a48eaeea 
>   ql/src/test/results/clientpositive/perf/tez/constraints/query5.q.out 
> 13288d28b4 
>   ql/src/test/results/clientpositive/perf/tez/constraints/query77.q.out 
> c2758b7033 
>   ql/src/test/results/clientpositive/perf/tez/constraints/query80.q.out 
> 72a54928c2 
>   ql/src/test/results/clientpositive/perf/tez/query14.q.out 00bc4cb026 
>   ql/src/test/results/clientpositive/perf/tez/query27.q.out 774c0fd192 
>   ql/src/test/results/clientpositive/perf/tez/query5.q.out 03980ac2c0 
>   ql/src/test/results/clientpositive/perf/tez/query77.q.out fcfc5a33bc 
>   ql/src/test/results/clientpositive/perf/tez/query80.q.out 3020b58781 
>   ql/src/test/results/clientpositive/topnkey_grouping_sets.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71932/diff/4/
> 
> 
> Testing
> ---
> 
> - New q test: topnkey_grouping_sets.q
> - Run `src/test/queries/clientpositive/perf/query*.q` tests with 
> TestTezPerfCliDriver, TestTezPerfConstraintsCliDriver
> 
> 
> Thanks,
> 
> Krisztian Kasa
> 
>

Re: Review Request 71761: HIVE-22489

[Krisztian Kasa]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71761/
---

(Updated Jan. 3, 2020, 11:50 a.m.)


Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.


Bugs: HIVE-22489
https://issues.apache.org/jira/browse/HIVE-22489


Repository: hive-git


Description
---

Reduce Sink operator orders nulls first
===
1. Set the default null sort order by hive config when creating Reduce Sink 
Desc.
2. Hash join uses 
`org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite`
 or `BinarySortableDeserializeRead` for selializing keys. For bigtable keys 
always ascending and nulls first ordering was hardcoded. This patch changes 
this behaviour to use the `Operator.getConf().TableDesc.getProperties()` (in 
this case `MapJoinOperator`) to setup ordering in `BinarySortableSerializeWrite`
3. Use null ordering set in ReduceRecordSource at Reduce phase when comparing 
keys in `CommonMergeJoinOperator` (This is the null ordering of the children 
Reduce Sink operators)


Diffs (updated)
-

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out 7c552621f2 
  contrib/src/test/results/clientpositive/udaf_example_group_concat.q.out 
6846720d95 
  hbase-handler/src/test/results/positive/hbase_queries.q.out a32ef81a7b 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 e997fa65cf 
  kudu-handler/src/test/results/positive/kudu_complex_queries.q.out 73fc3e514f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 
3974627a24 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordSource.java 
72446afeda 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinCommonOperator.java
 2380d936f2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
 f587517b08 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
 cdee3fd957 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
 e5d9fdae19 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
 29c531bd51 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
 a4cda921a5 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
 43f093d906 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
 8dce5b82d3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
 a35401d9b2 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringCommon.java
 1b108a8c14 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
 446feb2526 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
 c28ef9be2b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
 17bd5fda93 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
 4ab8902a3f 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
 21c355cb42 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java
 de1ee15c3b 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java
 42573f0898 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java
 829a03737d 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java
 18e1435019 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java
 da0e8365b1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java
 6c4d8a81d1 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java
 a6b754c7eb 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
 fdcd83dde7 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
 5c409e4573 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
a50ad78e8f 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 0f95d7788c 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java 
89b55001f0 
  

Review Request 71948: HIVE-22535: ACID: Check for already running compactions in Worker

[Laszlo Pinter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71948/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-22535: ACID: Check for already running compactions in Worker


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
99da86f9102da4df8b3531a3f4dc7f5d0dbb8309 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
7a0e32463d28007cff5526ae037cc1447e50a50b 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 
5aff71e0e981c429f85663300d3e5c21089529a9 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 
70ae85c458eae080a02df965174bf1207d9eaf8b 


Diff: https://reviews.apache.org/r/71948/diff/1/


Testing
---

Created new unit test.


Thanks,

Laszlo Pinter

Re: HIVE-2.4 release plans

[Mass Dosage]

+1 for this, or for a Hive 2.3.7 release. We are blocked from releasing
some of our projects which use Hive 2.3.x on Java >8 due to
https://issues.apache.org/jira/browse/HIVE-21508 which we helped get merged
but it hasn't been released yet. Similarly we'd like to be able to use some
Parquet related functionality which didn't work but is now fixed via
https://issues.apache.org/jira/browse/HIVE-22249 and also merged and ready
to be released.

Thanks,

Adrian

On Wed, 11 Dec 2019 at 15:25, Oleksiy S 
wrote:

> Hi all.
>
> Are there any plans for Hive-2.4 release?
>
> --
> Oleksiy
>

[jira] [Created] (HIVE-22692) Use only fixDecimalDataTypePhysicalVariations when vectorizing TopNKey operator

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-22692:
-

 Summary: Use only fixDecimalDataTypePhysicalVariations when 
vectorizing TopNKey operator
 Key: HIVE-22692
 URL: https://issues.apache.org/jira/browse/HIVE-22692
 Project: Hive
  Issue Type: Task
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa


Currently both 'fixDecimalDataTypePhysicalVariations' and 
'getVectorExpressionsUpConvertDecimal64'  are called when vectorizing TopNKey 
operator in 'Vectorizer.java'
{code}
vContext.markActualScratchColumns();
try {
  List keyColumns = topNKeyDesc.getKeyColumns();

  keyExpressions = 
vContext.getVectorExpressionsUpConvertDecimal64(keyColumns);
  fixDecimalDataTypePhysicalVariations(vContext, keyExpressions);

} finally {
  vContext.freeMarkedScratchColumns();
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)