-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71761/
-----------------------------------------------------------
(Updated Nov. 29, 2019, 3:03 p.m.)
Review request for hive, Jesús Camacho Rodríguez and Zoltan Haindrich.
Bugs: HIVE-22489
https://issues.apache.org/jira/browse/HIVE-22489
Repository: hive-git
Description (updated)
-------
Reduce Sink operator orders nulls first
=======================================
1. Set the default null sort order by hive config when creating Reduce Sink
Desc.
2. Hash join uses
`org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableSerializeWrite`
for selializing keys. For bigtable keys always ascending and nulls first
ordering was hardcoded. This patch changes this behaviour to use the
`Operator.getConf().TableDesc.getProperties()` (in this case `MapJoinOperator`)
to setup ordering in `BinarySortableSerializeWrite`
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerBigOnlyMultiKeyOperator.java
f587517b08
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinInnerMultiKeyOperator.java
cdee3fd957
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinLeftSemiMultiKeyOperator.java
e5d9fdae19
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/VectorMapJoinOuterMultiKeyOperator.java
29c531bd51
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
21c355cb42
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongCommon.java
de1ee15c3b
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMap.java
42573f0898
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashMultiSet.java
829a03737d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedLongHashSet.java
18e1435019
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringCommon.java
da0e8365b1
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMap.java
6c4d8a81d1
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashMultiSet.java
a6b754c7eb
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
fdcd83dde7
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
5c409e4573
ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java
a50ad78e8f
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
0f95d7788c
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
89b55001f0
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/opconventer/HiveGBOpConvUtil.java
46ddffd4fa
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/opconventer/HiveOpConverterUtils.java
9cc1712f45
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkReduceSinkMapJoinProc.java
ac5caa6135
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 60bfba826d
ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 2314f49631
ql/src/java/org/apache/hadoop/hive/ql/util/NullOrdering.java 46ff329981
ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java dd70524948
ql/src/test/queries/clientpositive/hashjoin.q PRE-CREATION
ql/src/test/results/clientpositive/autoColumnStats_5a.q.out 9e2606f7d9
ql/src/test/results/clientpositive/autoColumnStats_8.q.out 90039f828e
ql/src/test/results/clientpositive/auto_join_reordering_values.q.out
d9c7720da5
ql/src/test/results/clientpositive/bucket3.q.out d418750071
ql/src/test/results/clientpositive/cbo_rp_outer_join_ppr.q.out 605f1aec22
ql/src/test/results/clientpositive/columnstats_partlvl.q.out 3e2557455b
ql/src/test/results/clientpositive/cte_1.q.out 8a621cf872
ql/src/test/results/clientpositive/distinct_groupby.q.out d9e3cf9eff
ql/src/test/results/clientpositive/filter_aggr.q.out a4fe9405f9
ql/src/test/results/clientpositive/filter_join_breaktask.q.out ec5d4a14e1
ql/src/test/results/clientpositive/filter_union.q.out d2c167df7a
ql/src/test/results/clientpositive/groupby_grouping_sets_limit.q.out
d0eaf46d86
ql/src/test/results/clientpositive/groupby_map_ppr.q.out 66988ee04a
ql/src/test/results/clientpositive/groupby_map_ppr_multi_distinct.q.out
9f2a587ada
ql/src/test/results/clientpositive/groupby_ppr.q.out d84c649e2b
ql/src/test/results/clientpositive/groupby_ppr_multi_distinct.q.out
db358d9a53
ql/src/test/results/clientpositive/groupby_rollup_empty.q.out 1fcdf15976
ql/src/test/results/clientpositive/groupby_sort_1_23.q.out 13074dd1de
ql/src/test/results/clientpositive/groupby_sort_6.q.out eee2ffa8e8
ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 96e4b9d140
ql/src/test/results/clientpositive/groupingset_high_columns.q.out 3456719f7e
ql/src/test/results/clientpositive/infer_bucket_sort_grouping_operators.q.out
8fa79f708b
ql/src/test/results/clientpositive/join17.q.out 57c702e5de
ql/src/test/results/clientpositive/join35.q.out 5f91f28ad6
ql/src/test/results/clientpositive/join45.q.out 59276b6fdb
ql/src/test/results/clientpositive/join46.q.out 23604d8647
ql/src/test/results/clientpositive/join47.q.out 5e7d32cf3b
ql/src/test/results/clientpositive/join9.q.out be4ec4fb46
ql/src/test/results/clientpositive/join_emit_interval.q.out e8ca3e2ac7
ql/src/test/results/clientpositive/join_filters_overlap.q.out 4eab694c2a
ql/src/test/results/clientpositive/list_bucket_dml_1.q.out 3e05a866ea
ql/src/test/results/clientpositive/list_bucket_dml_11.q.out a5ffb418a2
ql/src/test/results/clientpositive/list_bucket_dml_12.q.out 045b81db71
ql/src/test/results/clientpositive/list_bucket_dml_13.q.out 6d1ae67339
ql/src/test/results/clientpositive/list_bucket_dml_2.q.out a8d414b371
ql/src/test/results/clientpositive/list_bucket_dml_3.q.out be21d65844
ql/src/test/results/clientpositive/list_bucket_dml_4.q.out f9e1c29ea5
ql/src/test/results/clientpositive/list_bucket_dml_5.q.out 43db28e8c4
ql/src/test/results/clientpositive/list_bucket_dml_6.q.out 40a110d0fb
ql/src/test/results/clientpositive/list_bucket_dml_7.q.out cdcbbf5bf4
ql/src/test/results/clientpositive/list_bucket_dml_8.q.out 29e45dc64a
ql/src/test/results/clientpositive/list_bucket_dml_9.q.out a95245fd2b
ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out
c398393b88
ql/src/test/results/clientpositive/llap/hashjoin.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
99cab932e5
ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out
7927cac9ba
ql/src/test/results/clientpositive/load_dyn_part8.q.out 4e34848601
ql/src/test/results/clientpositive/localtimezone.q.out d63309000e
ql/src/test/results/clientpositive/louter_join_ppr.q.out a898f7ed52
ql/src/test/results/clientpositive/merge3.q.out 7b3e3b6470
ql/src/test/results/clientpositive/outer_join_ppr.q.out 8bc0e4ba1c
ql/src/test/results/clientpositive/parquet_types_non_dictionary_encoding_vectorization.q.out
3299739f60
ql/src/test/results/clientpositive/parquet_vectorization_0.q.out d5a98db6dd
ql/src/test/results/clientpositive/parquet_vectorization_limit.q.out
61657ab9d9
ql/src/test/results/clientpositive/pcr.q.out 54a9585598
ql/src/test/results/clientpositive/pcs.q.out 661c09c329
ql/src/test/results/clientpositive/pointlookup2.q.out 6ea1a05242
ql/src/test/results/clientpositive/pointlookup3.q.out dfa625b23d
ql/src/test/results/clientpositive/ppd_join_filter.q.out 243f9bc2b8
ql/src/test/results/clientpositive/ppd_vc.q.out c863001400
ql/src/test/results/clientpositive/router_join_ppr.q.out 0d99feb6d7
ql/src/test/results/clientpositive/sample8.q.out 710897bffd
ql/src/test/results/clientpositive/setop_no_distinct.q.out 31ca7d4338
ql/src/test/results/clientpositive/smb_mapjoin_11.q.out b1fa4c1cbb
ql/src/test/results/clientpositive/smb_mapjoin_12.q.out 5ddded0c88
ql/src/test/results/clientpositive/smb_mapjoin_46.q.out 994805ca27
ql/src/test/results/clientpositive/smb_mapjoin_47.q.out 0cb4125a92
ql/src/test/results/clientpositive/subquery_notin_having.q.out aa1486a188
ql/src/test/results/clientpositive/test_teradatabinaryfile.q.out a6ba774f26
ql/src/test/results/clientpositive/timestamp.q.out 446e3263b8
ql/src/test/results/clientpositive/udf_explode.q.out bc76a2b741
ql/src/test/results/clientpositive/udf_width_bucket.q.out d1a208954f
ql/src/test/results/clientpositive/udtf_explode.q.out 955e08cdea
ql/src/test/results/clientpositive/union22.q.out 963408ebba
ql/src/test/results/clientpositive/union24.q.out feca9da9b9
ql/src/test/results/clientpositive/vector_grouping_sets.q.out 563304a248
ql/src/test/results/clientpositive/vectorization_sum_if_when.q.out 155d377997
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java
f58fb72524
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableUtils.java
PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/fast/BinarySortableSerializeWrite.java
b1ee7ecf14
Diff: https://reviews.apache.org/r/71761/diff/2/
Testing
-------
Run tests:
- TestExecDriver.java
- order_null.q
- sample10.q
- vector_char_2.q
- vector_order_null.q
- vector_windowing_gby2.q
Thanks,
Krisztian Kasa
