date:20210806

[hive] branch master updated: HIVE-25170: Fix wrong colExprMap generated by SemanticAnalyzer (#2331) (Wei Zhang reviewed by Zoltan Haindrich)

2021-08-06 Thread kgyrtkirk

This is an automated email from the ASF dual-hosted git repository.

kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new d22fa4a  HIVE-25170: Fix wrong colExprMap generated by 
SemanticAnalyzer (#2331) (Wei Zhang reviewed by Zoltan Haindrich)
d22fa4a is described below

commit d22fa4af5db01a99611790ed10cfddcab327823f
Author: Wei Zhang 
AuthorDate: Fri Aug 6 21:27:34 2021 +0800

HIVE-25170: Fix wrong colExprMap generated by SemanticAnalyzer (#2331) (Wei 
Zhang reviewed by Zoltan Haindrich)
---
 .../hadoop/hive/ql/parse/SemanticAnalyzer.java |   2 +-
 .../queries/clientpositive/constant_key_column.q   |  29 
 .../clientpositive/llap/constant_key_column.q.out  | 175 +
 3 files changed, 205 insertions(+), 1 deletion(-)

diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index a33d0fe..fdb6cc4 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -9021,7 +9021,7 @@ public class SemanticAnalyzer extends 
BaseSemanticAnalyzer {
 
 List keyColNames = rsdesc.getOutputKeyColumnNames();
 for (int i = 0 ; i < keyColNames.size(); i++) {
-  colExprMap.put(Utilities.ReduceField.KEY + "." + keyColNames.get(i), 
sortCols.get(i));
+  colExprMap.put(Utilities.ReduceField.KEY + "." + keyColNames.get(i), 
newSortCols.get(i));
 }
 List valueColNames = rsdesc.getOutputValueColumnNames();
 for (int i = 0 ; i < valueColNames.size(); i++) {
diff --git a/ql/src/test/queries/clientpositive/constant_key_column.q 
b/ql/src/test/queries/clientpositive/constant_key_column.q
new file mode 100644
index 000..28d3a16
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/constant_key_column.q
@@ -0,0 +1,29 @@
+--! qt:dataset:src
+SET hive.remove.orderby.in.subquery=false;
+
+-- SORT_QUERY_RESULTS
+
+EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+  SELECT 'constant' as constant_col, key, value
+  FROM src
+  DISTRIBUTE BY constant_col, key
+  SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10;
+
+SELECT constant_col, key, max(value)
+FROM
+(
+  SELECT 'constant' as constant_col, key, value
+  FROM src
+  DISTRIBUTE BY constant_col, key
+  SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10;
diff --git a/ql/src/test/results/clientpositive/llap/constant_key_column.q.out 
b/ql/src/test/results/clientpositive/llap/constant_key_column.q.out
new file mode 100644
index 000..a1a843d
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/constant_key_column.q.out
@@ -0,0 +1,175 @@
+PREHOOK: query: EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+  SELECT 'constant' as constant_col, key, value
+  FROM src
+  DISTRIBUTE BY constant_col, key
+  SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+ A masked pattern was here 
+POSTHOOK: query: EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+  SELECT 'constant' as constant_col, key, value
+  FROM src
+  DISTRIBUTE BY constant_col, key
+  SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+ A masked pattern was here 
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Tez
+ A masked pattern was here 
+  Edges:
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: src
+  Statistics: Num rows: 500 Data size: 89000 Basic stats: 
COMPLETE Column stats: COMPLETE
+  Select Operator
+expressions: key (type: string), value (type: string)
+outputColumnNames: _col1, _col2
+Statistics: Num rows: 500 Data size: 89000 Basic stats: 
COMPLETE Column stats: COMPLETE
+Reduce Output Operator
+  key expressions: _col1 (type: string), _col2 (type: 
string)
+  null sort order: zz
+  sort order: ++
+  Map-reduce partition columns: 'constant' (type: string), 
_col1 (type: string)
+  Statistics: Num rows: 500 Data size: 89000 Basic stats: 
COMPLETE Column stats: COMPLETE
+Execution mode: vectorized, llap
+LLAP IO: all inputs
+Reducer 2

[hive] branch master updated: HIVE-24902: Incorrect result after fold CASE into COALESCE (#2100) (Nemon Lou reviewed by Zoltan Haindrich)

2021-08-06 Thread kgyrtkirk

This is an automated email from the ASF dual-hosted git repository.

kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 69331f2  HIVE-24902: Incorrect result after fold CASE into COALESCE 
(#2100) (Nemon Lou reviewed by Zoltan Haindrich)
69331f2 is described below

commit 69331f2a26da639c6c011adc1bac8ce3b054fa8c
Author: loudongfeng 
AuthorDate: Fri Aug 6 21:23:19 2021 +0800

HIVE-24902: Incorrect result after fold CASE into COALESCE (#2100) (Nemon 
Lou reviewed by Zoltan Haindrich)
---
 .../ql/parse/type/ExprNodeDescExprFactory.java |  3 +-
 .../clientpositive/constant_prop_coalesce.q| 30 +++
 .../llap/constant_prop_coalesce.q.out  | 95 ++
 3 files changed, 127 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
index aad2cfe..a5e6ad5 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
@@ -809,7 +809,8 @@ public class ExprNodeDescExprFactory extends 
ExprFactory {
   Object thenVal = constThen.getValue();
   Object elseVal = constElse.getValue();
   if (thenVal instanceof Boolean && elseVal instanceof Boolean) {
-return true;
+//only convert to COALESCE when both branches are valid
+return !thenVal.equals(elseVal);
   }
 }
 return false;
diff --git a/ql/src/test/queries/clientpositive/constant_prop_coalesce.q 
b/ql/src/test/queries/clientpositive/constant_prop_coalesce.q
new file mode 100644
index 000..f81b4b9
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/constant_prop_coalesce.q
@@ -0,0 +1,30 @@
+explain
+select * from (
+select
+   case when b.a=1
+   then
+   cast (from_unixtime(unix_timestamp(cast(20210309 as 
string),'MMdd') - 86400,'MMdd') as bigint)
+   else
+   20210309
+  end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null;
+
+select * from (
+select
+   case when b.a=1
+   then
+   cast (from_unixtime(unix_timestamp(cast(20210309 as 
string),'MMdd') - 86400,'MMdd') as bigint)
+   else
+   20210309
+  end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null;
\ No newline at end of file
diff --git 
a/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out 
b/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out
new file mode 100644
index 000..d34fcb6
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out
@@ -0,0 +1,95 @@
+PREHOOK: query: explain
+select * from (
+select
+   case when b.a=1
+   then
+   cast (from_unixtime(unix_timestamp(cast(20210309 as 
string),'MMdd') - 86400,'MMdd') as bigint)
+   else
+   20210309
+  end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here 
+POSTHOOK: query: explain
+select * from (
+select
+   case when b.a=1
+   then
+   cast (from_unixtime(unix_timestamp(cast(20210309 as 
string),'MMdd') - 86400,'MMdd') as bigint)
+   else
+   20210309
+  end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here 
+STAGE DEPENDENCIES:
+  Stage-0 is a root stage
+
+STAGE PLANS:
+  Stage: Stage-0
+Fetch Operator
+  limit: -1
+  Processor Tree:
+TableScan
+  alias: _dummy_table
+  Row Limit Per Split: 1
+  Select Operator
+expressions: 2 (type: int), 1 (type: int), 2 (type: int)
+outputColumnNames: _col0, _col1, _col2
+UDTF Operator
+  function name: stack
+  Filter Operator
+predicate: CASE WHEN ((col0 = 1)) THEN (true) ELSE (true) END 
(type: boolean)
+Select Operator
+  expressions: CASE WHEN ((col0 = 1)) THEN (20210308L) ELSE 
(20210309L) END (type: bigint)
+  outputColumnNames: _col0
+  ListSink
+
+PREHOOK: query: select * from (
+select
+   case when b.a=1
+   then
+   cast (from_unixtime(unix_timestamp(cast(20210309 as 
string),'MMdd') - 86400,'MMdd') as bigint)
+   else
+   20210309
+

[hive] branch master updated (799d62d -> de33be2)

2021-08-06 Thread zabetak

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.


from 799d62d  HIVE-25394: Enable vectorization for TestIcebergCliDriver 
dynamic_partition_pruning.q (Adam Szita, reviewed by Peter Vary)
 add de33be2  HIVE-25364: NPE when estimating row count in external JDBC 
tables (Soumyakanti Das, reviewed by Stamatis Zampetakis)

No new revisions were added by this update.

Summary of changes:
 .../calcite/stats/HiveRelMdDistinctRowCount.java   |  28 +-
 .../clientpositive/external_jdbc_rowcount.q| 110 
 .../llap/external_jdbc_rowcount.q.out  | 291 +
 .../llap/external_jdbc_table_perf.q.out| 215 ---
 4 files changed, 528 insertions(+), 116 deletions(-)
 create mode 100644 ql/src/test/queries/clientpositive/external_jdbc_rowcount.q
 create mode 100644 
ql/src/test/results/clientpositive/llap/external_jdbc_rowcount.q.out

[hive] branch master updated: HIVE-25394: Enable vectorization for TestIcebergCliDriver dynamic_partition_pruning.q (Adam Szita, reviewed by Peter Vary)

2021-08-06 Thread szita

This is an automated email from the ASF dual-hosted git repository.

szita pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 799d62d  HIVE-25394: Enable vectorization for TestIcebergCliDriver 
dynamic_partition_pruning.q (Adam Szita, reviewed by Peter Vary)
799d62d is described below

commit 799d62d62643002d83986c02441f721bcfb1cc59
Author: Adam Szita <40628386+sz...@users.noreply.github.com>
AuthorDate: Fri Aug 6 13:34:22 2021 +0200

HIVE-25394: Enable vectorization for TestIcebergCliDriver 
dynamic_partition_pruning.q (Adam Szita, reviewed by Peter Vary)
---
 .../queries/positive/dynamic_partition_pruning.q   |  2 +-
 .../positive/dynamic_partition_pruning.q.out   | 96 ++
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git 
a/iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q 
b/iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q
index 28832da..8b469e2 100644
--- 
a/iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q
+++ 
b/iceberg/iceberg-handler/src/test/queries/positive/dynamic_partition_pruning.q
@@ -1,5 +1,5 @@
 --! qt:dataset:srcpart
-SET hive.vectorized.execution.enabled=false;
+SET hive.vectorized.execution.enabled=true;
 set hive.compute.query.using.stats=false;
 set hive.mapred.mode=nonstrict;
 set hive.explain.user=false;
diff --git 
a/iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
 
b/iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
index dc4ed7a..816b988 100644
--- 
a/iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
+++ 
b/iceberg/iceberg-handler/src/test/results/positive/dynamic_partition_pruning.q.out
@@ -115,7 +115,9 @@ STAGE PLANS:
 sort order: +
 Map-reduce partition columns: _col0 (type: string)
 Statistics: Num rows: 2 Data size: 368 Basic stats: 
COMPLETE Column stats: COMPLETE
+Execution mode: vectorized
 Reducer 2 
+Execution mode: vectorized
 Reduce Operator Tree:
   Group By Operator
 keys: KEY._col0 (type: string)
@@ -150,6 +152,7 @@ STAGE PLANS:
 Statistics: Num rows: 1 Data size: 472 Basic stats: 
COMPLETE Column stats: COMPLETE
 value expressions: _col0 (type: int), _col1 (type: 
struct), _col2 (type: bigint), _col3 (type: 
bigint), _col4 (type: binary), _col5 (type: int), _col6 (type: 
struct), _col7 (type: bigint), _col8 (type: 
binary)
 Reducer 3 
+Execution mode: vectorized
 Reduce Operator Tree:
   Group By Operator
 aggregations: max(VALUE._col0), avg(VALUE._col1), 
count(VALUE._col2), count(VALUE._col3), compute_bit_vector_hll(VALUE._col4), 
max(VALUE._col5), avg(VALUE._col6), count(VALUE._col7), 
compute_bit_vector_hll(VALUE._col8)
@@ -317,6 +320,7 @@ STAGE PLANS:
 sort order: +
 Map-reduce partition columns: _col0 (type: string)
 Statistics: Num rows: 2000 Data size: 188000 Basic 
stats: COMPLETE Column stats: COMPLETE
+Execution mode: vectorized
 Map 4 
 Map Operator Tree:
 TableScan
@@ -352,6 +356,7 @@ STAGE PLANS:
 Partition key expr: ds
 Statistics: Num rows: 1 Data size: 94 Basic stats: 
COMPLETE Column stats: COMPLETE
 Target Vertex: Map 1
+Execution mode: vectorized
 Reducer 2 
 Reduce Operator Tree:
   Merge Join Operator
@@ -373,6 +378,7 @@ STAGE PLANS:
 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: COMPLETE
 value expressions: _col0 (type: bigint)
 Reducer 3 
+Execution mode: vectorized
 Reduce Operator Tree:
   Group By Operator
 aggregations: count(VALUE._col0)
@@ -446,6 +452,7 @@ STAGE PLANS:
 sort order: +
 Map-reduce partition columns: _col0 (type: string)
 Statistics: Num rows: 2000 Data size: 188000 Basic 
stats: COMPLETE Column stats: COMPLETE
+Execution mode: vectorized
 Map 4 
 Map Operator Tree:
 TableScan
@@ -465,6 +472,7 @@ STAGE PLANS:
 sort order: +
 Map-reduce partition columns: _col0 (type: string)
 Statistics: Num rows: 1 Data size: 94 Basic stats: 
COMPLETE Column stats: COMPLETE
+Execution mode: vectorized
 Reducer 2 
 Reduce Operator Tree:
   Merge Join

[hive] branch master updated (2f46592 -> a97057f)

2021-08-06 Thread szita

This is an automated email from the ASF dual-hosted git repository.

szita pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.


from 2f46592  HIVE-25341. Reduce FileSystem calls in case drop database 
cascade. (#2491)
 add a97057f  HIVE-25422: Break up TestHiveIcebergStorageHandlerWithEngine 
test (Adam Szita, reviewed by Peter Vary and Marton Bod)

No new revisions were added by this update.

Summary of changes:
 .../HiveIcebergStorageHandlerWithEngineBase.java   |  198 ++
 .../iceberg/mr/hive/TestHiveIcebergCTAS.java   |  109 +
 .../mr/hive/TestHiveIcebergComplexTypeWrites.java  |  227 ++
 .../iceberg/mr/hive/TestHiveIcebergInserts.java|  336 +++
 .../iceberg/mr/hive/TestHiveIcebergMigration.java  |  182 ++
 .../iceberg/mr/hive/TestHiveIcebergPartitions.java |  247 ++
 .../mr/hive/TestHiveIcebergSchemaEvolution.java|  753 ++
 .../iceberg/mr/hive/TestHiveIcebergSelects.java|  207 ++
 .../iceberg/mr/hive/TestHiveIcebergStatistics.java |  171 ++
 .../TestHiveIcebergStorageHandlerWithEngine.java   | 2552 
 .../iceberg/mr/hive/TestHiveIcebergTimeTravel.java |  182 ++
 .../mr/hive/TestHiveIcebergTruncateTable.java  |  223 ++
 .../iceberg/mr/hive/TestHiveIcebergTypes.java  |  163 ++
 13 files changed, 2998 insertions(+), 2552 deletions(-)
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandlerWithEngineBase.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergCTAS.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergComplexTypeWrites.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergInserts.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergMigration.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergPartitions.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergSchemaEvolution.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergSelects.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStatistics.java
 delete mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergTimeTravel.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergTruncateTable.java
 create mode 100644 
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergTypes.java

[hive] branch master updated: HIVE-25170: Fix wrong colExprMap generated by SemanticAnalyzer (#2331) (Wei Zhang reviewed by Zoltan Haindrich)

[hive] branch master updated: HIVE-24902: Incorrect result after fold CASE into COALESCE (#2100) (Nemon Lou reviewed by Zoltan Haindrich)

[hive] branch master updated (799d62d -> de33be2)

[hive] branch master updated: HIVE-25394: Enable vectorization for TestIcebergCliDriver dynamic_partition_pruning.q (Adam Szita, reviewed by Peter Vary)

[hive] branch master updated (2f46592 -> a97057f)

5 matches

Site Navigation

Mail list logo

Footer information