[hive] branch master updated: HIVE-27431: Clean invalid properties in test module (#4408). (zhangbutao, reviewed by Denys Kuzmenko, Ayush Saxena)
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 12f7d5ea894 HIVE-27431: Clean invalid properties in test module
(#4408). (zhangbutao, reviewed by Denys Kuzmenko, Ayush Saxena)
12f7d5ea894 is described below
commit 12f7d5ea8941383e4ba73205919312a44133cbf5
Author: Butao Zhang
AuthorDate: Wed Aug 23 23:07:16 2023 +0800
HIVE-27431: Clean invalid properties in test module (#4408). (zhangbutao,
reviewed by Denys Kuzmenko, Ayush Saxena)
---
data/conf/hive-site-old.xml| 8
data/conf/hive-site.xml| 56 --
data/conf/iceberg/llap/hive-site.xml | 8
data/conf/iceberg/tez/hive-site.xml| 8
data/conf/llap/hive-site.xml | 20
data/conf/rlist/hive-site.xml | 48 ---
data/conf/tez/hive-site.xml| 8
.../src/test/resources/hive-site.xml | 13 -
8 files changed, 169 deletions(-)
diff --git a/data/conf/hive-site-old.xml b/data/conf/hive-site-old.xml
index 4e6ff161358..33c4f29306b 100644
--- a/data/conf/hive-site-old.xml
+++ b/data/conf/hive-site-old.xml
@@ -80,14 +80,6 @@
-
- hive.metastore.metadb.dir
- file://${build.dir}/test/data/metadb/
-
- Required by metastore server or if the uris argument below is not supplied
-
-
-
test.log.dir
${build.dir}/test/logs
diff --git a/data/conf/hive-site.xml b/data/conf/hive-site.xml
index e8a3798749e..b4ef1479a53 100644
--- a/data/conf/hive-site.xml
+++ b/data/conf/hive-site.xml
@@ -19,24 +19,12 @@
-
-hive.metastore.client.cache.maxSize
-10Mb
-Max size for hive metastore client local cache
-
-
hive.metastore.client.cache.enabled
true
This property enables a Caffeiene Cache for Metastore
client
-
-hive.metastore.client.cache.recordStats
-true
-This property enables recording metastore client cache stats
in DEBUG logs
-
-
hive.in.test
true
@@ -124,14 +112,6 @@
-
- hive.metastore.metadb.dir
- file://${test.tmp.dir}/metadb/
-
- Required by metastore server or if the uris argument below is not supplied
-
-
-
test.log.dir
${test.tmp.dir}/log/
@@ -226,16 +206,6 @@
Whether Hive ignores the mapjoin hint
-
- hive.mapjoin.max.gc.time.percentage
- 0.99
-
- Maximum percentage of wallclock time that the JVM can spend in GC.
- If this limit is exceeded, the local task will abort by itself.
- Tests may run in very stressed environment, so this number is set very high
to avoid false negatives.
-
-
-
hive.input.format
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
@@ -248,11 +218,6 @@
The default SerDe hive will use for the rcfile
format
-
- hive.stats.key.prefix.reserve.length
- 0
-
-
hive.conf.restricted.list
dummy.config.value
@@ -310,27 +275,6 @@
The Hive client authorization manager class name.
-
- hive.llap.io.cache.orc.size
- 8388608
-
-
-
- hive.llap.io.cache.orc.arena.size
- 8388608
-
-
-
- hive.llap.io.cache.orc.alloc.max
- 2097152
-
-
-
-
- hive.llap.io.cache.orc.alloc.min
- 32768
-
-
hive.llap.cache.allow.synthetic.fileid
true
diff --git a/data/conf/iceberg/llap/hive-site.xml
b/data/conf/iceberg/llap/hive-site.xml
index b01bd8e9f05..57982980abf 100644
--- a/data/conf/iceberg/llap/hive-site.xml
+++ b/data/conf/iceberg/llap/hive-site.xml
@@ -110,14 +110,6 @@
-
-hive.metastore.metadb.dir
-file://${test.tmp.dir}/metadb/
-
-Required by metastore server or if the uris argument below is not
supplied
-
-
-
test.log.dir
${test.tmp.dir}/log/
diff --git a/data/conf/iceberg/tez/hive-site.xml
b/data/conf/iceberg/tez/hive-site.xml
index 3154477563e..0dc31fed5ed 100644
--- a/data/conf/iceberg/tez/hive-site.xml
+++ b/data/conf/iceberg/tez/hive-site.xml
@@ -110,14 +110,6 @@
-
- hive.metastore.metadb.dir
- file://${test.tmp.dir}/metadb/
-
- Required by metastore server or if the uris argument below is not supplied
-
-
-
test.log.dir
${test.tmp.dir}/log/
diff --git a/data/conf/llap/hive-site.xml b/data/conf/llap/hive-site.xml
index dbd5d28042b..27609b7a2be 100644
--- a/data/conf/llap/hive-site.xml
+++ b/data/conf/llap/hive-site.xml
@@ -19,24 +19,12 @@
-
-hive.metastore.client.cache.maxSize
-10Mb
-Max size for hive metastore client local cache
-
-
hive.metastore.client.cache.enabled
true
This property enables a Caffeiene Cache for Metastore
client
-
-hive.metastore.client.cache.recordStats
-true
-This property enables recording metastore client cache stats
in DEBUG logs
-
-
hive.in.test
true
[hive] branch branch-3 updated: HIVE-27617: Backport of HIVE-18284: Fix NPE when inserting data with 'distribute by' clause with dynpart sort optimization
This is an automated email from the ASF dual-hosted git repository.
sankarh pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new a4194d7c4ba HIVE-27617: Backport of HIVE-18284: Fix NPE when inserting
data with 'distribute by' clause with dynpart sort optimization
a4194d7c4ba is described below
commit a4194d7c4baa6a498ce39012504580c40a637744
Author: Aman Raj <104416558+amanraj2...@users.noreply.github.com>
AuthorDate: Wed Aug 23 16:37:00 2023 +0530
HIVE-27617: Backport of HIVE-18284: Fix NPE when inserting data with
'distribute by' clause with dynpart sort optimization
Signed-off-by: Sankar Hariappan
Closes (#4597)
---
.../test/resources/testconfiguration.properties| 1 +
.../correlation/ReduceSinkDeDuplicationUtils.java | 19 ++
.../dynpart_sort_optimization_distribute_by.q | 26 ++
.../dynpart_sort_optimization_distribute_by.q.out | 245 ++
.../dynpart_sort_optimization_distribute_by.q.out | 279 +
.../dynpart_sort_optimization_distribute_by.q.out | 260 +++
6 files changed, 830 insertions(+)
diff --git a/itests/src/test/resources/testconfiguration.properties
b/itests/src/test/resources/testconfiguration.properties
index 4145b500574..a5bce33d74f 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -47,6 +47,7 @@ minitez.query.files.shared=delete_orig_table.q,\
topnkey.q,\
update_orig_table.q,\
vector_join_part_col_char.q,\
+ dynpart_sort_optimization_distribute_by.q,\
vector_non_string_partition.q,\
vector_topnkey.q
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
index 7ccd4a3725f..5b832b53f7b 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplicationUtils.java
@@ -17,10 +17,12 @@
*/
package org.apache.hadoop.hive.ql.optimizer.correlation;
+import java.lang.reflect.Field;
import java.util.ArrayList;
import java.util.List;
import java.util.Map.Entry;
+import com.google.common.collect.Lists;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.ql.exec.JoinOperator;
@@ -192,6 +194,23 @@ public class ReduceSinkDeDuplicationUtils {
TableDesc keyTable = PlanUtils.getReduceKeyTableDesc(new
ArrayList(), pRS
.getConf().getOrder(), pRS.getConf().getNullOrder());
pRS.getConf().setKeySerializeInfo(keyTable);
+ } else if (cRS.getConf().getKeyCols() != null &&
cRS.getConf().getKeyCols().size() > 0) {
+ArrayList keyColNames = Lists.newArrayList();
+for (ExprNodeDesc keyCol : pRS.getConf().getKeyCols()) {
+ String keyColName = keyCol.getExprString();
+ keyColNames.add(keyColName);
+}
+List fields =
PlanUtils.getFieldSchemasFromColumnList(pRS.getConf().getKeyCols(),
+keyColNames, 0, "");
+TableDesc keyTable = PlanUtils.getReduceKeyTableDesc(fields,
pRS.getConf().getOrder(),
+pRS.getConf().getNullOrder());
+ArrayList outputKeyCols = Lists.newArrayList();
+for (int i = 0; i < fields.size(); i++) {
+ outputKeyCols.add(fields.get(i).getName());
+}
+pRS.getConf().setOutputKeyColumnNames(outputKeyCols);
+pRS.getConf().setKeySerializeInfo(keyTable);
+
pRS.getConf().setNumDistributionKeys(cRS.getConf().getNumDistributionKeys());
}
}
return true;
diff --git
a/ql/src/test/queries/clientpositive/dynpart_sort_optimization_distribute_by.q
b/ql/src/test/queries/clientpositive/dynpart_sort_optimization_distribute_by.q
new file mode 100644
index 000..81ac82c2082
--- /dev/null
+++
b/ql/src/test/queries/clientpositive/dynpart_sort_optimization_distribute_by.q
@@ -0,0 +1,26 @@
+set hive.exec.dynamic.partition.mode=nonstrict;
+set hive.optimize.sort.dynamic.partition=true;
+set hive.vectorized.execution.enabled=false;
+
+
+create table table1 (col1 string, datekey int);
+insert into table1 values ('ROW1', 1), ('ROW2', 2), ('ROW3', 1);
+create table table2 (col1 string) partitioned by (datekey int);
+
+explain extended insert into table table2
+PARTITION(datekey)
+select col1,
+datekey
+from table1
+distribute by datekey;
+
+
+insert into table table2
+PARTITION(datekey)
+select col1,
+datekey
+from table1
+distribute by datekey;
+
+select * from table1;
+select * from table2;
\ No newline at end of file
diff --git
a/ql/src/test/results/clientpositive/dynpart_sort_optimization_distribute_by.q.out
b/ql/src/test/results/clientpositive/dynpart_sor
[hive] branch branch-3 updated: HIVE-27613: Backport of HIVE-22204: Beeline option to show/not show execution report (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
This is an automated email from the ASF dual-hosted git repository.
sankarh pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 5cbf98b0f93 HIVE-27613: Backport of HIVE-22204: Beeline option to
show/not show execution report (Jesus Camacho Rodriguez, reviewed by Ashutosh
Chauhan)
5cbf98b0f93 is described below
commit 5cbf98b0f93a4d07a9dae8cd370783c4ee411525
Author: Aman Raj <104416558+amanraj2...@users.noreply.github.com>
AuthorDate: Wed Aug 23 14:52:32 2023 +0530
HIVE-27613: Backport of HIVE-22204: Beeline option to show/not show
execution report (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
Signed-off-by: Sankar Hariappan
Closes (#4592)
---
.../java/org/apache/hive/beeline/BeeLineOpts.java | 9 +++
.../src/java/org/apache/hive/beeline/Commands.java | 20
beeline/src/main/resources/BeeLine.properties | 1 +
.../apache/hive/beeline/TestBeelineArgParsing.java | 28 ++
.../apache/hive/beeline/TestBeeLineWithArgs.java | 21
5 files changed, 75 insertions(+), 4 deletions(-)
diff --git a/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
b/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
index 3877b5cc30f..5967b4d7bc0 100644
--- a/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
+++ b/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
@@ -68,6 +68,7 @@ public class BeeLineOpts implements Completer {
private final BeeLine beeLine;
private boolean autosave = false;
private boolean silent = false;
+ private Boolean report = null;
private boolean color = false;
private boolean showHeader = true;
private boolean escapeCRLF = false;
@@ -569,6 +570,14 @@ public class BeeLineOpts implements Completer {
return silent;
}
+ public void setReport(boolean report) {
+this.report = report;
+ }
+
+ public Boolean isReport() {
+return report;
+ }
+
public void setAutosave(boolean autosave) {
this.autosave = autosave;
}
diff --git a/beeline/src/java/org/apache/hive/beeline/Commands.java
b/beeline/src/java/org/apache/hive/beeline/Commands.java
index f14564a81ac..425ea8b53c5 100644
--- a/beeline/src/java/org/apache/hive/beeline/Commands.java
+++ b/beeline/src/java/org/apache/hive/beeline/Commands.java
@@ -1026,8 +1026,10 @@ public class Commands {
int count = beeLine.print(rs);
long end = System.currentTimeMillis();
- beeLine.info(
- beeLine.loc("rows-selected", count) + " " +
beeLine.locElapsedTime(end - start));
+ if (showReport()) {
+beeLine.output(beeLine.loc("rows-selected", count) + " " +
beeLine.locElapsedTime(end - start),
+true, beeLine.getErrorStream());
+ }
} finally {
if (logThread != null) {
logThread.join(DEFAULT_QUERY_PROGRESS_THREAD_TIMEOUT);
@@ -1043,8 +1045,11 @@ public class Commands {
} else {
int count = stmnt.getUpdateCount();
long end = System.currentTimeMillis();
- beeLine.info(
- beeLine.loc("rows-affected", count) + " " +
beeLine.locElapsedTime(end - start));
+
+ if (showReport()) {
+beeLine.output(beeLine.loc("rows-affected", count) + " " +
beeLine.locElapsedTime(end - start),
+true, beeLine.getErrorStream());
+ }
}
} finally {
if (logThread != null) {
@@ -1068,6 +1073,13 @@ public class Commands {
return true;
}
+ private boolean showReport() {
+if (beeLine.getOpts().isReport() != null) {
+ return beeLine.getOpts().isReport();
+}
+return !beeLine.getOpts().isSilent();
+ }
+
/*
* Check if the input line is a multi-line command which needs to read
further
*/
diff --git a/beeline/src/main/resources/BeeLine.properties
b/beeline/src/main/resources/BeeLine.properties
index c41b3ed637e..a4e342d089b 100644
--- a/beeline/src/main/resources/BeeLine.properties
+++ b/beeline/src/main/resources/BeeLine.properties
@@ -189,6 +189,7 @@ cmd-usage: Usage: java org.apache.hive.cli.beeline.BeeLine
\n \
\ --maxWidth=MAXWIDTH the maximum width of the terminal\n \
\ --maxColumnWidth=MAXCOLWIDTHthe maximum width to use when displaying
columns\n \
\ --silent=[true/false] be more silent\n \
+\ --report=[true/false] show number of rows and execution time
after query execution\n \
\ --autosave=[true/false] automatically save preferences\n \
\ --outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv] format mode for
result display\n \
\ Note that csv, and tsv are deprecated - use
csv2, tsv2 instead\n \
diff --git
a/beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
b/beeline/src/tes
[hive] branch branch-3 updated: HIVE-27624: Backport of HIVE-26080: Upgrade accumulo-core to 1.10.1 (Ashish Sharma, reviewed by Adesh Rao)
This is an automated email from the ASF dual-hosted git repository.
sankarh pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 48d54fa6bba HIVE-27624: Backport of HIVE-26080: Upgrade accumulo-core
to 1.10.1 (Ashish Sharma, reviewed by Adesh Rao)
48d54fa6bba is described below
commit 48d54fa6bba38979365c81673012560af06f31f5
Author: Aman Raj <104416558+amanraj2...@users.noreply.github.com>
AuthorDate: Wed Aug 23 13:23:43 2023 +0530
HIVE-27624: Backport of HIVE-26080: Upgrade accumulo-core to 1.10.1 (Ashish
Sharma, reviewed by Adesh Rao)
Signed-off-by: Sankar Hariappan
Closes (#4606)
---
.../hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java | 2 +-
pom.xml | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java
b/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java
index 544717c7ab2..56b8a337962 100644
---
a/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java
+++
b/accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/predicate/PrimitiveComparisonFilter.java
@@ -55,7 +55,7 @@ public class PrimitiveComparisonFilter extends
WholeRowIterator {
@SuppressWarnings("unused")
private static final Logger LOG =
LoggerFactory.getLogger(PrimitiveComparisonFilter.class);
- public static final String FILTER_PREFIX =
"accumulo.filter.compare.iterator.";
+ public static final String FILTER_PREFIX = "accumuloFilterCompareIterator";
public static final String P_COMPARE_CLASS =
"accumulo.filter.iterator.p.compare.class";
public static final String COMPARE_OPT_CLASS =
"accumulo.filter.iterator.compare.opt.class";
public static final String CONST_VAL = "accumulo.filter.iterator.const.val";
diff --git a/pom.xml b/pom.xml
index b66a73711a6..b24f90da574 100644
--- a/pom.xml
+++ b/pom.xml
@@ -116,7 +116,7 @@
2.13.0
-1.7.3
+1.10.1
1.10.9
3.5.2
1.5.7
