[hive] branch master updated (57d15cb42f2 -> 8b94142d713)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from 57d15cb42f2 HIVE-27010: Reduce compilation time (#4005) add 8b94142d713 HIVE-26757: Add sfs+ofs support (#3779) (Michael Smith reviewed by Laszlo Bodor and Zoltan Haindrich) No new revisions were added by this update. Summary of changes: ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java | 3 +++ .../main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem | 1 + 2 files changed, 4 insertions(+)
[hive] branch master updated: HIVE-25879: MetaStoreDirectSql test query should not query the whole DBS table (#3348) (Miklos Szurap reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 4ce96653713 HIVE-25879: MetaStoreDirectSql test query should not query
the whole DBS table (#3348) (Miklos Szurap reviewed by Zoltan Haindrich)
4ce96653713 is described below
commit 4ce96653713325570a83704d2e131e284acdfe65
Author: mszurap
AuthorDate: Tue Jun 14 15:15:18 2022 +0200
HIVE-25879: MetaStoreDirectSql test query should not query the whole DBS
table (#3348) (Miklos Szurap reviewed by Zoltan Haindrich)
---
.../java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java| 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
index d24128c9618..6be4c3f84a0 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
@@ -316,6 +316,7 @@ class MetaStoreDirectSql {
}
private boolean runTestQuery() {
+boolean doTrace = LOG.isDebugEnabled();
Transaction tx = pm.currentTransaction();
boolean doCommit = false;
if (!tx.isActive()) {
@@ -323,10 +324,12 @@ class MetaStoreDirectSql {
doCommit = true;
}
// Run a self-test query. If it doesn't work, we will self-disable. What a
PITA...
-String selfTestQuery = "select \"DB_ID\" from " + DBS + "";
+String selfTestQuery = "select \"DB_ID\" from " + DBS + " WHERE
\"DB_ID\"=1";
try (QueryWrapper query = new
QueryWrapper(pm.newQuery("javax.jdo.query.SQL", selfTestQuery))) {
prepareTxn();
+ long start = doTrace ? System.nanoTime() : 0;
query.execute();
+ MetastoreDirectSqlUtils.timingTrace(doTrace, selfTestQuery, start,
doTrace ? System.nanoTime() : 0);
return true;
} catch (Throwable t) {
doCommit = false;
[hive] branch master updated: HIVE-26184: COLLECT_SET with GROUP BY is very slow when some keys are highly skewed (#3253) (okumin reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new c4f6eb2f914 HIVE-26184: COLLECT_SET with GROUP BY is very slow when
some keys are highly skewed (#3253) (okumin reviewed by Zoltan Haindrich)
c4f6eb2f914 is described below
commit c4f6eb2f91478152c89070a7455df9a1b8980c75
Author: okumin
AuthorDate: Tue Jun 14 00:26:36 2022 +0900
HIVE-26184: COLLECT_SET with GROUP BY is very slow when some keys are
highly skewed (#3253) (okumin reviewed by Zoltan Haindrich)
---
.../udf/generic/GenericUDAFMkCollectionEvaluator.java | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
b/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
index b05023dd37a..cffc7f76510 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
@@ -95,11 +95,27 @@ public class GenericUDAFMkCollectionEvaluator extends
GenericUDAFEvaluator
throw new RuntimeException("Buffer type unknown");
}
}
+
+private void reset() {
+ if (bufferType == BufferType.LIST) {
+container.clear();
+ } else if (bufferType == BufferType.SET) {
+// Don't reuse a container because HashSet#clear can be very slow. The
operation takes O(N)
+// and N is the capacity of the internal hash table. The internal
capacity grows based on
+// the number of elements and it never shrinks. Thus, HashSet#clear
takes O(N) every time
+// once a skewed key appears.
+// In order to avoid too many resizing in average cases, we set the
initial capacity to the
+// number of elements of the previous aggregation.
+container = new LinkedHashSet<>(container.size());
+ } else {
+throw new RuntimeException("Buffer type unknown");
+ }
+}
}
@Override
public void reset(AggregationBuffer agg) throws HiveException {
-((MkArrayAggregationBuffer) agg).container.clear();
+((MkArrayAggregationBuffer) agg).reset();
}
@Override
[hive] branch master updated: HIVE-25733: Add check-spelling/check-spelling (#2809) (Josh Soref reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 0099b14aa6a HIVE-25733: Add check-spelling/check-spelling (#2809) (Josh Soref reviewed by Zoltan Haindrich) 0099b14aa6a is described below commit 0099b14aa6a50d4470b057e93a95a7391b74add7 Author: Josh Soref <2119212+jso...@users.noreply.github.com> AuthorDate: Mon Jun 13 11:05:41 2022 -0400 HIVE-25733: Add check-spelling/check-spelling (#2809) (Josh Soref reviewed by Zoltan Haindrich) --- .github/actions/spelling/README.md| 17 ++ .github/actions/spelling/advice.md| 25 ++ .github/actions/spelling/allow.txt| 0 .github/actions/spelling/excludes.txt | 57 + .github/actions/spelling/expect.txt | 449 ++ .github/actions/spelling/only.txt | 1 + .github/actions/spelling/patterns.txt | 38 +++ .github/actions/spelling/reject.txt | 7 + .github/workflows/spelling.yml| 69 ++ 9 files changed, 663 insertions(+) diff --git a/.github/actions/spelling/README.md b/.github/actions/spelling/README.md new file mode 100644 index 000..749294b33fb --- /dev/null +++ b/.github/actions/spelling/README.md @@ -0,0 +1,17 @@ +# check-spelling/check-spelling configuration + +File | Purpose | Format | Info +-|-|-|- + +[allow.txt](allow.txt) | Add words to the dictionary | one word per line (only letters and `'`s allowed) | [allow](https://github.com/check-spelling/check-spelling/wiki/Configuration#allow) +[reject.txt](reject.txt) | Remove words from the dictionary (after allow) | grep pattern matching whole dictionary words | [reject](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-reject) +[excludes.txt](excludes.txt) | Files to ignore entirely | perl regular expression | [excludes](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-excludes) +[only.txt](only.txt) | Only check matching files (applied after excludes) | perl regular expression | [only](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-only) +[patterns.txt](patterns.txt) | Patterns to ignore from checked lines | perl regular expression (order matters, first match wins) | [patterns](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-patterns) +[expect.txt](expect.txt) | Expected words that aren't in the dictionary | one word per line (sorted, alphabetically) | [expect](https://github.com/check-spelling/check-spelling/wiki/Configuration#expect) +[advice.md](advice.md) | Supplement for GitHub comment when unrecognized words are found | GitHub Markdown | [advice](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples%3A-advice) + +Note: you can replace any of these files with a directory by the same name (minus the suffix) +and then include multiple files inside that directory (with that suffix) to merge multiple files together. diff --git a/.github/actions/spelling/advice.md b/.github/actions/spelling/advice.md new file mode 100644 index 000..c83423a8ef6 --- /dev/null +++ b/.github/actions/spelling/advice.md @@ -0,0 +1,25 @@ + +If the flagged items do not appear to be text + +If items relate to a ... +* well-formed pattern. + + If you can write a [pattern](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples:-patterns) that would match it, + try adding it to the `patterns.txt` file. + + Patterns are Perl 5 Regular Expressions - you can [test]( +https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your lines. + + Note that patterns can't match multiline strings. + +* binary file. + + Please add a file path to the `excludes.txt` file matching the containing file. + + File paths are Perl 5 Regular Expressions - you can [test]( +https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your files. + + `^` refers to the file's path from the root of the repository, so `^README\.md$` would exclude [README.md]( +../tree/HEAD/README.md) (on whichever branch you're using). + + diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt new file mode 100644 index 000..e69de29bb2d diff --git a/.github/actions/spelling/excludes.txt b/.github/actions/spelling/excludes.txt new file mode 100644 index 000..f15a0e0c9ca --- /dev/null +++ b/.github/actions/spelling/excludes.txt @@ -0,0 +1,57 @@ +# See https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples:-excludes +(?:^|/)(?i)COPYRIGHT +(?:^|/)(?i)LICEN[CS]E +(?:^|/)package(?:-lock|)\.json$ +(?:^|/)vendor/ +ignore$ +LICENSE +\.avi$ +\.avro$ +\.bz2$ +\.deflate$ +\.eot$ +\.gif$ +\.gz$ +\.ico$ +\.jar$ +\.jceks$ +\.jks$ +\.jpe?g$ +\.jpeg$ +\.jpg$ +\.keep$ +
[hive] branch master updated: HIVE-26300: Upgraded Jackson bom version to 2.12.6.1+ to avoid CVE-2020-36518 (#3351) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich and Ayush Saxena)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 568ded4b22a HIVE-26300: Upgraded Jackson bom version to 2.12.6.1+ to avoid CVE-2020-36518 (#3351) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich and Ayush Saxena) 568ded4b22a is described below commit 568ded4b22a020f4d2d3567f15b287b25a3f2b71 Author: Sai Hemanth Gantasala <68923650+saihemanth-cloud...@users.noreply.github.com> AuthorDate: Thu Jun 9 15:46:56 2022 +0530 HIVE-26300: Upgraded Jackson bom version to 2.12.6.1+ to avoid CVE-2020-36518 (#3351) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich and Ayush Saxena) --- pom.xml | 2 +- standalone-metastore/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index fb004de50cf..1c25c659b2a 100644 --- a/pom.xml +++ b/pom.xml @@ -142,7 +142,7 @@ 4.5.13 4.4.13 2.4.0 -2.12.0 +2.12.7 2.3.4 2.3.1 0.3.2 diff --git a/standalone-metastore/pom.xml b/standalone-metastore/pom.xml index 4935d0ef3e3..394763327a4 100644 --- a/standalone-metastore/pom.xml +++ b/standalone-metastore/pom.xml @@ -77,7 +77,7 @@ 19.0 3.1.0 2.6.1 -2.12.0 +2.12.7 5.5.1 4.13 5.6.2
[hive] branch master updated: HIVE-26268: Upgrade Snappy to 1.1.8.4 (#3326) (Sylwester Lachiewicz reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 6b1a876c3df HIVE-26268: Upgrade Snappy to 1.1.8.4 (#3326) (Sylwester Lachiewicz reviewed by Zoltan Haindrich) 6b1a876c3df is described below commit 6b1a876c3df7db54553e95552bcc2c99eccf8ae2 Author: Sylwester Lachiewicz AuthorDate: Wed Jun 8 18:02:29 2022 +0200 HIVE-26268: Upgrade Snappy to 1.1.8.4 (#3326) (Sylwester Lachiewicz reviewed by Zoltan Haindrich) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7b19fcb0217..fb004de50cf 100644 --- a/pom.xml +++ b/pom.xml @@ -194,7 +194,7 @@ 0.10.1 2.2.0 1.1 -1.1.4 +1.1.8.4 1.4 2.3 2.12.2
[hive] branch master updated: disable flaky mysql metastore test
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 34b24d55ade disable flaky mysql metastore test
34b24d55ade is described below
commit 34b24d55ade393673424f077b69add43bad9f731
Author: Zoltan Haindrich
AuthorDate: Wed May 25 11:41:41 2022 +
disable flaky mysql metastore test
---
Jenkinsfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index ade004a6c3e..79052c64e42 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -257,7 +257,7 @@ fi
}
def branches = [:]
- for (def d in ['derby','postgres','mysql','oracle']) {
+ for (def d in ['derby','postgres',/*'mysql',*/'oracle']) {
def dbType=d
def splitName = "init@$dbType"
branches[splitName] = {
[hive] branch master updated: HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot query partition data after rename table (#3255) (Zoltan Haindrich reviewed by Saihemanth Gantasala)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 778ce817fb HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot
query partition data after rename table (#3255) (Zoltan Haindrich reviewed by
Saihemanth Gantasala)
778ce817fb is described below
commit 778ce817fb50ca6dc8896ebaa258e434964d7639
Author: Zoltan Haindrich
AuthorDate: Wed May 11 17:10:51 2022 +0200
HIVE-26158: TRANSLATED_TO_EXTERNAL partition tables cannot query partition
data after rename table (#3255) (Zoltan Haindrich reviewed by Saihemanth
Gantasala)
---
.../clientpositive/translated_external_rename4.q | 21
.../llap/translated_external_rename4.q.out | 126 +
.../hive/metastore/utils/MetaStoreUtils.java | 22 +++-
.../hadoop/hive/metastore/HiveAlterHandler.java| 39 ---
.../metastore/MetastoreDefaultTransformer.java | 3 -
5 files changed, 189 insertions(+), 22 deletions(-)
diff --git a/ql/src/test/queries/clientpositive/translated_external_rename4.q
b/ql/src/test/queries/clientpositive/translated_external_rename4.q
new file mode 100644
index 00..30768d1298
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/translated_external_rename4.q
@@ -0,0 +1,21 @@
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set metastore.metadata.transformer.location.mode=prohibit;
+
+set hive.fetch.task.conversion=none;
+set hive.compute.query.using.stats=false;
+
+set hive.create.as.external.legacy=true;
+
+CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string);
+
+insert into part_test values ("11","th","20220101");
+insert into part_test values ("22","th","20220102");
+
+alter table part_test rename to part_test11;
+
+
+desc formatted part_test11;
+desc formatted part_test11 partition(dat="20220101");
diff --git
a/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
b/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
new file mode 100644
index 00..67c73b49a9
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/translated_external_rename4.q.out
@@ -0,0 +1,126 @@
+PREHOOK: query: CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@part_test
+POSTHOOK: query: CREATE TABLE part_test(
+c1 string
+,c2 string
+)PARTITIONED BY (dat string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@part_test
+PREHOOK: query: insert into part_test values ("11","th","20220101")
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_test
+POSTHOOK: query: insert into part_test values ("11","th","20220101")
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test@dat=20220101
+POSTHOOK: Lineage: part_test PARTITION(dat=20220101).c1 SCRIPT []
+POSTHOOK: Lineage: part_test PARTITION(dat=20220101).c2 SCRIPT []
+PREHOOK: query: insert into part_test values ("22","th","20220102")
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@part_test
+POSTHOOK: query: insert into part_test values ("22","th","20220102")
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test@dat=20220102
+POSTHOOK: Lineage: part_test PARTITION(dat=20220102).c1 SCRIPT []
+POSTHOOK: Lineage: part_test PARTITION(dat=20220102).c2 SCRIPT []
+PREHOOK: query: alter table part_test rename to part_test11
+PREHOOK: type: ALTERTABLE_RENAME
+PREHOOK: Input: default@part_test
+PREHOOK: Output: default@part_test
+POSTHOOK: query: alter table part_test rename to part_test11
+POSTHOOK: type: ALTERTABLE_RENAME
+POSTHOOK: Input: default@part_test
+POSTHOOK: Output: default@part_test
+POSTHOOK: Output: default@part_test11
+PREHOOK: query: desc formatted part_test11
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@part_test11
+POSTHOOK: query: desc formatted part_test11
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@part_test11
+# col_name data_type comment
+c1 string
+c2 string
+
+# Partition Information
+# col_name data_type comment
+datstring
[hive] branch master updated: HIVE-26135: Invalid Anti join conversion may cause missing results (#3205) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 59d462ad3e0 HIVE-26135: Invalid Anti join conversion may cause missing
results (#3205) (Zoltan Haindrich reviewed by Krisztian Kasa)
59d462ad3e0 is described below
commit 59d462ad3e023352ffbd57b4f2446e497a421252
Author: Zoltan Haindrich
AuthorDate: Tue Apr 26 17:22:49 2022 +0200
HIVE-26135: Invalid Anti join conversion may cause missing results (#3205)
(Zoltan Haindrich reviewed by Krisztian Kasa)
---
.../hive/ql/optimizer/calcite/HiveCalciteUtil.java | 18 +-
.../calcite/rules/HiveAntiSemiJoinRule.java| 23 +-
.../queries/clientpositive/antijoin_conversion.q | 22 ++
.../clientpositive/llap/antijoin_conversion.q.out | 280 +
4 files changed, 339 insertions(+), 4 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
index d925f159fba..160bfb86f6c 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java
@@ -630,7 +630,7 @@ public class HiveCalciteUtil {
}
};
- public static ImmutableList getPredsNotPushedAlready(RelNode inp,
List predsToPushDown) {
+ public static ImmutableList getPredsNotPushedAlready(RelNode inp,
List predsToPushDown) {
return getPredsNotPushedAlready(Sets.newHashSet(), inp,
predsToPushDown);
}
@@ -1238,6 +1238,22 @@ public class HiveCalciteUtil {
return false;
}
+ public static boolean hasAllExpressionsFromRightSide(RelNode joinRel,
List expressions) {
+List joinFields = joinRel.getRowType().getFieldList();
+int nTotalFields = joinFields.size();
+List leftFields =
(joinRel.getInputs().get(0)).getRowType().getFieldList();
+int nFieldsLeft = leftFields.size();
+ImmutableBitSet rightBitmap = ImmutableBitSet.range(nFieldsLeft,
nTotalFields);
+
+for (RexNode node : expressions) {
+ ImmutableBitSet inputBits = RelOptUtil.InputFinder.bits(node);
+ if (!rightBitmap.contains(inputBits)) {
+return false;
+ }
+}
+return true;
+ }
+
/**
* Extracts inputs referenced by aggregate operator.
*/
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAntiSemiJoinRule.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAntiSemiJoinRule.java
index 14a64c3d75c..3697ec2c4aa 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAntiSemiJoinRule.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAntiSemiJoinRule.java
@@ -20,12 +20,15 @@ package org.apache.hadoop.hive.ql.optimizer.calcite.rules;
import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.Strong;
import org.apache.calcite.rel.RelNode;
import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.Join;
import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexCall;
import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexVisitorImpl;
import org.apache.calcite.sql.SqlKind;
import org.apache.calcite.sql.fun.SqlStdOperatorTable;
import org.apache.hadoop.hive.ql.optimizer.calcite.HiveCalciteUtil;
@@ -36,8 +39,7 @@ import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
+import java.util.concurrent.atomic.AtomicBoolean;
/**
* Planner rule that converts a join plus filter to anti join.
@@ -136,7 +138,8 @@ public class HiveAntiSemiJoinRule extends RelOptRule {
for (RexNode filterNode : aboveFilters) {
if (filterNode.getKind() == SqlKind.IS_NULL) {
// Null filter from right side table can be removed and its a
pre-condition for anti join conversion.
-if (HiveCalciteUtil.hasAnyExpressionFromRightSide(join,
Collections.singletonList(filterNode))) {
+if (HiveCalciteUtil.hasAllExpressionsFromRightSide(join,
Collections.singletonList(filterNode))
+&& isStrong(((RexCall) filterNode).getOperands().get(0))) {
hasNullFilterOnRightSide = true;
} else {
filterList.add(filterNode);
@@ -157,4 +160,18 @@ public class HiveAntiSemiJoinRule extends RelOptRule {
}
return filterList;
}
+
+ private boolean isStrong(RexNode rexNode) {
+AtomicBoolean hasCast = new Atomi
[hive] branch master updated: disable flaky mapjoin_memcheck
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new ea935a0dd1 disable flaky mapjoin_memcheck ea935a0dd1 is described below commit ea935a0dd101118af10c6378093e3338a3120ff6 Author: Zoltan Haindrich AuthorDate: Tue Apr 12 17:28:25 2022 + disable flaky mapjoin_memcheck --- ql/src/test/queries/clientpositive/mapjoin_memcheck.q | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ql/src/test/queries/clientpositive/mapjoin_memcheck.q b/ql/src/test/queries/clientpositive/mapjoin_memcheck.q index 11cf2cbfc9..efac989c95 100644 --- a/ql/src/test/queries/clientpositive/mapjoin_memcheck.q +++ b/ql/src/test/queries/clientpositive/mapjoin_memcheck.q @@ -1,5 +1,6 @@ --! qt:dataset:src1 --! qt:dataset:src +--! qt:disabled:HIVE-26138 Fix mapjoin_memcheck set hive.mapred.mode=nonstrict; set hive.auto.convert.join = true; @@ -18,4 +19,4 @@ from src0 src1 inner join src0 src2 on src1.key = src2.key; select src1.key as k1, src1.value as v1, src2.key, src2.value from src0 src1 inner join src0 src2 on src1.key = src2.key; -drop table src0; \ No newline at end of file +drop table src0;
[hive] branch master updated: HIVE-25977: Enhance Compaction Cleaner to skip when there is nothing to do #2 (#2971) (Zoltan Haindrich reviewed by Karen Coppage and Denys Kuzmenko)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new d3542e1 HIVE-25977: Enhance Compaction Cleaner to skip when there is
nothing to do #2 (#2971) (Zoltan Haindrich reviewed by Karen Coppage and Denys
Kuzmenko)
d3542e1 is described below
commit d3542e1b35bbdbaafb52ad742a5168bd29549cee
Author: Zoltan Haindrich
AuthorDate: Wed Mar 9 13:04:11 2022 +0100
HIVE-25977: Enhance Compaction Cleaner to skip when there is nothing to do
#2 (#2971) (Zoltan Haindrich reviewed by Karen Coppage and Denys Kuzmenko)
---
.../txn/compactor/TestCleanerWithReplication.java | 4 +-
.../hadoop/hive/ql/txn/compactor/Cleaner.java | 38 ++---
.../hive/ql/txn/compactor/CompactorTest.java | 33 ++--
.../hadoop/hive/ql/txn/compactor/TestCleaner.java | 95 ++
4 files changed, 148 insertions(+), 22 deletions(-)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCleanerWithReplication.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCleanerWithReplication.java
index 429d55c..6353b37 100644
---
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCleanerWithReplication.java
+++
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCleanerWithReplication.java
@@ -143,7 +143,7 @@ public class TestCleanerWithReplication extends
CompactorTest {
addDeltaFile(t, null, 23L, 24L, 2);
addDeltaFile(t, null, 21L, 24L, 4);
-burnThroughTransactions(dbName, "camitc", 25);
+burnThroughTransactions(dbName, "camitc", 24);
CompactionRequest rqst = new CompactionRequest(dbName, "camitc",
CompactionType.MINOR);
compactInTxn(rqst);
@@ -161,7 +161,7 @@ public class TestCleanerWithReplication extends
CompactorTest {
addDeltaFile(t, p, 23L, 24L, 2);
addDeltaFile(t, p, 21L, 24L, 4);
-burnThroughTransactions(dbName, "camipc", 25);
+burnThroughTransactions(dbName, "camipc", 24);
CompactionRequest rqst = new CompactionRequest(dbName, "camipc",
CompactionType.MINOR);
rqst.setPartitionname("ds=today");
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
b/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
index 1e0dbf8..55a7802 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hive.ql.txn.compactor;
-import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.hive.common.ValidTxnList;
import org.apache.hadoop.hive.metastore.ReplChangeManager;
import org.apache.hadoop.hive.metastore.api.DataOperationType;
@@ -62,6 +61,7 @@ import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
import org.apache.hadoop.hive.ql.io.AcidUtils;
import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedBaseLight;
+import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedDelta;
import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedDeltaLight;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.StringUtils;
@@ -71,14 +71,15 @@ import java.io.IOException;
import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashSet;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.Callable;
+import java.util.Set;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
import static
org.apache.hadoop.hive.conf.Constants.COMPACTOR_CLEANER_THREAD_NAME_FORMAT;
@@ -87,8 +88,6 @@ import static
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_COMPACTOR_DELAY
import static org.apache.hadoop.hive.metastore.HMSHandler.getMSForConf;
import static
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.getDefaultCatalog;
-import com.codahale.metrics.Counter;
-
/**
* A class to clean directories after compactions. This will run in a
separate thread.
*/
@@ -425,8 +424,10 @@ public class Cleaner extends MetaStoreCompactorThread {
// Including obsolete directories for partitioned tables can result in
data loss.
obsoleteDirs = dir.getAbortedDirectories();
}
-if (obsoleteDirs.isEmpty() && !hasDataBelowWatermark(fs, path,
writeIdList.getHighWatermark())) {
- LOG.info(idWatermark(ci) + " nothing to remove below watermark " +
writeIdList.getHighWatermark() + &
[hive] branch master updated: HIVE-24390: Spelling common (#2810) (Josh Soref reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new adea623 HIVE-24390: Spelling common (#2810) (Josh Soref reviewed by
Zoltan Haindrich)
adea623 is described below
commit adea623294fd0a1a1502abc441794a61739cf8dc
Author: Josh Soref <2119212+jso...@users.noreply.github.com>
AuthorDate: Thu Mar 3 07:33:33 2022 -0500
HIVE-24390: Spelling common (#2810) (Josh Soref reviewed by Zoltan
Haindrich)
---
common/pom.xml | 2 +-
.../hadoop/hive/common/CompressionUtils.java | 10 +++---
.../org/apache/hadoop/hive/common/JvmMetrics.java | 2 +-
.../apache/hadoop/hive/common/JvmPauseMonitor.java | 2 +-
.../org/apache/hadoop/hive/common/LogUtils.java| 2 +-
.../apache/hadoop/hive/common/jsonexplain/Op.java | 12 +++
.../hadoop/hive/common/jsonexplain/Stage.java | 34 +-
.../hadoop/hive/common/jsonexplain/Vertex.java | 8 ++---
.../apache/hadoop/hive/common/type/Decimal128.java | 6 ++--
.../apache/hadoop/hive/common/type/HiveChar.java | 2 +-
.../hadoop/hive/common/type/SqlMathUtil.java | 2 +-
.../java/org/apache/hadoop/hive/conf/HiveConf.java | 42 +++---
.../java/org/apache/hadoop/hive/ql/ErrorMsg.java | 14
.../apache/hive/common/util/HiveStringUtils.java | 10 +++---
.../java/org/apache/hive/http/JMXJsonServlet.java | 2 +-
.../hive/http/Log4j2ConfiguratorServlet.java | 10 +++---
.../java/org/apache/hive/http/ProfileServlet.java | 2 +-
.../hadoop/hive/common/jsonexplain/TestStage.java | 42 +++---
.../hadoop/hive/common/jsonexplain/TestVertex.java | 2 +-
.../hadoop/hive/common/type/TestDecimal128.java| 16 -
.../hadoop/hive/conf/TestHiveConfRestrictList.java | 2 +-
.../hive/common/util/TestHiveStringUtils.java | 2 +-
.../clientnegative/stats_aggregator_error_2.q.out | 10 +++---
.../clientnegative/stats_publisher_error_1.q.out | 8 ++---
.../clientnegative/stats_publisher_error_2.q.out | 10 +++---
.../llap/stats_aggregator_error_1.q.out| 2 +-
.../llap/stats_publisher_error_1.q.out | 2 +-
.../hadoop/hive/common/metrics/common/Metrics.java | 4 +--
.../metrics/metrics2/JsonFileMetricsReporter.java | 2 +-
29 files changed, 132 insertions(+), 132 deletions(-)
diff --git a/common/pom.xml b/common/pom.xml
index 912de29..f89f273 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -28,7 +28,7 @@
-
+
org.apache.hive
hive-classification
diff --git
a/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
b/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
index a2e90e1..c5c050b 100644
--- a/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
+++ b/common/src/java/org/apache/hadoop/hive/common/CompressionUtils.java
@@ -117,7 +117,7 @@ public class CompressionUtils {
* @throws IOException
* @throws FileNotFoundException
*
- * @return The {@link List} of {@link File}s with the untared content.
+ * @return The {@link List} of {@link File}s with the untarred content.
* @throws ArchiveException
*/
public static List unTar(final String inputFileName, final String
outputDirName)
@@ -136,7 +136,7 @@ public class CompressionUtils {
* @throws IOException
* @throws FileNotFoundException
*
- * @return The {@link List} of {@link File}s with the untared content.
+ * @return The {@link List} of {@link File}s with the untarred content.
* @throws ArchiveException
*/
public static List unTar(final String inputFileName, final String
outputDirName,
@@ -145,7 +145,7 @@ public class CompressionUtils {
File inputFile = new File(inputFileName);
File outputDir = new File(outputDirName);
-final List untaredFiles = new LinkedList();
+final List untarredFiles = new LinkedList();
InputStream is = null;
try {
@@ -200,10 +200,10 @@ public class CompressionUtils {
IOUtils.copy(debInputStream, outputFileStream);
outputFileStream.close();
}
-untaredFiles.add(outputFile);
+untarredFiles.add(outputFile);
}
debInputStream.close();
- return untaredFiles;
+ return untarredFiles;
} finally {
if (is != null) is.close();
diff --git a/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java
b/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java
index b758abe..6edf396 100644
--- a/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java
+++ b/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java
@@ -128,7 +128,7 @@ public class JvmMetrics implements MetricsSource {
if (pauseMonitor != null) {
rb.addCounter(GcNumWarnThresholdEx
[hive] branch master updated (bf69b32 -> b63dab1)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from bf69b32 HIVE-25981: Avoid checking for archived parts in analyze table. (#3052). (Ayush Saxena, reviewed by Rajesh Balamohan and Peter Vary) add b63dab1 HIVE-23556: Support hive.metastore.limit.partition.request for get_partitions_ps (#3021) (Benny Zhu reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../hcatalog/listener/DummyRawStoreFailEvent.java | 6 .../hadoop/hive/metastore/conf/MetastoreConf.java | 4 ++- .../apache/hadoop/hive/metastore/HMSHandler.java | 31 .../apache/hadoop/hive/metastore/ObjectStore.java | 33 ++ .../org/apache/hadoop/hive/metastore/RawStore.java | 14 + .../hadoop/hive/metastore/cache/CachedStore.java | 6 .../metastore/DummyRawStoreControlledCommit.java | 6 .../metastore/DummyRawStoreForJdoConnection.java | 6 .../hadoop/hive/metastore/TestHiveMetaStore.java | 26 +++-- .../hadoop/hive/metastore/TestObjectStore.java | 12 .../hive/metastore/client/TestListPartitions.java | 11 11 files changed, 145 insertions(+), 10 deletions(-)
[hive] branch master updated (756a8fc -> 247d8aa)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 756a8fc HIVE-25960: Fix S3a recursive listing logic. (#3031). (Ayush Saxena reviewed by Laszlo Bodor) add 247d8aa HIVE-25874: Slow filter evaluation of nest struct fields in vectorized executions (#2952) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../vector/expressions/VectorUDFStructField.java | 3 +- .../clientpositive/vectorization_nested_struct.q | 21 ++ .../llap/vectorization_nested_struct.q.out | 81 ++ 3 files changed, 103 insertions(+), 2 deletions(-) create mode 100644 ql/src/test/queries/clientpositive/vectorization_nested_struct.q create mode 100644 ql/src/test/results/clientpositive/llap/vectorization_nested_struct.q.out
[hive] branch branch-3 updated: HIVE-25844: Exception deserialization error-s may cause beeline to terminate immediately (backport HIVE-24772) (#2918) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 1087c4e HIVE-25844: Exception deserialization error-s may cause
beeline to terminate immediately (backport HIVE-24772) (#2918) (Zoltan
Haindrich reviewed by Krisztian Kasa)
1087c4e is described below
commit 1087c4e14a2668eab914939f1f92dae2f049cf7e
Author: Zoltan Haindrich
AuthorDate: Tue Feb 22 15:29:52 2022 +0100
HIVE-25844: Exception deserialization error-s may cause beeline to
terminate immediately (backport HIVE-24772) (#2918) (Zoltan Haindrich reviewed
by Krisztian Kasa)
---
.../apache/hive/minikdc/TestJdbcWithMiniKdc.java | 28 +---
.../apache/hive/service/cli/HiveSQLException.java | 167 +
.../hive/service/cli/thrift/ThriftCLIService.java | 67 +
.../hive/service/cli/TestHiveSQLException.java | 112 +-
4 files changed, 79 insertions(+), 295 deletions(-)
diff --git
a/itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
b/itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
index e526b48..353d157 100644
---
a/itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
+++
b/itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java
@@ -221,22 +221,15 @@ public class TestJdbcWithMiniKdc {
* it's not allowed to impersonate
* @throws Exception
*/
- @Test
+ @Test(expected = HiveSQLException.class)
public void testNegativeTokenAuth() throws Exception {
miniHiveKdc.loginUser(MiniHiveKdc.HIVE_TEST_SUPER_USER);
hs2Conn = DriverManager.getConnection(miniHS2.getJdbcURL());
try {
// retrieve token and store in the cache
- String token = ((HiveConnection)hs2Conn).getDelegationToken(
+ ((HiveConnection)hs2Conn).getDelegationToken(
MiniHiveKdc.HIVE_TEST_USER_2, MiniHiveKdc.HIVE_SERVICE_PRINCIPAL);
-
- fail(MiniHiveKdc.HIVE_TEST_SUPER_USER + " shouldn't be allowed to
retrieve token for " +
- MiniHiveKdc.HIVE_TEST_USER_2);
-} catch (SQLException e) {
- // Expected error
- assertEquals("Unexpected type of exception class thrown",
HiveSQLException.class, e.getClass());
- assertTrue(e.getCause().getCause().getMessage().contains("is not allowed
to impersonate"));
} finally {
hs2Conn.close();
}
@@ -260,22 +253,11 @@ public class TestJdbcWithMiniKdc {
* impersonate the given user
* @throws Exception
*/
- @Test
+ @Test(expected = SQLException.class)
public void testNegativeProxyAuth() throws Exception {
miniHiveKdc.loginUser(MiniHiveKdc.HIVE_TEST_SUPER_USER);
-try {
- hs2Conn = DriverManager.getConnection(miniHS2.getJdbcURL("default",
- ";hive.server2.proxy.user=" + MiniHiveKdc.HIVE_TEST_USER_2));
- verifyProperty(SESSION_USER_NAME, MiniHiveKdc.HIVE_TEST_USER_2);
- fail(MiniHiveKdc.HIVE_TEST_SUPER_USER + " shouldn't be allowed proxy
connection for "
- + MiniHiveKdc.HIVE_TEST_USER_2);
-} catch (SQLException e) {
- // Expected error
- e.printStackTrace();
- assertTrue(e.getMessage().contains("Failed to validate proxy
privilege"));
- assertTrue(e.getCause().getCause().getCause().getMessage()
- .contains("is not allowed to impersonate"));
-}
+hs2Conn = DriverManager
+.getConnection(miniHS2.getJdbcURL("default",
";hive.server2.proxy.user=" + MiniHiveKdc.HIVE_TEST_USER_2));
}
/**
diff --git a/service/src/java/org/apache/hive/service/cli/HiveSQLException.java
b/service/src/java/org/apache/hive/service/cli/HiveSQLException.java
index 5f9ff43..f180830 100644
--- a/service/src/java/org/apache/hive/service/cli/HiveSQLException.java
+++ b/service/src/java/org/apache/hive/service/cli/HiveSQLException.java
@@ -19,39 +19,52 @@
package org.apache.hive.service.cli;
import java.sql.SQLException;
-import java.util.ArrayList;
+import java.util.Collections;
import java.util.List;
import org.apache.hive.service.rpc.thrift.TStatus;
import org.apache.hive.service.rpc.thrift.TStatusCode;
+import com.google.common.annotations.VisibleForTesting;
+
/**
- * HiveSQLException.
- *
+ * An exception that provides information on a Hive access error or other
+ * errors.
*/
public class HiveSQLException extends SQLException {
- /**
- *
- */
- private static final long serialVersionUID = -6095254671958748094L;
+ private static final long serialVersionUID = -6095254671958748095L;
+
+ @VisibleForTesting
+ public static final List DEFAULT_INFO =
+ Collections.singletonList("Server-side error; please check HS2 logs.");
/**
- *
+ * Constructor.
*/
publ
[hive] branch master updated: HIVE-25715: Provide nightly builds (#3013) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 362461f HIVE-25715: Provide nightly builds (#3013) (Zoltan Haindrich
reviewed by Krisztian Kasa)
362461f is described below
commit 362461fab1b79a01b6ab966606b5faa694d38d38
Author: Zoltan Haindrich
AuthorDate: Thu Feb 17 18:06:19 2022 +0100
HIVE-25715: Provide nightly builds (#3013) (Zoltan Haindrich reviewed by
Krisztian Kasa)
---
Jenkinsfile| 14 +++
beeline/pom.xml| 2 +-
common/pom.xml | 2 +-
dev-support/nightly| 45 ++
hcatalog/core/pom.xml | 4 +-
hcatalog/webhcat/java-client/pom.xml | 4 +-
hcatalog/webhcat/svr/pom.xml | 4 +-
iceberg/pom.xml| 4 +-
itests/pom.xml | 8 ++--
llap-server/pom.xml| 4 +-
metastore/pom.xml | 2 +-
pom.xml| 1 +
ql/pom.xml | 6 +--
service/pom.xml| 4 +-
.../metastore-tools/metastore-benchmarks/pom.xml | 3 +-
standalone-metastore/metastore-tools/pom.xml | 3 +-
standalone-metastore/pom.xml | 1 +
17 files changed, 86 insertions(+), 25 deletions(-)
diff --git a/Jenkinsfile b/Jenkinsfile
index 029d3a4..b079d51 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -292,6 +292,20 @@ mvn verify -DskipITests=false
-Dit.test=ITest${dbType.capitalize()} -Dtest=nosuc
}
}
}
+ branches['nightly-check'] = {
+ executorNode {
+stage('Prepare') {
+loadWS();
+}
+stage('Build') {
+sh '''#!/bin/bash
+set -e
+dev-support/nightly
+'''
+buildHive("install -Dtest=noMatches -Pdist -pl packaging -am")
+}
+ }
+ }
for (int i = 0; i < splits.size(); i++) {
def num = i
def split = splits[num]
diff --git a/beeline/pom.xml b/beeline/pom.xml
index a8d9159..131a3f6 100644
--- a/beeline/pom.xml
+++ b/beeline/pom.xml
@@ -65,7 +65,7 @@
org.apache.hive
hive-standalone-metastore-server
- ${project.version}
+ ${standalone-metastore.version}
diff --git a/common/pom.xml b/common/pom.xml
index f7c6c8a..912de29 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -46,7 +46,7 @@
org.apache.hive
hive-standalone-metastore-common
- ${project.version}
+ ${standalone-metastore.version}
diff --git a/dev-support/nightly b/dev-support/nightly
new file mode 100755
index 000..7045148
--- /dev/null
+++ b/dev-support/nightly
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+DATE="`date +%Y%m%d_%H%M%S`"
+HASH="`git rev-parse --short HEAD`"
+SUFFIX="nightly-$HASH-$DATE"
+
+V="`xmlstarlet sel -t -m /_:project/_:version -v . pom.xml`"
+NEW_HIVE="${NEW_HIVE:-${V/-*}-$SUFFIX}"
+V="`xmlstarlet sel -t -m /_:project/_:version -v . storage-api/pom.xml`"
+NEW_SA="${NEW_SA:-${V/-*}-$SUFFIX}"
+V="`xmlstarlet sel -t -m /_:project/_:version -v .
standalone-metastore/pom.xml`"
+NEW_MS="${NEW_MS:-${V/-*}-$SUFFIX}"
+
+
+mvn_versions_set="mvn versions:set versions:commit -DgenerateBackupPoms=false"
+
+$mvn_versions_set -B -DnewVersion=$NEW_HIVE
+$mvn_versions_set -B -DnewVersion=$NEW_SA -pl storage-api
+$mvn_versions_set -B -DnewVersion=$NEW_MS -pl standalone-metastore
+
+xmlstarlet edit -L -P --update "/_:project/_:properties/_:hive.version" \
+ --value $NEW_HIVE standalone-metastore/pom.xml
+
+xmlstarlet edit -L -P --update "/_:project/_:properties/_:storage-api.version"
\
+ --value $NEW_SA pom.xml standalone-metastore/pom.xml
+
+x
[hive] branch master updated: HIVE-25942: Upgrade commons-io version to 2.8.0 due to CVE-2021-29425. (#3008) (Syed Shameerur Rahman reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 43a08a8 HIVE-25942: Upgrade commons-io version to 2.8.0 due to CVE-2021-29425. (#3008) (Syed Shameerur Rahman reviewed by Zoltan Haindrich) 43a08a8 is described below commit 43a08a81335922f205753dccb1dce985bc4533cf Author: Syed Shameerur Rahman AuthorDate: Sun Feb 13 17:00:59 2022 +0530 HIVE-25942: Upgrade commons-io version to 2.8.0 due to CVE-2021-29425. (#3008) (Syed Shameerur Rahman reviewed by Zoltan Haindrich) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 761d573..ffd2433 100644 --- a/pom.xml +++ b/pom.xml @@ -125,7 +125,7 @@ 1.21 1.10 1.1 -2.6 +2.8.0 3.9 3.6.1 2.7.0
[hive] branch master updated (29c0e81 -> 4b7a948)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 29c0e81 HIVE-25926: Move all logging from AcidMetricService to AcidMetricLogger (Viktor Csomor, reviewed by Karen Coppage) add 4b7a948 HIVE-25766 java.util.NoSuchElementException in HiveFilterProjectTransposeRule if predicate has no InputRef (Alessandro Solimando reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../rules/HiveFilterProjectTransposeRule.java | 5 ++ .../cbo_filter_proj_transpose_noinputref.q | 9 .../cbo_filter_proj_transpose_noinputref.q.out | 54 ++ 3 files changed, 68 insertions(+) create mode 100644 ql/src/test/queries/clientpositive/cbo_filter_proj_transpose_noinputref.q create mode 100644 ql/src/test/results/clientpositive/llap/cbo_filter_proj_transpose_noinputref.q.out
[hive] branch master updated: HIVE-25883: Enhance Compaction Cleaner to skip when there is nothing to do (#2958) (Zoltan Haindrich reviewed by Denys Kuzmenko)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 5214369 HIVE-25883: Enhance Compaction Cleaner to skip when there is
nothing to do (#2958) (Zoltan Haindrich reviewed by Denys Kuzmenko)
5214369 is described below
commit 5214369d05ac02f36b6723c336cbdb953ea3c61c
Author: Zoltan Haindrich
AuthorDate: Sun Jan 23 08:49:30 2022 +0100
HIVE-25883: Enhance Compaction Cleaner to skip when there is nothing to do
(#2958) (Zoltan Haindrich reviewed by Denys Kuzmenko)
---
.../hadoop/hive/ql/txn/compactor/Cleaner.java | 57 +++---
.../hadoop/hive/ql/txn/compactor/TestCleaner.java | 116 +++--
2 files changed, 154 insertions(+), 19 deletions(-)
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
b/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
index db65250..8d724f6 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
@@ -17,7 +17,6 @@
*/
package org.apache.hadoop.hive.ql.txn.compactor;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hive.common.ValidTxnList;
import org.apache.hadoop.hive.metastore.ReplChangeManager;
import org.apache.hadoop.hive.metastore.api.DataOperationType;
@@ -52,6 +51,7 @@ import
org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil.ThrowingRunnable;
import
org.apache.hadoop.hive.ql.txn.compactor.metrics.DeltaFilesMetricReporter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.common.ValidWriteIdList;
@@ -60,6 +60,8 @@ import org.apache.hadoop.hive.common.StringableMap;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.txn.CompactionInfo;
import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedBaseLight;
+import org.apache.hadoop.hive.ql.io.AcidUtils.ParsedDeltaLight;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.StringUtils;
import org.apache.hive.common.util.Ref;
@@ -208,7 +210,7 @@ public class Cleaner extends MetaStoreCompactorThread {
}
Optional location =
Optional.ofNullable(ci.properties).map(StringableMap::new)
.map(config -> config.get("location"));
-
+
Callable cleanUpTask;
Table t = null;
Partition p = resolvePartition(ci);
@@ -248,12 +250,12 @@ public class Cleaner extends MetaStoreCompactorThread {
if (t != null) {
StorageDescriptor sd = resolveStorageDescriptor(t, p);
-cleanUpTask = () -> removeFiles(location.orElse(sd.getLocation()),
minOpenTxnGLB, ci,
+cleanUpTask = () -> removeFiles(location.orElse(sd.getLocation()),
minOpenTxnGLB, ci,
ci.partName != null && p == null);
} else {
cleanUpTask = () -> removeFiles(location.get(), ci);
}
-
+
Ref removedFiles = Ref.from(false);
if (runJobAsSelf(ci.runAs)) {
removedFiles.value = cleanUpTask.call();
@@ -328,7 +330,7 @@ public class Cleaner extends MetaStoreCompactorThread {
if (dropPartition) {
LockRequest lockRequest = createLockRequest(ci, 0, LockType.EXCL_WRITE,
DataOperationType.DELETE);
LockResponse res = null;
-
+
try {
res = txnHandler.lock(lockRequest);
if (res.getState() == LockState.ACQUIRED) {
@@ -349,7 +351,7 @@ public class Cleaner extends MetaStoreCompactorThread {
}
}
}
-
+
ValidTxnList validTxnList =
TxnUtils.createValidTxnListForCleaner(txnHandler.getOpenTxns(),
minOpenTxnGLB);
//save it so that getAcidState() sees it
@@ -388,7 +390,7 @@ public class Cleaner extends MetaStoreCompactorThread {
// Creating 'reader' list since we are interested in the set of 'obsolete'
files
ValidReaderWriteIdList validWriteIdList = getValidCleanerWriteIdList(ci,
validTxnList);
LOG.debug("Cleaning based on writeIdList: {}", validWriteIdList);
-
+
return removeFiles(location, validWriteIdList, ci);
}
/**
@@ -419,6 +421,10 @@ public class Cleaner extends MetaStoreCompactorThread {
// Including obsolete directories for partitioned tables can result in
data loss.
obsoleteDirs = dir.getAbortedDirectories();
}
+if (obsoleteDirs.isEmpty() && !hasDataBelowWatermark(fs, path,
writeIdList.getHighWatermark())) {
+ LOG.info(idWatermark(ci) + " nothing to remove below watermark " +
writeIdList.getHighWatermark() + ", ");
+ return true;
+}
StringBuilder extraDebugInfo = new
[hive] branch master updated: HIVE-25822: Unexpected result rows in case of outer join contains conditions only affecting one side (#2891) (Zoltan Haindrich reviewed by Stamatis Zampetakis, Aman Sinha
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 621bfd1 HIVE-25822: Unexpected result rows in case of outer join
contains conditions only affecting one side (#2891) (Zoltan Haindrich reviewed
by Stamatis Zampetakis, Aman Sinha)
621bfd1 is described below
commit 621bfd164f018063fe5e03d9f7a7d990ce22691a
Author: Zoltan Haindrich
AuthorDate: Tue Dec 21 11:50:45 2021 +0100
HIVE-25822: Unexpected result rows in case of outer join contains
conditions only affecting one side (#2891) (Zoltan Haindrich reviewed by
Stamatis Zampetakis, Aman Sinha)
---
.../java/org/apache/hadoop/hive/conf/HiveConf.java | 2 +
.../hive/ql/exec/CommonMergeJoinOperator.java | 80 --
.../clientpositive/outer_join_unexpected_rows.q| 17
.../test/results/clientpositive/llap/join46.q.out | 4 +-
.../results/clientpositive/llap/join_1to1.q.out| 64 +--
.../clientpositive/llap/join_emit_interval.q.out | 6 +-
.../results/clientpositive/llap/mapjoin2.q.out | 2 +-
.../llap/outer_join_unexpected_rows.q.out | 95 ++
.../llap/vector_full_outer_join2.q.out | 18 ++--
9 files changed, 208 insertions(+), 80 deletions(-)
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index fd95070..b0b9b4f 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -1836,6 +1836,8 @@ public class HiveConf extends Configuration {
HIVEALIAS("hive.alias", "", ""),
HIVEMAPSIDEAGGREGATE("hive.map.aggr", true, "Whether to use map-side
aggregation in Hive Group By queries"),
HIVEGROUPBYSKEW("hive.groupby.skewindata", false, "Whether there is skew
in data to optimize group by queries"),
+HIVE_JOIN_SHORTCUT_UNMATCHED_ROWS("hive.join.shortcut.unmatched.rows",
true,
+"Enables to shortcut processing of known filtered rows in merge joins.
internal use only. may affect correctness"),
HIVEJOINEMITINTERVAL("hive.join.emit.interval", 1000,
"How many rows in the right-most join operand Hive should buffer
before emitting the join result."),
HIVEJOINCACHESIZE("hive.join.cache.size", 25000,
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java
index e574fb9..0044a04 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java
@@ -42,12 +42,12 @@ import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.plan.CommonMergeJoinDesc;
import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
import org.apache.hadoop.hive.ql.plan.JoinCondDesc;
+import org.apache.hadoop.hive.ql.plan.JoinDesc;
import org.apache.hadoop.hive.ql.plan.OperatorDesc;
import org.apache.hadoop.hive.ql.plan.api.OperatorType;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
import
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.ObjectInspectorCopyOption;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
/*
@@ -74,6 +74,7 @@ public class CommonMergeJoinOperator extends
AbstractMapJoinOperator[] nextKeyWritables;
transient RowContainer>[] nextGroupStorage;
transient RowContainer>[] candidateStorage;
+ transient RowContainer>[] unmatchedStorage;
transient String[] tagToAlias;
private transient boolean[] fetchDone;
@@ -94,6 +95,7 @@ public class CommonMergeJoinOperator extends
AbstractMapJoinOperator> unmatchedRC = JoinUtil.getRowContainer(hconf,
+ rowContainerStandardObjectInspectors[pos], pos, bucketSize,
spillTableDesc, conf, !hasFilter(pos), reporter);
+ unmatchedStorage[pos] = unmatchedRC;
}
for (byte pos = 0; pos < order.length; pos++) {
@@ -240,6 +248,18 @@ public class CommonMergeJoinOperator extends
AbstractMapJoinOperator value = getFilteredValue(alias, row);
+
+if (isOuterJoinUnmatchedRow(tag, value)) {
+ int type = condn[0].getType();
+ if (tag == 0 && (type == JoinDesc.LEFT_OUTER_JOIN || type ==
JoinDesc.FULL_OUTER_JOIN)) {
+unmatchedStorage[tag].addRow(value);
+ }
+ if (tag == 1 && (type == JoinDesc.RIGHT_OUTER_JOIN || type ==
JoinDesc.FULL_OUTER_JOIN)) {
+unmatchedStorage[tag].addRow(value);
+ }
+ emitUnmatchedRows(tag, false);
+ re
[hive] branch master updated: HIVE-25782: Added client capabilites in the dry run call for the CTAS… (#2858) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new decd725 HIVE-25782: Added client capabilites in the dry run call for
the CTAS… (#2858) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich)
decd725 is described below
commit decd725f2b29c37fa265edfc7899623c99916d61
Author: Sai Hemanth Gantasala
<68923650+saihemanth-cloud...@users.noreply.github.com>
AuthorDate: Wed Dec 15 22:57:41 2021 -0800
HIVE-25782: Added client capabilites in the dry run call for the CTAS…
(#2858) (Sai Hemanth Gantasala reviewed by Zoltan Haindrich)
---
.../create_acid_table_with_transformer.q | 7 +
.../llap/create_acid_table_with_transformer.q.out | 11 ++
.../src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp | 30 ++---
.../src/gen/thrift/gen-cpp/ThriftHiveMetastore.h | 30 ++---
.../ThriftHiveMetastore_server.skeleton.cpp| 2 +-
.../hive/metastore/api/ThriftHiveMetastore.java| 146 ++---
.../metastore/ThriftHiveMetastoreClient.php| 8 +-
.../gen-php/metastore/ThriftHiveMetastoreIf.php| 4 +-
...ftHiveMetastore_translate_table_dryrun_args.php | 24 ++--
.../hive_metastore/ThriftHiveMetastore-remote | 2 +-
.../gen-py/hive_metastore/ThriftHiveMetastore.py | 34 ++---
.../src/gen/thrift/gen-rb/thrift_hive_metastore.rb | 14 +-
.../hadoop/hive/metastore/HiveMetaStoreClient.java | 8 +-
.../src/main/thrift/hive_metastore.thrift | 3 +-
.../apache/hadoop/hive/metastore/HMSHandler.java | 7 +-
.../metastore/HiveMetaStoreClientPreCatalog.java | 3 +-
16 files changed, 181 insertions(+), 152 deletions(-)
diff --git
a/ql/src/test/queries/clientpositive/create_acid_table_with_transformer.q
b/ql/src/test/queries/clientpositive/create_acid_table_with_transformer.q
new file mode 100644
index 000..2547549
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/create_acid_table_with_transformer.q
@@ -0,0 +1,7 @@
+set hive.support.concurrency=true;
+set hive.exec.dynamic.partition.mode=nonstrict;
+set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set
hive.metastore.client.capabilities=HIVEFULLACIDREAD,HIVEFULLACIDWRITE,HIVECACHEINVALIDATE,HIVEMANAGESTATS,HIVEMANAGEDINSERTWRITE,HIVEMANAGEDINSERTREAD;
+
+create table test stored as orc tblproperties ('transactional'='true') as
select from_unixtime(unix_timestamp("0002-01-01 09:57:21", "-MM-dd
HH:mm:ss"));
diff --git
a/ql/src/test/results/clientpositive/llap/create_acid_table_with_transformer.q.out
b/ql/src/test/results/clientpositive/llap/create_acid_table_with_transformer.q.out
new file mode 100644
index 000..550a020
--- /dev/null
+++
b/ql/src/test/results/clientpositive/llap/create_acid_table_with_transformer.q.out
@@ -0,0 +1,11 @@
+PREHOOK: query: create table test stored as orc tblproperties
('transactional'='true') as select from_unixtime(unix_timestamp("0002-01-01
09:57:21", "-MM-dd HH:mm:ss"))
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: database:default
+PREHOOK: Output: default@test
+POSTHOOK: query: create table test stored as orc tblproperties
('transactional'='true') as select from_unixtime(unix_timestamp("0002-01-01
09:57:21", "-MM-dd HH:mm:ss"))
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@test
+POSTHOOK: Lineage: test._c0 SIMPLE []
diff --git
a/standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
b/standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
index b07edc8..d08962f 100644
---
a/standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
+++
b/standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp
@@ -9684,8 +9684,8 @@ uint32_t
ThriftHiveMetastore_translate_table_dryrun_args::read(::apache::thrift:
{
case 1:
if (ftype == ::apache::thrift::protocol::T_STRUCT) {
- xfer += this->tbl.read(iprot);
- this->__isset.tbl = true;
+ xfer += this->request.read(iprot);
+ this->__isset.request = true;
} else {
xfer += iprot->skip(ftype);
}
@@ -9707,8 +9707,8 @@ uint32_t
ThriftHiveMetastore_translate_table_dryrun_args::write(::apache::thrift
::apache::thrift::protocol::TOutputRecursionTracker tracker(*oprot);
xfer +=
oprot->writeStructBegin("ThriftHiveMetastore_translate_table_dryrun_args");
- xfer += oprot->writeFieldBegin("tbl", ::apache::thr
[hive] branch master updated: HIVE-25804: Update log4j2 version to 2.16.0 to incorporate further CVE-2021-44228 hardening (#2874) (Csaba Juhász reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 6f29c1a HIVE-25804: Update log4j2 version to 2.16.0 to incorporate further CVE-2021-44228 hardening (#2874) (Csaba Juhász reviewed by Zoltan Haindrich) 6f29c1a is described below commit 6f29c1acf04fa1f1b84a6fb86f92bdc65b584567 Author: csjuhasz-c <66361392+csjuhas...@users.noreply.github.com> AuthorDate: Wed Dec 15 10:12:38 2021 +0100 HIVE-25804: Update log4j2 version to 2.16.0 to incorporate further CVE-2021-44228 hardening (#2874) (Csaba Juhász reviewed by Zoltan Haindrich) --- pom.xml | 2 +- standalone-metastore/pom.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index 710c1be..2fb29f6 100644 --- a/pom.xml +++ b/pom.xml @@ -178,7 +178,7 @@ 0.9.3 0.14.1 -2.15.0 +2.16.0 2.5.0 6.2.1.jre8 8.0.27 diff --git a/standalone-metastore/pom.xml b/standalone-metastore/pom.xml index bd331e3..34cdaab 100644 --- a/standalone-metastore/pom.xml +++ b/standalone-metastore/pom.xml @@ -91,7 +91,7 @@ 5.6.2 0.9.3 0.14.1 -2.15.0 +2.16.0 3.3.3 1.6.9
[hive] branch master updated: HIVE-25791: Improve SFS exception messages (#2859) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new ebb1e2f HIVE-25791: Improve SFS exception messages (#2859) (Zoltan
Haindrich reviewed by Krisztian Kasa)
ebb1e2f is described below
commit ebb1e2fa9914bcccecad261d53338933b699ccb1
Author: Zoltan Haindrich
AuthorDate: Wed Dec 15 08:49:18 2021 +0100
HIVE-25791: Improve SFS exception messages (#2859) (Zoltan Haindrich
reviewed by Krisztian Kasa)
---
.../org/apache/hadoop/hive/ql/QOutProcessor.java | 14 +++
.../hive/ql/qoption/QTestReplaceHandler.java | 2 +-
.../apache/hadoop/hive/ql/io/SingleFileSystem.java | 43 +-
.../hadoop/hive/ql/io/TestSingleFileSystem.java| 32
.../test/queries/clientnegative/sfs_nonexistent.q | 3 ++
.../results/clientnegative/sfs_nonexistent.q.out | 6 +++
6 files changed, 82 insertions(+), 18 deletions(-)
diff --git
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java
index 22aad8e..cdf599b 100644
--- a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java
+++ b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QOutProcessor.java
@@ -51,7 +51,7 @@ public class QOutProcessor {
public static final String HDFS_DATE_MASK = "### HDFS DATE ###";
public static final String HDFS_USER_MASK = "### USER ###";
public static final String HDFS_GROUP_MASK = "### GROUP ###";
-
+
public static final String MASK_PATTERN = " A masked pattern was here
";
public static final String PARTIAL_MASK_PATTERN = " A PARTIAL masked
pattern was here ";
private static final PatternReplacementPair MASK_STATS = new
PatternReplacementPair(
@@ -69,7 +69,7 @@ public class QOutProcessor {
public static class LineProcessingResult {
private String line;
private boolean partialMaskWasMatched = false;
-
+
public LineProcessingResult(String line) {
this.line = line;
}
@@ -78,7 +78,7 @@ public class QOutProcessor {
return line;
}
}
-
+
private final Pattern[] planMask = toPattern(new String[] {
".*[.][.][.] [0-9]* more.*",
"pk_-?[0-9]*_[0-9]*_[0-9]*",
@@ -211,9 +211,11 @@ public class QOutProcessor {
LineProcessingResult processLine(String line) {
LineProcessingResult result = new LineProcessingResult(line);
-
+
Matcher matcher = null;
+result.line = replaceHandler.processLine(result.line);
+
if (fsType == FsType.ENCRYPTED_HDFS) {
for (Pattern pattern : partialReservedPlanMask) {
matcher = pattern.matcher(result.line);
@@ -305,8 +307,6 @@ public class QOutProcessor {
}
}
-result.line = replaceHandler.processLine(result.line);
-
return result;
}
@@ -332,7 +332,7 @@ public class QOutProcessor {
ArrayList ppm = new ArrayList<>();
ppm.add(new
PatternReplacementPair(Pattern.compile("\\{\"writeid\":[1-9][0-9]*,\"bucketid\":"),
"{\"writeid\":### Masked writeid ###,\"bucketid\":"));
-
+
ppm.add(new PatternReplacementPair(Pattern.compile("attempt_[0-9_]+"),
"attempt_#ID#"));
ppm.add(new PatternReplacementPair(Pattern.compile("vertex_[0-9_]+"),
"vertex_#ID#"));
ppm.add(new PatternReplacementPair(Pattern.compile("task_[0-9_]+"),
"task_#ID#"));
diff --git
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestReplaceHandler.java
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestReplaceHandler.java
index aa2e3fd..06abe15 100644
---
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestReplaceHandler.java
+++
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/qoption/QTestReplaceHandler.java
@@ -55,7 +55,7 @@ public class QTestReplaceHandler implements
QTestOptionHandler {
throw new RuntimeException("illegal replacement expr: " + arguments + "
; expected something like /this/that/");
}
String sep = arguments.substring(0, 1);
-String[] parts = arguments.split(sep);
+String[] parts = arguments.split(Pattern.quote(sep));
if (parts.length != 3) {
throw new RuntimeException(
"unexpected replacement expr: " + arguments + " ; expected something
like /this/that/");
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
b/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
index e0e9bff..e073622 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.j
[hive] branch master updated (f2d29ef -> f7448be)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.
from f2d29ef HIVE-25768: Extend lifetime of query-level HMS response cache
(John Sherman, reviewed by Stephen Carlin, Krisztian Kasa)
add f7448be HIVE-25780: DistinctExpansion creates more than 64 grouping
sets II (#2849) (Zoltan Haindrich reviewed by Krisztian Kasa)
No new revisions were added by this update.
Summary of changes:
.../test/resources/testconfiguration.properties| 3 +-
.../rules/HiveExpandDistinctAggregatesRule.java| 20 +-
.../queries/clientpositive/multi_count_distinct.q | 13 +
.../{tez => llap}/multi_count_distinct.q.out | 345 ++---
4 files changed, 266 insertions(+), 115 deletions(-)
rename ql/src/test/results/clientpositive/{tez =>
llap}/multi_count_distinct.q.out (83%)
[hive] branch master updated: HIVE-24975: Fix a bug in ValidWriteIdList comparison in TxnIdUtils (#2641) (Sourabh Goyal reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 9f9844d HIVE-24975: Fix a bug in ValidWriteIdList comparison in
TxnIdUtils (#2641) (Sourabh Goyal reviewed by Zoltan Haindrich)
9f9844d is described below
commit 9f9844dbc881e2a9267c259b8c04e7787f7fadc4
Author: Sourabh Goyal
AuthorDate: Thu Dec 9 01:21:08 2021 -0800
HIVE-24975: Fix a bug in ValidWriteIdList comparison in TxnIdUtils (#2641)
(Sourabh Goyal reviewed by Zoltan Haindrich)
---
.../src/java/org/apache/hive/common/util/TxnIdUtils.java | 2 +-
.../src/test/org/apache/hive/common/util/TestTxnIdUtils.java | 11 +++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java
b/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java
index bd972d4..7de7464 100644
--- a/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java
+++ b/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java
@@ -67,7 +67,7 @@ public class TxnIdUtils {
}
} else {
if (b.getHighWatermark() != a.getInvalidWriteIds()[minLen] -1) {
-return Long.signum(b.getHighWatermark() -
(a.getInvalidWriteIds()[minLen] -1));
+return Long.signum((a.getInvalidWriteIds()[minLen] - 1) -
b.getHighWatermark());
}
if (allInvalidFrom(a.getInvalidWriteIds(), minLen,
a.getHighWatermark())) {
return 0;
diff --git
a/storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
b/storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
index ab5a472..1f646a3 100644
--- a/storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
+++ b/storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
@@ -19,6 +19,7 @@ package org.apache.hive.common.util;
import java.util.BitSet;
+import org.apache.hadoop.hive.common.ValidWriteIdList;
import org.apache.hadoop.hive.common.ValidReaderWriteIdList;
import org.junit.Test;
import static org.junit.Assert.assertEquals;
@@ -190,5 +191,15 @@ public class TestTxnIdUtils {
new ValidReaderWriteIdList("default.table2", new long[] {8,10,11}, new
BitSet(), 11)),
-1);
+ValidWriteIdList a =
+new ValidReaderWriteIdList("default.test:1:1:1:");
+ValidWriteIdList b =
+new ValidReaderWriteIdList("default.test:1:9223372036854775807::");
+
+// should return -1 since b is more recent
+assertEquals(TxnIdUtils.compare(a, b), -1);
+
+// should return 1 since b is more recent
+assertEquals(TxnIdUtils.compare(b, a), 1);
}
}
[hive] branch master updated (ed4ecfc -> 9a3d878)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from ed4ecfc HIVE-25788: Iceberg CTAS should honor location clause and have correct table properties (Marton Bod, reviewed by Adam Szita and Peter Vary) add 9a3d878 HIVE-25735: Improve statestimator in UDFWhen/UDFCase (#2814) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hadoop/hive/ql/udf/generic/GenericUDFCase.java | 8 +++--- .../hadoop/hive/ql/udf/generic/GenericUDFWhen.java | 30 -- .../clientpositive/llap/constant_prop_when.q.out | 4 +-- .../results/clientpositive/llap/innerjoin1.q.out | 16 ++-- .../materialized_view_create_rewrite_nulls.q.out | 6 ++--- .../clientpositive/llap/subquery_notin.q.out | 10 .../clientpositive/llap/subquery_select.q.out | 4 +-- .../llap/vector_between_columns.q.out | 4 +-- .../clientpositive/llap/vector_case_when_1.q.out | 12 - .../clientpositive/llap/vector_case_when_2.q.out | 24 - .../clientpositive/llap/vector_coalesce_2.q.out| 8 +++--- .../clientpositive/llap/vector_coalesce_3.q.out| 10 .../llap/vector_groupby_grouping_id1.q.out | 8 +++--- .../vector_groupby_grouping_sets_grouping.q.out| 4 +-- .../clientpositive/llap/vectorized_case.q.out | 16 ++-- .../perf/tpcds30tb/tez/query36.q.out | 4 +-- .../perf/tpcds30tb/tez/query39.q.out | 16 ++-- .../perf/tpcds30tb/tez/query70.q.out | 4 +-- .../perf/tpcds30tb/tez/query86.q.out | 4 +-- 19 files changed, 109 insertions(+), 83 deletions(-)
[hive] branch master updated: HIVE-25738: NullIf doesn't support complex types (#2816) (Zoltan Haindrich reviewed by Zhihua Deng and Stamatis Zampetakis)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 1cafaef HIVE-25738: NullIf doesn't support complex types (#2816)
(Zoltan Haindrich reviewed by Zhihua Deng and Stamatis Zampetakis)
1cafaef is described below
commit 1cafaef0456fd2714c20f57b7c81a4968f425d44
Author: Zoltan Haindrich
AuthorDate: Wed Dec 1 17:25:57 2021 +0100
HIVE-25738: NullIf doesn't support complex types (#2816) (Zoltan Haindrich
reviewed by Zhihua Deng and Stamatis Zampetakis)
---
.../hive/ql/udf/generic/GenericUDFNullif.java | 24 ++
.../hive/ql/udf/generic/TestGenericUDFNullif.java | 47 +++
ql/src/test/queries/clientnegative/nullif_union.q | 3 ++
ql/src/test/queries/clientpositive/udf_nullif.q| 9
.../test/results/clientnegative/nullif_union.q.out | 1 +
.../results/clientpositive/llap/udf_nullif.q.out | 54 ++
6 files changed, 130 insertions(+), 8 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNullif.java
b/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNullif.java
index a47882e..b99efa1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNullif.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNullif.java
@@ -23,6 +23,7 @@ import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
import
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
import
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.PrimitiveGrouping;
@@ -49,6 +50,17 @@ public class GenericUDFNullif extends GenericUDF {
returnOIResolver = new GenericUDFUtils.ReturnObjectInspectorResolver(true);
returnOIResolver.update(arguments[0]);
+switch (arguments[0].getCategory()) {
+case LIST:
+case MAP:
+case STRUCT:
+case PRIMITIVE:
+ break;
+case UNION:
+default:
+ throw new UDFArgumentTypeException(0, "Unsupported Argument type
category: " + arguments[0].getCategory());
+}
+
boolean isPrimitive = (arguments[0] instanceof PrimitiveObjectInspector);
if (isPrimitive)
{
@@ -86,17 +98,13 @@ public class GenericUDFNullif extends GenericUDF {
public Object evaluate(DeferredObject[] arguments) throws HiveException {
Object arg0 = arguments[0].get();
Object arg1 = arguments[1].get();
-Object value0 = null;
-if (arg0 != null) {
- value0 = returnOIResolver.convertIfNecessary(arg0, argumentOIs[0],
false);
-}
+Object value0 = returnOIResolver.convertIfNecessary(arg0, argumentOIs[0],
false);
if (arg0 == null || arg1 == null) {
return value0;
}
-PrimitiveObjectInspector compareOI = (PrimitiveObjectInspector)
returnOIResolver.get();
-if (PrimitiveObjectInspectorUtils.comparePrimitiveObjects(
-value0, compareOI,
-returnOIResolver.convertIfNecessary(arg1, argumentOIs[1], false),
compareOI)) {
+Object value1 = returnOIResolver.convertIfNecessary(arg1, argumentOIs[1],
false);
+ObjectInspector compareOI = returnOIResolver.get();
+if (ObjectInspectorUtils.compare(value0, compareOI, value1, compareOI) ==
0) {
return null;
}
return value0;
diff --git
a/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFNullif.java
b/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFNullif.java
index 281b0d5..2ff4408 100644
---
a/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFNullif.java
+++
b/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFNullif.java
@@ -18,6 +18,8 @@
package org.apache.hadoop.hive.ql.udf.generic;
+import static java.util.Arrays.asList;
+
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDF.DeferredJavaObject;
@@ -27,7 +29,9 @@ import org.apache.hadoop.hive.serde2.io.DateWritableV2;
import org.apache.hadoop.hive.serde2.lazy.LazyInteger;
import
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyPrimitiveObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import
org.apache.hadoop.hive.serde2.objectinspector.StandardListObjec
[hive] branch master updated: HIVE-25734: Wrongly-typed constant in case expression leads to incorrect empty result (#2815) ( Alessandro Solimando reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 17f7208 HIVE-25734: Wrongly-typed constant in case expression leads
to incorrect empty result (#2815) ( Alessandro Solimando reviewed by Zoltan
Haindrich)
17f7208 is described below
commit 17f72087bd0fc8b5d306e01277052cdcc87c8556
Author: Alessandro Solimando
AuthorDate: Tue Nov 30 09:54:33 2021 +0100
HIVE-25734: Wrongly-typed constant in case expression leads to incorrect
empty result (#2815) ( Alessandro Solimando reviewed by Zoltan Haindrich)
---
.../rules/HivePointLookupOptimizerRule.java| 35 +++-
.../calcite/translator/RexNodeConverter.java | 14 +-
.../rules/TestHivePointLookupOptimizerRule.java| 165 +-
.../calcite/translator/TestRexNodeConverter.java | 186 +
.../clientpositive/cbo_case_when_wrong_type.q | 10 ++
.../llap/cbo_case_when_wrong_type.q.out| 84 ++
.../perf/tpcds30tb/tez/query39.q.out | 8 +-
7 files changed, 484 insertions(+), 18 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
index bf69d3a..da6e9e7 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePointLookupOptimizerRule.java
@@ -29,6 +29,7 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import org.apache.calcite.plan.RelOptRule;
import org.apache.calcite.plan.RelOptRuleCall;
@@ -387,12 +388,12 @@ public abstract class HivePointLookupOptimizerRule
extends RelOptRule {
}
private static boolean isColumnExpr(RexNode node) {
- return !node.getType().isStruct() &&
HiveCalciteUtil.getInputRefs(node).size() > 0
+ return !node.getType().isStruct() &&
!HiveCalciteUtil.getInputRefs(node).isEmpty()
&& HiveCalciteUtil.isDeterministic(node);
}
private static boolean isConstExpr(RexNode node) {
- return !node.getType().isStruct() &&
HiveCalciteUtil.getInputRefs(node).size() == 0
+ return !node.getType().isStruct() &&
HiveCalciteUtil.getInputRefs(node).isEmpty()
&& HiveCalciteUtil.isDeterministic(node);
}
@@ -508,7 +509,7 @@ public abstract class HivePointLookupOptimizerRule extends
RelOptRule {
for (Entry, Collection> sa :
assignmentGroups.asMap().entrySet()) {
// skip opaque
-if (sa.getKey().size() == 0) {
+if (sa.getKey().isEmpty()) {
continue;
}
// not enough equalities should not be handled
@@ -593,6 +594,7 @@ public abstract class HivePointLookupOptimizerRule extends
RelOptRule {
// into a null value.
final Multimap inLHSExprToRHSNullableExprs =
LinkedHashMultimap.create();
final List operands = new
ArrayList<>(RexUtil.flattenAnd(call.getOperands()));
+
for (int i = 0; i < operands.size(); i++) {
RexNode operand = operands.get(i);
if (operand.getKind() == SqlKind.IN) {
@@ -614,7 +616,11 @@ public abstract class HivePointLookupOptimizerRule extends
RelOptRule {
inLHSExprToRHSNullableExprs.put(ref, constNode);
}
}
-inLHSExprToRHSExprs.get(ref).retainAll(expressions);
+Collection knownConstants = inLHSExprToRHSExprs.get(ref);
+if (!shareSameType(knownConstants, expressions)) {
+ return call;
+}
+knownConstants.retainAll(expressions);
} else {
for (int j = 1; j < inCall.getOperands().size(); j++) {
RexNode constNode = inCall.getOperands().get(j);
@@ -639,7 +645,12 @@ public abstract class HivePointLookupOptimizerRule extends
RelOptRule {
inLHSExprToRHSNullableExprs.put(c.exprNode, c.constNode);
}
if (inLHSExprToRHSExprs.containsKey(c.exprNode)) {
-
inLHSExprToRHSExprs.get(c.exprNode).retainAll(Collections.singleton(c.constNode));
+Collection knownConstants =
inLHSExprToRHSExprs.get(c.exprNode);
+Collection nextConstant =
Collections.singleton(c.constNode);
+if (!shareSameType(knownConstants, nextConstant)) {
+ return call;
+}
+knownConstants.retainAll(nextConstant);
} else {
inLHSExprToRHSExprs.put(c.exprNode, c.constNode);
}
@@ -655,6 +666,20 @@ public abstract class HivePointLookupOptimizerRule extends
RelOptRule
[hive] branch master updated (a8e5073 -> 3a610dc)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from a8e5073 HIVE-25710: Config used to enable non-blocking TRUNCATE is not properly propagated (Denys Kuzmenko, reviewed by Karen Coppage and Peter Vary) add 3a610dc HIVE-25749: Check if RelMetadataQuery.collations() returns null to avoid NPE (#2823) (Alessandro Solimando reviewed by Aman Sinha and Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/calcite/reloperators/HiveJoin.java | 12 +++- .../hive/ql/optimizer/calcite/rules/RelFieldTrimmer.java | 9 + 2 files changed, 12 insertions(+), 9 deletions(-)
[hive] branch master updated: HIVE-25561: Killed task should not commit file. (#2674) (zhengchenyu reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 63bb7b9 HIVE-25561: Killed task should not commit file. (#2674)
(zhengchenyu reviewed by Zoltan Haindrich)
63bb7b9 is described below
commit 63bb7b922cd5693ecdfc49859d905259625f01b2
Author: zhengchenyu
AuthorDate: Fri Nov 26 15:35:30 2021 +0800
HIVE-25561: Killed task should not commit file. (#2674) (zhengchenyu
reviewed by Zoltan Haindrich)
Co-authored-by: zhengchenyu001
---
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
index 2a4dcfb..0cbcc26 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
@@ -312,6 +312,7 @@ public class TezProcessor extends
AbstractLogicalIOProcessor {
perfLogger.perfLogEnd(CLASS_NAME, PerfLogger.TEZ_RUN_PROCESSOR);
} catch (Throwable t) {
+ rproc.setAborted(true);
originalThrowable = t;
} finally {
if (originalThrowable != null && (originalThrowable instanceof Error ||
[hive] branch master updated (cc4c3b7 -> 14095a2)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from cc4c3b7 HIVE-25656: Get materialized view state based on number of affected rows of transactions (Krisztian Kasa, reviewed by Zoltan Haindrich, Stephen Carlin) add 14095a2 HIVE-25721: Outer join result is wrong (#2798) (Yizhen Fan reviewed by Stamatis Zampetakis and Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hive/ql/exec/JoinUtil.java | 11 ++-- .../test/queries/clientpositive/sort_merge_join.q | 17 ++ .../clientpositive/llap/sort_merge_join.q.out | 60 ++ 3 files changed, 84 insertions(+), 4 deletions(-) create mode 100644 ql/src/test/queries/clientpositive/sort_merge_join.q create mode 100644 ql/src/test/results/clientpositive/llap/sort_merge_join.q.out
[hive] branch master updated: HIVE-24390: Spelling fixes - serde (#2802) (Josh Soref reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 989e72a HIVE-24390: Spelling fixes - serde (#2802) (Josh Soref
reviewed by Zoltan Haindrich)
989e72a is described below
commit 989e72a393356c5f91f96d1bab6455a4c75c77a7
Author: Josh Soref <2119212+jso...@users.noreply.github.com>
AuthorDate: Tue Nov 23 03:14:51 2021 -0500
HIVE-24390: Spelling fixes - serde (#2802) (Josh Soref reviewed by Zoltan
Haindrich)
---
.../src/java/org/apache/hadoop/hive/serde2/RandomTypeUtil.java | 6 +++---
.../java/org/apache/hadoop/hive/serde2/SerDeStatsStruct.java | 2 +-
serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java | 2 +-
.../org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java | 8
.../org/apache/hadoop/hive/serde2/avro/AvroSerializer.java | 2 +-
.../org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java | 2 +-
.../org/apache/hadoop/hive/serde2/io/HiveCharWritable.java | 2 +-
.../org/apache/hadoop/hive/serde2/json/HiveJsonReader.java | 2 +-
.../org/apache/hadoop/hive/serde2/json/HiveJsonWriter.java | 8
.../java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java | 2 +-
.../org/apache/hadoop/hive/serde2/lazy/LazyHiveDecimal.java| 2 +-
.../java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java | 2 +-
.../apache/hadoop/hive/serde2/lazy/fast/StringToDouble.java| 2 +-
.../apache/hadoop/hive/serde2/lazybinary/LazyBinaryArray.java | 6 +++---
.../apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe2.java | 2 +-
.../hive/serde2/objectinspector/ListObjectsEqualComparer.java | 2 +-
.../hive/serde2/objectinspector/ObjectInspectorConverters.java | 2 +-
.../hive/serde2/objectinspector/ObjectInspectorUtils.java | 4 ++--
.../primitive/PrimitiveObjectInspectorUtils.java | 2 +-
.../hive/serde2/teradata/TeradataBinaryDataInputStream.java| 2 +-
.../hive/serde2/teradata/TeradataBinaryDataOutputStream.java | 2 +-
.../hadoop/hive/serde2/teradata/TeradataBinarySerde.java | 4 ++--
.../hadoop/hive/serde2/thrift/TCTLSeparatedProtocol.java | 2 +-
.../hadoop/hive/serde2/typeinfo/TimestampLocalTZTypeInfo.java | 2 +-
.../org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java | 6 +++---
.../org/apache/hadoop/hive/serde2/SerdeRandomRowSource.java| 2 +-
.../src/test/org/apache/hadoop/hive/serde2/TestJsonSerDe.java | 4 ++--
.../apache/hadoop/hive/serde2/TestTCTLSeparatedProtocol.java | 10 +-
.../apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java | 2 +-
.../hive/serde2/avro/TestAvroObjectInspectorGenerator.java | 8
.../org/apache/hadoop/hive/serde2/avro/TestAvroSerdeUtils.java | 4 ++--
.../hive/serde2/binarysortable/TestBinarySortableFast.java | 2 +-
.../org/apache/hadoop/hive/serde2/lazy/TestLazySimpleFast.java | 2 +-
.../hadoop/hive/serde2/lazybinary/TestLazyBinaryFast.java | 2 +-
.../serde2/objectinspector/TestObjectInspectorConverters.java | 4 ++--
35 files changed, 59 insertions(+), 59 deletions(-)
diff --git a/serde/src/java/org/apache/hadoop/hive/serde2/RandomTypeUtil.java
b/serde/src/java/org/apache/hadoop/hive/serde2/RandomTypeUtil.java
index ad9de4c..c0e0583 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/RandomTypeUtil.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/RandomTypeUtil.java
@@ -131,7 +131,7 @@ public class RandomTypeUtil {
public static final long NANOSECONDS_PER_SECOND =
TimeUnit.SECONDS.toNanos(1);
public static final long MILLISECONDS_PER_SECOND =
TimeUnit.SECONDS.toMillis(1);
- public static final long NANOSECONDS_PER_MILLISSECOND =
TimeUnit.MILLISECONDS.toNanos(1);
+ public static final long NANOSECONDS_PER_MILLISECOND =
TimeUnit.MILLISECONDS.toNanos(1);
private static final ThreadLocal DATE_FORMAT =
new ThreadLocal() {
@@ -172,12 +172,12 @@ public class RandomTypeUtil {
case 2:
// Limit to milliseconds only...
optionalNanos = String.format(".%09d",
- Integer.valueOf(r.nextInt((int) MILLISECONDS_PER_SECOND)) *
NANOSECONDS_PER_MILLISSECOND);
+ Integer.valueOf(r.nextInt((int) MILLISECONDS_PER_SECOND)) *
NANOSECONDS_PER_MILLISECOND);
break;
case 3:
// Limit to below milliseconds only...
optionalNanos = String.format(".%09d",
- Integer.valueOf(r.nextInt((int) NANOSECONDS_PER_MILLISSECOND)));
+ Integer.valueOf(r.nextInt((int) NANOSECONDS_PER_MILLISECOND)));
break;
}
String timestampStr = String.format("%04d-%02d-%02d %02d:%02d:%02d%s",
diff --git a/serde/src/java/org/apache/hadoop/hive/serde2/SerDeStatsStruct.java
b/serde/src/java/org/apache/hadoop/hive/serde2/SerDeStatsStruct.java
index cf86e5e..8f0fa9d 100644
--- a/s
[hive] branch master updated: HIVE-25582: Empty result when using offset limit with MR (#2693) (Zhihua Deng reviewed by Laszlo Bodor and Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new cb23045 HIVE-25582: Empty result when using offset limit with MR
(#2693) (Zhihua Deng reviewed by Laszlo Bodor and Zoltan Haindrich)
cb23045 is described below
commit cb23045f92c62bc43ef5739532b486b524d99e03
Author: dengzh
AuthorDate: Tue Nov 23 16:02:58 2021 +0800
HIVE-25582: Empty result when using offset limit with MR (#2693) (Zhihua
Deng reviewed by Laszlo Bodor and Zoltan Haindrich)
---
.../test/resources/testconfiguration.properties| 1 +
.../apache/hadoop/hive/ql/exec/ObjectCache.java| 1 -
.../apache/hadoop/hive/ql/exec/mr/ObjectCache.java | 23 --
.../test/queries/clientpositive/offset_limit_mr.q | 12 +++
.../results/clientpositive/offset_limit_mr.q.out | 88 ++
5 files changed, 118 insertions(+), 7 deletions(-)
diff --git a/itests/src/test/resources/testconfiguration.properties
b/itests/src/test/resources/testconfiguration.properties
index 638af07..6b887f5 100644
--- a/itests/src/test/resources/testconfiguration.properties
+++ b/itests/src/test/resources/testconfiguration.properties
@@ -239,6 +239,7 @@ mr.query.files=\
masking_5.q,\
nonmr_fetch.q,\
nonreserved_keywords_input37.q,\
+ offset_limit_mr.q,\
parenthesis_star_by.q,\
partition_vs_table_metadata.q,\
row__id.q,\
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java
index cf04e1d..c9282b3 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java
@@ -48,7 +48,6 @@ public interface ObjectCache {
*
* @param
* @param key
- * function to generate the object if it's not there
* @return the last cached object with the key, null if none.
*/
public T retrieve(String key) throws HiveException;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java
index 5bb96e3..0acf6d7 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java
@@ -18,7 +18,9 @@
package org.apache.hadoop.hive.ql.exec.mr;
+import java.util.Map;
import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
@@ -26,36 +28,45 @@ import java.util.concurrent.TimeoutException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+
import org.apache.hadoop.hive.ql.metadata.HiveException;
/**
- * ObjectCache. No-op implementation on MR we don't have a means to reuse
- * Objects between runs of the same task.
+ * ObjectCache. Simple implementation on MR we don't have a means to reuse
+ * Objects between runs of the same task, this acts as a local cache.
*
*/
public class ObjectCache implements org.apache.hadoop.hive.ql.exec.ObjectCache
{
private static final Logger LOG =
LoggerFactory.getLogger(ObjectCache.class.getName());
+ private final Map cache = new ConcurrentHashMap<>();
+
@Override
public void release(String key) {
-// nothing to do
LOG.debug("{} no longer needed", key);
+cache.remove(key);
}
@Override
public T retrieve(String key) throws HiveException {
-return retrieve(key, null);
+return (T) cache.get(key);
}
@Override
public T retrieve(String key, Callable fn) throws HiveException {
+T value = (T) cache.get(key);
+if (value != null || fn == null) {
+ return value;
+}
try {
LOG.debug("Creating {}", key);
- return fn.call();
+ value = fn.call();
} catch (Exception e) {
throw new HiveException(e);
}
+T previous = (T) cache.putIfAbsent(key, value);
+return previous != null ? previous : value;
}
@Override
@@ -94,6 +105,6 @@ public class ObjectCache implements
org.apache.hadoop.hive.ql.exec.ObjectCache {
@Override
public void remove(String key) {
-// nothing to do
+cache.remove(key);
}
}
diff --git a/ql/src/test/queries/clientpositive/offset_limit_mr.q
b/ql/src/test/queries/clientpositive/offset_limit_mr.q
new file mode 100644
index 000..caba496
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/offset_limit_mr.q
@@ -0,0 +1,12 @@
+--! qt:dataset:src
+
+SELECT src.key, sum(substr(src.value,5)) FROM src GROUP BY src.key ORDER BY
src.key LIMIT 10,10;
+
+SELECT src.key, sum(substr(src.value,5)) FROM src GROUP BY src.key ORDER BY
src.key LIMIT 0,10;
+
+SELECT src.key, sum(substr(src.value,5)) FROM src GROUP BY src.key ORDER BY
src.key LIMIT 1,10;
+
+
[hive] branch master updated: HIVE-25680 : Authorize #get_table_meta HiveMetastore Server API to use any of the HiveMetastore Authorization model (#2770) (Syed Shameerur Rahman reviewed by Zoltan Hain
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 78b20c8 HIVE-25680 : Authorize #get_table_meta HiveMetastore Server
API to use any of the HiveMetastore Authorization model (#2770) (Syed Shameerur
Rahman reviewed by Zoltan Haindrich)
78b20c8 is described below
commit 78b20c803ef2e75a7fe830325df2c19c15b203a3
Author: Syed Shameerur Rahman
AuthorDate: Tue Nov 23 13:31:36 2021 +0530
HIVE-25680 : Authorize #get_table_meta HiveMetastore Server API to use any
of the HiveMetastore Authorization model (#2770) (Syed Shameerur Rahman
reviewed by Zoltan Haindrich)
---
...stMetastoreClientSideAuthorizationProvider.java | 145 +
.../apache/hadoop/hive/metastore/HMSHandler.java | 52
2 files changed, 197 insertions(+)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestMetastoreClientSideAuthorizationProvider.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestMetastoreClientSideAuthorizationProvider.java
new file mode 100644
index 000..dbd71cb
--- /dev/null
+++
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/TestMetastoreClientSideAuthorizationProvider.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.security;
+
+import com.google.common.collect.Lists;
+import org.apache.hadoop.hive.cli.CliSessionState;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.MetaStoreTestUtils;
+import org.apache.hadoop.hive.metastore.api.TableMeta;
+import org.apache.hadoop.hive.ql.DriverFactory;
+import org.apache.hadoop.hive.ql.IDriver;
+import
org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.shims.Utils;
+import org.apache.hadoop.security.UserGroupInformation;
+import static org.junit.Assert.assertEquals;
+
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.List;
+
+/**
+ * TestMetastoreClientSideAuthorizationProvider : Simple base test for
Metastore client side
+ * Authorization Providers. By default, tests DefaultHiveAuthorizationProvider
+ */
+public class TestMetastoreClientSideAuthorizationProvider {
+private HiveConf clientHiveConf;
+private HiveMetaStoreClient msc;
+private IDriver driver;
+private UserGroupInformation ugi;
+
+@Before
+public void setUp() throws Exception {
+
System.setProperty(HiveConf.ConfVars.METASTORE_PRE_EVENT_LISTENERS.varname,
+
"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener");
+
+int port = MetaStoreTestUtils.startMetaStoreWithRetry();
+
+clientHiveConf = new HiveConf(this.getClass());
+
+// Turn on client-side authorization
+
clientHiveConf.setBoolVar(HiveConf.ConfVars.HIVE_AUTHORIZATION_ENABLED,true);
+
clientHiveConf.set(HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER.varname,
+getAuthorizationProvider());
+
clientHiveConf.set(HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER.varname,
+InjectableDummyAuthenticator.class.getName());
+
clientHiveConf.set(HiveConf.ConfVars.HIVE_AUTHORIZATION_TABLE_OWNER_GRANTS.varname,
"");
+clientHiveConf.setVar(HiveConf.ConfVars.HIVEMAPREDMODE, "nonstrict");
+clientHiveConf.setVar(HiveConf.ConfVars.METASTOREURIS,
"thrift://localhost:" + port);
+
clientHiveConf.setIntVar(HiveConf.ConfVars.METASTORETHRIFTCONNECTIONRETRIES, 3);
+clientHiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname,
"false");
+
+clientHiveConf.set(HiveConf.ConfVars.PREEXECHOOKS.varname, "");
+clientHiveConf.set(HiveConf.ConfVars.POSTEXECHOOKS.varname, "");
+
+ugi = Utils.getUGI();
+
+Sessi
[hive] branch master updated: HIVE-25679: Use serdeContants#COLLECTION_DELIM in MultiDelimSerDe (#2768) (Franck Thang reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 15263be HIVE-25679: Use serdeContants#COLLECTION_DELIM in
MultiDelimSerDe (#2768) (Franck Thang reviewed by Zoltan Haindrich)
15263be is described below
commit 15263be5abe2050fe1f2cabf30ff77852728bcac
Author: Franck
AuthorDate: Mon Nov 22 12:57:18 2021 +0100
HIVE-25679: Use serdeContants#COLLECTION_DELIM in MultiDelimSerDe (#2768)
(Franck Thang reviewed by Zoltan Haindrich)
---
serde/src/java/org/apache/hadoop/hive/serde2/MultiDelimitSerDe.java | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git
a/serde/src/java/org/apache/hadoop/hive/serde2/MultiDelimitSerDe.java
b/serde/src/java/org/apache/hadoop/hive/serde2/MultiDelimitSerDe.java
index 42734af..0ec33f4 100644
--- a/serde/src/java/org/apache/hadoop/hive/serde2/MultiDelimitSerDe.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/MultiDelimitSerDe.java
@@ -65,8 +65,6 @@ import org.apache.hadoop.io.Writable;
public class MultiDelimitSerDe extends AbstractEncodingAwareSerDe {
private static final byte[] DEFAULT_SEPARATORS = {(byte) 1, (byte) 2, (byte)
3};
- // Due to HIVE-6404, define our own constant
- private static final String COLLECTION_DELIM = "collection.delim";
// actual delimiter(fieldDelimited) is replaced by REPLACEMENT_DELIM in row.
public static final String REPLACEMENT_DELIM_SEQUENCE = "\1";
@@ -107,8 +105,7 @@ public class MultiDelimitSerDe extends
AbstractEncodingAwareSerDe {
}
// get the collection separator and map key separator
-// TODO: use serdeConstants.COLLECTION_DELIM when the typo is fixed
-collSep = LazyUtils.getByte(properties.getProperty(COLLECTION_DELIM),
+collSep =
LazyUtils.getByte(properties.getProperty(serdeConstants.COLLECTION_DELIM),
DEFAULT_SEPARATORS[1]);
keySep =
LazyUtils.getByte(properties.getProperty(serdeConstants.MAPKEY_DELIM),
DEFAULT_SEPARATORS[2]);
[hive] branch master updated: HIVE-24849: Create external table timeouts when location has large number of files (#2567) (Sungwoo Park reviewed by Steve Loughran, Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new a814430 HIVE-24849: Create external table timeouts when location has
large number of files (#2567) (Sungwoo Park reviewed by Steve Loughran, Zoltan
Haindrich)
a814430 is described below
commit a81443033d31a64502219dec6e0b9b1802b865b0
Author: Sungwoo Park
AuthorDate: Mon Nov 22 20:23:25 2021 +0900
HIVE-24849: Create external table timeouts when location has large number
of files (#2567) (Sungwoo Park reviewed by Steve Loughran, Zoltan Haindrich)
---
.../org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java
b/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java
index 0531aee..03ade96 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/create/CreateTableDesc.java
@@ -927,7 +927,7 @@ public class CreateTableDesc implements DDLDesc,
Serializable {
// When replicating the statistics for a table will be obtained from the
source. Do not
// reset it on replica.
if (replicationSpec == null || !replicationSpec.isInReplicationScope()) {
- if (!this.isCTAS && (tbl.getPath() == null || (tbl.isEmpty() &&
!isExternal( {
+ if (!this.isCTAS && (tbl.getPath() == null || (!isExternal() &&
tbl.isEmpty( {
if (!tbl.isPartitioned() &&
conf.getBoolVar(HiveConf.ConfVars.HIVESTATSAUTOGATHER)) {
StatsSetupConst.setStatsStateForCreateTable(tbl.getTTable().getParameters(),
MetaStoreUtils.getColumnNames(tbl.getCols()),
StatsSetupConst.TRUE);
[hive] branch master updated: HIVE-25714: docker logs commands timeout regularily during testing (#2801) (Zoltan Haindrich reviewed by Stamatis Zampetakis)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 79098a7 HIVE-25714: docker logs commands timeout regularily during
testing (#2801) (Zoltan Haindrich reviewed by Stamatis Zampetakis)
79098a7 is described below
commit 79098a75baeffc683f73afb4788d4cb2b2484ad6
Author: Zoltan Haindrich
AuthorDate: Fri Nov 19 11:54:55 2021 +0100
HIVE-25714: docker logs commands timeout regularily during testing (#2801)
(Zoltan Haindrich reviewed by Stamatis Zampetakis)
---
.../apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java | 11 ++-
.../hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java | 3 ++-
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
index ba0c7ad..f328bfc 100644
---
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
+++
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java
@@ -36,7 +36,7 @@ import java.util.concurrent.TimeUnit;
/**
* The class is in charge of connecting and populating dockerized databases
for qtest.
- *
+ *
* The database should have at least one root user (admin/superuser) able to
modify every aspect of the system. The user
* either exists by default when the database starts or must created right
after startup.
*/
@@ -99,6 +99,7 @@ public abstract class AbstractExternalDB {
reader = new BufferedReader(new
InputStreamReader(proc.getErrorStream()));
final StringBuilder errLines = new StringBuilder();
reader.lines().forEach(s -> errLines.append(s).append('\n'));
+LOG.info("Result size: " + lines.length() + ";" + errLines.length());
return new ProcessResults(lines.toString(), errLines.toString(),
proc.exitValue());
}
@@ -120,7 +121,7 @@ public abstract class AbstractExternalDB {
ProcessResults pr;
do {
Thread.sleep(1000);
-pr = runCmd(buildLogCmd(), 5);
+pr = runCmd(buildLogCmd(), 30);
if (pr.rc != 0) {
throw new RuntimeException("Failed to get docker logs");
}
@@ -149,7 +150,7 @@ public abstract class AbstractExternalDB {
/**
* Return the name of the root user.
- *
+ *
* Override the method if the name of the root user must be different than
the default.
*/
protected String getRootUser() {
@@ -158,13 +159,13 @@ public abstract class AbstractExternalDB {
/**
* Return the password of the root user.
- *
+ *
* Override the method if the password must be different than the default.
*/
protected String getRootPassword() {
return "qtestpassword";
}
-
+
protected abstract String getJdbcUrl();
protected abstract String getJdbcDriver();
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
index f4e4388..4fc8d50 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/DatabaseRule.java
@@ -132,7 +132,7 @@ public abstract class DatabaseRule extends ExternalResource
{
ProcessResults pr;
do {
Thread.sleep(1000);
- pr = runCmd(buildLogCmd(), 5);
+ pr = runCmd(buildLogCmd(), 30);
if (pr.rc != 0) {
throw new RuntimeException("Failed to get docker logs");
}
@@ -185,6 +185,7 @@ public abstract class DatabaseRule extends ExternalResource
{
reader = new BufferedReader(new InputStreamReader(proc.getErrorStream()));
final StringBuilder errLines = new StringBuilder();
reader.lines().forEach(s -> errLines.append(s).append('\n'));
+LOG.info("Result size: " + lines.length() + ";" + errLines.length());
return new ProcessResults(lines.toString(), errLines.toString(),
proc.exitValue());
}
[hive] branch master updated (bfaaad4 -> 3f3d210)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from bfaaad4 HIVE-25531: Remove the core classified hive-exec artifact (Zoltan Haindrich, reviewed by Stamatis Zampetakis) add 3f3d210 HIVE-24390: Spelling fixes (#1674) (Josh Soref reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: RELEASE_NOTES.txt | 6 ++-- .../hive/accumulo/AccumuloHiveConstants.java | 2 +- .../hadoop/hive/accumulo/HiveAccumuloHelper.java | 2 +- .../accumulo/columns/ColumnMappingFactory.java | 2 +- .../accumulo/mr/HiveAccumuloTableInputFormat.java | 2 +- .../predicate/AccumuloPredicateHandler.java| 2 +- bin/hive | 2 +- .../java/org/apache/hadoop/hive/conf/HiveConf.java | 8 ++--- .../hive/common/metrics/TestLegacyMetrics.java | 6 ++-- hcatalog/bin/hcat | 2 +- hcatalog/src/docs/src/documentation/conf/cli.xconf | 2 +- .../docs/src/documentation/content/locationmap.xml | 2 +- .../src/documentation/content/xdocs/example.xml| 2 +- .../hive/hcatalog/api/repl/ReplicationTask.java| 2 +- .../hive/hcatalog/templeton/ExecServiceImpl.java | 8 ++--- itests/hive-minikdc/pom.xml| 2 +- .../hive/minikdc/JdbcWithMiniKdcSQLAuthTest.java | 2 +- .../parse/BaseReplicationScenariosAcidTables.java | 2 +- .../parse/TestReplicationScenariosAcidTables.java | 16 - ...estReplicationScenariosAcidTablesBootstrap.java | 2 +- .../parse/TestTableLevelReplicationScenarios.java | 4 +-- .../TestMultiAuthorizationPreEventListener.java| 2 +- .../TestJdbcWithSQLAuthorization.java | 6 ++-- .../hive/jdbc/miniHS2/StartMiniHS2Cluster.java | 2 +- itests/qtest-druid/pom.xml | 2 +- .../control/AbstractCoreBlobstoreCliDriver.java| 2 +- .../java/org/apache/hadoop/hive/ql/QTestUtil.java | 2 +- .../hadoop/hive/ql/hooks/CheckTableAccessHook.java | 2 +- .../java/org/apache/hive/jdbc/miniHS2/MiniHS2.java | 4 +-- .../llap/metrics/TestReadWriteLockMetrics.java | 2 +- .../hive/llap/shufflehandler/DirWatcher.java | 4 +-- .../llap/shufflehandler/FadvisedFileRegion.java| 2 +- .../hive/llap/shufflehandler/ShuffleHandler.java | 2 +- .../comparator/TestFirstInFirstOutComparator.java | 2 +- .../llap/shufflehandler/TestShuffleHandler.java| 6 ++-- .../hive/llap/tezplugins/LlapTaskCommunicator.java | 2 +- .../llap/tezplugins/LlapTaskSchedulerService.java | 6 ++-- parser/pom.xml | 4 +-- .../UDAFTemplates/VectorUDAFMinMax.txt | 42 +++--- .../UDAFTemplates/VectorUDAFMinMaxDecimal.txt | 42 +++--- .../VectorUDAFMinMaxIntervalDayTime.txt| 42 +++--- .../UDAFTemplates/VectorUDAFMinMaxString.txt | 42 +++--- .../UDAFTemplates/VectorUDAFMinMaxTimestamp.txt| 42 +++--- .../hive/llap/WritableByteChannelAdapter.java | 2 +- .../hadoop/hive/ql/ddl/table/AlterTableType.java | 4 +-- .../hadoop/hive/ql/exec/ReduceSinkOperator.java| 2 +- .../org/apache/hadoop/hive/ql/exec/TopNHash.java | 6 ++-- .../org/apache/hadoop/hive/ql/exec/Utilities.java | 10 +++--- .../apache/hadoop/hive/ql/exec/mr/MapRedTask.java | 2 +- .../ql/exec/persistence/HybridHashTableConf.java | 2 +- .../aggregates/VectorUDAFBloomFilter.java | 24 ++--- .../aggregates/VectorUDAFBloomFilterMerge.java | 24 ++--- .../optimized/VectorMapJoinOptimizedHashTable.java | 12 +++ .../org/apache/hadoop/hive/ql/io/AcidUtils.java| 2 +- .../hadoop/hive/ql/io/FlatFileInputFormat.java | 2 +- .../apache/hadoop/hive/ql/lock/CompileLock.java| 14 .../zookeeper/ZooKeeperHiveLockManager.java| 2 +- .../DynamicPartitionPruningOptimization.java | 2 +- .../hadoop/hive/ql/optimizer/GenMapRedUtils.java | 2 +- .../hadoop/hive/ql/optimizer/MapJoinProcessor.java | 2 +- .../calcite/CalciteSemanticException.java | 2 +- .../calcite/rules/HiveRelColumnsAlignment.java | 2 +- .../calcite/rules/HiveSubQueryRemoveRule.java | 2 +- .../optimizer/calcite/translator/ASTConverter.java | 2 +- .../calcite/translator/RexNodeConverter.java | 4 +-- .../correlation/ReduceSinkDeDuplication.java | 8 ++--- .../BucketingSortingInferenceOptimizer.java| 2 +- .../optimizer/physical/GenMRSkewJoinProcessor.java | 2 +- .../hadoop/hive/ql/parse/CalcitePlanner.java | 6 ++-- .../hive/ql/parse/ImportSemanticAnalyzer.java | 2 +- .../apache/hadoop/hive/ql/parse/QBSubQuery.java| 6 ++-- .../apache/hadoop/hive/ql/parse/RowResolver.java | 4 +-- .../hadoop/hive/ql/parse
[hive] branch master updated: HIVE-25531: Remove the core classified hive-exec artifact (Zoltan Haindrich, reviewed by Stamatis Zampetakis)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new bfaaad4 HIVE-25531: Remove the core classified hive-exec artifact (Zoltan Haindrich, reviewed by Stamatis Zampetakis) bfaaad4 is described below commit bfaaad4aa433963947d2a40020a432f19fc2b959 Author: Zoltan Haindrich AuthorDate: Thu Nov 18 16:17:58 2021 + HIVE-25531: Remove the core classified hive-exec artifact (Zoltan Haindrich, reviewed by Stamatis Zampetakis) Closes #2648 --- itests/qtest-accumulo/pom.xml | 6 -- itests/qtest-kudu/pom.xml | 6 -- ql/pom.xml| 21 - 3 files changed, 33 deletions(-) diff --git a/itests/qtest-accumulo/pom.xml b/itests/qtest-accumulo/pom.xml index 7c03159..41c125f 100644 --- a/itests/qtest-accumulo/pom.xml +++ b/itests/qtest-accumulo/pom.xml @@ -61,12 +61,6 @@ org.apache.hive hive-exec test - core - - - org.apache.hive - hive-exec - test tests diff --git a/itests/qtest-kudu/pom.xml b/itests/qtest-kudu/pom.xml index 5a894bd..c83369b 100644 --- a/itests/qtest-kudu/pom.xml +++ b/itests/qtest-kudu/pom.xml @@ -50,12 +50,6 @@ org.apache.hive hive-exec test - core - - - org.apache.hive - hive-exec - test tests diff --git a/ql/pom.xml b/ql/pom.xml index c82f965..4054bb3 100644 --- a/ql/pom.xml +++ b/ql/pom.xml @@ -1037,26 +1037,6 @@ - -core-jar -package - - jar - - - core - - - fallbackauthorizer-jar package @@ -1082,7 +1062,6 @@ shade -
[hive] branch master updated: HIVE-25443 : Arrow SerDe Cannot serialize/deserialize complex data types When there are more than 1024 values (#2581) (Syed Shameerur Rahman reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 8e693d1 HIVE-25443 : Arrow SerDe Cannot serialize/deserialize complex
data types When there are more than 1024 values (#2581) (Syed Shameerur Rahman
reviewed by Zoltan Haindrich)
8e693d1 is described below
commit 8e693d1b36e1ff0aacd802d16e1a3d0ec72ef04b
Author: Syed Shameerur Rahman
AuthorDate: Thu Nov 18 12:59:50 2021 +0530
HIVE-25443 : Arrow SerDe Cannot serialize/deserialize complex data types
When there are more than 1024 values (#2581) (Syed Shameerur Rahman reviewed by
Zoltan Haindrich)
---
.../hive/ql/io/arrow/ArrowColumnarBatchSerDe.java | 4 +-
.../hadoop/hive/ql/io/arrow/Deserializer.java | 3 ++
.../ql/io/arrow/TestArrowColumnarBatchSerDe.java | 43 ++
3 files changed, 48 insertions(+), 2 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java
b/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java
index fdef3b8..ceb794f 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/ArrowColumnarBatchSerDe.java
@@ -210,9 +210,9 @@ public class ArrowColumnarBatchSerDe extends AbstractSerDe {
static ListColumnVector toStructListVector(MapColumnVector mapVector) {
final StructColumnVector structVector;
final ListColumnVector structListVector;
-structVector = new StructColumnVector();
+structVector = new StructColumnVector(mapVector.childCount);
structVector.fields = new ColumnVector[] {mapVector.keys,
mapVector.values};
-structListVector = new ListColumnVector();
+structListVector = new ListColumnVector(mapVector.childCount, null);
structListVector.child = structVector;
structListVector.childCount = mapVector.childCount;
structListVector.isRepeating = mapVector.isRepeating;
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Deserializer.java
b/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Deserializer.java
index ac4d237..ce8488f 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Deserializer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Deserializer.java
@@ -391,6 +391,7 @@ class Deserializer {
private void readList(FieldVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo) {
final int size = arrowVector.getValueCount();
+hiveVector.ensureSize(size, false);
final ArrowBuf offsets = arrowVector.getOffsetBuffer();
final int OFFSET_WIDTH = 4;
@@ -412,6 +413,7 @@ class Deserializer {
private void readMap(FieldVector arrowVector, MapColumnVector hiveVector,
MapTypeInfo typeInfo) {
final int size = arrowVector.getValueCount();
+hiveVector.ensureSize(size, false);
final ListTypeInfo mapStructListTypeInfo = toStructListTypeInfo(typeInfo);
final ListColumnVector mapStructListVector =
toStructListVector(hiveVector);
final StructColumnVector mapStructVector = (StructColumnVector)
mapStructListVector.child;
@@ -430,6 +432,7 @@ class Deserializer {
private void readStruct(FieldVector arrowVector, StructColumnVector
hiveVector, StructTypeInfo typeInfo) {
final int size = arrowVector.getValueCount();
+hiveVector.ensureSize(size, false);
final List fieldTypeInfos =
typeInfo.getAllStructFieldTypeInfos();
final int fieldSize = arrowVector.getChildrenFromFields().size();
for (int i = 0; i < fieldSize; i++) {
diff --git
a/ql/src/test/org/apache/hadoop/hive/ql/io/arrow/TestArrowColumnarBatchSerDe.java
b/ql/src/test/org/apache/hadoop/hive/ql/io/arrow/TestArrowColumnarBatchSerDe.java
index d803063..a4b296b 100644
---
a/ql/src/test/org/apache/hadoop/hive/ql/io/arrow/TestArrowColumnarBatchSerDe.java
+++
b/ql/src/test/org/apache/hadoop/hive/ql/io/arrow/TestArrowColumnarBatchSerDe.java
@@ -17,6 +17,7 @@
*/
package org.apache.hadoop.hive.ql.io.arrow;
+import static
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_ARROW_BATCH_SIZE;
import com.google.common.base.Joiner;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
@@ -157,6 +158,7 @@ public class TestArrowColumnarBatchSerDe {
@Before
public void setUp() {
conf = new Configuration();
+conf.setInt(HIVE_ARROW_BATCH_SIZE.varname, 1025);
}
private static ByteWritable byteW(int value) {
@@ -1024,4 +1026,45 @@ public class TestArrowColumnarBatchSerDe {
initAndSerializeAndDeserialize(schema, toList(DECIMAL_ROWS));
}
+ @Test
+ public void testListBooleanWithMoreThan1024Values() throws SerDeException {
+String[][] schema = {
+{"boolean_list", "array"},
+};
+
+Object[][] rows = new Object[1025][1];
+for (int i = 0; i < 102
[hive] branch master updated: HIVE-25095: Beeline/hive -e command can't deal with query with trailing quote (#2526) (Robbie Zhang reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 651b0ef HIVE-25095: Beeline/hive -e command can't deal with query
with trailing quote (#2526) (Robbie Zhang reviewed by Zoltan Haindrich)
651b0ef is described below
commit 651b0ef7f1e4c6b0e4696ccd75e032ab965006bc
Author: Robbie Zhang
AuthorDate: Thu Nov 18 18:22:07 2021 +1100
HIVE-25095: Beeline/hive -e command can't deal with query with trailing
quote (#2526) (Robbie Zhang reviewed by Zoltan Haindrich)
Co-authored-by: Robbie Zhang
---
beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java | 4 +++-
beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java | 6 ++
pom.xml | 2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git
a/beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
b/beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
index 26e118a..26a892f 100644
--- a/beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
+++ b/beeline/src/test/org/apache/hive/beeline/TestBeelineArgParsing.java
@@ -193,11 +193,13 @@ public class TestBeelineArgParsing {
public void testQueryScripts() throws Exception {
TestBeeline bl = new TestBeeline();
String args[] = new String[] {"-u", "url", "-n", "name",
- "-p", "password", "-d", "driver", "-e", "select1", "-e", "select2"};
+ "-p", "password", "-d", "driver", "-e", "select1", "-e", "select2",
+ "-e", "select \"hive\""};
Assert.assertEquals(0, bl.initArgs(args));
Assert.assertTrue(bl.connectArgs.equals("url name password driver"));
Assert.assertTrue(bl.queries.contains("select1"));
Assert.assertTrue(bl.queries.contains("select2"));
+Assert.assertTrue(bl.queries.contains("select \"hive\""));
}
/**
diff --git a/beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
b/beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
index 1e68405..5ea4d11 100644
--- a/beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
+++ b/beeline/src/test/org/apache/hive/beeline/cli/TestHiveCli.java
@@ -168,6 +168,12 @@ public class TestHiveCli {
}
@Test
+ public void testSqlFromCmdWithEmbeddedQuotes() {
+verifyCMD(null, "hive", out,
+new String[] { "-e", "select \"hive\"" }, ERRNO_OK, true);
+ }
+
+ @Test
public void testInvalidOptions() {
verifyCMD(null,
"The '-e' and '-f' options cannot be specified simultaneously", err,
diff --git a/pom.xml b/pom.xml
index c5542e4..3f28653 100644
--- a/pom.xml
+++ b/pom.xml
@@ -118,7 +118,7 @@
5.2.4
3.2.0-release
5.2.4
-1.2
+1.4
1.15
3.2.2
4.1
[hive] branch master updated: disable unstable tests
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 2cdfa8d disable unstable tests
2cdfa8d is described below
commit 2cdfa8d3e6241c28803d804a8042f784c032d061
Author: Zoltan Haindrich
AuthorDate: Wed Nov 17 17:37:45 2021 +
disable unstable tests
---
.../apache/hadoop/hive/ql/parse/TestScheduledReplicationScenarios.java | 2 ++
.../org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics.java | 1 +
ql/src/test/queries/clientpositive/replication_metrics_ingest.q | 1 +
3 files changed, 4 insertions(+)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestScheduledReplicationScenarios.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestScheduledReplicationScenarios.java
index cca89b9..e0b389e 100644
---
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestScheduledReplicationScenarios.java
+++
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestScheduledReplicationScenarios.java
@@ -254,6 +254,7 @@ public class TestScheduledReplicationScenarios extends
BaseReplicationScenariosA
}
@Test
+ @Ignore("HIVE-25720")
public void testCompleteFailoverWithReverseBootstrap() throws Throwable {
String withClause = "'" + HiveConf.ConfVars.HIVE_IN_TEST + "' = 'true','"
+ HiveConf.ConfVars.REPL_RETAIN_PREV_DUMP_DIR + "'='true'" ;
@@ -400,6 +401,7 @@ public class TestScheduledReplicationScenarios extends
BaseReplicationScenariosA
}
@Test
+ @Ignore("HIVE-25720")
public void testSetPolicyId() throws Throwable {
String withClause =
" WITH('" + HiveConf.ConfVars.HIVE_IN_TEST + "' = 'true'" + ",'"
diff --git
a/ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics.java
b/ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics.java
index 51fe394..13c97fb 100644
---
a/ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics.java
+++
b/ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCompactionMetrics.java
@@ -178,6 +178,7 @@ public class TestCompactionMetrics extends CompactorTest {
}
@Test
+ @org.junit.Ignore("HIVE-25716")
public void testOldestReadyForCleaningAge() throws Exception {
conf.setIntVar(HiveConf.ConfVars.COMPACTOR_MAX_NUM_DELTA, 1);
diff --git a/ql/src/test/queries/clientpositive/replication_metrics_ingest.q
b/ql/src/test/queries/clientpositive/replication_metrics_ingest.q
index a710b00..a03aa86 100644
--- a/ql/src/test/queries/clientpositive/replication_metrics_ingest.q
+++ b/ql/src/test/queries/clientpositive/replication_metrics_ingest.q
@@ -1,3 +1,4 @@
+--! qt:disabled:HIVE-25719
--! qt:authorizer
--! qt:scheduledqueryservice
--! qt:sysdb
[hive] branch master updated: HIVE-25670: Avoid getTable() calls for foreign key tables not used in… (#2763) (Steve Carlin reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 47ba530 HIVE-25670: Avoid getTable() calls for foreign key tables not
used in… (#2763) (Steve Carlin reviewed by Zoltan Haindrich)
47ba530 is described below
commit 47ba530464c6941b4fc0f2882fd09ce1683808d6
Author: scarlin-cloudera <55709772+scarlin-cloud...@users.noreply.github.com>
AuthorDate: Wed Nov 17 08:23:42 2021 -0800
HIVE-25670: Avoid getTable() calls for foreign key tables not used in…
(#2763) (Steve Carlin reviewed by Zoltan Haindrich)
RelOptHiveTable currently fetches the Table information for all
referential constraint tables. However, it only needs to fetch the tables
that are used in the query.
---
.../ql/metadata/HiveMaterializedViewsRegistry.java | 5 +-
.../hive/ql/optimizer/calcite/RelOptHiveTable.java | 47 +++--
.../hadoop/hive/ql/parse/CalcitePlanner.java | 5 +-
.../apache/hadoop/hive/ql/parse/ParseContext.java | 6 +-
.../hadoop/hive/ql/parse/ParsedQueryTables.java| 29 +
.../apache/hadoop/hive/ql/parse/QueryTables.java | 76 ++
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 10 +--
7 files changed, 142 insertions(+), 36 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
b/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
index 9af4d0a..fba0a02 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java
@@ -66,6 +66,7 @@ import
org.apache.hadoop.hive.ql.optimizer.calcite.translator.TypeConverter;
import org.apache.hadoop.hive.ql.parse.CBOPlan;
import org.apache.hadoop.hive.ql.parse.CalcitePlanner;
import org.apache.hadoop.hive.ql.parse.ParseUtils;
+import org.apache.hadoop.hive.ql.parse.QueryTables;
import org.apache.hadoop.hive.ql.parse.RowResolver;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.serde2.SerDeException;
@@ -459,7 +460,7 @@ public final class HiveMaterializedViewsRegistry {
// for materialized views.
RelOptHiveTable optTable = new RelOptHiveTable(null,
cluster.getTypeFactory(), fullyQualifiedTabName,
rowType, viewTable, nonPartitionColumns, partitionColumns, new
ArrayList<>(),
- conf, null, new HashMap<>(), new HashMap<>(), new HashMap<>(), new
AtomicInteger());
+ conf, null, new QueryTables(true), new HashMap<>(), new HashMap<>(),
new AtomicInteger());
DruidTable druidTable = new DruidTable(new DruidSchema(address, address,
false),
dataSource, RelDataTypeImpl.proto(rowType), metrics,
DruidTable.DEFAULT_TIMESTAMP_COLUMN,
intervals, null, null);
@@ -474,7 +475,7 @@ public final class HiveMaterializedViewsRegistry {
// for materialized views.
RelOptHiveTable optTable = new RelOptHiveTable(null,
cluster.getTypeFactory(), fullyQualifiedTabName,
rowType, viewTable, nonPartitionColumns, partitionColumns, new
ArrayList<>(),
- conf, null, new HashMap<>(), new HashMap<>(), new HashMap<>(), new
AtomicInteger());
+ conf, null, new QueryTables(true), new HashMap<>(), new HashMap<>(),
new AtomicInteger());
tableRel = new HiveTableScan(cluster,
cluster.traitSetOf(HiveRelNode.CONVENTION), optTable,
viewTable.getTableName(), null, false, false);
}
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
index 8f9b78c..385fe9a 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
@@ -67,6 +67,7 @@ import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
import
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ExprNodeConverter;
import org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner;
import org.apache.hadoop.hive.ql.parse.ColumnStatsList;
+import org.apache.hadoop.hive.ql.parse.ParsedQueryTables;
import org.apache.hadoop.hive.ql.parse.PrunedPartitionList;
import org.apache.hadoop.hive.ql.plan.ColStatistics;
import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
@@ -82,6 +83,7 @@ import org.slf4j.LoggerFactory;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
public class RelOptHiveTable implements RelOptTable {
@@ -102,11 +104,12 @@ public class RelOptHiveTable implements RelOptTable {
privat
[hive] branch master updated: HIVE-25692: ExceptionHandler may mask checked exceptions (#2782) (Zoltan Haindrich reviewed by Zhihua Deng and Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 8f2dc79 HIVE-25692: ExceptionHandler may mask checked exceptions
(#2782) (Zoltan Haindrich reviewed by Zhihua Deng and Krisztian Kasa)
8f2dc79 is described below
commit 8f2dc79e7f9d6355234fd2694caa33cbb5d6733c
Author: Zoltan Haindrich
AuthorDate: Wed Nov 17 09:58:38 2021 +0100
HIVE-25692: ExceptionHandler may mask checked exceptions (#2782) (Zoltan
Haindrich reviewed by Zhihua Deng and Krisztian Kasa)
---
.../hadoop/hive/metastore/ExceptionHandler.java| 33 --
.../apache/hadoop/hive/metastore/HMSHandler.java | 10 +--
.../hive/metastore/TestExceptionHandler.java | 2 --
3 files changed, 31 insertions(+), 14 deletions(-)
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ExceptionHandler.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ExceptionHandler.java
index 8bffa97..ffac608 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ExceptionHandler.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ExceptionHandler.java
@@ -44,8 +44,7 @@ public final class ExceptionHandler {
/**
* Throws if the input exception is the instance of the input class
*/
- public ExceptionHandler
- throwIfInstance(Class t) throws T {
+ public ExceptionHandler throwIfInstance(Class t)
throws T {
if (t.isInstance(e)) {
throw t.cast(e);
}
@@ -55,13 +54,29 @@ public final class ExceptionHandler {
/**
* Throws if the input exception is the instance of the one in the input
classes
*/
- public ExceptionHandler
- throwIfInstance(Class ...te) throws T {
-if (te != null) {
- for (Class t : te) {
-throwIfInstance(t);
- }
-}
+ public
+ ExceptionHandler throwIfInstance(
+ Class te1,
+ Class te2) throws T1, T2 {
+throwIfInstance(te1);
+throwIfInstance(te2);
+return this;
+ }
+
+ /**
+ * Throws if the input exception is the instance of the one in the input
classes
+ */
+ public
+ ExceptionHandler throwIfInstance(
+ Class te1,
+ Class te2,
+ Class te3) throws T1, T2, T3 {
+throwIfInstance(te1);
+throwIfInstance(te2);
+throwIfInstance(te3);
return this;
}
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
index c2b166b..a2211a4 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
@@ -3245,7 +3245,7 @@ public class HMSHandler extends FacebookBase implements
IHMSHandler {
}
private void drop_table_with_environment_context(final String dbname, final
String name, final boolean deleteData,
- final EnvironmentContext envContext, boolean dropPartitions) throws
MetaException {
+ final EnvironmentContext envContext, boolean dropPartitions) throws
MetaException, NoSuchObjectException {
String[] parsedDbName = parseDbName(dbname, conf);
startTableFunction("drop_table", parsedDbName[CAT_NAME],
parsedDbName[DB_NAME], name);
@@ -5190,8 +5190,12 @@ public class HMSHandler extends FacebookBase implements
IHMSHandler {
@Override
public boolean equals(Object o) {
- if (this == o) return true;
- if (o == null || getClass() != o.getClass()) return false;
+ if (this == o) {
+return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+return false;
+ }
PathAndDepth that = (PathAndDepth) o;
return depth == that.depth && Objects.equals(path, that.path);
}
diff --git
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestExceptionHandler.java
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestExceptionHandler.java
index 9e79e40..435c085 100644
---
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestExceptionHandler.java
+++
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestExceptionHandler.java
@@ -19,7 +19,6 @@
package org.apache.hadoop.hive.metastore;
import java.io.IOException;
-
import org.junit.Test;
import org.apache.hadoop.hive.metastore.api.InvalidOperationException;
@@ -120,5 +119,4 @@ public class TestExceptionHandler {
assertTrue(e.getMessage().equals(ix.getMessage()));
}
}
-
}
[hive] branch master updated: disable unstable tests
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new dbec774 disable unstable tests
dbec774 is described below
commit dbec7744f04a9389e1bb168e85b5e8f73d357011
Author: Zoltan Haindrich
AuthorDate: Wed Nov 17 08:02:17 2021 +
disable unstable tests
---
contrib/src/test/queries/clientpositive/url_hook.q | 1 +
.../apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java | 1 +
2 files changed, 2 insertions(+)
diff --git a/contrib/src/test/queries/clientpositive/url_hook.q
b/contrib/src/test/queries/clientpositive/url_hook.q
index b8f4c9f..cc95729 100644
--- a/contrib/src/test/queries/clientpositive/url_hook.q
+++ b/contrib/src/test/queries/clientpositive/url_hook.q
@@ -1,3 +1,4 @@
+--! qt:disabled:HIVE-25712
--! qt:dataset:src
add jar
${system:maven.local.repository}/org/apache/hive/hive-contrib/${system:hive.version}/hive-contrib-${system:hive.version}.jar;
SHOW TABLES 'src';
diff --git
a/llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
b/llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
index 4c13c73..3a7e6d0 100644
---
a/llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
+++
b/llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java
@@ -603,6 +603,7 @@ public class TestLlapTaskSchedulerService {
@Test(timeout = 1)
+ @org.junit.Ignore("HIVE-25713")
public void testPreemption() throws InterruptedException, IOException {
Priority priority1 = Priority.newInstance(1);
[hive] branch master updated: HIVE-25697: Upgrade commons-compress to 1.21 (#2788) (Ramesh Kumar Thangarajan reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 49a4397 HIVE-25697: Upgrade commons-compress to 1.21 (#2788) (Ramesh Kumar Thangarajan reviewed by Zoltan Haindrich) 49a4397 is described below commit 49a43979ec0ad43bb3ab38da96f8bae97fe50e23 Author: Ramesh Kumar AuthorDate: Mon Nov 15 08:03:15 2021 -0800 HIVE-25697: Upgrade commons-compress to 1.21 (#2788) (Ramesh Kumar Thangarajan reviewed by Zoltan Haindrich) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 7a25fc1..cd654d0 100644 --- a/pom.xml +++ b/pom.xml @@ -124,7 +124,7 @@ 1.15 3.2.2 4.1 -1.19 +1.21 1.10 1.1 2.6
[hive] branch master updated: HIVE-25365: Insufficient privileges to show partitions when partition columns are authorized (#2515) (Zhihua Deng reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 209b5e3 HIVE-25365: Insufficient privileges to show partitions when
partition columns are authorized (#2515) (Zhihua Deng reviewed by Zoltan
Haindrich)
209b5e3 is described below
commit 209b5e37b43ddbe6ffcd37815d9600d24fa77882
Author: dengzh
AuthorDate: Mon Nov 15 17:00:48 2021 +0800
HIVE-25365: Insufficient privileges to show partitions when partition
columns are authorized (#2515) (Zhihua Deng reviewed by Zoltan Haindrich)
---
.../partition/show/ShowPartitionAnalyzer.java | 3 +
.../hadoop/hive/ql/parse/TestColumnAccess.java | 65 ++
2 files changed, 44 insertions(+), 24 deletions(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/show/ShowPartitionAnalyzer.java
b/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/show/ShowPartitionAnalyzer.java
index a4158d6..40500f1 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/show/ShowPartitionAnalyzer.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/show/ShowPartitionAnalyzer.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.hive.ql.hooks.ReadEntity;
import org.apache.hadoop.hive.ql.metadata.Table;
import org.apache.hadoop.hive.ql.parse.ASTNode;
import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
+import org.apache.hadoop.hive.ql.parse.ColumnAccessInfo;
import org.apache.hadoop.hive.ql.parse.HiveParser;
import org.apache.hadoop.hive.ql.parse.HiveTableName;
import org.apache.hadoop.hive.ql.parse.RowResolver;
@@ -76,6 +77,8 @@ public class ShowPartitionAnalyzer extends
BaseSemanticAnalyzer {
Table table = getTable(HiveTableName.of(tableName));
inputs.add(new ReadEntity(table));
+setColumnAccessInfo(new ColumnAccessInfo());
+table.getPartColNames().forEach(col ->
getColumnAccessInfo().add(table.getCompleteName(), col));
ExprNodeDesc filter = getShowPartitionsFilter(table, ast);
String orderBy = getShowPartitionsOrder(table, ast);
diff --git a/ql/src/test/org/apache/hadoop/hive/ql/parse/TestColumnAccess.java
b/ql/src/test/org/apache/hadoop/hive/ql/parse/TestColumnAccess.java
index c4e336a..305c170 100644
--- a/ql/src/test/org/apache/hadoop/hive/ql/parse/TestColumnAccess.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/parse/TestColumnAccess.java
@@ -41,6 +41,7 @@ public class TestColumnAccess {
Driver driver = createDriver();
driver.run("create table t1(id1 int, name1 string)");
driver.run("create table t2(id2 int, id1 int, name2 string)");
+driver.run("create table t3(id1 int) partitioned by (`date` int, p0
string)");
driver.run("create view v1 as select * from t1");
}
@@ -49,6 +50,7 @@ public class TestColumnAccess {
Driver driver = createDriver();
driver.run("drop table t1");
driver.run("drop table t2");
+driver.run("drop table t3");
driver.run("drop view v1");
}
@@ -64,16 +66,16 @@ public class TestColumnAccess {
List cols =
columnAccessInfo.getTableToColumnAccessMap().get("default@t1");
Assert.assertNotNull(cols);
Assert.assertEquals(2, cols.size());
-Assert.assertNotNull(cols.contains("id1"));
-Assert.assertNotNull(cols.contains("name1"));
+Assert.assertTrue(cols.contains("id1"));
+Assert.assertTrue(cols.contains("name1"));
// check access columns from readEntity
Map> tableColsMap =
getColsFromReadEntity(plan.getInputs());
cols = tableColsMap.get("default@t1");
Assert.assertNotNull(cols);
Assert.assertEquals(2, cols.size());
-Assert.assertNotNull(cols.contains("id1"));
-Assert.assertNotNull(cols.contains("name1"));
+Assert.assertTrue(cols.contains("id1"));
+Assert.assertTrue(cols.contains("name1"));
}
@Test
@@ -88,14 +90,14 @@ public class TestColumnAccess {
List cols =
columnAccessInfo.getTableToColumnAccessMap().get("default@t1");
Assert.assertNotNull(cols);
Assert.assertEquals(2, cols.size());
-Assert.assertNotNull(cols.contains("id1"));
-Assert.assertNotNull(cols.contains("name1"));
+Assert.assertTrue(cols.contains("id1"));
+Assert.assertTrue(cols.contains("name1"));
cols = columnAccessInfo.getTableToColumnAccessMap().get("default@t2");
Assert.assertNotNull(cols);
Assert.assertEquals(3, cols.size());
-Assert.assertNotNull(cols.contains("id2"));
-Assert.assertNotNull(cols.contains("id1"));
-Assert.assertNotNull(cols.contains("name1"));
+Assert.assertTrue(cols.contains
[hive] branch master updated: HIVE-25636: Bump Xerce2 to 2.12.1. (#2740) (Takanobu Asanuma reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new 24b84ea HIVE-25636: Bump Xerce2 to 2.12.1. (#2740) (Takanobu Asanuma reviewed by Zoltan Haindrich) 24b84ea is described below commit 24b84ea1ef7ce36bfabdaae2a2f6d03e71cb9c9d Author: Takanobu Asanuma AuthorDate: Mon Nov 15 17:56:05 2021 +0900 HIVE-25636: Bump Xerce2 to 2.12.1. (#2740) (Takanobu Asanuma reviewed by Zoltan Haindrich) --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index b4e6ca4..7a25fc1 100644 --- a/pom.xml +++ b/pom.xml @@ -211,7 +211,7 @@ 1.1.4 1.4 1.5 -2.9.1 +2.12.1 2.2.1 3.5.5 1.1
[hive] branch master updated: HIVE-25681: Drop support for multi-threaded qtest execution via QTestRunnerUtils (#2769) (Stamatis Zampetakis reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 03a2aa8 HIVE-25681: Drop support for multi-threaded qtest execution
via QTestRunnerUtils (#2769) (Stamatis Zampetakis reviewed by Zoltan Haindrich)
03a2aa8 is described below
commit 03a2aa85c0cc6dc47ae90db0694a8442b376d90c
Author: Stamatis Zampetakis
AuthorDate: Mon Nov 15 09:54:49 2021 +0100
HIVE-25681: Drop support for multi-threaded qtest execution via
QTestRunnerUtils (#2769) (Stamatis Zampetakis reviewed by Zoltan Haindrich)
1. Remove QTestRunnerUtils#queryListRunnerMultiThreaded and related code
2. Remove (disabled) unit tests for this API (TestMTQueries).
The code is not being used by the current test infrastructure and this
is unlikely to change in the near future (especially after HIVE-22942).
Remove the code to facilitate code evolution and maintenance.
---
.../org/apache/hadoop/hive/ql/TestMTQueries.java | 63 ---
.../apache/hadoop/hive/ql/QTestRunnerUtils.java| 115 +
2 files changed, 2 insertions(+), 176 deletions(-)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java
deleted file mode 100644
index f2c81fd..000
---
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hadoop.hive.ql;
-
-import java.io.File;
-
-import org.junit.Ignore;
-import org.junit.Test;
-import static org.junit.Assert.fail;
-
-/**
- * Suite for testing running of queries in multi-threaded mode.
- */
-@Ignore("Ignore until HIVE-23138 is finished")
-public class TestMTQueries extends BaseTestQueries {
-
- public TestMTQueries() {
-File logDirFile = new File(logDir);
-if (!(logDirFile.exists() || logDirFile.mkdirs())) {
- fail("Could not create " + logDir);
-}
- }
-
- @Test
- public void testMTQueries1() throws Exception {
-String[] testNames = new String[] {"join2.q", "groupby1.q", "input1.q",
"input19.q"};
-
-File[] qfiles = setupQFiles(testNames);
-QTestUtil[] qts = QTestRunnerUtils.queryListRunnerSetup(qfiles, resDir,
logDir, "q_test_init_src_with_stats.sql",
- "q_test_cleanup_src_with_stats.sql");
-for (QTestUtil util : qts) {
- util.postInit();
- // derby fails creating multiple stats aggregator concurrently
- util.getConf().setBoolean("hive.exec.submitviachild", true);
- util.getConf().setBoolean("hive.exec.submit.local.task.via.child", true);
- util.getConf().setBoolean("hive.vectorized.execution.enabled", true);
- util.getConf().set("hive.stats.dbclass", "fs");
- util.getConf().set("hive.mapred.mode", "nonstrict");
- util.getConf().set("hive.stats.column.autogather", "false");
- util.newSession();
-}
-boolean success = QTestRunnerUtils.queryListRunnerMultiThreaded(qfiles,
qts);
-if (!success) {
- fail("One or more queries failed");
-}
- }
-}
diff --git
a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestRunnerUtils.java
b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestRunnerUtils.java
index ffd9952..6996ff7 100644
--- a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestRunnerUtils.java
+++ b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestRunnerUtils.java
@@ -21,66 +21,8 @@ package org.apache.hadoop.hive.ql;
import java.io.File;
import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.hive.ql.QTestMiniClusters.MiniClusterType;
public class QTestRunnerUtils {
- public static final String DEFAULT_INIT_SCRIPT = "q_test_init.sql";
- public static final String DEFAULT_CLEANUP_SCRIPT = "q_test_cleanup.sql";
[hive] branch master updated: HIVE-25634: Eclipse compiler bumps into AIOBE during ObjectStore compilation (#2754) (Zhihua Deng reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 3826ffe HIVE-25634: Eclipse compiler bumps into AIOBE during
ObjectStore compilation (#2754) (Zhihua Deng reviewed by Zoltan Haindrich)
3826ffe is described below
commit 3826ffedd7563ec393e44f92841a37bdd8fabe39
Author: dengzh
AuthorDate: Fri Nov 12 21:55:27 2021 +0800
HIVE-25634: Eclipse compiler bumps into AIOBE during ObjectStore
compilation (#2754) (Zhihua Deng reviewed by Zoltan Haindrich)
---
.../org/apache/hadoop/hive/metastore/ObjectStore.java| 16 ++--
1 file changed, 6 insertions(+), 10 deletions(-)
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
index ea70e94..603c65a 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
@@ -14175,12 +14175,13 @@ public class ObjectStore implements RawStore,
Configurable {
@Override
public ScheduledQueryPollResponse
scheduledQueryPoll(ScheduledQueryPollRequest request) throws MetaException {
ensureScheduledQueriesEnabled();
-String namespace = request.getClusterNamespace();
boolean commited = false;
ScheduledQueryPollResponse ret = new ScheduledQueryPollResponse();
-try (QueryWrapper q = new QueryWrapper(pm.newQuery(MScheduledQuery.class,
-"nextExecution <= now && enabled && clusterNamespace == ns &&
activeExecution == null"))) {
+Query q = null;
+try {
openTransaction();
+ q = pm.newQuery(MScheduledQuery.class,
+ "nextExecution <= now && enabled && clusterNamespace == ns &&
activeExecution == null");
q.setSerializeRead(true);
q.declareParameters("java.lang.Integer now, java.lang.String ns");
q.setOrdering("nextExecution");
@@ -14190,7 +14191,6 @@ public class ObjectStore implements RawStore,
Configurable {
return new ScheduledQueryPollResponse();
}
MScheduledQuery schq = results.get(0);
- Integer plannedExecutionTime = schq.getNextExecution();
schq.setNextExecution(computeNextExecutionTime(schq.getSchedule()));
MScheduledExecution execution = new MScheduledExecution();
@@ -14212,12 +14212,8 @@ public class ObjectStore implements RawStore,
Configurable {
LOG.debug("Caught jdo exception; exclusive", e);
commited = false;
} finally {
- if (commited) {
-return ret;
- } else {
-rollbackTransaction();
-return new ScheduledQueryPollResponse();
- }
+ rollbackAndCleanup(commited, q);
+ return commited ? ret : new ScheduledQueryPollResponse();
}
}
[hive] branch master updated: HIVE-25569: Enable table definition over a single file(SFS) (#2680) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 6e152aa HIVE-25569: Enable table definition over a single file(SFS)
(#2680) (Zoltan Haindrich reviewed by Krisztian Kasa)
6e152aa is described below
commit 6e152aa28bc5116bf9210f9deb0f95d2d73183f7
Author: Zoltan Haindrich
AuthorDate: Tue Nov 2 11:17:27 2021 +0100
HIVE-25569: Enable table definition over a single file(SFS) (#2680) (Zoltan
Haindrich reviewed by Krisztian Kasa)
---
.../apache/hadoop/hive/ql/io/SingleFileSystem.java | 346 +
.../services/org.apache.hadoop.fs.FileSystem | 9 +
.../hadoop/hive/ql/io/TestSingleFileSystem.java| 159 ++
ql/src/test/queries/clientpositive/sfs.q | 17 +
ql/src/test/results/clientpositive/llap/sfs.q.out | 103 ++
5 files changed, 634 insertions(+)
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
b/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
new file mode 100644
index 000..e0e9bff
--- /dev/null
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java
@@ -0,0 +1,346 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.io;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.util.Progressable;
+
+/**
+ * Implements an abstraction layer to show files in a single directory.
+ *
+ * Suppose the filesystem has a directory in which there are multiple files:
+ * file://somedir/f1.txt
+ * file://somedir/f2.txt
+ *
+ * In case of Hive the content of a directory may be inside a table.
+ * To give a way to show a single file as a single file in a directory it
could be specified:
+ *
+ * sfs+file://somedir/f1.txt/#SINGLEFILE#
+ *
+ * This will be a directory containing only the f1.txt and nothing else.
+ *
+ */
+/*
+ * Thru out this file there are paths of both the overlay filesystem and the
underlying fs.
+ * To avoid confusion between these path types - all paths which are in the
overlay fs are refered
+ * with the upper keyword - and paths on the underlying fs are identified with
the lower keyword.
+ *
+ * For example:
+ *'sfs+file:///foo/bar/#SINGLEFILE#/bar' is an upper path
+ *'file:///foo/bar' is a lower path
+ */
+public abstract class SingleFileSystem extends FileSystem {
+
+ public static class HDFS extends SingleFileSystem {
+ }
+
+ public static class S3A extends SingleFileSystem {
+ }
+
+ public static class ABFS extends SingleFileSystem {
+ }
+
+ public static class ABFSS extends SingleFileSystem {
+ }
+
+ public static class ADL extends SingleFileSystem {
+ }
+
+ public static class GS extends SingleFileSystem {
+ }
+
+ public static class O3FS extends SingleFileSystem {
+ }
+
+ public static class PFILE extends SingleFileSystem {
+ }
+
+ public static class FILE extends SingleFileSystem {
+ }
+
+ private static final String SINGLEFILE = "#SINGLEFILE#";
+
+ private URI uri;
+ private Configuration conf;
+ private Path workDir;
+
+ public String getScheme() {
+return "sfs+" + getClass().getSimpleName().toLowerCase();
+ }
+
+ @Override
+ public void initialize(URI uri, Configuration conf) throws IOException {
+super.initialize(uri, conf);
+this.uri = uri;
+this.conf = conf;
+ }
+
+ @Override
+ public URI getUri() {
+return uri;
+ }
+
+ @Override
+ public FSDataInputStream open(Path upperPath, int bufferSize) throws
IOException {
+SfsInfo info = new SfsInfo(upperPath);
+switch (info.type) {
+case LEAF_FILE:
+ return
info.lowerTargetPath.getFi
[hive] branch master updated: HIVE-25528: Avoid recalculating types after CBO on second AST pass (#2722) (Stephen Carlin reviewed by Alessandro Solimando, Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 50f54d2 HIVE-25528: Avoid recalculating types after CBO on second AST
pass (#2722) (Stephen Carlin reviewed by Alessandro Solimando, Zoltan Haindrich)
50f54d2 is described below
commit 50f54d24d4fc367ced045b9b8f25b2a0e358634e
Author: scarlin-cloudera <55709772+scarlin-cloud...@users.noreply.github.com>
AuthorDate: Wed Oct 27 10:58:20 2021 -0700
HIVE-25528: Avoid recalculating types after CBO on second AST pass (#2722)
(Stephen Carlin reviewed by Alessandro Solimando, Zoltan Haindrich)
At compile time, Hive parses the query into ASTNodes. For CBO, the
ASTNodes get converted into Calcite RexNodes and then get converted
back into ASTNodes. After the optimization step is done, the types
chosen for each step should be the final type. This commit eliminates
any type changes done on the conversion back to the ASTNode.
A couple of ptest files changed with this commit. The reason for the
change is because there was an extra constant fold done on the conversion
back to the ASTNode. This constant fold should have been done at
optimization time. A Jira will be filed for this, but there is no adverse
effect for this issue.
---
parser/pom.xml | 5 +
.../org/apache/hadoop/hive/ql/parse/ASTNode.java | 13 +
.../optimizer/calcite/translator/ASTConverter.java | 4 ++--
.../calcite/translator/SqlFunctionConverter.java | 4 +++-
.../hive/ql/parse/type/TypeCheckProcFactory.java | 17 +
.../hive/ql/udf/generic/GenericUDFBaseNumeric.java | 22 +-
.../perf/tpcds30tb/tez/query23.q.out | 4 ++--
.../clientpositive/perf/tpcds30tb/tez/query6.q.out | 4 ++--
8 files changed, 57 insertions(+), 16 deletions(-)
diff --git a/parser/pom.xml b/parser/pom.xml
index 41fee3b..b02476a 100644
--- a/parser/pom.xml
+++ b/parser/pom.xml
@@ -62,6 +62,11 @@
${junit.version}
test
+
+ org.apache.hive
+ hive-serde
+ ${project.version}
+
diff --git a/parser/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java
b/parser/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java
index f51de08..802872f 100644
--- a/parser/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java
+++ b/parser/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java
@@ -33,6 +33,7 @@ import org.antlr.runtime.tree.Tree;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hive.common.StringInternUtils;
import org.apache.hadoop.hive.ql.lib.Node;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
/**
*
@@ -46,6 +47,10 @@ public class ASTNode extends CommonTree implements
Node,Serializable {
private transient ASTNode rootNode;
private transient boolean isValidASTStr;
private transient boolean visited = false;
+ // At parsing type, the typeInfo isn't known. However, Hive has logic that
converts
+ // the CBO plan back into ASTNode objects, and at this point, the typeInfo
has
+ // been calculated by the optimizer.
+ private transient TypeInfo typeInfo;
private static final Interner TOKEN_CACHE =
Interners.newWeakInterner();
@@ -155,6 +160,14 @@ public class ASTNode extends CommonTree implements
Node,Serializable {
this.origin = origin;
}
+ public void setTypeInfo(TypeInfo typeInfo) {
+this.typeInfo = typeInfo;
+ }
+
+ public TypeInfo getTypeInfo() {
+return typeInfo;
+ }
+
public String dump() {
StringBuilder sb = new StringBuilder("\n");
dump(sb);
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
index 7073bca..78ecd17 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
@@ -803,7 +803,7 @@ public class ASTConverter {
astNodeLst.add(operand.accept(this));
}
return SqlFunctionConverter.buildAST(SqlStdOperatorTable.NOT,
-
Collections.singletonList(SqlFunctionConverter.buildAST(SqlStdOperatorTable.IS_NOT_DISTINCT_FROM,
astNodeLst)));
+
Collections.singletonList(SqlFunctionConverter.buildAST(SqlStdOperatorTable.IS_NOT_DISTINCT_FROM,
astNodeLst, call.getType())), call.getType());
case CAST:
assert(call.getOperands().size() == 1);
if (call.getType().isStruct() ||
@@ -850,7 +850,7 @@ public class ASTConverter {
if (isFlat(call)) {
return SqlFunctionConverter.buildAST(op, astNodeLst, 0);
} else {
-return SqlFunctionConverter.buildAS
[hive] branch master updated: HIVE-25630: Transformer fixes (#2738) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 9a51d84 HIVE-25630: Transformer fixes (#2738) (Zoltan Haindrich
reviewed by Krisztian Kasa)
9a51d84 is described below
commit 9a51d84ad3dfb4628078ecd57c8cedbfbc2e4efe
Author: Zoltan Haindrich
AuthorDate: Wed Oct 27 16:22:24 2021 +0200
HIVE-25630: Transformer fixes (#2738) (Zoltan Haindrich reviewed by
Krisztian Kasa)
---
.../translated_external_createexisting.q | 13 ++
.../clientpositive/translated_external_alter.q | 8 ++
.../clientpositive/translated_external_rename3.q | 26
.../translated_external_createexisting.q.out | 50 +++
.../llap/translated_external_alter.q.out | 16 +++
.../llap/translated_external_rename3.q.out | 154 +
.../hadoop/hive/metastore/conf/MetastoreConf.java | 7 +-
.../hive/metastore/utils/MetaStoreUtils.java | 2 +-
.../hadoop/hive/metastore/ExceptionHandler.java| 2 +-
.../apache/hadoop/hive/metastore/HMSHandler.java | 5 +
.../metastore/MetastoreDefaultTransformer.java | 12 +-
.../hive/metastore/TestMetastoreTransformer.java | 141 +++
.../client/TestTablesCreateDropAlterTruncate.java | 1 +
13 files changed, 430 insertions(+), 7 deletions(-)
diff --git
a/ql/src/test/queries/clientnegative/translated_external_createexisting.q
b/ql/src/test/queries/clientnegative/translated_external_createexisting.q
new file mode 100644
index 000..c2dab16
--- /dev/null
+++ b/ql/src/test/queries/clientnegative/translated_external_createexisting.q
@@ -0,0 +1,13 @@
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set metastore.metadata.transformer.location.mode=prohibit;
+
+set hive.fetch.task.conversion=none;
+set hive.compute.query.using.stats=false;
+
+create table t (a integer);
+
+-- table should be translated
+desc formatted t;
+
+create table t (a integer);
+
diff --git a/ql/src/test/queries/clientpositive/translated_external_alter.q
b/ql/src/test/queries/clientpositive/translated_external_alter.q
new file mode 100644
index 000..7010782
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/translated_external_alter.q
@@ -0,0 +1,8 @@
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set metastore.metadata.transformer.location.mode=seqsuffix;
+
+set hive.fetch.task.conversion=none;
+set hive.compute.query.using.stats=false;
+
+create table caseSensitive (a integer);
+alter table casesEnsitivE set tblproperties('some'='one');
diff --git a/ql/src/test/queries/clientpositive/translated_external_rename3.q
b/ql/src/test/queries/clientpositive/translated_external_rename3.q
new file mode 100644
index 000..7ccce0d
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/translated_external_rename3.q
@@ -0,0 +1,26 @@
+set
metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
+set metastore.metadata.transformer.location.mode=force;
+
+set hive.fetch.task.conversion=none;
+set hive.compute.query.using.stats=false;
+
+create external table t (a integer);
+insert into t values(1);
+alter table t rename to t2;
+
+-- this TRANSLATED table will have its location shared with the pre-existing
t2 table
+create table t (a integer);
+insert into t values(2);
+
+-- the rows from bot T and T2 can be seen from both tables
+select assert_true(count(1) = 2) from t;
+select assert_true(count(1) = 2) from t2;
+
+select * from t;
+select * from t2;
+
+-- the location of both T and T2 is the same
+desc formatted t;
+desc formatted t2;
+
+
diff --git
a/ql/src/test/results/clientnegative/translated_external_createexisting.q.out
b/ql/src/test/results/clientnegative/translated_external_createexisting.q.out
new file mode 100644
index 000..3550da4
--- /dev/null
+++
b/ql/src/test/results/clientnegative/translated_external_createexisting.q.out
@@ -0,0 +1,50 @@
+PREHOOK: query: create table t (a integer)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t
+POSTHOOK: query: create table t (a integer)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t
+PREHOOK: query: desc formatted t
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@t
+POSTHOOK: query: desc formatted t
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@t
+# col_name data_type comment
+a int
+
+# Detailed Table Information
+Database: default
+ A masked pattern was here
+Retention: 0
+ A masked pattern was here
[hive] branch master updated: HIVE-25633: Prevent shutdown of MetaStore scheduled worker ThreadPool (reviewed by Eugene Chung and Krisztian Kasa) (#2737)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new bdab34c HIVE-25633: Prevent shutdown of MetaStore scheduled worker
ThreadPool (reviewed by Eugene Chung and Krisztian Kasa) (#2737)
bdab34c is described below
commit bdab34c32f9ebba3a9f8efe40adab5a47d75af99
Author: Zoltan Haindrich
AuthorDate: Wed Oct 27 09:19:34 2021 +0200
HIVE-25633: Prevent shutdown of MetaStore scheduled worker ThreadPool
(reviewed by Eugene Chung and Krisztian Kasa) (#2737)
---
.../src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java | 1 -
1 file changed, 1 deletion(-)
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
index 7c46857..a75091b 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
@@ -991,7 +991,6 @@ public class HMSHandler extends FacebookBase implements
IHMSHandler {
public void shutdown() {
cleanupRawStore();
PerfLogger.getPerfLogger(false).cleanupPerfLogMetrics();
-ThreadPool.shutdown();
}
@Override
[hive] branch master updated (a9462c2 -> 116a6be)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from a9462c2 HIVE-25553: Support Map data-type natively in Arrow format (Sruthi M, reviewed by Sankar Hariappan) add 116a6be Revert "HIVE-25553: Support Map data-type natively in Arrow format (Sruthi M, reviewed by Sankar Hariappan)" No new revisions were added by this update. Summary of changes: data/files/datatypes.txt | 4 +- .../org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java | 16 ++--- .../hive/jdbc/TestJdbcWithMiniLlapArrow.java | 83 ++ .../hive/jdbc/TestJdbcWithMiniLlapVectorArrow.java | 83 ++ pom.xml| 2 +- .../hive/llap/WritableByteChannelAdapter.java | 2 +- .../hive/ql/io/arrow/ArrowColumnarBatchSerDe.java | 17 +++-- .../apache/hadoop/hive/ql/io/arrow/Serializer.java | 42 --- 8 files changed, 107 insertions(+), 142 deletions(-)
[hive] branch master updated: HIVE-25532: Fixing authorization for Kill Query command. (#2649) (Abhay Chennagiri reviewed by Saihemanth Gantasala and Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new f53bb7c HIVE-25532: Fixing authorization for Kill Query command.
(#2649) (Abhay Chennagiri reviewed by Saihemanth Gantasala and Zoltan Haindrich)
f53bb7c is described below
commit f53bb7cefe64cd652b48bb802eaf0716f84fa592
Author: achennagiri <77031092+achennag...@users.noreply.github.com>
AuthorDate: Wed Oct 13 04:54:09 2021 -0700
HIVE-25532: Fixing authorization for Kill Query command. (#2649) (Abhay
Chennagiri reviewed by Saihemanth Gantasala and Zoltan Haindrich)
---
.../plugin/TestHiveAuthorizerCheckInvocation.java | 40 +-
.../apache/hive/service/server/KillQueryImpl.java | 7 +++-
2 files changed, 45 insertions(+), 2 deletions(-)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/authorization/plugin/TestHiveAuthorizerCheckInvocation.java
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/authorization/plugin/TestHiveAuthorizerCheckInvocation.java
index 13656c5..ee6925d 100644
---
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/authorization/plugin/TestHiveAuthorizerCheckInvocation.java
+++
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/security/authorization/plugin/TestHiveAuthorizerCheckInvocation.java
@@ -40,6 +40,7 @@ import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
import org.apache.hadoop.hive.metastore.utils.TestTxnDbUtil;
import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.QueryState;
import org.apache.hadoop.hive.ql.exec.Registry;
import org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
import org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider;
@@ -47,6 +48,10 @@ import
org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator;
import
org.apache.hadoop.hive.ql.security.authorization.plugin.HivePrivilegeObject.HivePrivilegeObjectType;
import org.apache.hadoop.hive.ql.session.SessionState;
import org.apache.hadoop.hive.ql.stats.StatsUtils;
+import org.apache.hive.service.cli.operation.OperationManager;
+import org.apache.hive.service.server.KillQueryImpl;
+import org.apache.hive.service.server.KillQueryZookeeperManager;
+
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
@@ -64,6 +69,7 @@ public class TestHiveAuthorizerCheckInvocation {
private final Logger LOG =
LoggerFactory.getLogger(this.getClass().getName());;
protected static HiveConf conf;
protected static Driver driver;
+ protected static SessionState ss;
private static final String tableName =
TestHiveAuthorizerCheckInvocation.class.getSimpleName()
+ "Table";
private static final String viewName =
TestHiveAuthorizerCheckInvocation.class.getSimpleName()
@@ -102,10 +108,17 @@ public class TestHiveAuthorizerCheckInvocation {
conf.setVar(ConfVars.HIVE_TXN_MANAGER, DbTxnManager.class.getName());
conf.setBoolVar(ConfVars.HIVE_QUERY_RESULTS_CACHE_ENABLED, true);
conf.setVar(HiveConf.ConfVars.HIVEMAPREDMODE, "nonstrict");
+conf.setBoolVar(ConfVars.HIVE_TEST_AUTHORIZATION_SQLSTD_HS2_MODE, true);
+conf.setBoolVar(ConfVars.HIVE_ZOOKEEPER_KILLQUERY_ENABLE, false);
TestTxnDbUtil.prepDb(conf);
-SessionState.start(conf);
+SessionState ss = SessionState.start(conf);
+OperationManager operationManager = Mockito.mock(OperationManager.class);
+KillQueryZookeeperManager killQueryZookeeperManager =
Mockito.mock(KillQueryZookeeperManager.class);
+KillQueryImpl killQueryImpl = new KillQueryImpl(operationManager,
killQueryZookeeperManager);
+ss.setKillQuery(killQueryImpl);
+
driver = new Driver(conf);
runCmd("create table " + tableName
+ " (i int, j int, k string) partitioned by (city string, `date`
string) ");
@@ -676,4 +689,29 @@ public class TestHiveAuthorizerCheckInvocation {
inputsCapturer.getValue(), outputsCapturer.getValue());
}
+ /**
+ * Unit test for HIVE-25532.
+ * Checks if the right privilege objects are being sent when a kill query
call is made.
+ * @throws Exception
+ */
+ @Test
+ public void testKillQueryAuthorization() throws Exception {
+int queryStatus = driver.compile("select " + viewName + ".i, " + tableName
+ ".city from "
++ viewName + " join " + tableName + " on " + viewName + ".city = "
+ tableName
++ ".city where " + tableName + ".k = 'X'", true);
+assertEquals(0, queryStatus);
+
+resetAuthorizer();
+QueryState queryState = driver.getQueryState();
+String queryId = queryState.getQueryId();
+int killQueryStatus = driv
[hive] branch master updated: HIVE-25485: Transform selects of literals under a UNION ALL to inline table scan (#2608) (Zoltan Haindrich reviewed by Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 4efb565 HIVE-25485: Transform selects of literals under a UNION ALL
to inline table scan (#2608) (Zoltan Haindrich reviewed by Krisztian Kasa)
4efb565 is described below
commit 4efb565b36ef740fd3a932cfcb07590fc3e93a40
Author: Zoltan Haindrich
AuthorDate: Wed Sep 22 11:31:30 2021 +0200
HIVE-25485: Transform selects of literals under a UNION ALL to inline table
scan (#2608) (Zoltan Haindrich reviewed by Krisztian Kasa)
---
.../hive/jdbc/BaseJdbcWithMiniLlap.java.orig | 747 -
.../java/org/apache/hive/jdbc/TestJdbcDriver2.java | 2 +-
.../calcite/rules/HiveRelDecorrelator.java | 2 +-
.../HiveUnionSimpleSelectsToInlineTableRule.java | 244 +++
.../calcite/translator/SqlFunctionConverter.java | 2 +-
.../hadoop/hive/ql/parse/CalcitePlanner.java | 4 +
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 6 +-
.../test/queries/clientpositive/union_literals.q | 103 +++
.../results/clientpositive/llap/udf_likeall.q.out | 4 +-
.../results/clientpositive/llap/udf_likeany.q.out | 4 +-
.../clientpositive/llap/udf_sort_array_by.q.out| 6 +-
.../clientpositive/llap/union_literals.q.out | 454 +
.../clientpositive/llap/vectorized_mapjoin3.q.out | 229 +++
13 files changed, 925 insertions(+), 882 deletions(-)
diff --git
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java.orig
b/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java.orig
deleted file mode 100644
index 4c46db9..000
---
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/BaseJdbcWithMiniLlap.java.orig
+++ /dev/null
@@ -1,747 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hive.jdbc;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertArrayEquals;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-import java.io.File;
-import java.math.BigDecimal;
-import java.net.URL;
-import java.sql.Connection;
-import java.sql.DriverManager;
-import java.sql.ResultSet;
-import java.sql.SQLException;
-import java.sql.Statement;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.mapred.InputSplit;
-import org.apache.hadoop.mapred.JobConf;
-import org.apache.hadoop.mapred.RecordReader;
-
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
-import org.apache.hadoop.hive.llap.FieldDesc;
-import org.apache.hadoop.hive.llap.Row;
-import org.apache.hadoop.hive.llap.Schema;
-import org.apache.hadoop.io.NullWritable;
-
-import org.apache.hive.jdbc.miniHS2.MiniHS2;
-import org.apache.hive.jdbc.miniHS2.MiniHS2.MiniClusterType;
-import org.apache.hadoop.hive.common.type.Date;
-import org.apache.hadoop.hive.common.type.Timestamp;
-import org.apache.hadoop.hive.llap.LlapBaseInputFormat;
-
-import org.junit.After;
-import org.junit.AfterClass;
-import org.junit.Before;
-import org.junit.Test;
-import org.apache.hadoop.mapred.InputFormat;
-
-/**
- * Specialize this base class for different serde's/formats
- * {@link #beforeTest(boolean) beforeTest} should be called
- * by sub-classes in a {@link org.junit.BeforeClass} initializer
- */
-public abstract class BaseJdbcWithMiniLlap {
-
- private static String dataFileDir;
- private static Path kvDataFilePath;
- private static Path dataTypesFilePath;
- private static Path over10KFilePath;
-
- protected static MiniHS2 miniHS2 = null;
- protected static HiveConf conf = null;
- protected static Connection hs2Conn = null;
-
- // This method should be called by sub-classes in a @BeforeClass initializer
- public static MiniHS2 beforeTest(HiveConf inputConf) throws E
[hive] branch master updated: HIVE-19647: use bitvectors in IN operators (#2598) (Soumyakanti Das reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 60b656d HIVE-19647: use bitvectors in IN operators (#2598)
(Soumyakanti Das reviewed by Zoltan Haindrich)
60b656d is described below
commit 60b656da37c04ad6db7a12324e0b9ee079a80f84
Author: Soumyakanti Das
AuthorDate: Tue Sep 21 03:23:14 2021 -0700
HIVE-19647: use bitvectors in IN operators (#2598) (Soumyakanti Das
reviewed by Zoltan Haindrich)
---
.../java/org/apache/hadoop/hive/conf/HiveConf.java | 2 +
.../stats/annotation/HiveMurmur3Adapter.java | 85 +++
.../stats/annotation/StatsRulesProcFactory.java| 50 +++-
.../apache/hadoop/hive/ql/plan/ColStatistics.java | 9 +
.../apache/hadoop/hive/ql/stats/StatsUtils.java| 7 +
.../hive/ql/plan/mapping/TestStatEstimations.java | 39 ++-
.../queries/clientpositive/in_bitvector_filter.q | 22 ++
.../clientpositive/llap/in_bitvector_filter.q.out | 274 +
8 files changed, 485 insertions(+), 3 deletions(-)
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index f61b903..903a803 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -2914,6 +2914,8 @@ public class HiveConf extends Configuration {
"UDTFs change the number of rows of the output. A common UDTF is the
explode() method that creates\n" +
"multiple rows for each element in the input array. This factor is
applied to the number of\n" +
"output rows and output size."),
+HIVE_STATS_USE_BITVECTORS("hive.stats.use.bitvectors", false,
+ "Enables to use bitvectors for estimating selectivity."),
HIVE_STATS_MAX_NUM_STATS("hive.stats.max.num.stats", (long) 1,
"When the number of stats to be updated is huge, this value is used to
control the number of \n" +
" stats to be sent to HMS for update."),
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/HiveMurmur3Adapter.java
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/HiveMurmur3Adapter.java
new file mode 100644
index 000..0baaa62
--- /dev/null
+++
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/HiveMurmur3Adapter.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.stats.annotation;
+
+import java.nio.ByteBuffer;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.io.DateWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector.PrimitiveCategory;
+import
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hive.common.util.Murmur3;
+
+/**
+* This class could be used to map Hive values type to Murmur3 hash values.
+*/
+public class HiveMurmur3Adapter {
+
+ private PrimitiveCategory type;
+ private PrimitiveObjectInspector inputOI;
+
+ public HiveMurmur3Adapter(PrimitiveObjectInspector oi) throws HiveException {
+this.inputOI = oi;
+type = oi.getTypeInfo().getPrimitiveCategory();
+ }
+
+ private final ByteBuffer LONG_BUFFER = ByteBuffer.allocate(Long.BYTES);
+
+ public long murmur3(Object objVal) throws HiveException {
+Object p = objVal;
+switch (type) {
+ case BYTE:
+ case SHORT:
+ case INT:
+ case LONG:
+ case TIMESTAMP: {
+long val = PrimitiveObjectInspectorUtils.getLong(objVal, inputOI);
+LONG_BUFFER.putLong(0, val);
+return Murmur3.hash64(LONG_BUFFER.array());
+ }
+ case FLOAT:
+ case DOUBLE: {
+double val = PrimitiveObjectInspectorUtils.getDouble(objVal
[hive] branch master updated (3883710 -> 2065dcb)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 3883710 HIVE-25378: Enable removal of old builds on hive ci (#2652) (Zoltan Haindrich reviewed by Krisztian Kasa) add 2065dcb HIVE-25508: Partitioned tables created with CTAS queries doesnt have lineage informations (#2626) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hadoop/hive/ql/parse/SemanticAnalyzer.java | 5 ++-- ql/src/test/queries/clientpositive/lineage5.q | 4 +++ ql/src/test/results/clientpositive/llap/ctas.q.out | 1 + .../results/clientpositive/llap/lineage5.q.out | 33 ++ .../materialized_view_authorization_sqlstd.q.out | 2 ++ .../llap/materialized_view_create.q.out| 1 + .../llap/materialized_view_partition_cluster.q.out | 10 +++ .../llap/materialized_view_partitioned.q.out | 3 ++ .../llap/materialized_view_partitioned_2.q.out | 4 +++ ...lized_view_partitioned_create_rewrite_agg.q.out | 2 ++ ...zed_view_partitioned_create_rewrite_agg_2.q.out | 2 ++ .../materialized_view_rewrite_no_join_opt_2.q.out | 17 +++ .../llap/materialized_view_rewrite_part_1.q.out| 17 +++ .../llap/materialized_view_rewrite_part_2.q.out| 9 ++ ...terialized_view_rewrite_ssb_grouping_sets.q.out | 18 .../clientpositive/llap/partition_ctas.q.out | 6 .../clientpositive/llap/partition_discovery.q.out | 1 + .../llap/temp_table_partition_ctas.q.out | 1 + 18 files changed, 133 insertions(+), 3 deletions(-) create mode 100644 ql/src/test/queries/clientpositive/lineage5.q create mode 100644 ql/src/test/results/clientpositive/llap/lineage5.q.out
[hive] branch master updated (d066ec4 -> 3883710)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from d066ec4 HIVE-24766: Fix TestScheduledReplication (#2575)(Haymant Mangla, reviewed by Arko Sharma) add 3883710 HIVE-25378: Enable removal of old builds on hive ci (#2652) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: Jenkinsfile | 30 -- 1 file changed, 28 insertions(+), 2 deletions(-)
[hive] branch master updated (a08b83a -> a20ee05)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from a08b83a HIVE-25406: Fetch writeId from insert-only transactional tables (Krisztian Kasa, reviewed by Zoltan Haindrich) add a20ee05 HIVE-25404: Inserts inside merge statements are rewritten incorrectly for partitioned tables (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/parse/MergeSemanticAnalyzer.java | 1 - .../clientpositive/merge_partitioned_insert.q | 20 .../llap/merge_partitioned_insert.q.out| 129 + 3 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 ql/src/test/queries/clientpositive/merge_partitioned_insert.q create mode 100644 ql/src/test/results/clientpositive/llap/merge_partitioned_insert.q.out
[hive] branch master updated: HIVE-25170: Fix wrong colExprMap generated by SemanticAnalyzer (#2331) (Wei Zhang reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new d22fa4a HIVE-25170: Fix wrong colExprMap generated by
SemanticAnalyzer (#2331) (Wei Zhang reviewed by Zoltan Haindrich)
d22fa4a is described below
commit d22fa4af5db01a99611790ed10cfddcab327823f
Author: Wei Zhang
AuthorDate: Fri Aug 6 21:27:34 2021 +0800
HIVE-25170: Fix wrong colExprMap generated by SemanticAnalyzer (#2331) (Wei
Zhang reviewed by Zoltan Haindrich)
---
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 2 +-
.../queries/clientpositive/constant_key_column.q | 29
.../clientpositive/llap/constant_key_column.q.out | 175 +
3 files changed, 205 insertions(+), 1 deletion(-)
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index a33d0fe..fdb6cc4 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -9021,7 +9021,7 @@ public class SemanticAnalyzer extends
BaseSemanticAnalyzer {
List keyColNames = rsdesc.getOutputKeyColumnNames();
for (int i = 0 ; i < keyColNames.size(); i++) {
- colExprMap.put(Utilities.ReduceField.KEY + "." + keyColNames.get(i),
sortCols.get(i));
+ colExprMap.put(Utilities.ReduceField.KEY + "." + keyColNames.get(i),
newSortCols.get(i));
}
List valueColNames = rsdesc.getOutputValueColumnNames();
for (int i = 0 ; i < valueColNames.size(); i++) {
diff --git a/ql/src/test/queries/clientpositive/constant_key_column.q
b/ql/src/test/queries/clientpositive/constant_key_column.q
new file mode 100644
index 000..28d3a16
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/constant_key_column.q
@@ -0,0 +1,29 @@
+--! qt:dataset:src
+SET hive.remove.orderby.in.subquery=false;
+
+-- SORT_QUERY_RESULTS
+
+EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+ SELECT 'constant' as constant_col, key, value
+ FROM src
+ DISTRIBUTE BY constant_col, key
+ SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10;
+
+SELECT constant_col, key, max(value)
+FROM
+(
+ SELECT 'constant' as constant_col, key, value
+ FROM src
+ DISTRIBUTE BY constant_col, key
+ SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10;
diff --git a/ql/src/test/results/clientpositive/llap/constant_key_column.q.out
b/ql/src/test/results/clientpositive/llap/constant_key_column.q.out
new file mode 100644
index 000..a1a843d
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/constant_key_column.q.out
@@ -0,0 +1,175 @@
+PREHOOK: query: EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+ SELECT 'constant' as constant_col, key, value
+ FROM src
+ DISTRIBUTE BY constant_col, key
+ SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+ A masked pattern was here
+POSTHOOK: query: EXPLAIN
+SELECT constant_col, key, max(value)
+FROM
+(
+ SELECT 'constant' as constant_col, key, value
+ FROM src
+ DISTRIBUTE BY constant_col, key
+ SORT BY constant_col, key, value
+) a
+GROUP BY constant_col, key
+ORDER BY constant_col, key
+LIMIT 10
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+ A masked pattern was here
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+Tez
+ A masked pattern was here
+ Edges:
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+ A masked pattern was here
+ Vertices:
+Map 1
+Map Operator Tree:
+TableScan
+ alias: src
+ Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
+ Select Operator
+expressions: key (type: string), value (type: string)
+outputColumnNames: _col1, _col2
+Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
+Reduce Output Operator
+ key expressions: _col1 (type: string), _col2 (type:
string)
+ null sort order: zz
+ sort order: ++
+ Map-reduce partition columns: 'constant' (type: string),
_col1 (type: string)
+ Statistics: Num rows: 500 Data size: 89000 Basic stats:
COMPLETE Column stats: COMPLETE
+Execution mode: vectorized, llap
+LLA
[hive] branch master updated: HIVE-24902: Incorrect result after fold CASE into COALESCE (#2100) (Nemon Lou reviewed by Zoltan Haindrich)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 69331f2 HIVE-24902: Incorrect result after fold CASE into COALESCE
(#2100) (Nemon Lou reviewed by Zoltan Haindrich)
69331f2 is described below
commit 69331f2a26da639c6c011adc1bac8ce3b054fa8c
Author: loudongfeng
AuthorDate: Fri Aug 6 21:23:19 2021 +0800
HIVE-24902: Incorrect result after fold CASE into COALESCE (#2100) (Nemon
Lou reviewed by Zoltan Haindrich)
---
.../ql/parse/type/ExprNodeDescExprFactory.java | 3 +-
.../clientpositive/constant_prop_coalesce.q| 30 +++
.../llap/constant_prop_coalesce.q.out | 95 ++
3 files changed, 127 insertions(+), 1 deletion(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
index aad2cfe..a5e6ad5 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/parse/type/ExprNodeDescExprFactory.java
@@ -809,7 +809,8 @@ public class ExprNodeDescExprFactory extends
ExprFactory {
Object thenVal = constThen.getValue();
Object elseVal = constElse.getValue();
if (thenVal instanceof Boolean && elseVal instanceof Boolean) {
-return true;
+//only convert to COALESCE when both branches are valid
+return !thenVal.equals(elseVal);
}
}
return false;
diff --git a/ql/src/test/queries/clientpositive/constant_prop_coalesce.q
b/ql/src/test/queries/clientpositive/constant_prop_coalesce.q
new file mode 100644
index 000..f81b4b9
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/constant_prop_coalesce.q
@@ -0,0 +1,30 @@
+explain
+select * from (
+select
+ case when b.a=1
+ then
+ cast (from_unixtime(unix_timestamp(cast(20210309 as
string),'MMdd') - 86400,'MMdd') as bigint)
+ else
+ 20210309
+ end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null;
+
+select * from (
+select
+ case when b.a=1
+ then
+ cast (from_unixtime(unix_timestamp(cast(20210309 as
string),'MMdd') - 86400,'MMdd') as bigint)
+ else
+ 20210309
+ end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null;
\ No newline at end of file
diff --git
a/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out
b/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out
new file mode 100644
index 000..d34fcb6
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/constant_prop_coalesce.q.out
@@ -0,0 +1,95 @@
+PREHOOK: query: explain
+select * from (
+select
+ case when b.a=1
+ then
+ cast (from_unixtime(unix_timestamp(cast(20210309 as
string),'MMdd') - 86400,'MMdd') as bigint)
+ else
+ 20210309
+ end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here
+POSTHOOK: query: explain
+select * from (
+select
+ case when b.a=1
+ then
+ cast (from_unixtime(unix_timestamp(cast(20210309 as
string),'MMdd') - 86400,'MMdd') as bigint)
+ else
+ 20210309
+ end
+as col
+from
+(select stack(2,1,2) as (a))
+ as b
+) t
+where t.col is not null
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+ A masked pattern was here
+STAGE DEPENDENCIES:
+ Stage-0 is a root stage
+
+STAGE PLANS:
+ Stage: Stage-0
+Fetch Operator
+ limit: -1
+ Processor Tree:
+TableScan
+ alias: _dummy_table
+ Row Limit Per Split: 1
+ Select Operator
+expressions: 2 (type: int), 1 (type: int), 2 (type: int)
+outputColumnNames: _col0, _col1, _col2
+UDTF Operator
+ function name: stack
+ Filter Operator
+predicate: CASE WHEN ((col0 = 1)) THEN (true) ELSE (true) END
(type: boolean)
+Select Operator
+ expressions: CASE WHEN ((col0 = 1)) THEN (20210308L) ELSE
(20210309L) END (type: bigint)
+ outputColumnNames: _col0
+ ListSink
+
+PREHOOK: query: select * from (
+select
+ case when b.a=1
+ then
+ cast (from_unixtime(unix_timestamp(cast(20210309 as
string),'MMdd') - 86400,'MMdd') as bigint)
+ else
+
[hive] branch master updated (e6d98b8 -> 69c97c2)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from e6d98b8 HIVE-25014: Optimise ObjectStore::updateTableColumnStatistics (Rajesh Balamohan, reviewed by Mahesh Kumar Behera) add 69c97c2 HIVE-25370: Improve SharedWorkOptimizer performance (#2536) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/SharedWorkOptimizer.java | 105 - .../hive/ql/optimizer/TestSharedWorkOptimizer.java | 72 +- 2 files changed, 130 insertions(+), 47 deletions(-)
[hive] branch master updated (67f58d0 -> e07a750)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 67f58d0 HIVE-25281: Add optional fields to enable returning filemetadata (Vihang Karajgaonkar, reviewed by Kishen Das) add e07a750 HIVE-25358: Remove reviewer pattern (#2506) (Jesus Camacho Rodriguez reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .github/assign-by-files.yml | 3 --- 1 file changed, 3 deletions(-)
[hive] branch master updated (76c49b9 -> a7baee7)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.
from 76c49b9 HIVE-25276: Enable automatic statistics generation for
Iceberg tables (Peter Vary reviewed by Marton Bod and Adam Szita)
add a7baee7 HIVE-20071: Migrate to jackson 2.x and prevent usage (#2464)
(Zoltan Haindrich reviewed by Krisztian Kasa)
No new revisions were added by this update.
Summary of changes:
.../hadoop/hive/common/jsonexplain/Vertex.java | 34
.../java/org/apache/hive/http/JMXJsonServlet.java | 29 +-
.../hive/http/Log4j2ConfiguratorServlet.java | 3 +-
.../messaging/json/JSONAddPartitionMessage.java| 2 +-
.../messaging/json/JSONAlterPartitionMessage.java | 2 +-
.../messaging/json/JSONAlterTableMessage.java | 2 +-
.../messaging/json/JSONCreateDatabaseMessage.java | 2 +-
.../messaging/json/JSONCreateFunctionMessage.java | 2 +-
.../messaging/json/JSONCreateTableMessage.java | 2 +-
.../messaging/json/JSONDropDatabaseMessage.java| 2 +-
.../messaging/json/JSONDropFunctionMessage.java| 2 +-
.../messaging/json/JSONDropPartitionMessage.java | 2 +-
.../messaging/json/JSONDropTableMessage.java | 2 +-
.../hcatalog/messaging/json/JSONInsertMessage.java | 2 +-
.../messaging/json/JSONMessageDeserializer.java| 7 +--
.../hive/hcatalog/templeton/JsonBuilder.java | 12 +++--
.../hcatalog/templeton/SimpleWebException.java | 5 +-
.../apache/hive/hcatalog/templeton/TestDesc.java | 3 +-
.../hive/hcatalog/templeton/TestWebHCatE2e.java| 8 +--
.../{serde2 => ql/log/syslog}/TestSysLogSerDe.java | 1 -
.../org/apache/hive/jdbc/TestActivePassiveHA.java | 9 ++--
.../hive/service/cli/session/TestQueryDisplay.java | 61 ++
.../hive/llap/cli/status/AppStatusBuilder.java | 3 +-
.../llap/cli/status/LlapStatusServiceDriver.java | 19 +++
.../llap/daemon/services/impl/LlapWebServices.java | 7 +--
.../services/impl/SystemConfigurationServlet.java | 5 +-
pom.xml| 1 +
.../org/apache/hadoop/hive/ql/QueryDisplay.java| 24 +++--
.../org/apache/hadoop/hive/ql/ddl/ShowUtils.java | 20 +++
.../formatter/JsonShowResourcePlanFormatter.java | 7 +--
.../ql/exec/repl/ranger/RangerBaseModelObject.java | 15 +++---
.../exec/repl/ranger/RangerExportPolicyList.java | 16 +++---
.../hive/ql/exec/repl/ranger/RangerPolicy.java | 36 +++--
.../hive/ql/exec/repl/ranger/RangerPolicyList.java | 11 ++--
.../hadoop/hive/ql/exec/tez/AmPluginNode.java | 6 +--
.../hadoop/hive/ql/exec/tez/TezSessionState.java | 4 +-
.../apache/hadoop/hive/ql/exec/tez/WmEvent.java| 5 +-
.../hadoop/hive/ql/exec/tez/WmTezSession.java | 15 +++---
.../hadoop/hive/ql/exec/tez/WorkloadManager.java | 11 ++--
.../org/apache/hadoop/hive/ql/io/AcidUtils.java| 5 +-
.../metadata/formatting/JsonMetaDataFormatter.java | 5 +-
.../hadoop/hive/ql/parse/repl/ReplState.java | 14 +++--
.../hive/ql/parse/repl/dump/io/JsonWriter.java | 7 +--
.../parse/repl/dump/log/state/AtlasDumpBegin.java | 3 +-
.../ql/parse/repl/dump/log/state/AtlasDumpEnd.java | 2 +-
.../repl/dump/log/state/BootstrapDumpBegin.java| 2 +-
.../repl/dump/log/state/BootstrapDumpEnd.java | 2 +-
.../repl/dump/log/state/BootstrapDumpFunction.java | 2 +-
.../repl/dump/log/state/BootstrapDumpTable.java| 2 +-
.../repl/dump/log/state/IncrementalDumpBegin.java | 2 +-
.../repl/dump/log/state/IncrementalDumpEnd.java| 2 +-
.../repl/dump/log/state/IncrementalDumpEvent.java | 2 +-
.../parse/repl/dump/log/state/RangerDumpBegin.java | 2 +-
.../parse/repl/dump/log/state/RangerDumpEnd.java | 2 +-
.../parse/repl/load/log/state/AtlasLoadBegin.java | 2 +-
.../ql/parse/repl/load/log/state/AtlasLoadEnd.java | 2 +-
.../repl/load/log/state/BootstrapLoadBegin.java| 2 +-
.../repl/load/log/state/BootstrapLoadEnd.java | 2 +-
.../repl/load/log/state/BootstrapLoadFunction.java | 2 +-
.../repl/load/log/state/BootstrapLoadTable.java| 2 +-
.../ql/parse/repl/load/log/state/DataCopyEnd.java | 2 +-
.../repl/load/log/state/IncrementalLoadBegin.java | 2 +-
.../repl/load/log/state/IncrementalLoadEnd.java| 2 +-
.../repl/load/log/state/IncrementalLoadEvent.java | 2 +-
.../parse/repl/load/log/state/RangerLoadBegin.java | 2 +-
.../parse/repl/load/log/state/RangerLoadEnd.java | 3 +-
.../org/apache/hadoop/hive/ql/udf/UDFJson.java | 26 -
.../hive/ql/udf/generic/GenericUDTFJSONTuple.java | 20 +++
.../java/org/apache/hadoop/hive/ql/wm/Trigger.java | 2 +-
.../org/apache/hadoop/hive/ql/wm/WmContext.java| 15 +++---
.../metadata/formatting/TestJsonRPFormatter.java | 5 +-
.../hadoop/hive/serde2/avro/TypeInfoToSchema.java | 2 -
.../hive/service/cli/operation/SQLOperation.java |
[hive] branch master updated (ff27abe -> 273593e)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from ff27abe HIVE-25311: Slow compilation of union operators with >100 branches (#2456) (Zoltan Haindrich reviewed by Krisztian Kasa) add 273593e HIVE-25209: SELECT query with SUM function producing unexpected result (#2360) (Soumyakanti Das reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../hadoop/hive/ql/optimizer/StatsOptimizer.java | 6 ++ ql/src/test/queries/clientpositive/select_sum.q| 17 .../results/clientpositive/llap/select_sum.q.out | 90 ++ 3 files changed, 113 insertions(+) create mode 100644 ql/src/test/queries/clientpositive/select_sum.q create mode 100644 ql/src/test/results/clientpositive/llap/select_sum.q.out
[hive] branch master updated (66c72f7 -> ff27abe)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 66c72f7 HIVE-25256: Support ALTER TABLE CHANGE COLUMN for Iceberg (Marton Bod, reviewed by Peter Vary and Adam Szita) add ff27abe HIVE-25311: Slow compilation of union operators with >100 branches (#2456) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../apache/hadoop/hive/ql/parse/GenTezUtils.java | 74 +- 1 file changed, 72 insertions(+), 2 deletions(-)
[hive] branch master updated (53e7426 -> d59c2a3)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 53e7426 HIVE-25312: Upgrade netty to 4.1.65.Final (Zoltan Haindrich reviewed by Panagiotis Garefalakis) add d59c2a3 HIVE-25313: Upgrade commons-codec to 1.15 (#2454) (Zoltan Haindrich reviewed by Panagiotis Garefalakis) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hive] branch master updated (8a4fa84 -> 53e7426)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 8a4fa84 HIVE-25299: Casting timestamp to numeric data types is incorrect for non-UTC timezones (Adesh Rao, reviewed by Ashish Kumar, Sankar Hariappan) add 53e7426 HIVE-25312: Upgrade netty to 4.1.65.Final (Zoltan Haindrich reviewed by Panagiotis Garefalakis) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hive] branch master updated (2639508 -> 7eb731a)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 2639508 HIVE-25213: Implement List getTables() for existing connectors (Dantong Dong reviewed Naveen Gangam) add 7eb731a HIVE-25278: HiveProjectJoinTransposeRule may do invalid transformations with windowing expressions (ADDENDUM) No new revisions were added by this update. Summary of changes: ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hive] branch master updated (1301673 -> a7bd3a4)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 1301673 HIVE-25252: All new compaction metrics should be lower case (Antal Sinkovits, reviewed by Karen Coppage) add a7bd3a4 HIVE-25278: HiveProjectJoinTransposeRule may do invalid transformations with windowing expressions (#2423) (Zoltan Haindrich reviewed by Krisztian Kasa and Jesus Camacho Rodriguez) No new revisions were added by this update. Summary of changes: .../rules/HiveProjectJoinTransposeRule.java| 3 +- .../clientpositive/cbo_proj_join_transpose.q | 21 ++ .../llap/cbo_proj_join_transpose.q.out | 84 ++ 3 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 ql/src/test/queries/clientpositive/cbo_proj_join_transpose.q create mode 100644 ql/src/test/results/clientpositive/llap/cbo_proj_join_transpose.q.out
[hive] branch master updated (765bb0f -> e8c2275)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 765bb0f disable unstable test add e8c2275 HIVE-24901: Re-enable tests in TestBeeLineWithArgs (#2087) (Zhihua Deng reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/hive/beeline/TestBeeLineWithArgs.java| 4 1 file changed, 4 deletions(-)
[hive] branch master updated (865664b -> 765bb0f)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 865664b disable unstable tests add 765bb0f disable unstable test No new revisions were added by this update. Summary of changes: ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java | 1 + 1 file changed, 1 insertion(+)
[hive] branch master updated (6aceb75 -> 865664b)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 6aceb75 HIVE-25259: Tweak delta metrics with custom MBean for Prometheus (Denys Kuzmenko, reviewed by Karen Coppage) add 865664b disable unstable tests No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hive/ql/txn/compactor/TestMmCompactorOnTez.java| 1 + ql/src/test/queries/clientpositive/external_jdbc_table3.q| 1 + ql/src/test/queries/clientpositive/external_jdbc_table4.q| 1 + 3 files changed, 3 insertions(+)
[hive] branch master updated (68daf79 -> 5bfb061)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 68daf79 HIVE-25101: Remove HBase libraries from Hive distribution (#2259) (Istvan Toth reviewed by Zoltan Haindrich) add 5bfb061 HIVE-25279: Fix q.outs caused by concurrent commits of HIVE-25240 and HIVE-25229 (#2424) (Peter Vary reviewed by Jesus Camacho Rodriguez) No new revisions were added by this update. Summary of changes: ql/src/test/results/clientpositive/llap/masking_mv_by_text_2.q.out | 2 ++ .../clientpositive/llap/materialized_view_rewrite_by_text_6.q.out | 2 ++ .../clientpositive/llap/materialized_view_rewrite_by_text_7.q.out | 2 ++ .../clientpositive/llap/materialized_view_rewrite_by_text_8.q.out | 2 ++ 4 files changed, 8 insertions(+)
[hive] branch master updated (a7e937e -> 68daf79)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from a7e937e HIVE-25085: MetaStore Clients no longer shared across sessions. (#2238) (Steve Carlin reviewed by Kishen Das and Zoltan Haindrich) add 68daf79 HIVE-25101: Remove HBase libraries from Hive distribution (#2259) (Istvan Toth reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: bin/hive | 85 +- common/pom.xml | 4 + druid-handler/pom.xml | 5 ++ hbase-handler/pom.xml | 8 ++ itests/hive-unit/pom.xml | 2 + llap-server/pom.xml| 15 .../llap/cli/service/AsyncTaskCopyAuxJars.java | 21 +++--- .../llap/cli/service/LlapServiceCommandLine.java | 14 pom.xml| 7 ++ ql/pom.xml | 13 +++- .../hive/ql/exec/spark/HiveSparkClientFactory.java | 6 +- 11 files changed, 129 insertions(+), 51 deletions(-)
[hive] branch master updated (2aafcdd -> a7e937e)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 2aafcdd HIVE-25216: Vectorized reading of ORC tables via Iceberg (Adam Szita, reviewed by Peter Vary) add a7e937e HIVE-25085: MetaStore Clients no longer shared across sessions. (#2238) (Steve Carlin reviewed by Kishen Das and Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../org/apache/hadoop/hive/ql/metadata/Hive.java | 8 ++ .../hadoop/hive/ql/session/SessionState.java | 22 +-- .../hive/service/cli/operation/SQLOperation.java | 7 - .../hive/service/cli/session/HiveSessionImpl.java | 32 ++ 4 files changed, 42 insertions(+), 27 deletions(-)
[hive] branch master updated (d0bbe76 -> ad925ea)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from d0bbe76 HIVE-25204 : Reduce overhead of adding notification log for update partition column statistics. (Mahesh Kumar Behera, reviewed by Aasha Medhi) add ad925ea disable tests No new revisions were added by this update. Summary of changes: .../apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerWithEngine.java | 1 + .../test/java/org/apache/hadoop/hive/ql/TestWarehouseExternalDir.java| 1 + .../apache/hadoop/hive/ql/parse/TestReplicationScenariosAcidTables.java | 1 + 3 files changed, 3 insertions(+)
[hive] branch master updated (7eb5ab0 -> d323351)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 7eb5ab0 HIVE-25238: Make SSL cipher suites configurable for Hive Web UI and HS2 (#2385) (Yongzhi Chen, reviewed by Naveen Gangam) add d323351 disable unstable tests No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/hive/jdbc/TestWMMetricsWithTrigger.java | 1 + .../java/org/apache/hive/service/TestHS2ImpersonationWithRemoteMS.java | 1 + .../apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java | 1 + ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java | 1 + 4 files changed, 4 insertions(+)
[hive] branch master updated (6a7d4ba -> 588d44d)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 6a7d4ba HIVE-24991: Enable fetching deleted rows in vectorized mode (Krisztian Kasa, reviewed by Panos Garefalakis) add 588d44d HIVE-25224: Multi insert statements involving tables with different bucketing_versions results in error (#2381) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/BucketVersionPopulator.java | 48 - .../ql/optimizer/TestBucketVersionPopulator.java | 70 +++ .../multi_insert_bucketing_version.q | 17 ++ .../llap/multi_insert_bucketing_version.q.out | 232 + 4 files changed, 360 insertions(+), 7 deletions(-) create mode 100644 ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestBucketVersionPopulator.java create mode 100644 ql/src/test/queries/clientpositive/multi_insert_bucketing_version.q create mode 100644 ql/src/test/results/clientpositive/llap/multi_insert_bucketing_version.q.out
[hive] branch master updated (973b386 -> e3e827b)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 973b386 HIVE-24707: Apply Sane Default for Tez Containers as Last Resort (#1933) add e3e827b HIVE-25138: Auto disable scheduled queries after repeated failures (#2343) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/schq/TestScheduledQueryService.java| 2 +- .../gen/thrift/gen-cpp/hive_metastore_types.cpp| 8 +- .../src/gen/thrift/gen-cpp/hive_metastore_types.h | 3 +- .../hadoop/hive/metastore/api/QueryState.java | 5 +- .../gen/thrift/gen-php/metastore/QueryState.php| 3 + .../src/gen/thrift/gen-py/hive_metastore/ttypes.py | 3 + .../src/gen/thrift/gen-rb/hive_metastore_types.rb | 5 +- .../hadoop/hive/metastore/conf/MetastoreConf.java | 16 +++ .../src/main/thrift/hive_metastore.thrift | 1 + .../apache/hadoop/hive/metastore/ObjectStore.java | 87 - .../hive/metastore/model/MScheduledQuery.java | 3 + .../client/TestMetastoreScheduledQueries.java | 138 + 12 files changed, 264 insertions(+), 10 deletions(-)
[hive] branch master updated (a143f49 -> 33dfe71)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.
from a143f49 HIVE-25162: Add support for CREATE TABLE ... STORED BY
ICEBERG statements (#2317) (Laszlo Pinter, reviewed by Marton Bod and Peter
Vary)
add 33dfe71 HIVE-24920: TRANSLATED_TO_EXTERNAL tables may write to the
same location (#2191) (Zoltan Haindrich reviewed by Naveen Gangam)
No new revisions were added by this update.
Summary of changes:
itests/qtest/pom.xml | 1 +
.../clientnegative/translated_external_rename.q| 11 ++
.../clientpositive/translated_external_rename.q| 20 +++
.../clientpositive/translated_external_rename2.q | 20 +++
.../translated_external_rename.q.out | 31
...lyze.q.out => translated_external_rename.q.out} | 119 +-
...yze.q.out => translated_external_rename2.q.out} | 119 +-
.../hadoop/hive/metastore/conf/MetastoreConf.java | 13 ++
.../hive/metastore/utils/MetaStoreUtils.java | 5 +
.../apache/hadoop/hive/metastore/HMSHandler.java | 5 +-
.../metastore/IMetaStoreMetadataTransformer.java | 4 +-
.../metastore/MetastoreDefaultTransformer.java | 183 +
.../hive/metastore/client/TestGetPartitions.java | 16 +-
13 files changed, 426 insertions(+), 121 deletions(-)
create mode 100644
ql/src/test/queries/clientnegative/translated_external_rename.q
create mode 100644
ql/src/test/queries/clientpositive/translated_external_rename.q
create mode 100644
ql/src/test/queries/clientpositive/translated_external_rename2.q
create mode 100644
ql/src/test/results/clientnegative/translated_external_rename.q.out
copy ql/src/test/results/clientpositive/llap/{schq_analyze.q.out =>
translated_external_rename.q.out} (53%)
copy ql/src/test/results/clientpositive/llap/{schq_analyze.q.out =>
translated_external_rename2.q.out} (53%)
[hive] branch master updated (985226c -> b95fe31)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 985226c HIVE-25100: Use default values of Iceberg client pool configuration. (#2258) (Laszlo Pinter, reviewed by Adam Szita and Marton Bod) add b95fe31 HIVE-25044: Parallel edge fixer may not be able to process semijoin edges (#2207) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/ParallelEdgeFixer.java | 69 +- .../clientpositive/perf/tpcds30tb/tez/query2.q.out | 36 +++ .../perf/tpcds30tb/tez/query59.q.out | 12 +++- .../perf/tpcds30tb/tez/query71.q.out | 24 +--- 4 files changed, 119 insertions(+), 22 deletions(-)
[hive] branch master updated (7373ca8 -> ac3317f)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 7373ca8 HIVE-24954: MetastoreTransformer is disabled during testing (#2139) (Zoltan Haindrich reviewed by Krisztian Kasa) add ac3317f HIVE-25029: Remove travis builds (#2192) (Zoltan Haindrich reviewed by Peter Vary) No new revisions were added by this update. Summary of changes: .travis.yml | 40 1 file changed, 40 deletions(-) delete mode 100644 .travis.yml
[hive] branch master updated (cc3f4d1 -> 7373ca8)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.
from cc3f4d1 HIVE-24524: LLAP ShuffleHandler: upgrade to Netty4 and remove
Netty3 dependency from hive where it's possible (#1778) (Laszlo Bodor reviewed
by Panagiotis Garefalakis)
add 7373ca8 HIVE-24954: MetastoreTransformer is disabled during testing
(#2139) (Zoltan Haindrich reviewed by Krisztian Kasa)
No new revisions were added by this update.
Summary of changes:
data/conf/hivemetastore-site.xml | 4 ++
data/conf/llap/hivemetastore-site.xml | 6 ++
.../clientpositive/translated_external_qopt.q | 4 ++
.../llap/translated_external_qopt.q.out} | 27
.../apache/hadoop/hive/metastore/HMSHandler.java | 6 +-
.../hive/metastore/NonCatCallsWithCatalog.java | 73 --
.../hadoop/hive/metastore/TestHiveMetaStore.java | 1 +
.../hive/metastore/TestHiveMetaStoreTxns.java | 14 +++--
.../hive/metastore/TestMetaStoreEventListener.java | 2 +-
.../hive/metastore/TestPartitionManagement.java| 1 +
.../hive/metastore/TestRetryingHMSHandler.java | 2 -
.../client/TestTablesCreateDropAlterTruncate.java | 25 +---
12 files changed, 117 insertions(+), 48 deletions(-)
create mode 100644
ql/src/test/queries/clientpositive/translated_external_qopt.q
copy ql/src/test/results/{clientnegative/alter_file_format.q.out =>
clientpositive/llap/translated_external_qopt.q.out} (60%)
[hive] branch master updated (b2533be -> 9b3d10a)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from b2533be disable script_broken_pipe2 add 9b3d10a HIVE-24986: Support aggregates on columns present in rollups (#2159) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hadoop/hive/ql/parse/CalcitePlanner.java | 3 - .../hadoop/hive/ql/parse/SemanticAnalyzer.java | 78 -- ql/src/test/queries/clientnegative/groupby_cube2.q | 5 - .../test/queries/clientnegative/groupby_rollup1.q | 5 - .../test/queries/clientnegative/groupby_rollup2.q | 5 - .../test/queries/clientpositive/groupby_rollup2.q | 22 ++ .../clientpositive/vector_groupby_rollup2.q| 32 +++ .../results/clientnegative/groupby_cube2.q.out | 1 - .../results/clientnegative/groupby_rollup1.q.out | 1 - .../results/clientnegative/groupby_rollup2.q.out | 1 - .../clientpositive/llap/groupby_rollup2.q.out | 265 + .../llap/vector_groupby_rollup2.q.out | 265 + 12 files changed, 584 insertions(+), 99 deletions(-) delete mode 100644 ql/src/test/queries/clientnegative/groupby_cube2.q delete mode 100644 ql/src/test/queries/clientnegative/groupby_rollup1.q delete mode 100644 ql/src/test/queries/clientnegative/groupby_rollup2.q create mode 100644 ql/src/test/queries/clientpositive/groupby_rollup2.q create mode 100644 ql/src/test/queries/clientpositive/vector_groupby_rollup2.q delete mode 100644 ql/src/test/results/clientnegative/groupby_cube2.q.out delete mode 100644 ql/src/test/results/clientnegative/groupby_rollup1.q.out delete mode 100644 ql/src/test/results/clientnegative/groupby_rollup2.q.out create mode 100644 ql/src/test/results/clientpositive/llap/groupby_rollup2.q.out create mode 100644 ql/src/test/results/clientpositive/llap/vector_groupby_rollup2.q.out
[hive] branch master updated (391e9fc -> b2533be)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 391e9fc HIVE-24980: Add timeout for failed and did not initiate compaction cleanup (Karen Coppage, reviewed by Denys Kuzmenko) add b2533be disable script_broken_pipe2 No new revisions were added by this update. Summary of changes: ql/src/test/queries/clientnegative/script_broken_pipe2.q | 1 + 1 file changed, 1 insertion(+)
[hive] branch master updated (2eb0e00 -> 46ddd5a)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. omit 2eb0e00 HIVE-24396: Additional feedback incorporated (Naveen Gangam) Removed ReplicationSpec for connectors Notification event for alter connector removed some code. omit 34720cf HIVE-24396: Remaining comments from the feedback (Naveen Gangam) omit cd90398 HIVE-24396: Conflict from rebase to master omit 53edee9 HIVE-24396: Changes from additional feedback from code review (Naveen Gangam) omit 07ebf02 HIVE-24396: qtest failure (Naveen Gangam) omit 14283d3 HIVE-24396: Cleanup and one test failure (Naveen Gangam) omit b9150e1 HIVE-24396: Incorporating feedback from the initial review (Naveen Gangam) omit 5f5ec66 HIVE-24396: Duplicate SQL statements in derby upgrade script (Naveen Gangam) omit d7a8eb7 HIVE-24396: Some changes with formatters after the rebase (Naveen Gangam) omit 506621c HIVE-24396: Fix for NPE in get_database_core with null catalog name omit 2b4fa4e HIVE-24396: Database name for remote table should be set to hive dbname not the scoped dbname omit 4319d29 HIVE-24396: Fix in connector provider to return null instead of blank Table omit 5c98e30 HIVE-24396: get_table_core() to return null instead of exception omit 4779e86 HIVE-24396: Fix to CachedStore to make DBs NATIVE and fix to create_table_core on null DBs omit 3e41782 HIVE-24396: Unhandled longvarchar and integer types for derby omit 60ae013 HIVE-24396: Addressing test failures omit 5a2236e HIVE-24396: Build failure due to duplicate db definitions omit 9937963 HIVE-24396 Moving create/drop/alter APIs to the interface. Reverting fix for case sensitivity omit 0b60db9 Retain case on table names during query processing omit d66d5fc fix for 2 additional test failures omit 23aafb7 Build issue with EventMessage omit 3de6032 HIVE-24396: refactored code to Abstract class and providers share common code omit a2a592f Test failures with tez driver and duplicate error codes omit df8cb11 HIVE-24396: Follow up test failure fixes omit 683d0ae HIVE-24396: qtest failures, regenerate them because of new columns omit 7504491 HIVE-24396: Fix for drop database for remote databases omit 6369daa Missed change from the rebase omit 7d91a9a HIVE-24396: Added schema changes for Oracle Made DBS.TYPE NOT NULL in all scripts Added Type support to DatabaseBuilder Added Unit test for DataConnector Added Unit test REMOTE Database Fixed test failures in TestSchemaToolForMetaStore omit 1523cd4 HIVE-24396: getTable/getTables API not expected to throw NoSuchObjectException omit b30173d Adding schema changes for mysql and postgres as well omit 013a693 Adding a qtest and fixing type for default db omit c6ed378 NullPointerException in CreateDatabaseOperation due to last change omit 0b9a4f4 HIVE-24396: Build failure in itests due to unimplemented interface methods omit 80013b4 Deleted commented out code and fixed location and IO classes omit 91f1ccd Added provider for postgres, refactored bunch of classes omit d65307d Implemented getTable and getTableNames for MYSQL (working) omit 87c74ec Adding DDL support for connectors (create/drop/show/desc/alter) omit 69e5417 External metastore: clean after rebase This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/master (46ddd5a) \ O -- O -- O (2eb0e00) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../java/org/apache/hadoop/hive/ql/ErrorMsg.java | 3 - .../hcatalog/listener/DummyRawStoreFailEvent.java |27 - .../hadoop/hive/ql/parse/AlterClauseParser.g |31 - .../apache/hadoop/hive/ql/parse/CreateDDLParser.g |42 - .../apache/hadoop/hive/ql/parse/HiveLexerParent.g | 6 - .../org/apache/hadoop/hive/ql/parse/HiveParser.g |35 +- .../hadoop/hive/ql/parse/IdentifiersParser.g | 8 +- pom.xml| 3 - .../database/create/CreateDatabaseAnalyzer.java|37 +- .../ql/ddl/database/create/CreateDatabaseDesc.java |41 +- .../database/create/CreateDatabaseOperation.java |24 +- .../ql/ddl/database/desc/DescDatabaseDesc.java | 6 +- .../ddl/database/desc/DescDatabaseFormatter.java |23 +- .../ddl/database/desc/DescDatabaseOperation.java |27 +- .../alter/AbstractAlterDataConnectorAnalyzer.java |42 - .../alter/AbstractAlterDataConnectorDe
[hive] branch master updated (61d5c64 -> eed78df)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 61d5c64 HIVE-24718: Moving to file based iteration for copying data (Arko Sharma, reviewed by Pravin Kumar Sinha ) add eed78df HIVE-24853. HMS leaks queries in case of timeout. (#2044) (Ayush Saxena reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../cache/TestCachedStoreUpdateUsingEvents.java| 4 + .../hadoop/hive/metastore/HiveAlterHandler.java| 7 +- .../hadoop/hive/metastore/MetaStoreDirectSql.java | 273 - .../hive/metastore/MetastoreDirectSqlUtils.java| 76 +++--- .../hive/metastore/TestHiveAlterHandler.java | 6 + 5 files changed, 213 insertions(+), 153 deletions(-)
[hive] branch master updated (61087be -> 2c92427)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 61087be HIVE-24865: Implement Respect/Ignore Nulls in first/last_value (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez) add 2c92427 HIVE-24841: Parallel edge fixer may run into NPE when RS is missing a duplicate column from the output schema (#2035) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hadoop/hive/ql/optimizer/ParallelEdgeFixer.java | 15 ++- .../hadoop/hive/ql/optimizer/SharedWorkOptimizer.java | 14 -- 2 files changed, 14 insertions(+), 15 deletions(-)
[hive] branch master updated (6e8936f -> f0b5732)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 6e8936f HIVE-24543: Support SAML 2.0 as an authentication mechanism (Vihang Karajgaonkar, reviewed by Naveen Gangam) add f0b5732 HIVE-24812: Disable sharedworkoptimizer remove semijoin by default (#2006) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../java/org/apache/hadoop/hive/conf/HiveConf.java | 2 +- kafka-handler/pom.xml | 2 +- .../test/queries/clientpositive/sharedwork_semi.q | 1 + .../llap/dynamic_semijoin_reduction_2.q.out| 94 -- .../perf/tpcds30tb/tez/cbo_query24.q.out | 2 +- .../perf/tpcds30tb/tez/query16.q.out | 115 --- .../clientpositive/perf/tpcds30tb/tez/query2.q.out | 125 .../perf/tpcds30tb/tez/query24.q.out | 225 +- .../perf/tpcds30tb/tez/query39.q.out | 167 +- .../perf/tpcds30tb/tez/query59.q.out | 231 +++--- .../perf/tpcds30tb/tez/query65.q.out | 148 +++-- .../perf/tpcds30tb/tez/query81.q.out | 181 +++ .../perf/tpcds30tb/tez/query94.q.out | 115 --- .../perf/tpcds30tb/tez/query95.q.out | 346 - 14 files changed, 1076 insertions(+), 678 deletions(-)
[hive] branch master updated (8b2c31a -> 0ba8b6f)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 8b2c31a disable TestZookeeperLockManager#testMetrics add 0ba8b6f HIVE-24839: SubStrStatEstimator.estimate throws NullPointerException (#2034) (Robbie Zhang reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../test/resources/testconfiguration.properties| 3 +- .../org/apache/hadoop/hive/ql/udf/UDFSubstr.java | 6 ++- .../queries/clientpositive/substr_estimate_range.q | 5 +++ .../llap/substr_estimate_range.q.out | 46 ++ .../clientpositive/tez/substr_estimate_range.q.out | 46 ++ 5 files changed, 103 insertions(+), 3 deletions(-) create mode 100644 ql/src/test/queries/clientpositive/substr_estimate_range.q create mode 100644 ql/src/test/results/clientpositive/llap/substr_estimate_range.q.out create mode 100644 ql/src/test/results/clientpositive/tez/substr_estimate_range.q.out
[hive] branch master updated (e1870c1 -> 8b0542f)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from e1870c1 HIVE-24783: Store currentNotificationID on target during repl load operation (Haymant Mangla, reviewed by Pravin Kumar Sinha) add 8b0542f HIVE-24827: Hive aggregation query returns incorrect results for non text files. (#2018) (Ayush Saxena reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../java/org/apache/hive/jdbc/TestJdbcDriver2.java | 26 ++ .../org/apache/hadoop/hive/ql/exec/Utilities.java | 21 ++--- .../apache/hadoop/hive/ql/plan/TableScanDesc.java | 5 - 3 files changed, 48 insertions(+), 4 deletions(-)
[hive] branch master updated (ab69e64 -> 5145468)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from ab69e64 HIVE-24821: Restrict parallel edge creation for invertable RS operators (#2015) (Zoltan Haindrich reviewed by Krisztian Kasa) add 5145468 HIVE-24829: CorrelationUtilities#replaceReduceSinkWithSelectOperator misses KEY mappings (#2021) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/NonBlockingOpDeDupProc.java | 8 .../ql/optimizer/correlation/CorrelationUtilities.java | 17 +++-- 2 files changed, 23 insertions(+), 2 deletions(-)
[hive] branch master updated (54ef4ff -> ab69e64)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 54ef4ff HIVE-24822: Materialized View rebuild loses materializationTime property value (Krisztian Kasa, reviewed by Jesus Camacho Rodriguez) add ab69e64 HIVE-24821: Restrict parallel edge creation for invertable RS operators (#2015) (Zoltan Haindrich reviewed by Krisztian Kasa) No new revisions were added by this update. Summary of changes: .../hive/ql/optimizer/ParallelEdgeFixer.java | 47 +- .../hive/ql/optimizer/SharedWorkOptimizer.java | 29 - .../hive/ql/optimizer/graph/OperatorGraph.java | 15 --- .../results/clientpositive/llap/auto_join2.q.out | 4 +- .../results/clientpositive/llap/auto_join28.q.out | 8 ++-- .../results/clientpositive/llap/auto_join29.q.out | 12 +++--- .../results/clientpositive/llap/auto_join7.q.out | 4 +- .../llap/correlationoptimizer3.q.out | 4 +- .../clientpositive/llap/explainuser_2.q.out| 10 ++--- .../clientpositive/llap/explainuser_4.q.out| 4 +- .../clientpositive/llap/join_max_hashtable.q.out | 12 +++--- ql/src/test/results/clientpositive/llap/mrr.q.out | 4 +- .../llap/reduce_deduplicate_exclude_join.q.out | 4 +- .../clientpositive/llap/skewjoin_mapjoin7.q.out| 4 +- .../clientpositive/llap/swo_vertex_merge.q.out | 4 +- .../llap/tez_dynpart_hashjoin_1.q.out | 12 +++--- .../llap/tez_vector_dynpart_hashjoin_1.q.out | 8 ++-- .../clientpositive/llap/vector_nullsafe_join.q.out | 24 +-- .../llap/vectorized_nested_mapjoin.q.out | 4 +- .../perf/tpcds30tb/tez/query18.q.out | 4 +- .../perf/tpcds30tb/tez/query23.q.out | 4 +- .../perf/tpcds30tb/tez/query24.q.out | 4 +- .../perf/tpcds30tb/tez/query44.q.out | 4 +- .../perf/tpcds30tb/tez/query47.q.out | 4 +- .../perf/tpcds30tb/tez/query57.q.out | 4 +- .../perf/tpcds30tb/tez/query64.q.out | 16 .../perf/tpcds30tb/tez/query75.q.out | 12 +++--- .../perf/tpcds30tb/tez/query76.q.out | 16 .../perf/tpcds30tb/tez/query81.q.out | 4 +- .../perf/tpcds30tb/tez/query83.q.out | 4 +- .../perf/tpcds30tb/tez/query85.q.out | 4 +- .../perf/tpcds30tb/tez/query92.q.out | 4 +- .../perf/tpcds30tb/tez/query95.q.out | 4 +- .../clientpositive/tez/explainanalyze_4.q.out | 4 +- 34 files changed, 172 insertions(+), 133 deletions(-)
[hive] branch master updated (665c443 -> a896e5f)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 665c443 HIVE-24751: Kill trigger in workload manager fails with "No privilege" exception when authorization is disabled (Nikhil Gupta, reviewed by Ashish Sharma, Sankar Hariappan) add a896e5f HIVE-24823: Fix ide error in BasePartitionEvaluato (#2014) (Zoltan Haindrich reviewed by Rajesh Balamohan) No new revisions were added by this update. Summary of changes: .../hive/ql/udf/ptf/BasePartitionEvaluator.java| 75 +++--- 1 file changed, 51 insertions(+), 24 deletions(-)
[hive] branch master updated (0e9c0ce -> 75fec20)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 0e9c0ce HIVE-24808: Cache Parsed Dates (David Mollitor reviewed by Miklos Gergely) add 75fec20 HIVE-24809: Build failure while resolving javax.el dependency (#2003) (Stamatis Zampetakis reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: itests/pom.xml | 12 standalone-metastore/metastore-tools/pom.xml | 4 2 files changed, 16 insertions(+)
[hive] branch master updated (64af144 -> 1c7f2cc)
This is an automated email from the ASF dual-hosted git repository. kgyrtkirk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git. from 64af144 HIVE-24726: Track required data for cache hydration (Antal Sinkovits, reviewed by Adam Szita) add 1c7f2cc HIVE-24645: Call configure for UDFs after fetch task conversion (#1876) (John Sherman reviewed by Zoltan Haindrich) No new revisions were added by this update. Summary of changes: .../test/resources/testconfiguration.properties| 1 + .../hive/ql/udf/generic/TestConfigureUDF.java | 27 +-- .../hive/ql/exec/ExprNodeGenericFuncEvaluator.java | 38 ++ .../apache/hadoop/hive/ql/exec/MapredContext.java | 10 ++ ql/src/test/queries/clientpositive/udf_configure.q | 8 + .../clientpositive/llap/udf_configure.q.out| 34 +++ 6 files changed, 108 insertions(+), 10 deletions(-) copy ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBucketNumber.java => itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/TestConfigureUDF.java (72%) create mode 100644 ql/src/test/queries/clientpositive/udf_configure.q create mode 100644 ql/src/test/results/clientpositive/llap/udf_configure.q.out
[hive] branch master updated (5430dda -> 2fb4899)
This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git.
from 5430dda HIVE-24625 CTAS with TBLPROPERTIES ('transactional'='false')
loads data into incorrect directory (amagyar, rajkrrsingh)
add 2fb4899 disable TransactionalKafkaWriterTest
No new revisions were added by this update.
Summary of changes:
.../test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java | 1 +
1 file changed, 1 insertion(+)
