(hive) branch master updated: HIVE-28252: AssertionError when using HiveTableScan with a HepPlanner cluster (Stamatis Zampetakis reviewed by Krisztian Kasa)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 0e84fe2000c HIVE-28252: AssertionError when using HiveTableScan with a 
HepPlanner cluster (Stamatis Zampetakis reviewed by Krisztian Kasa)
0e84fe2000c is described below

commit 0e84fe2000c026afd0a49f4e7c7dd5f54fe7b1ec
Author: Stamatis Zampetakis 
AuthorDate: Fri May 10 16:29:54 2024 +0200

HIVE-28252: AssertionError when using HiveTableScan with a HepPlanner 
cluster (Stamatis Zampetakis reviewed by Krisztian Kasa)

The HepPlanner does not use any RelTraitDef so the default traitset for the 
respective cluster is always empty.

1. Relax the assertion to not throw an error when the cluster does not 
contain a ConventionTraitDef (HepPlanner).
2. Add tests to ensure that it is possible to create a HiveTableScan with 
VolcanoPlanner and HepPlanner and the expected convention is returned.

Close apache/hive#5244
---
 .../calcite/reloperators/HiveTableScan.java|  2 +-
 .../calcite/reloperators/TestHiveTableScan.java| 75 ++
 2 files changed, 76 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
index 26ceedac8c5..2c3e4c6e6e3 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableScan.java
@@ -150,7 +150,7 @@ public class HiveTableScan extends TableScan implements 
HiveRelNode {
   String alias, String concatQbIDAlias, RelDataType newRowtype, boolean 
useQBIdInDigest, boolean insideView,
   HiveTableScanTrait tableScanTrait) {
 super(cluster, TraitsUtil.getDefaultTraitSet(cluster), table);
-assert getConvention() == HiveRelNode.CONVENTION;
+assert getTraitSet().containsIfApplicable(HiveRelNode.CONVENTION);
 this.tblAlias = alias;
 this.concatQbIDAlias = concatQbIDAlias;
 this.hiveTableScanRowType = newRowtype;
diff --git 
a/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/TestHiveTableScan.java
 
b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/TestHiveTableScan.java
new file mode 100644
index 000..3e2bc6e1b44
--- /dev/null
+++ 
b/ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/TestHiveTableScan.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.optimizer.calcite.reloperators;
+
+import org.apache.calcite.jdbc.JavaTypeFactoryImpl;
+import org.apache.calcite.plan.ConventionTraitDef;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.volcano.VolcanoPlanner;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.type.RelDataType;
+import org.apache.calcite.rel.type.RelDataTypeFactory;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.parse.QueryTables;
+
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class TestHiveTableScan {
+
+  @Test
+  public void testGetConventionWithWithHepPlannerCluster() {
+RelNode scan = createScanWithPlanner(new 
HepPlanner(HepProgram.builder().build()));
+assertNull(scan.getConvention());
+  }
+
+  @Test
+  public void testGetConventionWithVolcanoPlannerCluster() {
+RelNode scan = createScanWithPlanne

(calcite) branch main updated: [CALCITE-6384] Add ASF header to buildcache.yml, gradle-wrapper-validation.yml

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new aa8d81bf1f [CALCITE-6384] Add ASF header to buildcache.yml, 
gradle-wrapper-validation.yml
aa8d81bf1f is described below

commit aa8d81bf1ff39e3632aeb856fc4cc247ce9727e5
Author: Stamatis Zampetakis 
AuthorDate: Wed Apr 24 11:06:34 2024 +0200

[CALCITE-6384] Add ASF header to buildcache.yml, 
gradle-wrapper-validation.yml
---
 .github/workflows/buildcache.yml| 14 ++
 .github/workflows/gradle-wrapper-validation.yml | 14 ++
 2 files changed, 28 insertions(+)

diff --git a/.github/workflows/buildcache.yml b/.github/workflows/buildcache.yml
index 0169caf47c..bb5d7b4736 100644
--- a/.github/workflows/buildcache.yml
+++ b/.github/workflows/buildcache.yml
@@ -1,3 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 name: Seed build cache
 
 on:
diff --git a/.github/workflows/gradle-wrapper-validation.yml 
b/.github/workflows/gradle-wrapper-validation.yml
index 8bfd0dcf51..d1cfb42029 100644
--- a/.github/workflows/gradle-wrapper-validation.yml
+++ b/.github/workflows/gradle-wrapper-validation.yml
@@ -1,3 +1,17 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 name: "Validate Gradle Wrapper"
 on: [push, pull_request]
 

(calcite) branch main updated: [CALCITE-6385] LintTest fails when run in source distribution

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new f78dd940d7 [CALCITE-6385] LintTest fails when run in source 
distribution
f78dd940d7 is described below

commit f78dd940d745a5242c35546308ebf74a97f26610
Author: Stamatis Zampetakis 
AuthorDate: Thu Apr 25 12:22:54 2024 +0200

[CALCITE-6385] LintTest fails when run in source distribution

The testContributorsFileIsSorted and testMailmapFile tests fail cause
they rely on git and there is no git info available in source
distribution.

In fact the tests don't really need git and the paths to the .mailmap
and contributor.yml files can be specified explicitly. This removes
the git requirement and also makes the tests more efficient since there
is no process calls and lookup involved.
---
 build.gradle.kts  |  1 +
 .../test/java/org/apache/calcite/test/LintTest.java   | 19 ++-
 2 files changed, 7 insertions(+), 13 deletions(-)

diff --git a/build.gradle.kts b/build.gradle.kts
index 579d4bd907..0f4ff553db 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -826,6 +826,7 @@ allprojects {
 passProperty("user.country", "tr")
 passProperty("user.timezone", "UTC")
 passProperty("calcite.avatica.version", 
props.string("calcite.avatica.version"))
+passProperty("gradle.rootDir", rootDir.toString())
 val props = System.getProperties()
 for (e in props.propertyNames() as 
`java.util`.Enumeration) {
 if (e.startsWith("calcite.") || e.startsWith("avatica.")) {
diff --git a/core/src/test/java/org/apache/calcite/test/LintTest.java 
b/core/src/test/java/org/apache/calcite/test/LintTest.java
index 1e36cfd020..5ba82b4250 100644
--- a/core/src/test/java/org/apache/calcite/test/LintTest.java
+++ b/core/src/test/java/org/apache/calcite/test/LintTest.java
@@ -37,6 +37,8 @@ import java.io.File;
 import java.io.IOException;
 import java.io.PrintWriter;
 import java.io.StringWriter;
+import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.List;
@@ -46,10 +48,6 @@ import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 import java.util.stream.Stream;
 
-import static com.google.common.collect.Iterables.getOnlyElement;
-
-import static org.apache.calcite.util.Util.filter;
-
 import static org.hamcrest.CoreMatchers.is;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.empty;
@@ -64,6 +62,7 @@ class LintTest {
* space. */
   private static final Pattern CALCITE_PATTERN =
   Pattern.compile("^(\\[CALCITE-[0-9]{1,4}][ ]).*");
+  private static final Path ROOT_PATH = 
Paths.get(System.getProperty("gradle.rootDir"));
 
   @SuppressWarnings("Convert2MethodRef") // JDK 8 requires lambdas
   private Puffin.Program makeProgram() {
@@ -397,10 +396,7 @@ class LintTest {
   /** Ensures that the {@code contributors.yml} file is sorted by name. */
   @Test void testContributorsFileIsSorted() throws IOException {
 final ObjectMapper mapper = new YAMLMapper();
-final List files = TestUnsafe.getTextFiles();
-final File contributorsFile =
-getOnlyElement(
-filter(files, f -> f.getName().equals("contributors.yml")));
+final File contributorsFile = 
ROOT_PATH.resolve("site/_data/contributors.yml").toFile();
 JavaType listType =
 mapper.getTypeFactory()
 .constructCollectionType(List.class, Contributor.class);
@@ -416,12 +412,9 @@ class LintTest {
 
   /** Ensures that the {@code .mailmap} file is sorted. */
   @Test void testMailmapFile() {
-final List files = TestUnsafe.getTextFiles();
-final File contributorsFile =
-getOnlyElement(
-filter(files, f -> f.getName().equals(".mailmap")));
+final File mailmapFile = ROOT_PATH.resolve(".mailmap").toFile();
 final List lines = new ArrayList<>();
-forEachLineIn(contributorsFile, line -> {
+forEachLineIn(mailmapFile, line -> {
   if (!line.startsWith("#")) {
 lines.add(line);
   }

(hive) 02/02: HIVE-28207: NullPointerException is thrown when checking column uniqueness (Shohei Okumiya reviewed by Stamatis Zampetakis, Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 2d855b27d31db6476f18870651db6987816bb5e3
Author: okumin 
AuthorDate: Sun Apr 21 22:55:34 2024 +0900

HIVE-28207: NullPointerException is thrown when checking column uniqueness 
(Shohei Okumiya reviewed by Stamatis Zampetakis, Ayush Saxena)

Close apache/hive#5207
---
 .../calcite/rules/HiveAggregateSplitRule.java  |   4 +-
 .../calcite/rules/HiveJoinConstraintsRule.java |   3 +-
 .../optimizer/calcite/stats/HiveRelMdRowCount.java |   3 +-
 .../queries/clientpositive/cbo_join_constraints.q  |  11 ++
 .../cbo_row_count_non_simple_filter.q  |  26 
 .../clientpositive/llap/cbo_join_constraints.q.out |  78 ++
 .../llap/cbo_row_count_non_simple_filter.q.out | 159 +
 7 files changed, 281 insertions(+), 3 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
index b956201f133..8534981b19f 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveAggregateSplitRule.java
@@ -86,7 +86,9 @@ public class HiveAggregateSplitRule extends RelOptRule {
   aggregateCall.name));
 }
 
-if 
(aggregate.getCluster().getMetadataQuery().areColumnsUnique(aggregate.getInput(),
 bottomAggregateGroupSet)) {
+final Boolean isUnique =
+
aggregate.getCluster().getMetadataQuery().areColumnsUnique(aggregate.getInput(),
 bottomAggregateGroupSet);
+if (isUnique != null && isUnique) {
   // Nothing to do, probably already pushed
   return;
 }
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinConstraintsRule.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinConstraintsRule.java
index d60d0ee3d84..58eb2011ed0 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinConstraintsRule.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinConstraintsRule.java
@@ -274,7 +274,8 @@ public class HiveJoinConstraintsRule extends RelOptRule {
   }
 }
 
-if (!call.getMetadataQuery().areColumnsUnique(nonFkInput, 
fkJoinColBuilder.build())) {
+final Boolean isUnique = 
call.getMetadataQuery().areColumnsUnique(nonFkInput, fkJoinColBuilder.build());
+if (isUnique == null || !isUnique) {
   return;
 }
 
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
index b4c649871ea..254a5ed8c83 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java
@@ -638,7 +638,8 @@ public class HiveRelMdRowCount extends RelMdRowCount {
 private boolean isSimple(ImmutableBitSet condBits, Filter filter, 
RelMetadataQuery mq) {
   // Returns whether the filter is only applied on the key columns
   if (constraintsBased) {
-return mq.areColumnsUnique(filter, condBits);
+final Boolean isUnique = mq.areColumnsUnique(filter, condBits);
+return isUnique != null && isUnique;
   }
   return isKey(condBits, filter, mq);
 }
diff --git a/ql/src/test/queries/clientpositive/cbo_join_constraints.q 
b/ql/src/test/queries/clientpositive/cbo_join_constraints.q
new file mode 100644
index 000..2bba038d621
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/cbo_join_constraints.q
@@ -0,0 +1,11 @@
+CREATE TABLE `store_sales` (`ss_item_sk` bigint);
+CREATE TABLE `household_demographics` (`hd_demo_sk` bigint);
+CREATE TABLE `item` (`i_item_sk` bigint);
+ALTER TABLE `store_sales` ADD CONSTRAINT `pk_ss` PRIMARY KEY (`ss_item_sk`) 
DISABLE NOVALIDATE RELY;
+ALTER TABLE `item` ADD CONSTRAINT `pk_i` PRIMARY KEY (`i_item_sk`) DISABLE 
NOVALIDATE RELY;
+ALTER TABLE `store_sales` ADD CONSTRAINT `ss_i` FOREIGN KEY (`ss_item_sk`) 
REFERENCES `item`(`i_item_sk`) DISABLE NOVALIDATE RELY;
+
+EXPLAIN CBO
+SELECT i_item_sk
+FROM store_sales, household_demographics, item
+WHERE ss_item_sk = i_item_sk;
diff --git 
a/ql/src/test/queries/clientpositive/cbo_row_count_non_simple_filter.q 
b/ql/src/test/queries/clientpositive/cbo_row_count_non_simple_filter.q
new file mode 100644
index 000..6169370587e
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/cbo_row_count_non_simple_filter.q
@@ -0,0 +1,26 @@
+CREATE TABLE `store_sales` (`ss_item_sk` bigint, `ss_hdemo_sk` bigint);
+CREATE TABLE `household_demographics` (`hd_demo_sk` bigint, 
`hd_income

(hive) branch master updated (1a969f6642d -> 2d855b27d31)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 1a969f6642d HIVE-28190: Fix MaterializationRebuild lock heartbeat 
(Zsolt Miskolczi, reviewed by Attila Turoczy, Denys Kuzmenko, Krisztian Kasa, 
Zoltan Ratkai)
 new 5f9cb486e4d HIVE-28155 HIVE-28157 HIVE-28158 HIVE-28159 HIVE-28160: 
LICENSE and source header improvements (Stamatis Zampetakis reviewed by Zhihua 
Deng)
 new 2d855b27d31 HIVE-28207: NullPointerException is thrown when checking 
column uniqueness (Shohei Okumiya reviewed by Stamatis Zampetakis, Ayush Saxena)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .arcconfig |  15 --
 .github/actions/spelling/README.md |  18 +++
 .github/actions/spelling/advice.md |  18 +++
 .github/assign-by-files.yml|  16 +++
 .github/workflows/auto-assign.yml  |  16 +++
 .github/workflows/docker-GA-images.yml |  16 +++
 .github/workflows/spelling.yml |  16 +++
 .github/workflows/stale.yml|  16 +++
 LICENSE|  37 -
 README.md  |  18 +++
 .../apache/hive/beeline/TestTableOutputFormat.java |  14 +-
 bin/ext/tokentool.sh   |  17 +++
 data/conf/fair-scheduler-test.xml  |  16 +++
 data/conf/iceberg/llap/tez-site.xml|  17 +++
 data/conf/iceberg/tez/tez-site.xml |  17 +++
 data/conf/llap/tez-site.xml|  17 +++
 data/conf/tez/tez-site.xml |  17 +++
 druid-handler/README.md|  18 +++
 hbase-handler/README.md|  20 ++-
 iceberg/checkstyle/checkstyle-suppressions.xml |  16 +++
 iceberg/checkstyle/checkstyle.xml  |  17 +++
 iceberg/iceberg-catalog/pom.xml|  13 ++
 iceberg/iceberg-handler/pom.xml|  13 ++
 iceberg/patched-iceberg-api/pom.xml|  13 ++
 iceberg/patched-iceberg-core/pom.xml   |  13 ++
 itests/bin/generate-cli-splits.sh  |  17 +++
 itests/bin/validateTestConfiguration.sh|  15 ++
 .../TestDDLWithRemoteMetastoreSecondNamenode.java  |  13 +-
 .../hadoop/hive/ql/TestWarehouseExternalDir.java   |  13 +-
 itests/qtest-iceberg/pom.xml   |  13 ++
 kafka-handler/README.md|  18 +++
 llap-server/src/main/resources/package.py  |  16 +++
 llap-server/src/main/resources/templates.py|  17 +++
 packaging/src/docker/README.md |  18 +++
 packaging/src/docker/docker-compose.yml|  16 +++
 packaging/src/license/licenses.xml |  18 +++
 packaging/src/main/assembly/src.xml|   1 -
 pom.xml|  18 ++-
 .../hive/ql/exec/HiveTotalOrderPartitioner.java|   3 -
 .../hadoop/hive/ql/exec/PartitionKeySampler.java   |   3 -
 .../hive/ql/exec/tez/TezTotalOrderPartitioner.java |   3 -
 .../calcite/rules/HiveAggregateSplitRule.java  |   4 +-
 .../calcite/rules/HiveJoinConstraintsRule.java |   3 +-
 .../optimizer/calcite/stats/HiveRelMdRowCount.java |   3 +-
 .../ql/optimizer/physical/SamplingOptimizer.java   |   3 -
 .../io/orc/TestOrcFileStripeMergeRecordReader.java |  12 +-
 .../TestNegativePartitionPrunerCompactExpr.java|  12 +-
 .../TestPositivePartitionPrunerCompactExpr.java|  12 +-
 .../hadoop/hive/ql/parse/TestSplitSample.java  |  13 +-
 .../queries/clientpositive/cbo_join_constraints.q  |  11 ++
 .../cbo_row_count_non_simple_filter.q  |  26 
 .../clientpositive/llap/cbo_join_constraints.q.out |  78 ++
 .../llap/cbo_row_count_non_simple_filter.q.out | 159 +
 .../hadoop/hive/serde2/MultiDelimitSerDe.java  |   2 -
 .../hive/serde2/lazy/fast/StringToDouble.java  |  17 ---
 .../src/resources/hive-webapps/static/js/llap.js   |  17 +++
 .../resources/hive-webapps/static/js/logconf.js|  17 +++
 .../src/resources/hive-webapps/static/js/tab.js|   2 -
 .../main/resources/datanucleus-log4j.properties|  17 +++
 .../metastore-tools/metastore-benchmarks/README.md |  18 +++
 .../src/main/resources/log4j.properties|  16 +++
 standalone-metastore/pom.xml   |   1 -
 62 files changed, 968 insertions(+), 101 deletions(-)
 delete mode 100644 .arcconfig
 create mode 100644 ql/src/test/queries/clientpositive/cbo_join_constraints.q
 create mode 100644 
ql/src/test/qu

(hive) 01/02: HIVE-28155 HIVE-28157 HIVE-28158 HIVE-28159 HIVE-28160: LICENSE and source header improvements (Stamatis Zampetakis reviewed by Zhihua Deng)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 5f9cb486e4de27129ab0428a71e9cbb0d49fd087
Author: Stamatis Zampetakis 
AuthorDate: Wed Mar 27 10:47:13 2024 +0100

HIVE-28155 HIVE-28157 HIVE-28158 HIVE-28159 HIVE-28160: LICENSE and source 
header improvements (Stamatis Zampetakis reviewed by Zhihua Deng)

HIVE-28158: Add ASF license header in non-java files

According to the a ASF policy all source files should contain an ASF 
header. Currently there are a lot of source files that do not contain the ASF 
header. The changes here aim to address the following categories:

Must have:

* Python files (.py)
* Bash/Shell script files (.sh)
* Javascript files (.js)

Should have:

* Maven files (pom.xml)
* GitHub workflows and Docker files (.yml)

Good to have:

* Hive/Tez/Yarn and other configuration files (.xml)
* Log4J property files (.properties)
* Markdown files (.md)

HIVE-28160: Improve LICENSE for jquery and glyphicons-halflings fonts/icons

The following files do not have an ASF header and it unclear to reason 
about the license that applies to them and if it complies to the ASF policy.

jquery.min.js (missing from LICENSE)
jquery.sparkline.min.js (typo in LICENSE)
glyphicons-halflings-regular.svg
glyphicons-halflings-regular.woff
glyphicons-halflings-regular.ttf
glyphicons-halflings-regular.eot

The Glyphicons are available under multiple licenses including commercial 
ones so additional clarifications are needed to ensure that the ones Hive is 
using are under AL2.

HIVE-28159: Remove copyright notice from ASF headers

Source files should not contain ASF copyright notice in the header as per 
https://www.apache.org/legal/src-headers.html#headers

HIVE-28157: Drop unused Arcanist configuration file

The Arcanist configuration file .arcconfig was last modified in 2011 
(HIVE-2588) and it has not been used for a very long time.

The configuration points to Facebook instance of the Phabricator 
(https://reviews.facebook.net/) that is no longer in service.

HIVE-28155: StringToDouble.java violates the ASF 3rd party license 
guidelines

1. Remove ASF header since the original file is under BSD-3 license
2. Add rat exclusion
3. Add entry in LICENSE file

Upgrade Rat plugin verion from 0.13 to 0.16.1 for better coverage and error 
messages

Close apache/hive#5168
---
 .arcconfig | 15 -
 .github/actions/spelling/README.md | 18 +++
 .github/actions/spelling/advice.md | 18 +++
 .github/assign-by-files.yml| 16 ++
 .github/workflows/auto-assign.yml  | 16 ++
 .github/workflows/docker-GA-images.yml | 16 ++
 .github/workflows/spelling.yml | 16 ++
 .github/workflows/stale.yml| 16 ++
 LICENSE| 37 +-
 README.md  | 18 +++
 .../apache/hive/beeline/TestTableOutputFormat.java | 14 
 bin/ext/tokentool.sh   | 17 ++
 data/conf/fair-scheduler-test.xml  | 16 ++
 data/conf/iceberg/llap/tez-site.xml| 17 ++
 data/conf/iceberg/tez/tez-site.xml | 17 ++
 data/conf/llap/tez-site.xml| 17 ++
 data/conf/tez/tez-site.xml | 17 ++
 druid-handler/README.md| 18 +++
 hbase-handler/README.md| 20 +++-
 iceberg/checkstyle/checkstyle-suppressions.xml | 16 ++
 iceberg/checkstyle/checkstyle.xml  | 17 ++
 iceberg/iceberg-catalog/pom.xml| 13 
 iceberg/iceberg-handler/pom.xml| 13 
 iceberg/patched-iceberg-api/pom.xml| 13 
 iceberg/patched-iceberg-core/pom.xml   | 13 
 itests/bin/generate-cli-splits.sh  | 17 ++
 itests/bin/validateTestConfiguration.sh| 15 +
 .../TestDDLWithRemoteMetastoreSecondNamenode.java  | 13 
 .../hadoop/hive/ql/TestWarehouseExternalDir.java   | 13 
 itests/qtest-iceberg/pom.xml   | 13 
 kafka-handler/README.md| 18 +++
 llap-server/src/main/resources/package.py  | 16 ++
 llap-server/src/main/resources/templates.py| 17 ++
 packaging/src/docker/README.md | 18 +++
 packaging/src/docker/docker

(hive-site) branch main updated: HIVE-28177: Announce Hive 1.x EOL (Stamatis Zampetakis reviewed by Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-site.git


The following commit(s) were added to refs/heads/main by this push:
 new 8b1a898  HIVE-28177: Announce Hive 1.x EOL (Stamatis Zampetakis 
reviewed by Ayush Saxena)
8b1a898 is described below

commit 8b1a898c4ded604bdc06897429c59a3f69f03a6d
Author: Stamatis Zampetakis 
AuthorDate: Wed Apr 17 09:58:59 2024 +0200

HIVE-28177: Announce Hive 1.x EOL (Stamatis Zampetakis reviewed by Ayush 
Saxena)

Close apache/hive-site#14
---
 content/general/downloads.md | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/content/general/downloads.md b/content/general/downloads.md
index 0c0e019..8de6bca 100644
--- a/content/general/downloads.md
+++ b/content/general/downloads.md
@@ -33,6 +33,12 @@ directory.
 
 ## News
 ---
+*  11 April 2024: EOL for release 1.x line
+
+The Apache Hive Community has voted to declare the 1.x release line as End of 
Life (EOL). This means no further updates or releases will be made for this 
series.
+
+We urge all Hive 1.x users to upgrade to the latest versions promptly to 
benefit from new features and ongoing support.
+
 *  29 March 2024: release 4.0.0 available
 * This release works with Hadoop 3.3.6, Tez 0.10.3
 * You can look at the complete [JIRA change log for this 
release][HIVE_4_0_0_CL].

svn commit: r68598 - /release/hive/hive-4.0.0-beta-1/

[zabetak]

Author: zabetak
Date: Wed Apr 17 07:23:37 2024
New Revision: 68598

Log:
Archiving release Apache Hive 4.0.0-beta-1

Removed:
release/hive/hive-4.0.0-beta-1/

svn commit: r68596 - /release/hive/hive-4.0.0-alpha-1/

[zabetak]

Author: zabetak
Date: Wed Apr 17 07:23:14 2024
New Revision: 68596

Log:
Archiving release Apache Hive 4.0.0-alpha-1

Removed:
release/hive/hive-4.0.0-alpha-1/

svn commit: r68597 - /release/hive/hive-4.0.0-alpha-2/

[zabetak]

Author: zabetak
Date: Wed Apr 17 07:23:22 2024
New Revision: 68597

Log:
Archiving release Apache Hive 4.0.0-alpha-2

Removed:
release/hive/hive-4.0.0-alpha-2/

svn commit: r68595 - /release/hive/hive-3.1.2/

[zabetak]

Author: zabetak
Date: Wed Apr 17 07:22:18 2024
New Revision: 68595

Log:
Archiving release Apache Hive 3.1.2

Removed:
release/hive/hive-3.1.2/

svn commit: r68594 - /release/hive/hive-1.2.2/

[zabetak]

Author: zabetak
Date: Wed Apr 17 07:21:45 2024
New Revision: 68594

Log:
Archiving release Apache Hive 1.2.2

Removed:
release/hive/hive-1.2.2/

(hive) branch master updated (855e4055675 -> a537000af73)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 855e4055675 HIVE-28148: Implement array_compact UDF to remove all 
nulls from an array (#5161) (Taraka Rama Rao Lethavadla reviewed by Sourabh 
Badhya)
 new e6e6b12d5fe HIVE-28144: Remove overly verbose debug messages from 
MetastoreDirectSqlUtils (Stamatis Zampetakis reviewed by Butao Zhang)
 new a537000af73 HIVE-28046: Use serdeConstants fields instead of string 
literals in hive-exec module (Michal Lorek reviewed by Stamatis Zampetakis)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../hadoop/hive/ql/exec/ColumnStatsUpdateTask.java | 27 ---
 .../apache/hadoop/hive/ql/exec/DDLPlanUtils.java   |  3 +-
 .../ql/exec/vector/VectorExpressionDescriptor.java | 24 +++---
 .../hive/ql/exec/vector/VectorizationContext.java  | 86 +++---
 .../hive/ql/exec/vector/VectorizedBatchUtil.java   |  3 +-
 .../exec/vector/expressions/CastDateToBoolean.java |  3 +-
 .../vector/expressions/CastDateToTimestamp.java|  3 +-
 .../vector/expressions/CastDoubleToTimestamp.java  |  3 +-
 .../vector/expressions/CastTimestampToBoolean.java |  3 +-
 .../vector/expressions/CastTimestampToDouble.java  |  3 +-
 .../vector/expressions/CastTimestampToLong.java|  3 +-
 .../expressions/DateColSubtractDateColumn.java |  5 +-
 .../expressions/DateColSubtractDateScalar.java |  5 +-
 .../expressions/DateScalarSubtractDateColumn.java  |  5 +-
 .../IfExprDoubleColumnDoubleColumn.java|  5 +-
 .../hive/ql/exec/vector/ptf/VectorPTFOperator.java | 11 +--
 .../org/apache/hadoop/hive/ql/io/IOConstants.java  |  2 +-
 .../hadoop/hive/ql/io/RCFileOutputFormat.java  |  3 +-
 .../org/apache/hadoop/hive/ql/io/orc/OrcSerde.java |  4 +-
 .../org/apache/hadoop/hive/ql/io/orc/OrcUnion.java |  5 +-
 .../parquet/convert/HiveCollectionConverter.java   |  4 +-
 .../hive/ql/io/parquet/serde/ParquetHiveSerDe.java |  2 +-
 .../hive/ql/io/sarg/ConvertAstToSearchArg.java |  3 +-
 .../hive/ql/optimizer/SimpleFetchOptimizer.java|  3 +-
 .../calcite/translator/SqlFunctionConverter.java   | 28 +++
 .../hive/ql/optimizer/physical/Vectorizer.java | 55 +++---
 .../hadoop/hive/ql/parse/BaseSemanticAnalyzer.java |  4 +-
 .../hive/ql/parse/rewrite/MergeRewriter.java   |  3 +-
 .../hive/ql/parse/type/TypeCheckProcFactory.java   |  8 +-
 .../apache/hadoop/hive/ql/plan/PartitionDesc.java  |  2 +-
 .../org/apache/hadoop/hive/ql/plan/PlanUtils.java  |  9 ++-
 .../hadoop/hive/ql/processors/DfsProcessor.java|  3 +-
 .../ql/processors/LlapCacheResourceProcessor.java  |  5 +-
 .../processors/LlapClusterResourceProcessor.java   | 13 ++--
 .../hive/ql/udf/esri/serde/BaseJsonSerDe.java  |  3 +-
 .../hive/ql/udf/generic/GenericUDFBetween.java |  3 +-
 .../hive/ql/udf/ptf/ValueBoundaryScanner.java  | 25 ---
 .../hive/metastore/MetastoreDirectSqlUtils.java|  5 +-
 38 files changed, 209 insertions(+), 175 deletions(-)

(hive) 01/02: HIVE-28144: Remove overly verbose debug messages from MetastoreDirectSqlUtils (Stamatis Zampetakis reviewed by Butao Zhang)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit e6e6b12d5fe13e8cbca99e8f3a5d5f13438b2400
Author: Stamatis Zampetakis 
AuthorDate: Fri Mar 22 11:05:08 2024 +0100

HIVE-28144: Remove overly verbose debug messages from 
MetastoreDirectSqlUtils (Stamatis Zampetakis reviewed by Butao Zhang)

When BITVECTOR or KLL stats are disabled/not present in the metastore the 
following message may appear way too often in the HMS logs.
```
2024-03-22T01:50:57,849 DEBUG [CachedStore-CacheUpdateService: Thread-240] 
metastore.MetastoreDirectSqlUtils: Expected blob type but got java.lang.String
```
In fact in some cases, the message appears more than once for every single 
partition that is present in the table(s) being queried. When the number of 
partitions is important it can easily clog the logs with redundant and useless 
information.

To put things in perspective while running the cbo_query10.q on the 
statistics of TPC-DS30TB dataset the message occupies more than 50% (26MB) of 
the total log file (46MB).

The presence of the message does not tells us much on its own. In 
conjunction with the code we can infer that we are not fetching BITVECTOR/KLL 
stats from the metastore but this could be done in a different place without 
having to print the same message 170K times.

Removing this message saves disk space, avoids frequent log rotation, and 
improves the overall readability of the log file.

There is another redundant message which appears when transforming a 
database value to Boolean. The message is redundant since it is followed 
directly by an exception so there is no reason to have both. This message may 
not appear as often as the previous one but given that it doesn't add much 
value it can also be removed.

Close apache/hive#5159
---
 .../org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java| 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
index 067e415d725..8a608a030ee 100644
--- 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
+++ 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java
@@ -554,7 +554,6 @@ class MetastoreDirectSqlUtils {
 return true;
   }
 }
-LOG.debug("Value is of type {}", value.getClass());
 throw new MetaException("Cannot extract boolean from column value " + 
value);
   }
 
@@ -590,8 +589,8 @@ class MetastoreDirectSqlUtils {
   return (byte[]) value;
 }
else {
-  // this may happen when enablebitvector is false
-  LOG.debug("Expected blob type but got " + value.getClass().getName());
+  // 
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getStatsList(enableBitVector,enableKll)
+  // We get here when enableBitvector or enableKll is false
   return null;
 }
   }

(hive) 02/02: HIVE-28046: Use serdeConstants fields instead of string literals in hive-exec module (Michal Lorek reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit a537000af730e994a37b0f60b37007e3caf8f1c7
Author: mlorek 
AuthorDate: Tue Jan 30 14:28:23 2024 +

HIVE-28046: Use serdeConstants fields instead of string literals in 
hive-exec module (Michal Lorek reviewed by Stamatis Zampetakis)

The procedure to replace literals with constans is outlined below:
1. Generate a sed script file using serdeConstants.java for the possible 
replacements
grep "public static final java.lang.String" 
./serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/serdeConstants.java
 | sed 's/.\+java\.lang\.String//'| sed 's/ //g' | sed 
's/\(.*\)=\(.*\);/s#\2#serdeConstants.\1#/' | sed 's/\./\\./g'  > sedrepscript

2. Apply the script file to all java files in production code
find ql/src/java -name "*.java" -exec sed -i -f sedrepscript {} \;

3. Manual review of change files adding import when necessary and reverting 
irrelevant changes

using serdeConstants fields instead of string literals for 'columns' and 
'columns.types'

Co-authored-by: Stamatis Zampetakis 

Close apache/hive#5072
---
 .../hadoop/hive/ql/exec/ColumnStatsUpdateTask.java | 27 ---
 .../apache/hadoop/hive/ql/exec/DDLPlanUtils.java   |  3 +-
 .../ql/exec/vector/VectorExpressionDescriptor.java | 24 +++---
 .../hive/ql/exec/vector/VectorizationContext.java  | 86 +++---
 .../hive/ql/exec/vector/VectorizedBatchUtil.java   |  3 +-
 .../exec/vector/expressions/CastDateToBoolean.java |  3 +-
 .../vector/expressions/CastDateToTimestamp.java|  3 +-
 .../vector/expressions/CastDoubleToTimestamp.java  |  3 +-
 .../vector/expressions/CastTimestampToBoolean.java |  3 +-
 .../vector/expressions/CastTimestampToDouble.java  |  3 +-
 .../vector/expressions/CastTimestampToLong.java|  3 +-
 .../expressions/DateColSubtractDateColumn.java |  5 +-
 .../expressions/DateColSubtractDateScalar.java |  5 +-
 .../expressions/DateScalarSubtractDateColumn.java  |  5 +-
 .../IfExprDoubleColumnDoubleColumn.java|  5 +-
 .../hive/ql/exec/vector/ptf/VectorPTFOperator.java | 11 +--
 .../org/apache/hadoop/hive/ql/io/IOConstants.java  |  2 +-
 .../hadoop/hive/ql/io/RCFileOutputFormat.java  |  3 +-
 .../org/apache/hadoop/hive/ql/io/orc/OrcSerde.java |  4 +-
 .../org/apache/hadoop/hive/ql/io/orc/OrcUnion.java |  5 +-
 .../parquet/convert/HiveCollectionConverter.java   |  4 +-
 .../hive/ql/io/parquet/serde/ParquetHiveSerDe.java |  2 +-
 .../hive/ql/io/sarg/ConvertAstToSearchArg.java |  3 +-
 .../hive/ql/optimizer/SimpleFetchOptimizer.java|  3 +-
 .../calcite/translator/SqlFunctionConverter.java   | 28 +++
 .../hive/ql/optimizer/physical/Vectorizer.java | 55 +++---
 .../hadoop/hive/ql/parse/BaseSemanticAnalyzer.java |  4 +-
 .../hive/ql/parse/rewrite/MergeRewriter.java   |  3 +-
 .../hive/ql/parse/type/TypeCheckProcFactory.java   |  8 +-
 .../apache/hadoop/hive/ql/plan/PartitionDesc.java  |  2 +-
 .../org/apache/hadoop/hive/ql/plan/PlanUtils.java  |  9 ++-
 .../hadoop/hive/ql/processors/DfsProcessor.java|  3 +-
 .../ql/processors/LlapCacheResourceProcessor.java  |  5 +-
 .../processors/LlapClusterResourceProcessor.java   | 13 ++--
 .../hive/ql/udf/esri/serde/BaseJsonSerDe.java  |  3 +-
 .../hive/ql/udf/generic/GenericUDFBetween.java |  3 +-
 .../hive/ql/udf/ptf/ValueBoundaryScanner.java  | 25 ---
 37 files changed, 207 insertions(+), 172 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java
index 8b6c8d6b1bd..c4ae676d7a9 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java
@@ -55,6 +55,7 @@ import org.apache.hadoop.hive.ql.metadata.Table;
 import org.apache.hadoop.hive.ql.parse.SemanticException;
 import org.apache.hadoop.hive.ql.plan.ColumnStatsUpdateWork;
 import org.apache.hadoop.hive.ql.plan.api.StageType;
+import org.apache.hadoop.hive.serde.serdeConstants;
 import org.apache.hadoop.hive.serde2.io.DateWritableV2;
 import org.apache.hadoop.hive.serde2.io.TimestampWritableV2;
 import org.slf4j.Logger;
@@ -101,9 +102,11 @@ public class ColumnStatsUpdateTask extends 
Task {
 
 ColumnStatisticsData statsData = new ColumnStatisticsData();
 
-if (columnType.equalsIgnoreCase("long") || 
columnType.equalsIgnoreCase("tinyint")
-|| columnType.equalsIgnoreCase("smallint") || 
columnType.equalsIgnoreCase("int")
-|| columnType.equalsIgnoreCase("bigint")) {
+if (columnType.equalsIgnoreCase("long")
+|| columnType.equalsIgnoreCase(serdeConstants.TINYINT_T

(hive-site) branch gh-pages deleted (was 95b1c83)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/hive-site.git


 was 95b1c83  Updated supported Hadoop version for Hive 4.0.0-alpha-2

This change permanently discards the following revisions:

 discard 95b1c83  Updated supported Hadoop version for Hive 4.0.0-alpha-2
 discard 59d1147  Hive 4.0.0-alpha-2 release
 discard 8744476  Apache Hive 3.1.3 release (Naveen Gangam)
 discard 6b491aa  Hive 4.0.0-alpha-1 release.
 discard ba13381  dummy commit
 discard 408fadd  empty commit

(hive) branch master updated (c1d592f4f98 -> 9a0ce4e1589)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from c1d592f4f98 HIVE-28069: Iceberg: Implement Merge task functionality 
for Iceberg tables (#5076) (Sourabh Badhya reviewed by Denys Kuzmenko, 
Krisztian Kasa)
 add 9a0ce4e1589 HIVE-28140: Remove unnecessary shading of test jars in 
Kafka module to speedup build (Stamatis Zampetakis reviewed by Butao Zhang)

No new revisions were added by this update.

Summary of changes:
 kafka-handler/pom.xml | 1 -
 1 file changed, 1 deletion(-)

(hive) 02/03: HIVE-28115: Upgrade mockito-core from 3.3.3 to 3.4.4 in standalone-metastore (Raghav Aggarwal reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit cf49f0eaeadd297e444fe6da42849c1e39bb869d
Author: Raghav Aggarwal 
AuthorDate: Tue Mar 12 16:51:52 2024 +0530

HIVE-28115: Upgrade mockito-core from 3.3.3 to 3.4.4 in 
standalone-metastore (Raghav Aggarwal reviewed by Stamatis Zampetakis)

The change simply aligns the version with the one used in the main pom.xml 
file.

Close apache/hive#5125
---
 standalone-metastore/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/standalone-metastore/pom.xml b/standalone-metastore/pom.xml
index e56d7b6197c..12d2481a5c4 100644
--- a/standalone-metastore/pom.xml
+++ b/standalone-metastore/pom.xml
@@ -88,7 +88,7 @@
 0.9.3
 0.16.0
 2.18.0
-3.3.3
+3.4.4
 1.8.5
 3.24.4
 1.51.0

(hive) 03/03: HIVE-27848: Refactor Initiator hierarchy into CompactorUtil and fix failure in TestCrudCompactorOnTez (Taraka Rama Rao Lethavadla reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit df45194268ffb10f9f7aae0ab4e3aec35a7b31d8
Author: Taraka Rama Rao Lethavadla 
AuthorDate: Tue Nov 7 19:06:27 2023 +0530

HIVE-27848: Refactor Initiator hierarchy into CompactorUtil and fix failure 
in TestCrudCompactorOnTez (Taraka Rama Rao Lethavadla reviewed by Stamatis 
Zampetakis)

The work started initially to fix the 
TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing.

However, while changing the code to address the failure, the inheritance 
based design for the Initator that was chosen in HIVE-27598 revealed some 
weaknesses briefly outlined below.

Due to inheritance the InitiatorBase class becomes a Thread something that 
doesn't really make sense and it comes with additional overhead every time we 
instantiate it. Moreover, the only class that currently extends InitiatorBase 
is the Initiator and it's difficult to imagine how we can make other extensions 
from InitiatorBase; the code becomes complex and any subtle change in 
InitiatorBase may have unpredictable effects on Initiator. Having a "Base" 
class that is not really meant to [...]

For the reasons above the focus of the work changed from just re-enabling 
the test to improving and addressing the shortcomings of the inheritance based 
design of Initiator.

Close apache/hive#4859
---
 .../ql/txn/compactor/TestCrudCompactorOnTez.java   |  48 +++-
 .../compact/AlterTableCompactOperation.java|  83 +++---
 .../org/apache/hadoop/hive/ql/io/AcidUtils.java|   7 +
 .../hive/ql/txn/compactor/CompactorUtil.java   | 250 +
 .../hadoop/hive/ql/txn/compactor/Initiator.java|   6 +-
 .../hive/ql/txn/compactor/InitiatorBase.java   | 307 -
 6 files changed, 357 insertions(+), 344 deletions(-)

diff --git 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
index f64fac9038d..26302e6c143 100644
--- 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
+++ 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
@@ -77,7 +77,6 @@ import org.apache.orc.StripeInformation;
 import org.apache.orc.TypeDescription;
 import org.apache.orc.impl.RecordReaderImpl;
 import org.junit.Assert;
-import org.junit.Ignore;
 import org.junit.Test;
 import org.mockito.ArgumentCaptor;
 import org.mockito.Mockito;
@@ -602,7 +601,6 @@ public class TestCrudCompactorOnTez extends 
CompactorOnTezTest {
   }
 
   @Test
-  @Ignore("HIVE-27848")
   public void secondCompactionShouldBeRefusedBeforeEnqueueing() throws 
Exception {
 conf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, true);
 
@@ -648,6 +646,52 @@ public class TestCrudCompactorOnTez extends 
CompactorOnTezTest {
 Assert.assertEquals("ready for cleaning", compacts.get(0).getState());
   }
 
+  @Test
+  public void secondCompactionShouldBeRefusedBeforeEnqueueingForPartition() 
throws Exception {
+conf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, true);
+
+final String dbName = "default";
+final String tableName = "compaction_test";
+executeStatementOnDriver("drop table if exists " + tableName, driver);
+executeStatementOnDriver("CREATE TABLE " + tableName + "(id string, value 
string) partitioned by(pt string) CLUSTERED BY(id) "
++ "INTO 10 BUCKETS STORED AS ORC 
TBLPROPERTIES('transactional'='true')", driver);
+executeStatementOnDriver("alter table " + tableName + " add 
partition(pt='test')",driver);
+executeStatementOnDriver("INSERT INTO TABLE " + tableName + " 
partition(pt='test') values ('1','one'),('2','two'),('3','three'),"
++ 
"('4','four'),('5','five'),('6','six'),('7','seven'),('8','eight'),('9','nine'),('10','ten'),"
++ 
"('11','eleven'),('12','twelve'),('13','thirteen'),('14','fourteen'),('15','fifteen'),('16','sixteen'),"
++ 
"('17','seventeen'),('18','eighteen'),('19','nineteen'),('20','twenty')", 
driver);
+
+executeStatementOnDriver("insert into " + tableName + " 
partition(pt='test') values ('21', 'value21'),('84', 'value84'),"
++ "('66', 'value66'),('54', 'value54')", driver);
+executeStatementOnDriver(
+"insert into " + tableName + " partition(pt='test') values ('22', 
'value22'),('34', 'value34')," + "('35', 'value35')", driver);
+executeStatementOnDriver("insert into " + tableName + " 
partition(pt='test') value

(hive) 01/03: HIVE-27953: Retire https://apache.github.io sites and remove obsolete content/actions (Simhadri Govindappa reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 33737a123d929af4d1d4c623c9031c68a316c583
Author: SimhadriG 
AuthorDate: Thu Mar 7 01:04:18 2024 +0530

HIVE-27953: Retire https://apache.github.io sites and remove obsolete 
content/actions (Simhadri Govindappa reviewed by Stamatis Zampetakis)

1. Remove docs directory which holds the obsolete content 
https://apache.github.io/hive/
2. Remove GitHub Pages config from .asf.yaml

Close apache/hive#5119
---
 .asf.yaml|   2 -
 docs/Dockerfile  |  51 --
 docs/Gemfile |   3 -
 docs/README.md   |  24 ---
 docs/_config.yml |  17 --
 docs/_includes/footer.html   |  14 --
 docs/_includes/header.html   |   5 -
 docs/_includes/sidenav.html  |  50 --
 docs/_includes/top.html  |  21 ---
 docs/_layouts/default.html   |  19 --
 docs/css/hive.css| 365 ---
 docs/doap_Hive.rdf   |  58 ---
 docs/downloads.md| 211 --
 docs/favicon.ico | Bin 1150 -> 0 bytes
 docs/hcatalog_downloads.md   |  43 -
 docs/images/feather_small.gif| Bin 7500 -> 0 bytes
 docs/images/hive-logo.jpg| Bin 2498 -> 0 bytes
 docs/images/hive_logo_medium.jpg | Bin 4372 -> 0 bytes
 docs/index.md|  62 ---
 docs/issue_tracking.md   |  31 
 docs/javadoc.md  |  32 
 docs/mailing_lists.md|  78 -
 docs/people.md   | 156 -
 docs/privacy_policy.md   |  48 -
 docs/version_control.md  |  27 ---
 25 files changed, 1317 deletions(-)

diff --git a/.asf.yaml b/.asf.yaml
index 84c207f56d9..426b19e9b4d 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -17,8 +17,6 @@
 github:
   description: "Apache Hive"
   homepage: https://hive.apache.org/
-  ghp_branch: master
-  ghp_path: /docs
   labels:
 - hive
 - java
diff --git a/docs/Dockerfile b/docs/Dockerfile
deleted file mode 100644
index 1889955fd3b..000
--- a/docs/Dockerfile
+++ /dev/null
@@ -1,51 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Hive site builder
-#
-
-FROM ubuntu:18.04
-MAINTAINER Hive team 
-
-RUN ln -fs /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
-RUN apt-get update
-RUN apt-get install -y \
-  g++ \
-  gcc \
-  git \
-  libssl-dev \
-  libz-dev \
-  make \
-  ruby-dev \
-  rubygems \
-  tzdata
-RUN gem install \
-  bundler \
-  liquid \
-  listen \
-  rouge
-RUN gem install jekyll -v 3.8.6
-RUN gem install github-pages
-
-RUN useradd -ms /bin/bash hive
-COPY . /home/hive/site
-RUN chown -R hive:hive /home/hive
-USER hive
-WORKDIR /home/hive/site
-
-EXPOSE 4000
-CMD bundle exec jekyll serve -H 0.0.0.0
-
diff --git a/docs/Gemfile b/docs/Gemfile
deleted file mode 100644
index 1c529c9ce18..000
--- a/docs/Gemfile
+++ /dev/null
@@ -1,3 +0,0 @@
-source 'https://rubygems.org'
-gem 'rouge'
-gem 'jekyll', "~> 3.8.3"
diff --git a/docs/README.md b/docs/README.md
deleted file mode 100644
index 6100928ca72..000
--- a/docs/README.md
+++ /dev/null
@@ -1,24 +0,0 @@
-# Apache Hive docs site
-
-This directory contains the code for the Apache Hive web site,
-[hive.apache.org](https://hive.apache.org/). The easiest way to build
-the site is to use docker to use a standard environment.
-
-## Run the docker container with the preview of the site.
-
-1. `docker build -t hive-site .`
-2. `CONTAINER=$(docker run -d -p 4000:4000 hive-site)`
-
-## Browsing
-
-Look at the site by navigating to
-[http://0.0.0.0:4000/](http://0.0.0.0:4000/) .
-
-## Pushing to site
-
-Commit and push the changes to the main branch. The site is automatically 
deployed
-from the site directory.
-
-## Shutting down the docker container
-
-1. `docker stop $CONTAINER`
\ No newline at end of file
diff --git a/docs/_config.yml b/docs/_config.yml
deleted file mode 100644
index 68bc8f0b591..000
--- a/docs/_config.yml
+++ /dev/null
@

(hive) branch master updated (ee3d362c6d8 -> df45194268f)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from ee3d362c6d8 HIVE-28125: HIVE-28125: Bump 
org.apache.zookeeper:zookeeper from 3.8.3 to 3.8.4 (#5130). (dependabot, 
Reviewed by Ayush Saxena)
 new 33737a123d9 HIVE-27953: Retire https://apache.github.io sites and 
remove obsolete content/actions (Simhadri Govindappa reviewed by Stamatis 
Zampetakis)
 new cf49f0eaead HIVE-28115: Upgrade mockito-core from 3.3.3 to 3.4.4 in 
standalone-metastore (Raghav Aggarwal reviewed by Stamatis Zampetakis)
 new df45194268f HIVE-27848: Refactor Initiator hierarchy into 
CompactorUtil and fix failure in TestCrudCompactorOnTez (Taraka Rama Rao 
Lethavadla reviewed by Stamatis Zampetakis)

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .asf.yaml  |   2 -
 docs/Dockerfile|  51 ---
 docs/Gemfile   |   3 -
 docs/README.md |  24 --
 docs/_config.yml   |  17 -
 docs/_includes/footer.html |  14 -
 docs/_includes/header.html |   5 -
 docs/_includes/sidenav.html|  50 ---
 docs/_includes/top.html|  21 --
 docs/_layouts/default.html |  19 --
 docs/css/hive.css  | 365 -
 docs/doap_Hive.rdf |  58 
 docs/downloads.md  | 211 
 docs/favicon.ico   | Bin 1150 -> 0 bytes
 docs/hcatalog_downloads.md |  43 ---
 docs/images/feather_small.gif  | Bin 7500 -> 0 bytes
 docs/images/hive-logo.jpg  | Bin 2498 -> 0 bytes
 docs/images/hive_logo_medium.jpg   | Bin 4372 -> 0 bytes
 docs/index.md  |  62 
 docs/issue_tracking.md |  31 --
 docs/javadoc.md|  32 --
 docs/mailing_lists.md  |  78 -
 docs/people.md | 156 -
 docs/privacy_policy.md |  48 ---
 docs/version_control.md|  27 --
 .../ql/txn/compactor/TestCrudCompactorOnTez.java   |  48 ++-
 .../compact/AlterTableCompactOperation.java|  83 +++--
 .../org/apache/hadoop/hive/ql/io/AcidUtils.java|   7 +
 .../hive/ql/txn/compactor/CompactorUtil.java   | 250 ++
 .../hadoop/hive/ql/txn/compactor/Initiator.java|   6 +-
 .../hive/ql/txn/compactor/InitiatorBase.java   | 307 -
 standalone-metastore/pom.xml   |   2 +-
 32 files changed, 358 insertions(+), 1662 deletions(-)
 delete mode 100644 docs/Dockerfile
 delete mode 100644 docs/Gemfile
 delete mode 100644 docs/README.md
 delete mode 100644 docs/_config.yml
 delete mode 100644 docs/_includes/footer.html
 delete mode 100644 docs/_includes/header.html
 delete mode 100644 docs/_includes/sidenav.html
 delete mode 100644 docs/_includes/top.html
 delete mode 100644 docs/_layouts/default.html
 delete mode 100644 docs/css/hive.css
 delete mode 100644 docs/doap_Hive.rdf
 delete mode 100644 docs/downloads.md
 delete mode 100644 docs/favicon.ico
 delete mode 100644 docs/hcatalog_downloads.md
 delete mode 100644 docs/images/feather_small.gif
 delete mode 100644 docs/images/hive-logo.jpg
 delete mode 100644 docs/images/hive_logo_medium.jpg
 delete mode 100644 docs/index.md
 delete mode 100644 docs/issue_tracking.md
 delete mode 100644 docs/javadoc.md
 delete mode 100644 docs/mailing_lists.md
 delete mode 100644 docs/people.md
 delete mode 100644 docs/privacy_policy.md
 delete mode 100644 docs/version_control.md
 delete mode 100644 
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/InitiatorBase.java

(incubator-xtable) branch main updated: Enforce ASF license header in non-java files via spotless (#359)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git


The following commit(s) were added to refs/heads/main by this push:
 new 7c7b54b3 Enforce ASF license header in non-java files via spotless 
(#359)
7c7b54b3 is described below

commit 7c7b54b3870791578b4689b29cf8794866eefa52
Author: wuchunfu <319355...@qq.com>
AuthorDate: Fri Mar 15 09:58:20 2024 +0100

Enforce ASF license header in non-java files via spotless (#359)
---
 .asf.yaml  | 18 ++---
 .azure-pipelines/workflows/compile-and-test.yml| 16 
 .azure-pipelines/workflows/maven.yml   | 16 
 .github/dependabot.yml | 26 +++---
 .github/workflows/azure_ci_check.yml   | 29 ---
 .github/workflows/package-deploy.yml   | 16 
 .github/workflows/scheduled_workflow.yml   | 29 ---
 .github/workflows/site-build.yml   | 16 
 .github/workflows/site-deploy.yml  | 16 
 .mvn/extensions.xml| 28 +++
 api/pom.xml| 18 -
 core/pom.xml   | 16 
 core/src/test/resources/junit-platform.properties  | 16 
 core/src/test/resources/log4j2.xml | 20 -
 demo/docker-compose.yaml   | 16 
 demo/presto/catalog/delta.properties   | 16 
 demo/presto/catalog/hive.properties| 16 
 demo/presto/catalog/hudi.properties| 16 
 demo/presto/catalog/iceberg.properties | 16 
 demo/presto/config.properties  | 16 
 demo/presto/jvm.config | 16 
 demo/presto/node.properties| 16 
 demo/start_demo.sh | 20 -
 demo/trino/catalog/delta.properties| 16 
 demo/trino/catalog/hive.properties | 16 
 demo/trino/catalog/iceberg.properties  | 16 
 hudi-support/extensions/pom.xml| 18 -
 .../src/test/resources/junit-platform.properties   | 16 
 hudi-support/pom.xml   | 18 -
 hudi-support/utils/pom.xml | 18 -
 .../src/test/resources/junit-platform.properties   | 16 
 lombok.config  | 18 -
 pom.xml| 92 +-
 style/text-license-header  | 16 
 style/xml-license-header   | 16 
 utilities/pom.xml  | 16 
 utilities/src/main/resources/log4j2.xml| 20 -
 .../main/resources/onetable-client-defaults.yaml   | 36 ++---
 .../main/resources/onetable-hadoop-defaults.xml| 18 -
 39 files changed, 702 insertions(+), 92 deletions(-)

diff --git a/.asf.yaml b/.asf.yaml
index 2693b39d..7179cfbf 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -1,19 +1,19 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
 #
-# http://www.apache.org/licenses/LICENSE-2.0
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
+#
 github:
   description: "Apache XTable (incubating) is a cross-table converter for 
lakehouse table formats that facilitates interoperability across data 
processing systems and query engines."
   homepage: https://xtable.apache.org/
diff --git a/.azure-pipelines/workflows/compile-and-test.yml 
b/.azure-pipelines/workflows/compile-and-test.yml
index b3d10620..0df997d4 100644
--- a/.azure-pipelines/workflows/compile-and-test.yml
+++ b/.azure-pipelines/workflows/compil

(incubator-xtable) branch main updated (ebcb9b5 -> 7bd832d)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git


from ebcb9b5  Reenable CI using Azure Pipelines (#368)
 add 7bd832d  Upgrade spotless-maven-plugin from 2.27.2 to 2.43.0 (#371)

No new revisions were added by this update.

Summary of changes:
 core/src/main/java/io/onetable/delta/DeltaSourceClient.java | 1 +
 core/src/test/java/io/onetable/delta/TestDeltaSync.java | 1 +
 pom.xml | 4 ++--
 3 files changed, 4 insertions(+), 2 deletions(-)

(hive) branch master updated: HIVE-28063: Drop PerfLogger#setPerfLogger method and unused fields/methods (Stamatis Zampetakis reviewed by Laszlo Bodor, Attila Turoczy)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 49e65bdd7fd HIVE-28063: Drop PerfLogger#setPerfLogger method and 
unused fields/methods (Stamatis Zampetakis reviewed by Laszlo Bodor, Attila 
Turoczy)
49e65bdd7fd is described below

commit 49e65bdd7fd47adffbc59091eaea1618d90c253a
Author: Stamatis Zampetakis 
AuthorDate: Tue Feb 6 10:41:17 2024 +0100

HIVE-28063: Drop PerfLogger#setPerfLogger method and unused fields/methods 
(Stamatis Zampetakis reviewed by Laszlo Bodor, Attila Turoczy)

The PerfLogger#setPerfLogger is redundant and error-prone.

The small number of current uses could be replaced by simply calling the 
respective getter (which implicitly changes the underlying ThreadLocal 
variable).

Ideally thread local variable should never be set after obtaining the 
initial value. Moreover, allowing any caller to change the thread local 
variable can affect the correctness of the program.

Dropping this method improves the encapsulation and readability of the 
class.

The org.apache.hadoop.hive.metastore.metrics.PerfLogger has various unused 
fields/methods that can be removed as well to improve encapsulation, 
readability, and maintenance.

Close apache/hive#5066
---
 .../org/apache/hadoop/hive/ql/log/PerfLogger.java  |  4 --
 .../apache/hadoop/hive/ql/log/PerfLoggerTest.java  |  2 -
 .../hive/service/cli/operation/SQLOperation.java   |  2 +-
 .../cli/operation/hplsql/HplSqlOperation.java  |  2 +-
 .../hadoop/hive/metastore/metrics/PerfLogger.java  | 61 --
 5 files changed, 2 insertions(+), 69 deletions(-)

diff --git a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 
b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
index d6fcc0bda75..cc57e9b42b0 100644
--- a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
+++ b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
@@ -119,10 +119,6 @@ public class PerfLogger {
 return result;
   }
 
-  public static void setPerfLogger(PerfLogger resetPerfLogger) {
-perfLogger.set(resetPerfLogger);
-  }
-
   /**
* Call this function when you start to measure time spent by a piece of 
code.
* @param callerName the logging object to be used.
diff --git a/common/src/test/org/apache/hadoop/hive/ql/log/PerfLoggerTest.java 
b/common/src/test/org/apache/hadoop/hive/ql/log/PerfLoggerTest.java
index d844a983a7b..d06bfa87356 100644
--- a/common/src/test/org/apache/hadoop/hive/ql/log/PerfLoggerTest.java
+++ b/common/src/test/org/apache/hadoop/hive/ql/log/PerfLoggerTest.java
@@ -55,7 +55,6 @@ public class PerfLoggerTest {
 AtomicInteger count = new AtomicInteger(0);
 // getEndTimes in a loop
 executorService.execute(() -> {
-  PerfLogger.setPerfLogger(pl);
   try {
 count.incrementAndGet();
 snooze(100);
@@ -76,7 +75,6 @@ public class PerfLoggerTest {
   executorService.execute(() -> {
 try {
   int cnt = count.incrementAndGet();
-  PerfLogger.setPerfLogger(pl);
   for (int i = 0; i < 64; ++i) {
 pl.perfLogBegin("test", PerfLogger.COMPILE + "_ "+  cnt + "_" + i);
 snooze(50);
diff --git 
a/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
b/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
index 4aa142f52f3..d7deba7b274 100644
--- a/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
+++ b/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java
@@ -322,7 +322,7 @@ public class SQLOperation extends ExecuteStatementOperation 
{
   Hive.set(parentHive);
   // TODO: can this result in cross-thread reuse of session state?
   SessionState.setCurrentSessionState(parentSessionState);
-  PerfLogger.setPerfLogger(SessionState.getPerfLogger());
+  SessionState.getPerfLogger();
   if (!embedded) {
 LogUtils.registerLoggingContext(queryState.getConf());
   }
diff --git 
a/service/src/java/org/apache/hive/service/cli/operation/hplsql/HplSqlOperation.java
 
b/service/src/java/org/apache/hive/service/cli/operation/hplsql/HplSqlOperation.java
index 2bd829fa806..16d16792aed 100644
--- 
a/service/src/java/org/apache/hive/service/cli/operation/hplsql/HplSqlOperation.java
+++ 
b/service/src/java/org/apache/hive/service/cli/operation/hplsql/HplSqlOperation.java
@@ -199,7 +199,7 @@ public class HplSqlOperation extends 
ExecuteStatementOperation implements Result
 assert (!parentHive.allowClose());
 Hive.set(parentHive);
 SessionState.setCurrentSessionState(parentSessionState);
-PerfLogger.setPerfLogger(SessionState.getPerfLogger(

(hive) branch master updated: HIVE-28083: Enable HMS client/query cache for EXPLAIN queries (Soumyakanti Das reviewed by John Sherman, Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 053c6c3463b HIVE-28083: Enable HMS client/query cache for EXPLAIN 
queries (Soumyakanti Das reviewed by John Sherman, Stamatis Zampetakis)
053c6c3463b is described below

commit 053c6c3463b8a49f75f28716124b1522fedcb26c
Author: Soumyakanti Das 
AuthorDate: Tue Feb 20 12:22:40 2024 -0800

HIVE-28083: Enable HMS client/query cache for EXPLAIN queries (Soumyakanti 
Das reviewed by John Sherman, Stamatis Zampetakis)

Close apache/hive#5092
---
 .../hadoop/hive/ql/metadata/HiveMetaStoreClientWithLocalCache.java | 5 ++---
 .../org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java   | 7 +++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreClientWithLocalCache.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreClientWithLocalCache.java
index 015895da966..750d674aa4d 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreClientWithLocalCache.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMetaStoreClientWithLocalCache.java
@@ -61,7 +61,6 @@ import org.apache.hadoop.hive.metastore.api.TableStatsResult;
 import org.apache.hadoop.hive.metastore.api.TableValidWriteIds;
 import org.apache.hadoop.hive.metastore.api.UniqueConstraintsRequest;
 import org.apache.hadoop.hive.metastore.api.UniqueConstraintsResponse;
-import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
 import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.util.IncrementalObjectSizeEstimator;
 import 
org.apache.hadoop.hive.ql.util.IncrementalObjectSizeEstimator.ObjectEstimator;
@@ -98,8 +97,8 @@ public class HiveMetaStoreClientWithLocalCache extends 
HiveMetaStoreClient imple
 // init cache only once
 if (!INITIALIZED.get()) {
   LOG.info("Initializing local cache in HiveMetaStoreClient...");
-  maxSize = MetastoreConf.getSizeVar(conf, 
MetastoreConf.ConfVars.MSC_CACHE_MAX_SIZE);
-  recordStats = MetastoreConf.getBoolVar(conf, 
MetastoreConf.ConfVars.MSC_CACHE_RECORD_STATS);
+  maxSize = HiveConf.getSizeVar(conf, 
HiveConf.ConfVars.MSC_CACHE_MAX_SIZE);
+  recordStats = HiveConf.getBoolVar(conf, 
HiveConf.ConfVars.MSC_CACHE_RECORD_STATS);
   initSizeEstimator();
   initCache();
   LOG.info("Local cache initialized in HiveMetaStoreClient: {}", 
mscLocalCache);
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
index ddd42d1590a..ac0b3ead9f5 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
@@ -282,4 +282,11 @@ public class ExplainSemanticAnalyzer extends 
BaseSemanticAnalyzer {
 }
 return super.skipAuthorization();
   }
+
+  @Override
+  public void startAnalysis() {
+if 
(conf.getBoolVar(HiveConf.ConfVars.HIVE_OPTIMIZE_HMS_QUERY_CACHE_ENABLED)) {
+  queryState.createHMSCache();
+}
+  }
 }

(hive) branch master updated: HIVE-28064: Add cause to ParseException for diagnosability purposes (Stamatis Zampetakis reviewed by okumin, Butao Zhang)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 6e061e65595 HIVE-28064: Add cause to ParseException for diagnosability 
purposes (Stamatis Zampetakis reviewed by okumin, Butao Zhang)
6e061e65595 is described below

commit 6e061e6559522c8a060c1b55439ada0001bf5e5d
Author: Stamatis Zampetakis 
AuthorDate: Tue Feb 6 12:59:24 2024 +0100

HIVE-28064: Add cause to ParseException for diagnosability purposes 
(Stamatis Zampetakis reviewed by okumin, Butao Zhang)

The ParseException contains high level information about problems 
encountered during parsing but currently the stacktrace is pretty shallow. The 
end-user gets a hint about what the error might be but the developer has no way 
to tell how far we went into parsing the given statement and which grammar rule 
failed to pass.

Add RecognitionException (when available) as cause in ParseException to 
provide better insights around the origin of the problem and grammar rules that 
were invoked till the crash.

Close apache/hive#5067
---
 .../java/org/apache/hadoop/hive/ql/parse/ParseDriver.java| 12 ++--
 .../java/org/apache/hadoop/hive/ql/parse/ParseException.java | 11 +++
 2 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java 
b/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java
index 7e54bdf95d5..c99895756d0 100644
--- a/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java
+++ b/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseDriver.java
@@ -122,7 +122,7 @@ public class ParseDriver {
 try {
   r = parser.statement();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 
 if (lexer.getErrors().size() == 0 && parser.errors.size() == 0) {
@@ -152,7 +152,7 @@ public class ParseDriver {
 try {
   r = parser.hint();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 
 if (lexer.getErrors().size() == 0 && parser.errors.size() == 0) {
@@ -191,7 +191,7 @@ public class ParseDriver {
 try {
   r = parser.selectClause();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 
 if (lexer.getErrors().size() == 0 && parser.errors.size() == 0) {
@@ -215,7 +215,7 @@ public class ParseDriver {
 try {
   r = parser.expression();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 
 if (lexer.getErrors().size() == 0 && parser.errors.size() == 0) {
@@ -238,7 +238,7 @@ public class ParseDriver {
 try {
   r = parser.triggerExpressionStandalone();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 if (lexer.getErrors().size() != 0) {
   throw new ParseException(lexer.getErrors());
@@ -258,7 +258,7 @@ public class ParseDriver {
 try {
   r = parser.triggerActionExpressionStandalone();
 } catch (RecognitionException e) {
-  throw new ParseException(parser.errors);
+  throw new ParseException(parser.errors, e);
 }
 if (lexer.getErrors().size() != 0) {
   throw new ParseException(lexer.getErrors());
diff --git 
a/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseException.java 
b/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseException.java
index 7d945adf0d3..5b2d17a19e7 100644
--- a/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseException.java
+++ b/parser/src/java/org/apache/hadoop/hive/ql/parse/ParseException.java
@@ -18,8 +18,6 @@
 
 package org.apache.hadoop.hive.ql.parse;
 
-import java.util.ArrayList;
-
 /**
  * ParseException.
  *
@@ -27,13 +25,18 @@ import java.util.ArrayList;
 public class ParseException extends Exception {
 
   private static final long serialVersionUID = 1L;
-  ArrayList errors;
+  private final Iterable errors;
 
-  public ParseException(ArrayList errors) {
+  public ParseException(Iterable errors) {
 super();
 this.errors = errors;
   }
 
+  public ParseException(Iterable errors, Throwable cause) {
+super(cause);
+this.errors = errors;
+  }
+
   @Override
   public String getMessage() {
 

(hive) branch master updated (8b735924b4d -> af545603951)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 8b735924b4d HIVE-28038: Disable fallback to jdo for DeadlineException 
(#5040)(Wechar Yu, reviewed by Butao Zhang)
 add af545603951 HIVE-26818: Beeline module misses transitive dependencies 
due to shading (Stamatis Zampetakis reviewed by Attila Turoczy, Ayush Saxena, 
Kokila N)

No new revisions were added by this update.

Summary of changes:
 beeline/pom.xml  | 1 +
 itests/hive-unit/pom.xml | 6 --
 itests/util/pom.xml  | 4 
 3 files changed, 1 insertion(+), 10 deletions(-)

(hive) branch master updated (f3439697343 -> 0a133b3ec38)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from f3439697343 HIVE-27637: Compare highest write ID of compaction records 
when trying to perform abort cleanup (#4740) (Zsolt Miskolczi reviewed by 
Attila Turoczy, Sourabh Badhya)
 add 0a133b3ec38 HIVE-28053: Incorrect shading configuration for beeline 
jar-with-dependencies (Stamatis Zampetakis reviewed by Alessandro Solimando, 
Attila Turoczy)

No new revisions were added by this update.

Summary of changes:
 beeline/pom.xml | 5 +
 1 file changed, 1 insertion(+), 4 deletions(-)

(hive-dev-box) 01/03: Add NOTICE & LICENSE files

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-dev-box.git

commit e98a8aee0cbca055b7aace8e50e2b7443963129c
Author: Ayush Saxena 
AuthorDate: Tue Jan 23 11:45:27 2024 +0530

Add NOTICE & LICENSE files
---
 LICENSE | 353 
 NOTICE  |   5 +
 2 files changed, 358 insertions(+)

diff --git a/LICENSE b/LICENSE
new file mode 100644
index 000..6ccb61a
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,353 @@
+
+ Apache License
+   Version 2.0, January 2017
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, ha

(hive-dev-box) branch main updated (bb4ce68 -> 094c7b5)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/hive-dev-box.git


from bb4ce68  Downgrade jdk8 to 1.8.0_262-b19 to avoid timestamp issue in 
HIVE-28014 (#16)
 new e98a8ae  Add NOTICE & LICENSE files
 new 3b5f940  Add DISCLAIMER-WIP
 new 094c7b5  Add Apache License Header

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 DISCLAIMER-WIP |   8 +
 LICENSE| 353 +
 NOTICE |   5 +
 bin/apdiff |  16 ++
 bin/bl |  15 ++
 bin/conf   |  15 ++
 bin/deploy_custom_jars |  16 ++
 bin/deploy_jars|  16 ++
 bin/dev_eclipse|  15 ++
 bin/gerrit_pr  |  15 ++
 bin/github_pr  |  15 ++
 bin/hive_launch|  15 ++
 bin/hive_load_test_data|  15 ++
 bin/hive_patch_development |  15 ++
 bin/hs2_debug  |  15 ++
 bin/init_hive_src  |  15 ++
 bin/init_tsdb  |  16 ++
 bin/mrproper   |  15 ++
 bin/mvn|  15 ++
 bin/psql   |  16 ++
 bin/rebase |  16 ++
 bin/reinit_metastore   |  15 ++
 bin/safe_bl|  15 ++
 bin/send_custom_jars   |  15 ++
 bin/spawn_shell_after  |  15 ++
 bin/srcs   |  15 ++
 bin/sw |  15 ++
 bin/tez_debug  |  15 ++
 bin/wait_port  |  15 ++
 buildAll   |  16 ++
 conf/artifactory.config.latest.xml |  13 ++
 enter.bash |  16 ++
 hdb|  15 ++
 hooks/build|  15 ++
 run.bash   |  16 ++
 settings.xml   |  13 ++
 start_artifactory.bash |  15 ++
 tools/build_cleanup|  15 ++
 tools/cdpcli   |  16 ++
 tools/docker_entrypoint|  15 ++
 tools/docker_entrypoint.bazaar |  15 ++
 tools/docker_entrypoint.executor   |  15 ++
 tools/entrypoint.impala|  15 ++
 tools/i_sort   |  15 ++
 tools/iii  |  16 ++
 tools/impala_bootstrap |  16 ++
 tools/install_basics   |  15 ++
 tools/install_conf |  15 ++
 tools/install_executor |  16 ++
 tools/install_executor2|  16 ++
 tools/install_executor3|  16 ++
 tools/install_java_zulu|  15 ++
 tools/install_mysql.bash   |  15 ++
 tools/install_psql.bash|  15 ++
 tools/install_sdk.bash |  15 ++
 tools/install_texturepacker|  15 ++
 tools/install_toolbox  |  16 ++
 tools/install_x2go |  15 ++
 tools/install_xmlstarlet   |  15 ++
 tools/python3_default  |  16 ++
 tools/y|  15 ++
 61 files changed, 1249 insertions(+)
 create mode 100644 DISCLAIMER-WIP
 create mode 100644 LICENSE
 create mode 100644 NOTICE

(hive-dev-box) 03/03: Add Apache License Header

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-dev-box.git

commit 094c7b5cf7204893c58e9623f112b7cfda0a7169
Author: Ayush Saxena 
AuthorDate: Tue Jan 23 11:57:18 2024 +0530

Add Apache License Header
---
 bin/apdiff | 16 
 bin/bl | 15 +++
 bin/conf   | 15 +++
 bin/deploy_custom_jars | 16 
 bin/deploy_jars| 16 
 bin/dev_eclipse| 15 +++
 bin/gerrit_pr  | 15 +++
 bin/github_pr  | 15 +++
 bin/hive_launch| 15 +++
 bin/hive_load_test_data| 15 +++
 bin/hive_patch_development | 15 +++
 bin/hs2_debug  | 15 +++
 bin/init_hive_src  | 15 +++
 bin/init_tsdb  | 16 
 bin/mrproper   | 15 +++
 bin/mvn| 15 +++
 bin/psql   | 16 
 bin/rebase | 16 
 bin/reinit_metastore   | 15 +++
 bin/safe_bl| 15 +++
 bin/send_custom_jars   | 15 +++
 bin/spawn_shell_after  | 15 +++
 bin/srcs   | 15 +++
 bin/sw | 15 +++
 bin/tez_debug  | 15 +++
 bin/wait_port  | 15 +++
 buildAll   | 16 
 conf/artifactory.config.latest.xml | 13 +
 enter.bash | 16 
 hdb| 15 +++
 hooks/build| 15 +++
 run.bash   | 16 
 settings.xml   | 13 +
 start_artifactory.bash | 15 +++
 tools/build_cleanup| 15 +++
 tools/cdpcli   | 16 
 tools/docker_entrypoint| 15 +++
 tools/docker_entrypoint.bazaar | 15 +++
 tools/docker_entrypoint.executor   | 15 +++
 tools/entrypoint.impala| 15 +++
 tools/i_sort   | 15 +++
 tools/iii  | 16 
 tools/impala_bootstrap | 16 
 tools/install_basics   | 15 +++
 tools/install_conf | 15 +++
 tools/install_executor | 16 
 tools/install_executor2| 16 
 tools/install_executor3| 16 
 tools/install_java_zulu| 15 +++
 tools/install_mysql.bash   | 15 +++
 tools/install_psql.bash| 15 +++
 tools/install_sdk.bash | 15 +++
 tools/install_texturepacker| 15 +++
 tools/install_toolbox  | 16 
 tools/install_x2go | 15 +++
 tools/install_xmlstarlet   | 15 +++
 tools/python3_default  | 16 
 tools/y| 15 +++
 58 files changed, 883 insertions(+)

diff --git a/bin/apdiff b/bin/apdiff
index f5942a9..a52b133 100755
--- a/bin/apdiff
+++ b/bin/apdiff
@@ -1,4 +1,20 @@
 #!/bin/bash -e
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 upstream="`git config --local --get extra.upstream || echo apache/master`"
 git diff `git merge-base HEAD $upstream` "$@"
 
diff --git a/bin/bl b/bin/bl
index 03da60b..ff5523f 100755
--- a/bin/bl
+++ b/bin/bl
@@ -1,3 +1,18 @@
 #!/bin/bash
 
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agre

(hive-dev-box) 02/03: Add DISCLAIMER-WIP

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-dev-box.git

commit 3b5f9401d3975e713790924f4b80153674ec8551
Author: Ayush Saxena 
AuthorDate: Tue Jan 23 11:46:22 2024 +0530

Add DISCLAIMER-WIP
---
 DISCLAIMER-WIP | 8 
 1 file changed, 8 insertions(+)

diff --git a/DISCLAIMER-WIP b/DISCLAIMER-WIP
new file mode 100644
index 000..981792d
--- /dev/null
+++ b/DISCLAIMER-WIP
@@ -0,0 +1,8 @@
+Some of this project’s releases may not be fully compliant with ASF
+policy. For example, releases may have incomplete or un-reviewed licensing
+conditions.
+
+If you are planning to incorporate this work into your product/project, please
+be aware that you will need to conduct a thorough licensing review to determine
+the overall implications of including this work. For the current status of this
+project reach out Apache Hive dev mailing lists.
\ No newline at end of file

(hive-dev-box) branch main created (now bb4ce68)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/hive-dev-box.git


  at bb4ce68  Downgrade jdk8 to 1.8.0_262-b19 to avoid timestamp issue in 
HIVE-28014 (#16)

This branch includes the following new commits:

 new 6891ce4  initial
 new 0bffbfe  updates
 new 9b9ec00  updates
 new 65471f4  make nodemanger.service work
 new 8161023  rm
 new e1a7e74  fixes to launching
 new 340ec53  reduce tezam mem
 new fa0c8c4  cleanup; raise memory
 new 93007f5  rename scripts
 new 9337045  set default shell
 new b38ad35  add bl autostart
 new 91d1959  rename; disable disk
 new 4468449  wait for nodemanager to go down on re-launch
 new 3d0a9db  alternate conf desc
 new 7c2f902  getting started with psql
 new e9aca9b  use newer hive/hadoop
 new 84004c1  fix log_dir
 new 687feb7  cleanup; use conf for hive-conf; add psql
 new 5f68684  move yarn-site to conf
 new da44520  move old conf.xml-s
 new 8f2df40  fix switch back to derby
 new b23ccf1  add psql install
 new 02e7680  extend install
 new 3f2d043  add hive-dev support
 new b08bcab  fixes
 new 69b3a4b  renames/cleanup
 new e9e5516  fix typo
 new 6e01567  fixes; add readme
 new 0c33066  remove old scripts
 new bb9a051  make hive-sources optional
 new dba58a8  fix psql default access
 new ef4f424  mysql
 new ff19437  add oracle
 new 86801e6  use buster
 new 686083a  small fixes
 new bb267d0  dont preinstall mysql in basics
 new 72d174f  add initial x2go installer
 new 132ca88  add x2go support; add authkey
 new 33f3f26  use zulu instead of the same thru sdkman
 new 9efe5e5  cleanup/add eclipse
 new d0fa579  add trial of making an eclipse ws/etc
 new a0469ba  add artifactory; build/etc
 new 9e14257  fix typo; install toolbox if available
 new 11123e1  Update Vagrantfile
 new 7fae92e  Merge pull request #1 from jfsii/patch-1
 new e5b7699  interpret -d instead of passing all args
 new 2735457  patch development; init eclipse
 new 5b38147  updates to readme; add host_dir
 new 2f0f9a2  add macosx specific stuff
 new 9fb6308  updates
 new 07c4c86  add artifactory url
 new ba6ac30  fix macos display
 new a23a149  DISPLAY=host.docker.internal:0
 new cc2a1c1  cleanup readme
 new 82a6bb3  fix ws creation; add test-data
 new 1fdebdd  fix new branch creation
 new 5f0eee2  fix mistakes
 new ba2d8c7  handle HIVE_SOURCES env
 new 1041709  map toolbox config if exists
 new 4f2b96c  cleanup
 new 74a691f  fix locale; add docker
 new 8b980d5  use shared network
 new 7f9001d  add missing reset
 new 7835dae  minor updates: no snapshot update, tez container size, hs2 xmx
 new 0224f7f  add protobuf/visualvm install
 new f0143af  fixes
 new 745dc1c  add hdb
 new 47bd1aa  add more specific eclipse installer
 new 6c5b0e8  minor updates
 new c6738eb  leave bash as pid#1 it will reap zombies :D
 new a03e0db  install wait-for-port; move package installation to basics; 
use a mysql container
 new 0c6cf28  dont change installed components after first start
 new 244f603  progress on psql
 new f95ffb9  add kdiff3
 new c788cf5  use toolbox 0.2.5
 new 8d1927f  remove vagrantfile - as it is not maintained much anymore
 new 1d19618  add srcs
 new 71a3122  lowercase dbname; enable postgres
 new acd54c5  re-enable oracle support
 new 7b18eac  add note about tez libs; change hive-dev pattern
 new 2a9f40e  add thrift installer
 new 0001353  add aliases
 new 5c91cbf  fix no-arg metastore reinit
 new 416b8fa  fix typo
 new d4526e2  update readme
 new a022fb0  add more conf values
 new 378f259  toolbox to 0.2.6
 new 073e841  0.2.7
 new 6bc3a23  add hdb bash_competion; update readme a bit
 new 0d111a4  extend srcs
 new 330a318  kill build time cloning; fix non-cloned remote issue in srcs
 new 5ffaf53  add update
 new eccbf1e  add dev_eclipse
 new 7ad74b5  remove github user arg
 new 11bee2c  updates
 new 82add4a  avoid some hdb bash completion error during ssh
 new 04da8b4  checkout branch; dont hide downloads
 new d486bc0  toolbox 0.2.8
 new 94fc584  fix sw tez-dev
 new a919131  set tez to use hadoop-libs
 new dc44e03  Echo git config into default dsl instead of executing it
 new 6eee389  patcher
 new a1639ab  move thrift related packages install; add derby install to sw
 new 08b6f59  fix hdb command without arguments
 new 7ff7e1a  toolbox-0.2.9
 new ee487a8  use extra.upstream to refer to upstream branch; add rebase cmd
 new 6b8f9b5  add graphviz
 new 3716d25  some small updates
 new 5b9242f  bump maven to 3.6.3; add archive mirror
 new e5076e6  Merge branch 'master

(calcite) branch main updated (e2c84a6485 -> d260a7dd6d)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


from e2c84a6485 [CALCITE-6111] Explicit cast from expression to numeric 
type doesn't check overflow
 new 58edb0e85f [CALCITE-6098] Update LICENSE and NOTICE for Jekyll website 
template
 new d260a7dd6d Site: Remove committer by request process

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 LICENSE   |  9 +
 NOTICE|  2 --
 site/develop/index.md | 46 --
 3 files changed, 9 insertions(+), 48 deletions(-)

(calcite) 01/02: [CALCITE-6098] Update LICENSE and NOTICE for Jekyll website template

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit 58edb0e85f01580961f9fb07f171686166a0da34
Author: Stamatis Zampetakis 
AuthorDate: Wed Nov 8 14:32:28 2023 +0100

[CALCITE-6098] Update LICENSE and NOTICE for Jekyll website template

The NOTICE file contains the following statement:
```
The web site includes files generated by Jekyll.
```

However, there is nothing in the LICENSE of Jekyll, that requires such
attribution.

According to the instructions of composing the NOTICE file for ASF
projects we shouldn't add anything in there that is not legally
required.

The files comprisings the template of the web site i.e.,
site/_includes,
site/_layouts,
site/_sass,
site/css

were copied and slightly adapted from the ORC repository, which in turn
copied the template from the Jekyll repository. There have been various
small additions in the orignal template (some .html files under the
respective directories are completely new) which could potentially
licensed under AL2 but it would complicate the LICENSE documentation
without significant gain. It's better if the consumers of the source
distribution know that this is a Jekyll template licensed under MIT.

Close apache/calcite#3510
---
 LICENSE | 9 +
 NOTICE  | 2 --
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/LICENSE b/LICENSE
index efcd7e51a2..a87594d500 100644
--- a/LICENSE
+++ b/LICENSE
@@ -182,6 +182,15 @@ Additional License files can be found in the 'licenses' 
folder located in the sa
 
 MIT
 * font-awesome:font-awesome-code:4.2.0
+site/_sass/_font-awesome.scss
 * cobyism:gridism:
+site/_sass/_gridism.scss
 * jekyll:jekyll:
+site/css/screen.scss
+site/_includes/*.html
+site/_layouts/*.html
+site/news/releases/index.html
+site/_sass/_mixins.scss
+site/_sass/_pygments.scss
 * normalize:normalize:3.0.2
+site/_sass/_normalize.scss
diff --git a/NOTICE b/NOTICE
index f4562f0e62..d5061e4093 100644
--- a/NOTICE
+++ b/NOTICE
@@ -8,5 +8,3 @@ This product is based on source code originally developed
 by DynamoBI Corporation, LucidEra Inc., SQLstream Inc. and others
 under the auspices of the Eigenbase Foundation
 and released as the LucidDB project.
-
-The web site includes files generated by Jekyll.

(calcite) 02/02: Site: Remove committer by request process

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit d260a7dd6db541191e84c38135d2d436510dfa08
Author: Stamatis Zampetakis 
AuthorDate: Fri Jan 12 11:18:41 2024 +0100

Site: Remove committer by request process

The committer by request process was introduced after a discussion [1] in 
the dev list.

However, internal discussions on private@calcite showed that not everyone 
in the PMC is fully inline with this special process.

Moreover, the process is against the notion of becoming committer based on 
"merit" [2].

[1] https://lists.apache.org/thread/nm4zgddkyjb0qzonv8pgcnn6h76x2orb
[2] https://www.apache.org/foundation/governance/pmcs.html#merit

Close apache/calcite#3638
---
 site/develop/index.md | 46 --
 1 file changed, 46 deletions(-)

diff --git a/site/develop/index.md b/site/develop/index.md
index 180048afa9..3bacd91e32 100644
--- a/site/develop/index.md
+++ b/site/develop/index.md
@@ -359,49 +359,3 @@ You can contribute by testing the code, helping verify a 
release, writing docume
 the web site, or just by answering questions on the list.
 
 After you have made several useful contributions we may invite you to become a 
[committer](https://community.apache.org/contributors/).
-The most common way of becoming a committer is by contributing regularly to 
the project. In some
-exceptional cases you can bypass the invitation process as described below.
-
-If you are a committer in another **ASF project** (such as Drill, Flink, Hive, 
Ignite, Phoenix, etc.)
-and you are familiar with the Calcite codebase you can request explicitly from 
the Calcite PMC to
-consider your for committership. You can do that by sending an email to 
[priv...@calcite.apache.org](mailto:priv...@calcite.apache.org)
-including the following information (template below):
-* Apache ID, you must have one if you are committer;
-* links to commits, discussions, presentations, blogs, etc., demonstrating 
your experience with
-Calcite.
-
-{% highlight text %}
-Subject: [REQUEST] New committer: Stamatis Zampetakis
-To: priv...@calcite.apache.org
-
-Hi all,
-
-My name is Stamatis Zampetakis and my Apache ID is zabetak:
-https://home.apache.org/phonebook.html?uid=zabetak
-
-I am a Hive committer and have been working with Calcite for the past 2 years.
-I contributed various improvements and fixes in the cost based optimizer of
-Hive, which relies on Calcite, and I am pretty familiar with the Calcite
-codebase. Below you can find a list of contributions in the Hive repo that are
-related with Calcite:
-
-* 
https://github.com/apache/hive/commit/f29cb2245c97102975ea0dd73783049eaa0947a0
-* 
https://github.com/apache/hive/commit/efae863fe010ed5c4b7de1874a336ed93b3c60b8
-* 
https://github.com/apache/hive/commit/587c698fa25ca6da46d9c02e4199689426fec40f
-* 
https://github.com/apache/hive/commit/9087fa93cd785223f4f2552ec836e7580c78830a
-* 
https://github.com/apache/hive/commit/0616bcaa2436ccbf388b635bfea160b47849553c
-* 
https://github.com/apache/hive/commit/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d
-* 
https://github.com/apache/hive/commit/a0faf5ecb196a20cfef64d554df54961e8c074a7
-* 
https://github.com/apache/hive/commit/4f4cbeda00d5ebb7d0b8cedee5daa2c03df4a755
-* 
https://github.com/apache/hive/commit/6f2b8883c44edcf57538f3b1da2c5a599b0c5862
-* 
https://github.com/apache/hive/commit/170d5b4c3edf2daaa47ef3299277d44def4d39a7
-
-I would like to become a Calcite committer and help the project as much as I 
can.
-
-Best,
-Stamatis
-
-{% endhighlight %}
-
-After receiving your email the PMC will evaluate your request and get back to 
you in 1-2 weeks
-(usually a vote for adding a new committer takes ~7days).

(hive) branch master-tez092 deleted (was 750daa4a639)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master-tez092
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 750daa4a639 HIVE-20605 : Merge branch 'master' into master-tez092

This change permanently discards the following revisions:

 discard 750daa4a639 HIVE-20605 : Merge branch 'master' into master-tez092
 discard c55347d5283 HIVE-20605 : Merge branch 'master' into master-tez092
 discard 8151911b4d2 HIVE-18952 : Tez session disconnect and reconnect on HS2 
HA failover (Sergey Shelukhin, reviewed by Eric Wohlstadter)
 discard e5448bf08e1 HIVE-20547 : HS2: support Tez sessions started by someone 
else (part 1) (Sergey Shelukhin, reviewed by Prasanth Jayachandran)

(hive) branch dependabot/maven/standalone-metastore/org.apache.derby-derby-10.17.1.0 deleted (was 4e1155ae294)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch 
dependabot/maven/standalone-metastore/org.apache.derby-derby-10.17.1.0
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 4e1155ae294 Bump org.apache.derby:derby in /standalone-metastore

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch dependabot/maven/org.apache.derby-derby-10.17.1.0 deleted (was 4b66e1a50e0)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch 
dependabot/maven/org.apache.derby-derby-10.17.1.0
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 4b66e1a50e0 Bump org.apache.derby:derby from 10.14.2.0 to 10.17.1.0

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch vectorization deleted (was 09aecc093af)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch vectorization
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 09aecc093af Merged in with trunk

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch dependabot/maven/org.apache.commons-commons-compress-1.24.0 deleted (was 758bccb684f)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch 
dependabot/maven/org.apache.commons-commons-compress-1.24.0
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 758bccb684f Bump org.apache.commons:commons-compress from 1.23.0 to 
1.24.0

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch revert-4501-OptimizeGetPartitionAPI deleted (was 1e845012fa7)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-4501-OptimizeGetPartitionAPI
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 1e845012fa7 Revert "HIVE-27505: Added batching with retry mechanism in 
Hive.getAllPartitionsOf call. (#4501)"

This change permanently discards the following revisions:

 discard 1e845012fa7 Revert "HIVE-27505: Added batching with retry mechanism in 
Hive.getAllPartitionsOf call. (#4501)"

(hive) branch revert-4247-hive-23256 deleted (was 6abf512b70c)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-4247-hive-23256
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 6abf512b70c Revert "HIVE-23256: Fix flaky test 
TestMiniLlapLocalCliDriver[schema_evol_par_vec_table_dictionary_encoding.q] 
(Mayank Kunwar, reviewed Ayush Saxena, Zsolt Miskolczi)"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch revert-3624-HIVE-26567 deleted (was 09282ddceba)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-3624-HIVE-26567
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 09282ddceba Revert "Hive Security - Upgrade Apache Log4j to 2.18.0 due 
to critical CVEs (#3624)"

This change permanently discards the following revisions:

 discard 09282ddceba Revert "Hive Security - Upgrade Apache Log4j to 2.18.0 due 
to critical CVEs (#3624)"

(hive) branch ptf-windowing deleted (was 700d34f1230)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch ptf-windowing
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 700d34f1230 Merged in with latest trunk.

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch revert-4621-HIVE-27113 deleted (was bc8967f339f)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-4621-HIVE-27113
in repository https://gitbox.apache.org/repos/asf/hive.git


 was bc8967f339f Revert "HIVE-27113: Increasing default for 
hive.thrift.client.max.message.size to 2GB (#4621)(Riju Trivedi, Reviewed by 
Attila Turoczy, Sai Hemanth Gantasala)"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch revert-4452-HIVE-57988-BetweenBugFix deleted (was 60da8311561)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-4452-HIVE-57988-BetweenBugFix
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 60da8311561 Revert "Hive-57988: Fix for the BETWEEN parameter issue 
and adjusted selectivity estimation in BETWEEN case for NULLs (Ryan Johnson, 
reviewed by Krisztian Kasa)"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch revert-4306-HIVE-27330 deleted (was 1dede71eac0)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch revert-4306-HIVE-27330
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 1dede71eac0 Revert "HIVE-27330: Compaction entry dequeue order (Laszlo 
Vegh, reviewed by Attila Turoczy)"

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch cbo deleted (was 8cac2aa0450)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch cbo
in repository https://gitbox.apache.org/repos/asf/hive.git


 was 8cac2aa0450 HIVE-10400 : CBO (Calcite Return Path): Exception when 
column name contains dot or colon characters [CBO branch] (Jesus Camacho 
Rodriguez)

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch HIVE-4115 deleted (was c84c2184fe5)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch HIVE-4115
in repository https://gitbox.apache.org/repos/asf/hive.git


 was c84c2184fe5 Merging r1489797 through r1490612 into HIVE-4115

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

(hive) branch master updated: HIVE-27999: Run Sonar analysis using Java 17 (Wechar Yu reviewed by Stamatis Zampetakis, Attila Turoczy)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 1760304401f HIVE-27999: Run Sonar analysis using Java 17 (Wechar Yu 
reviewed by Stamatis Zampetakis, Attila Turoczy)
1760304401f is described below

commit 1760304401ff65d2f1fd7276d8a2bc241babb18a
Author: wecharyu 
AuthorDate: Mon Jan 15 06:24:33 2024 +

HIVE-27999: Run Sonar analysis using Java 17 (Wechar Yu reviewed by 
Stamatis Zampetakis, Attila Turoczy)

1. Use executor image built from 
https://github.com/kgyrtkirk/hive-dev-box/commit/c67d8af69b869bab1ccc7013a8bccec2b2410518
 with JDK 17 support
2. Remove unecessary ping

Close apache/hive#5004
---
 Jenkinsfile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/Jenkinsfile b/Jenkinsfile
index 0f99c42cc4a..fc11bb3299e 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -112,7 +112,7 @@ def sonarAnalysis(args) {
   """+args+" -DskipTests -Dit.skipTests -Dmaven.javadoc.skip"
 
   sh """#!/bin/bash -e
-  sw java 11 && . /etc/profile.d/java.sh
+  sw java 17 && . /etc/profile.d/java.sh
   export MAVEN_OPTS=-Xmx5G
   """+mvnCmd
   }
@@ -121,7 +121,7 @@ def sonarAnalysis(args) {
 def hdbPodTemplate(closure) {
   podTemplate(
   containers: [
-containerTemplate(name: 'hdb', image: 'kgyrtkirk/hive-dev-box:executor', 
ttyEnabled: true, command: 'tini -- cat',
+containerTemplate(name: 'hdb', image: 'wecharyu/hive-dev-box:executor', 
ttyEnabled: true, command: 'tini -- cat',
 alwaysPullImage: true,
 resourceRequestCpu: '1800m',
 resourceLimitCpu: '8000m',
@@ -287,7 +287,6 @@ set -x
 echo 127.0.0.1 dev_$dbType | sudo tee -a /etc/hosts
 . /etc/profile.d/confs.sh
 sw hive-dev $PWD
-ping -c2 dev_$dbType
 export DOCKER_NETWORK=host
 export DBNAME=metastore
 reinit_metastore $dbType

(calcite) branch main updated (b2be7e1dac -> e71d59fe92)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


from b2be7e1dac [CALCITE-6044] RelMetadataQuery should regard single-row 
relational expressions as unique
 new 402740d970 [CALCITE-6096] Remove obsolete html5shiv and respond 
entries from LICENSE
 new e71d59fe92 [CALCITE-6097] Gridism CSS dependency is mispelled in 
LICENSE

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 LICENSE|  4 +---
 release/build.gradle.kts   |  4 +---
 .../config/licenses/{gridsim => gridism}/LICENSE   |  0
 src/main/config/licenses/html5shiv/LICENSE | 22 --
 src/main/config/licenses/respond/LICENSE   | 20 
 5 files changed, 2 insertions(+), 48 deletions(-)
 rename src/main/config/licenses/{gridsim => gridism}/LICENSE (100%)
 delete mode 100644 src/main/config/licenses/html5shiv/LICENSE
 delete mode 100644 src/main/config/licenses/respond/LICENSE

(calcite) 01/02: [CALCITE-6096] Remove obsolete html5shiv and respond entries from LICENSE

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit 402740d970e162dd0a83172d509199c02a2a23a3
Author: Stamatis Zampetakis 
AuthorDate: Wed Nov 8 12:30:14 2023 +0100

[CALCITE-6096] Remove obsolete html5shiv and respond entries from LICENSE

The LICENSE file includes the following entries:

* cobyism:html5shiv:3.7.2
* respond:respond:1.4.2

implying that we are bundling the respective dependencies in the
project.

These dependencies correspond to minified javascript files that were
removed completely from the project some time ago:

https://github.com/apache/calcite/commit/7f5e9b8b7e6b4afd8e4f21524aa3c4ce8b7ddb61

Close apache/calcite#3508
---
 LICENSE|  2 --
 release/build.gradle.kts   |  2 --
 src/main/config/licenses/html5shiv/LICENSE | 22 --
 src/main/config/licenses/respond/LICENSE   | 20 
 4 files changed, 46 deletions(-)

diff --git a/LICENSE b/LICENSE
index 2f69b1fe06..af260a722f 100644
--- a/LICENSE
+++ b/LICENSE
@@ -181,9 +181,7 @@ Additional License files can be found in the 'licenses' 
folder located in the sa
 - Software produced outside the ASF which is available under other licenses 
(not Apache-2.0)
 
 MIT
-* cobyism:html5shiv:3.7.2
 * font-awesome:font-awesome-code:4.2.0
 * gridsim:gridsim:
 * jekyll:jekyll:
 * normalize:normalize:3.0.2
-* respond:respond:1.4.2
diff --git a/release/build.gradle.kts b/release/build.gradle.kts
index add0d9b677..4d26eed000 100644
--- a/release/build.gradle.kts
+++ b/release/build.gradle.kts
@@ -108,8 +108,6 @@ val getLicenses by 
tasks.registering(GatherLicenseTask::class) {
 // Gridism: A simple, responsive, and handy CSS grid by @cobyism
 // https://github.com/cobyism/gridism
 addDependency(":gridsim:", SpdxLicense.MIT)
-addDependency("cobyism:html5shiv:3.7.2", SpdxLicense.MIT)
-addDependency(":respond:1.4.2", SpdxLicense.MIT)
 }
 
 val license by tasks.registering(Apache2LicenseRenderer::class) {
diff --git a/src/main/config/licenses/html5shiv/LICENSE 
b/src/main/config/licenses/html5shiv/LICENSE
deleted file mode 100644
index 2817a980db..00
--- a/src/main/config/licenses/html5shiv/LICENSE
+++ /dev/null
@@ -1,22 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2014 Alexander Farkas (aFarkas)
-
-Permission is hereby granted, free  of charge, to any person obtaining
-a  copy  of this  software  and  associated  documentation files  (the
-"Software"), to  deal in  the Software without  restriction, including
-without limitation  the rights to  use, copy, modify,  merge, publish,
-distribute,  sublicense, and/or sell  copies of  the Software,  and to
-permit persons to whom the Software  is furnished to do so, subject to
-the following conditions:
-
-The  above  copyright  notice  and  this permission  notice  shall  be
-included in all copies or substantial portions of the Software.
-
-THE  SOFTWARE IS  PROVIDED  "AS  IS", WITHOUT  WARRANTY  OF ANY  KIND,
-EXPRESS OR  IMPLIED, INCLUDING  BUT NOT LIMITED  TO THE  WARRANTIES OF
-MERCHANTABILITY,FITNESSFORA   PARTICULARPURPOSEAND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE,  ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/src/main/config/licenses/respond/LICENSE 
b/src/main/config/licenses/respond/LICENSE
deleted file mode 100644
index c7264e7a84..00
--- a/src/main/config/licenses/respond/LICENSE
+++ /dev/null
@@ -1,20 +0,0 @@
-Copyright (c) 2013 Scott Jehl
-
-Permission is hereby granted, free  of charge, to any person obtaining
-a  copy  of this  software  and  associated  documentation files  (the
-"Software"), to  deal in  the Software without  restriction, including
-without limitation  the rights to  use, copy, modify,  merge, publish,
-distribute,  sublicense, and/or sell  copies of  the Software,  and to
-permit persons to whom the Software  is furnished to do so, subject to
-the following conditions:
-
-The  above  copyright  notice  and  this permission  notice  shall  be
-included in all copies or substantial portions of the Software.
-
-THE  SOFTWARE IS  PROVIDED  "AS  IS", WITHOUT  WARRANTY  OF ANY  KIND,
-EXPRESS OR  IMPLIED, INCLUDING  BUT NOT LIMITED  TO THE  WARRANTIES OF
-MERCHANTABILITY,FITNESSFORA   PARTICULARPURPOSEAND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
-LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
-OF CONTRACT, TORT OR OTHERWISE,  ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

(calcite) 02/02: [CALCITE-6097] Gridism CSS dependency is mispelled in LICENSE

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit e71d59fe92c5b39293830ed3416a4db8ac8a369a
Author: Stamatis Zampetakis 
AuthorDate: Wed Nov 8 13:22:24 2023 +0100

[CALCITE-6097] Gridism CSS dependency is mispelled in LICENSE

The website uses  the gridism.css style for presentation purposes. The
file is present in git and also in the source distribution of Calcite
so it should have a proper reference in the LICENSE file.

There is an entry in the LICENSE file but it is mispelled: gridsim vs
gridism

Close apache/calcite#3509
---
 LICENSE   | 2 +-
 release/build.gradle.kts  | 2 +-
 src/main/config/licenses/{gridsim => gridism}/LICENSE | 0
 3 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/LICENSE b/LICENSE
index af260a722f..efcd7e51a2 100644
--- a/LICENSE
+++ b/LICENSE
@@ -182,6 +182,6 @@ Additional License files can be found in the 'licenses' 
folder located in the sa
 
 MIT
 * font-awesome:font-awesome-code:4.2.0
-* gridsim:gridsim:
+* cobyism:gridism:
 * jekyll:jekyll:
 * normalize:normalize:3.0.2
diff --git a/release/build.gradle.kts b/release/build.gradle.kts
index 4d26eed000..26a4b109fb 100644
--- a/release/build.gradle.kts
+++ b/release/build.gradle.kts
@@ -107,7 +107,7 @@ val getLicenses by 
tasks.registering(GatherLicenseTask::class) {
 addDependency(":normalize:3.0.2", SpdxLicense.MIT)
 // Gridism: A simple, responsive, and handy CSS grid by @cobyism
 // https://github.com/cobyism/gridism
-addDependency(":gridsim:", SpdxLicense.MIT)
+addDependency("cobyism:gridism:", SpdxLicense.MIT)
 }
 
 val license by tasks.registering(Apache2LicenseRenderer::class) {
diff --git a/src/main/config/licenses/gridsim/LICENSE 
b/src/main/config/licenses/gridism/LICENSE
similarity index 100%
rename from src/main/config/licenses/gridsim/LICENSE
rename to src/main/config/licenses/gridism/LICENSE

(calcite) branch main updated: Site: Switch PMC Chair to Benchao Li

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new a816267808 Site: Switch PMC Chair to Benchao Li
a816267808 is described below

commit a816267808f2aea93ef5831783d9d7f0f3dd7d3f
Author: Stamatis Zampetakis 
AuthorDate: Thu Dec 21 10:21:00 2023 +0100

Site: Switch PMC Chair to Benchao Li
---
 site/_data/contributors.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/site/_data/contributors.yml b/site/_data/contributors.yml
index 9f650f48e9..528602b4c2 100644
--- a/site/_data/contributors.yml
+++ b/site/_data/contributors.yml
@@ -50,7 +50,7 @@
   githubId: libenchao
   pronouns: he/him
   org: ByteDance
-  role: PMC
+  role: PMC Chair
 - name: Bertil Chapuis
   apacheId: bchapuis
   githubId: bchapuis
@@ -313,7 +313,7 @@
   githubId: zabetak
   pronouns: he/him
   org: Cloudera
-  role: PMC Chair
+  role: PMC
   homepage: https://people.apache.org/~zabetak/
 - name: Steven Noels
   apacheId: stevenn

(hive) 02/02: HIVE-27919: Constant reduction in CBO does not work for FROM_UNIXTIME, DATE_ADD, DATE_SUB, TO_UNIX_TIMESTAMP (Stamatis Zampetakis reviewed by Akshat Mathur, Krisztian Kasa)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 36ce858163a19e29eafe4a8d3307191bc28fc175
Author: Stamatis Zampetakis 
AuthorDate: Fri Dec 8 13:22:56 2023 +0100

HIVE-27919: Constant reduction in CBO does not work for FROM_UNIXTIME, 
DATE_ADD, DATE_SUB, TO_UNIX_TIMESTAMP (Stamatis Zampetakis reviewed by Akshat 
Mathur, Krisztian Kasa)

Constant reduction does not work because the functions are declared as
dynamic (isDynamicFunction returns true). However, the dynamic
declaration is wrong cause none of the above depends on context
variables; they all operate on concrete parameters and require one or
more inputs.

Moreover, DATE_ADD, DATE_SUB, and FROM_UNIXTIME are not time functions,
so it is wrong to extend the SqlAbsctractTimeFunction class. The
overrides in SqlAbsctractTimeFunction are not correct/relevant to these
functions so the changes here address this as well.

Overview of the changes:
1. Turn DATE_ADD, DATE_SUB, and FROM_UNIXTIME to regular SqlFunctions
and pass the correct return type inference strategy. The operand type
inference and type checker can remain null as they were before since
they are not used currently in Hive.
2. Change the type family for FROM_UNIXTIME to reflect that the
function returns a string (and not date or time).
3. Create and pass an appropriate operand checker for FROM_UNIXTIME
(minor since it's not used at the moment).
4. Remove isDynamicFunction override from TO_UNIX_TIMESTAMP
(which is wrong), to enable constant reduction.
5. Finalize classes and make them non-instantiable

Constant reduction in CBO allows some further optimizations to kick-in
such as the removal of the (always true) filter operator in
constant_prop_coalesce.q.out and the transformation to
dynamic partition hash join (DPHJ) in tez_dynpart_hashjoin_4.q.out.

Note, that without the changes here the DPHJ transformation for the
query in tez_dynpart_hashjoin_4.q fails due to inconsistencies on the
way constant folding is performed at the Operator (physical) layer
(HIVE-27658).

Close apache/hive#4932
---
 .../reloperators/HiveDateAddSqlOperator.java   |  15 +--
 .../reloperators/HiveDateSubSqlOperator.java   |  15 +--
 .../reloperators/HiveFromUnixTimeSqlOperator.java  |  29 --
 .../HiveToUnixTimestampSqlOperator.java|  12 +--
 .../queries/clientpositive/cbo_constantfolding.q   |   5 +
 .../clientpositive/llap/cbo_constantfolding.q.out  |  60 
 .../llap/constant_prop_coalesce.q.out  |  10 +-
 .../llap/tez_dynpart_hashjoin_4.q.out  | 101 ++---
 8 files changed, 165 insertions(+), 82 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateAddSqlOperator.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateAddSqlOperator.java
index af9b12ee6f7..ef865e4d22c 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateAddSqlOperator.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateAddSqlOperator.java
@@ -18,13 +18,16 @@
 
 package org.apache.hadoop.hive.ql.optimizer.calcite.reloperators;
 
-import org.apache.calcite.sql.fun.SqlAbstractTimeFunction;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.SqlFunction;
+import org.apache.calcite.sql.SqlFunctionCategory;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.type.ReturnTypes;
 
-public class HiveDateAddSqlOperator extends SqlAbstractTimeFunction {
-  public static final HiveDateAddSqlOperator INSTANCE = new 
HiveDateAddSqlOperator();
+public final class HiveDateAddSqlOperator {
+  public static final SqlFunction INSTANCE =
+  new SqlFunction("DATE_ADD", SqlKind.OTHER_FUNCTION, 
ReturnTypes.DATE_NULLABLE, null, null,
+  SqlFunctionCategory.TIMEDATE);
 
-  protected HiveDateAddSqlOperator() {
-super("DATE_ADD", SqlTypeName.DATE);
+  private HiveDateAddSqlOperator() {
   }
 }
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateSubSqlOperator.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateSubSqlOperator.java
index 4f737126f02..d1c00211189 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateSubSqlOperator.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveDateSubSqlOperator.java
@@ -18,13 +18,16 @@
 
 package org.apache.hadoop.hive.ql.optimizer.calcite.reloperators;
 
-import org.apache.calcite.sql.fun.SqlAbstractTimeFunction;
-import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.calcite.sql.

(hive) 01/02: HIVE-27963: Build failure when license-maven-plugin downloads bsd-license.php (Akshat Mathur reviewed by Stamatis Zampetakis, Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 6671e1a91f288d8552840a2517912e5820d0204b
Author: akshat0395 
AuthorDate: Wed Dec 20 14:01:53 2023 +0530

HIVE-27963: Build failure when license-maven-plugin downloads 
bsd-license.php (Akshat Mathur reviewed by Stamatis Zampetakis, Ayush Saxena)

1. Add BSD-2-CLAUSE in licenseUrlFileNames patterns to unify downloaded 
files and avoid the build failure.
2. Upgrade plugin version from 2.1.0 to 2.3.0 (unrelated to the failure but 
still beneficial)

Close apache/hive#4963
---
 packaging/pom.xml | 4 
 pom.xml   | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/packaging/pom.xml b/packaging/pom.xml
index 274dcc7d08b..9a25658475d 100644
--- a/packaging/pom.xml
+++ b/packaging/pom.xml
@@ -158,6 +158,10 @@
   \Qhttps://opensource.org/licenses/Apache-2.0\E
   \Qhttp://www.apache.org/licenses/\E
 
+
+  https?://(www\.)?opensource.org/licenses/bsd-license.php
+  https?://(www\.)?opensource.org/licenses/BSD-2-Clause
+
 
   https?://(www\.)?opensource.org/licenses/BSD-3-Clause
 
diff --git a/pom.xml b/pom.xml
index 62ba8c78d61..b6959e10cee 100644
--- a/pom.xml
+++ b/pom.xml
@@ -100,7 +100,7 @@
 3.5.0
 3.0.0-M4
 2.7.10
-2.1.0
+2.3.0
 
 1.10.1
 1.10.13

(hive) branch master updated (f265cc25905 -> 36ce858163a)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from f265cc25905 HIVE-27876 Incorrect query results on tables with 
ClusterBy & SortBy (Ramesh Kumar Thangarajan, reviewed by Krisztian Kasa, 
Attila Turoczy)
 new 6671e1a91f2 HIVE-27963: Build failure when license-maven-plugin 
downloads bsd-license.php (Akshat Mathur reviewed by Stamatis Zampetakis, Ayush 
Saxena)
 new 36ce858163a HIVE-27919: Constant reduction in CBO does not work for 
FROM_UNIXTIME, DATE_ADD, DATE_SUB, TO_UNIX_TIMESTAMP (Stamatis Zampetakis 
reviewed by Akshat Mathur, Krisztian Kasa)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 packaging/pom.xml  |   4 +
 pom.xml|   2 +-
 .../reloperators/HiveDateAddSqlOperator.java   |  15 +--
 .../reloperators/HiveDateSubSqlOperator.java   |  15 +--
 .../reloperators/HiveFromUnixTimeSqlOperator.java  |  29 --
 .../HiveToUnixTimestampSqlOperator.java|  12 +--
 .../queries/clientpositive/cbo_constantfolding.q   |   5 +
 .../clientpositive/llap/cbo_constantfolding.q.out  |  60 
 .../llap/constant_prop_coalesce.q.out  |  10 +-
 .../llap/tez_dynpart_hashjoin_4.q.out  | 101 ++---
 10 files changed, 170 insertions(+), 83 deletions(-)
 create mode 100644 ql/src/test/queries/clientpositive/cbo_constantfolding.q
 create mode 100644 
ql/src/test/results/clientpositive/llap/cbo_constantfolding.q.out

(hive) branch master updated: HIVE-24219: Disable flaky TestStreaming (Stamatis Zampetakis reviewed by Sourabh Badhya)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 553ee3b908f HIVE-24219: Disable flaky TestStreaming (Stamatis 
Zampetakis reviewed by Sourabh Badhya)
553ee3b908f is described below

commit 553ee3b908f89841713044a8e5440b6e19c65103
Author: Stamatis Zampetakis 
AuthorDate: Wed Dec 13 13:30:32 2023 +0100

HIVE-24219: Disable flaky TestStreaming (Stamatis Zampetakis reviewed by 
Sourabh Badhya)

Recent failure: http://ci.hive.apache.org/job/hive-flaky-check/772

Close apache/hive#4944
---
 streaming/src/test/org/apache/hive/streaming/TestStreaming.java | 1 +
 1 file changed, 1 insertion(+)

diff --git a/streaming/src/test/org/apache/hive/streaming/TestStreaming.java 
b/streaming/src/test/org/apache/hive/streaming/TestStreaming.java
index 37c64429b78..f68c396de7e 100644
--- a/streaming/src/test/org/apache/hive/streaming/TestStreaming.java
+++ b/streaming/src/test/org/apache/hive/streaming/TestStreaming.java
@@ -118,6 +118,7 @@ import org.junit.rules.TemporaryFolder;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+@Ignore("HIVE-24219")
 public class TestStreaming {
   private static final Logger LOG = 
LoggerFactory.getLogger(TestStreaming.class);
 

(hive) branch master updated: HIVE-27943: NPE in VectorMapJoinCommonOperator.setUpHashTable when running query with join on date (Stamatis Zampetakis reviewed by Attila Turoczy, Krisztian Kasa)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 1455f6201b0 HIVE-27943: NPE in 
VectorMapJoinCommonOperator.setUpHashTable when running query with join on date 
(Stamatis Zampetakis reviewed by Attila Turoczy, Krisztian Kasa)
1455f6201b0 is described below

commit 1455f6201b0f7b061361bc9acc23cb810ff02483
Author: Stamatis Zampetakis 
AuthorDate: Thu Dec 7 16:20:17 2023 +0100

HIVE-27943: NPE in VectorMapJoinCommonOperator.setUpHashTable when running 
query with join on date (Stamatis Zampetakis reviewed by Attila Turoczy, 
Krisztian Kasa)

Close apache/hive#4929
---
 .../VectorMapJoinOptimizedCreateHashTable.java |   1 +
 ql/src/test/queries/clientpositive/mapjoin_date.q  |   8 +
 .../results/clientpositive/llap/mapjoin_date.q.out | 186 +
 3 files changed, 195 insertions(+)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
index 4171f2038c2..dff1f51da60 100644
--- 
a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
+++ 
b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedCreateHashTable.java
@@ -55,6 +55,7 @@ public class VectorMapJoinOptimizedCreateHashTable {
 case SHORT:
 case INT:
 case LONG:
+case DATE:
   switch (hashTableKind) {
   case HASH_MAP:
 hashTable = new VectorMapJoinOptimizedLongHashMap(
diff --git a/ql/src/test/queries/clientpositive/mapjoin_date.q 
b/ql/src/test/queries/clientpositive/mapjoin_date.q
new file mode 100644
index 000..8126c017a90
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/mapjoin_date.q
@@ -0,0 +1,8 @@
+set hive.auto.convert.join=true;
+
+CREATE TABLE person (fname string, birthDate date);
+INSERT INTO person VALUES ('Victor', '2023-11-27'), ('Alexandre', 
'2023-11-28');
+
+EXPLAIN VECTORIZATION DETAIL SELECT * FROM person p1 INNER JOIN person p2 ON 
p1.birthDate=p2.birthDate;
+
+SELECT * FROM person p1 INNER JOIN person p2 ON p1.birthDate=p2.birthDate;
diff --git a/ql/src/test/results/clientpositive/llap/mapjoin_date.q.out 
b/ql/src/test/results/clientpositive/llap/mapjoin_date.q.out
new file mode 100644
index 000..c5dfc75a5f3
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/mapjoin_date.q.out
@@ -0,0 +1,186 @@
+PREHOOK: query: CREATE TABLE person (fname string, birthDate date)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@person
+POSTHOOK: query: CREATE TABLE person (fname string, birthDate date)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@person
+PREHOOK: query: INSERT INTO person VALUES ('Victor', '2023-11-27'), 
('Alexandre', '2023-11-28')
+PREHOOK: type: QUERY
+PREHOOK: Input: _dummy_database@_dummy_table
+PREHOOK: Output: default@person
+POSTHOOK: query: INSERT INTO person VALUES ('Victor', '2023-11-27'), 
('Alexandre', '2023-11-28')
+POSTHOOK: type: QUERY
+POSTHOOK: Input: _dummy_database@_dummy_table
+POSTHOOK: Output: default@person
+POSTHOOK: Lineage: person.birthdate SCRIPT []
+POSTHOOK: Lineage: person.fname SCRIPT []
+PREHOOK: query: EXPLAIN VECTORIZATION DETAIL SELECT * FROM person p1 INNER 
JOIN person p2 ON p1.birthDate=p2.birthDate
+PREHOOK: type: QUERY
+PREHOOK: Input: default@person
+ A masked pattern was here 
+POSTHOOK: query: EXPLAIN VECTORIZATION DETAIL SELECT * FROM person p1 INNER 
JOIN person p2 ON p1.birthDate=p2.birthDate
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@person
+ A masked pattern was here 
+PLAN VECTORIZATION:
+  enabled: true
+  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
+STAGE DEPENDENCIES:
+  Stage-1 is a root stage
+  Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+  Stage: Stage-1
+Tez
+ A masked pattern was here 
+  Edges:
+Map 1 <- Map 2 (BROADCAST_EDGE)
+ A masked pattern was here 
+  Vertices:
+Map 1 
+Map Operator Tree:
+TableScan
+  alias: p1
+  filterExpr: birthdate is not null (type: boolean)
+  probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_25_container, 
bigKeyColName:birthdate, smallTablePos:1, keyRatio:0.0
+  Statistics: Num rows: 2 Data size: 296 Basic stats: COMPLETE 
Column stats: COMPLETE
+  TableScan Vectorization:
+  native: true
+  vectorizationSchemaColumns: [0:fname:string, 
1:birthdate:date, 2:ROW__ID:struct, 
3:ROW__IS__DELETED:boolean]
+  Filter Opera

(hive) branch master updated: HIVE-27658: Error resolving join keys during conversion to dynamic partition hashjoin (Stamatis Zampetakis reviewed by Denys Kuzmenko)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new f396676b09d HIVE-27658: Error resolving join keys during conversion to 
dynamic partition hashjoin (Stamatis Zampetakis reviewed by Denys Kuzmenko)
f396676b09d is described below

commit f396676b09d9dd706b4bff4e1c1d999f9f3b1d2f
Author: Stamatis Zampetakis 
AuthorDate: Thu Dec 7 17:56:08 2023 +0100

HIVE-27658: Error resolving join keys during conversion to dynamic 
partition hashjoin (Stamatis Zampetakis reviewed by Denys Kuzmenko)

Sometimes when the compiler attempts to convert a Join to a
Dynamic Partition HashJoin (DPHJ) and certain assumptions about the
shape of the plan do not hold a SemanticException is thrown.

The DPHJ is a performance optimization so there is no reason to raise a
fatal error when the conversion cannot be performed. It is preferable
to simply skip the conversion and use a regular join instead of
blocking completely the query. The `MapJoinProcessor.getMapJoinDesc`
method already returns null in certain cases, so it is safe to add
another exit condition.

Overview of changes:
1. Return null when join key resolution fails and simply skip conversion
to DPHJ.
2. Log a warning instead of throwing a fatal SemanticException.
3. Enrich error message with more information to improve diagnosability.

Bringing the plan into a shape that will allow the DPHJ conversion is
still meaningful but can be tracked independently with other tickets.

Close apache/hive#4930
---
 .../hadoop/hive/ql/optimizer/MapJoinProcessor.java |   3 +-
 .../clientpositive/tez_dynpart_hashjoin_4.q|  24 +++
 .../llap/tez_dynpart_hashjoin_4.q.out  | 210 +
 3 files changed, 236 insertions(+), 1 deletion(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 
b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
index e922ce47796..adf4fbe1b21 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
@@ -1306,7 +1306,8 @@ public class MapJoinProcessor extends Transform {
 List keyExprList =
 ExprNodeDescUtils.resolveJoinKeysAsRSColumns(mapEntry.getValue(), 
rsParent);
 if (keyExprList == null) {
-  throw new SemanticException("Error resolving join keys");
+  LOG.warn("Error resolving join keys {} in {} {}", 
mapEntry.getValue(), rsParent, rsParent.getColumnExprMap());
+  return null;
 }
 newKeyExprMap.put(pos, keyExprList);
   }
diff --git a/ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_4.q 
b/ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_4.q
new file mode 100644
index 000..d15307a42de
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_4.q
@@ -0,0 +1,24 @@
+CREATE TABLE table_a (start_date date, product_id int);
+
+ALTER TABLE table_a UPDATE STATISTICS 
SET('numRows'='2','rawDataSize'='0' );
+ALTER TABLE table_a UPDATE STATISTICS FOR COLUMN product_id 
SET('lowValue'='100','highValue'='1','numNulls'='0','numDVs'='30'
 );
+ALTER TABLE table_a UPDATE STATISTICS FOR COLUMN start_date 
SET('lowValue'='1','highValue'='2','numNulls'='0','numDVs'='2500' );
+
+CREATE TABLE table_b (start_date date, product_id int, product_sk string);
+
+ALTER TABLE table_b UPDATE STATISTICS 
SET('numRows'='1','rawDataSize'='0' );
+ALTER TABLE table_b UPDATE STATISTICS FOR COLUMN product_id 
SET('lowValue'='100','highValue'='1','numNulls'='0','numDVs'='30'
 );
+ALTER TABLE table_b UPDATE STATISTICS FOR COLUMN start_date 
SET('lowValue'='1','highValue'='2','numNulls'='0','numDVs'='500' );
+ALTER TABLE table_b UPDATE STATISTICS FOR COLUMN product_sk SET 
('numDVs'='30','numNulls'='0','avgColLen'='10','maxColLen'='10');
+
+set hive.optimize.dynamic.partition.hashjoin=true;
+set hive.auto.convert.join=true;
+set hive.auto.convert.join.noconditionaltask.size=18000;
+
+EXPLAIN
+SELECT TC.CONST_DATE, TB.PRODUCT_SK
+FROM TABLE_A TA
+INNER JOIN (SELECT TO_DATE(FROM_UNIXTIME(1701088643)) AS CONST_DATE) TC
+ON TA.START_DATE = TC.CONST_DATE
+INNER JOIN TABLE_B TB
+ON TB.START_DATE = TC.CONST_DATE AND TA.PRODUCT_ID = TB.PRODUCT_ID;
diff --git 
a/ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_4.q.out 
b/ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_4.q.out
new file mode 100644
index 000..5ca79e22b2b
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/tez_dynpart_hashjoin_4.q.out
@@ -0,0 +1,210 @@
+PREHOOK: query: CREATE TABLE table_a (start_date date, product_id int)
+PREHOO

(hive) branch master updated: HIVE-27846: HIVE-27849: Tests under hive-unit module are not running (Stamatis Zampetakis reviewed by Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 698194183a2 HIVE-27846: HIVE-27849: Tests under hive-unit module are 
not running (Stamatis Zampetakis reviewed by Ayush Saxena)
698194183a2 is described below

commit 698194183a2cb4ee6bc528ee71462cbae2c2c7ad
Author: Stamatis Zampetakis 
AuthorDate: Thu Nov 2 12:52:41 2023 +0100

HIVE-27846: HIVE-27849: Tests under hive-unit module are not running 
(Stamatis Zampetakis reviewed by Ayush Saxena)

1. Drop jupiter-engine dependency from hive-unit module preventing
tests from running.
2. Remove redundant jupiter annotations from TestStorageSchemaReader to
avoid dependencies to jupiter API.
3. Revert ivy upgrade cause since it leads to failures in replication
tests:
* TestReplAcrossInstancesWithJsonMessageFormat;
* TestReplicationScenariosAcrossInstances tests.
4. Disable 
TestCrudCompactorOnTez#secondCompactionShouldBeRefusedBeforeEnqueueing
test till HIVE-27848 is resolved.

Close apache/hive#4850
---
 itests/hive-unit/pom.xml| 5 -
 .../org/apache/hadoop/hive/metastore/TestStorageSchemaReader.java   | 6 ++
 .../apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java | 2 ++
 pom.xml | 2 +-
 4 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/itests/hive-unit/pom.xml b/itests/hive-unit/pom.xml
index 09d4f5712f9..29ece216722 100644
--- a/itests/hive-unit/pom.xml
+++ b/itests/hive-unit/pom.xml
@@ -208,11 +208,6 @@
   junit
   test
 
-
-  org.junit.jupiter
-  junit-jupiter-engine
-  test
-
 
   org.apache.hadoop
   hadoop-hdfs
diff --git 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestStorageSchemaReader.java
 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestStorageSchemaReader.java
index c6e1f1a4b0d..fadd793bc23 100644
--- 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestStorageSchemaReader.java
+++ 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestStorageSchemaReader.java
@@ -36,8 +36,6 @@ import org.apache.thrift.TException;
 import org.junit.After;
 import org.junit.Before;
 import org.junit.Test;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
 
 import java.util.HashMap;
 import java.util.List;
@@ -59,7 +57,7 @@ public class TestStorageSchemaReader {
   Map jdbcTableParams = new HashMap<>();
   Map jdbcSerdeParams = new HashMap<>();
 
-  @Before @BeforeEach public void setUp() throws Exception {
+  @Before public void setUp() throws Exception {
 dbName = "sampleDb";
 hiveConf = new HiveConf(this.getClass());
 new DatabaseBuilder().setName(dbName).create(new 
HiveMetaStoreClient(hiveConf), hiveConf);
@@ -83,7 +81,7 @@ public class TestStorageSchemaReader {
 jdbcTableParams.put("storage_handler", 
"org.apache.hive.storage.jdbc.JdbcStorageHandler");
   }
 
-  @After @AfterEach public void tearDown() throws Exception {
+  @After public void tearDown() throws Exception {
 new HiveMetaStoreClient(hiveConf).dropDatabase(dbName, true, true, true);
   }
 
diff --git 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
index e72811e103b..295bbdd09f7 100644
--- 
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
+++ 
b/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java
@@ -77,6 +77,7 @@ import org.apache.orc.StripeInformation;
 import org.apache.orc.TypeDescription;
 import org.apache.orc.impl.RecordReaderImpl;
 import org.junit.Assert;
+import org.junit.Ignore;
 import org.junit.Test;
 import org.mockito.ArgumentCaptor;
 import org.mockito.Mockito;
@@ -601,6 +602,7 @@ public class TestCrudCompactorOnTez extends 
CompactorOnTezTest {
   }
 
   @Test
+  @Ignore("HIVE-27848")
   public void secondCompactionShouldBeRefusedBeforeEnqueueing() throws 
Exception {
 conf.setBoolVar(HiveConf.ConfVars.COMPACTOR_CRUD_QUERY_BASED, true);
 
diff --git a/pom.xml b/pom.xml
index bf8a5d3e5fd..c163ce1d3a2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -151,7 +151,7 @@
 
 4.5.13
 4.4.13
-2.5.2
+2.5.1
 2.13.5
 2.3.4
 2.4.1

(calcite) branch main updated: [CALCITE-6011] Add FilterWindowTransposeRule to push a Filter past a Window

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new bdafeec27c [CALCITE-6011] Add FilterWindowTransposeRule to push a 
Filter past a Window
bdafeec27c is described below

commit bdafeec27c529d0089434a4d8b0fc7a1e49efcd1
Author: shenlang 
AuthorDate: Fri Sep 22 23:23:08 2023 +0800

[CALCITE-6011] Add FilterWindowTransposeRule to push a Filter past a Window

Close apache/calcite#3439
---
 .../java/org/apache/calcite/plan/RelOptRules.java  |   3 +-
 .../org/apache/calcite/rel/rules/CoreRules.java|   4 +
 .../rel/rules/FilterWindowTransposeRule.java   | 131 +
 .../org/apache/calcite/test/RelOptRulesTest.java   |  61 ++
 .../org/apache/calcite/test/RelOptRulesTest.xml|  97 +++
 core/src/test/resources/sql/winagg.iq  |  41 +++
 6 files changed, 336 insertions(+), 1 deletion(-)

diff --git a/core/src/main/java/org/apache/calcite/plan/RelOptRules.java 
b/core/src/main/java/org/apache/calcite/plan/RelOptRules.java
index 46f39fc326..816008f34e 100644
--- a/core/src/main/java/org/apache/calcite/plan/RelOptRules.java
+++ b/core/src/main/java/org/apache/calcite/plan/RelOptRules.java
@@ -89,7 +89,8 @@ public class RelOptRules {
   CoreRules.EXCHANGE_REMOVE_CONSTANT_KEYS,
   CoreRules.SORT_EXCHANGE_REMOVE_CONSTANT_KEYS,
   CoreRules.SAMPLE_TO_FILTER,
-  CoreRules.FILTER_SAMPLE_TRANSPOSE);
+  CoreRules.FILTER_SAMPLE_TRANSPOSE,
+  CoreRules.FILTER_WINDOW_TRANSPOSE);
 
   static final List ABSTRACT_RULES =
   ImmutableList.of(CoreRules.AGGREGATE_ANY_PULL_UP_CONSTANTS,
diff --git a/core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java 
b/core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java
index 77c19d58ab..24a104f151 100644
--- a/core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java
+++ b/core/src/main/java/org/apache/calcite/rel/rules/CoreRules.java
@@ -321,6 +321,10 @@ public class CoreRules {
   public static final FilterSetOpTransposeRule FILTER_SET_OP_TRANSPOSE =
   FilterSetOpTransposeRule.Config.DEFAULT.toRule();
 
+  /** Rule that pushes a {@link Filter} past a {@link 
org.apache.calcite.rel.core.Window}. */
+  public static final FilterWindowTransposeRule FILTER_WINDOW_TRANSPOSE =
+  FilterWindowTransposeRule.Config.DEFAULT.toRule();
+
   /** Rule that reduces constants inside a {@link LogicalFilter}.
*
* @see #JOIN_REDUCE_EXPRESSIONS
diff --git 
a/core/src/main/java/org/apache/calcite/rel/rules/FilterWindowTransposeRule.java
 
b/core/src/main/java/org/apache/calcite/rel/rules/FilterWindowTransposeRule.java
new file mode 100644
index 00..bc38dafd03
--- /dev/null
+++ 
b/core/src/main/java/org/apache/calcite/rel/rules/FilterWindowTransposeRule.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.rel.rules;
+
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Window;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+
+import com.google.common.collect.ImmutableList;
+
+import org.immutables.value.Value;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Planner rule that pushes a {@link org.apache.calcite.rel.core.Filter}
+ * past a {@link org.apache.calcite.rel.core.Window}.
+ *
+ *  If {@code Filter} condition used columns belongs {@code Window} 
partition keys,
+ * then we could push the condition past the {@code Window}.
+ *
+ *  For example:
+ *  {@code
+ *LogicalProject(NAME=[$0], DEPTNO=[$1], EXPR$2=[$2])
+ * LogicalFilter(condition=[>($1, 0)])
+ *   LogicalProject(NAME=[$1], DEPTNO=[$0], EXPR$2=[$2])
+ * LogicalWindow(window#0=[window(partition {0} aggs [COUNT()])])
+ *   LogicalTableScan(table

(calcite) branch main updated: [CALCITE-6075] Site: Cloning source code from GitHub using git protocol fails

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new ef82a6cc6b [CALCITE-6075] Site: Cloning source code from GitHub using 
git protocol fails
ef82a6cc6b is described below

commit ef82a6cc6bc565f6dc596c1d2aca7221fd6cdc49
Author: caicancai <77189278+caican...@users.noreply.github.com>
AuthorDate: Fri Oct 27 09:55:12 2023 +0800

[CALCITE-6075] Site: Cloning source code from GitHub using git protocol 
fails

GitHub removed the unecrypted git protocoli a while ago:

https://github.blog/changelog/2022-03-15-removed-unencrypted-git-protocol-and-certain-ssh-keys/

Close apache/calcite#3488
---
 site/_docs/howto.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/_docs/howto.md b/site/_docs/howto.md
index e179f21c49..476ee2b779 100644
--- a/site/_docs/howto.md
+++ b/site/_docs/howto.md
@@ -58,7 +58,7 @@ Create a local copy of the GitHub repository,
 then build using the included Gradle wrapper:
 
 {% highlight bash %}
-$ git clone git://github.com/apache/calcite.git
+$ git clone https://github.com/apache/calcite.git
 $ cd calcite
 $ ./gradlew build
 {% endhighlight %}

[hive] 01/02: HIVE-27777: CBO fails on multi insert overwrites with common group expression (Steve Carlin reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 4cbd9579c6cb30a3e0dbe6f86464202d6b935b78
Author: Steve Carlin 
AuthorDate: Sun Oct 8 16:50:03 2023 -0700

HIVE-2: CBO fails on multi insert overwrites with common group 
expression (Steve Carlin reviewed by Stamatis Zampetakis)

The following statement is failing at compilation time when CBO is
enabled.

FROM (select key, f1 FROM tbl1 where key=5) a
INSERT OVERWRITE TABLE tbl2 partition(key=5)
select f1 WHERE key > 0 GROUP by f1
INSERT OVERWRITE TABLE tbl2 partition(key=6)
select f1 WHERE key > 0 GROUP by f1;

The failure happens when there is a filter to a constant value in
the FROM clause, the value is referenced in the filter in the INSERT
OVERWRITE, and there is a common group existing across the insert
overwrites.

CBO is pulling up the key = 5 expression into the select clause as a
constant (i.e. select 5 key, f1 FROM tbl1 where key = 5).  After it
gets converted back into AST and then re-compiled, there is code in
the common group method that expects all columns to be non-constants
which is causing the failiure.

Close apache/hive#4783
---
 .../hadoop/hive/ql/parse/SemanticAnalyzer.java |   2 +-
 .../hadoop/hive/ql/plan/ExprNodeConstantDesc.java  |   9 +
 .../queries/clientpositive/multi_insert_gby5.q |   6 +
 .../clientpositive/llap/multi_insert_gby5.q.out| 207 +
 4 files changed, 223 insertions(+), 1 deletion(-)

diff --git a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
index 5be52904b5f..250f7c2fcbc 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
@@ -6097,7 +6097,7 @@ public class SemanticAnalyzer extends 
BaseSemanticAnalyzer {
 for (Map.Entry entry : nodeOutputs.entrySet()) {
   ASTNode parameter = entry.getKey();
   ExprNodeDesc expression = entry.getValue();
-  if (!(expression instanceof ExprNodeColumnDesc)) {
+  if (!(expression instanceof ExprNodeColumnDesc) && 
!ExprNodeConstantDesc.isFoldedFromCol(expression)) {
 continue;
   }
   if (ExprNodeDescUtils.indexOf(expression, reduceValues) >= 0) {
diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java 
b/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
index 268aa1a2faa..f5e3828e2cd 100755
--- a/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDesc.java
@@ -22,6 +22,7 @@ import java.io.Serializable;
 import java.util.List;
 
 import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.commons.lang3.StringUtils;
 import org.apache.hadoop.hive.common.StringInternUtils;
 import org.apache.hadoop.hive.serde.serdeConstants;
 import org.apache.hadoop.hive.serde2.objectinspector.ConstantObjectInspector;
@@ -213,6 +214,14 @@ public class ExprNodeConstantDesc extends ExprNodeDesc 
implements Serializable {
 return true;
   }
 
+  public static boolean isFoldedFromCol(ExprNodeDesc expr) {
+if (!(expr instanceof ExprNodeConstantDesc)) {
+  return false;
+}
+ExprNodeConstantDesc constantExpr = (ExprNodeConstantDesc) expr;
+return StringUtils.isNotEmpty(constantExpr.foldedFromCol);
+  }
+
   @Override
   public int hashCode() {
 int superHashCode = super.hashCode();
diff --git a/ql/src/test/queries/clientpositive/multi_insert_gby5.q 
b/ql/src/test/queries/clientpositive/multi_insert_gby5.q
new file mode 100644
index 000..be856d7669b
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/multi_insert_gby5.q
@@ -0,0 +1,6 @@
+CREATE TABLE target1 (tc int);
+CREATE TABLE target2 (tc int);
+
+EXPLAIN FROM (SELECT 100 as sa, 200 as sb) source
+INSERT OVERWRITE TABLE target1 SELECT sa WHERE sb > 0 GROUP BY sa
+INSERT OVERWRITE TABLE target2 SELECT sa GROUP BY sa;
diff --git a/ql/src/test/results/clientpositive/llap/multi_insert_gby5.q.out 
b/ql/src/test/results/clientpositive/llap/multi_insert_gby5.q.out
new file mode 100644
index 000..1345395e0ee
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/multi_insert_gby5.q.out
@@ -0,0 +1,207 @@
+PREHOOK: query: CREATE TABLE target1 (tc int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@target1
+POSTHOOK: query: CREATE TABLE target1 (tc int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@target1
+PREHOOK: query: CREATE TABLE target2 (tc int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@target2
+POSTHOOK: query: 

[hive] branch master updated (b02cef4fe94 -> 089e2727cb9)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from b02cef4fe94 HIVE-27731: Iceberg: Perform metadata delete for queries 
with static filters (#4748) (Sourabh Badhya reviewed by Denys Kuzmenko, 
Krisztian Kasa)
 new 4cbd9579c6c HIVE-2: CBO fails on multi insert overwrites with 
common group expression (Steve Carlin reviewed by Stamatis Zampetakis)
 new 089e2727cb9 HIVE-27745: Add unit test to ensure short version is 
inline with full version (Kirti Ruge reviewed by Stamatis Zampetakis)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../hadoop/hive/ql/parse/SemanticAnalyzer.java |   2 +-
 .../hadoop/hive/ql/plan/ExprNodeConstantDesc.java  |   9 +
 .../queries/clientpositive/multi_insert_gby5.q |   6 +
 .../clientpositive/llap/multi_insert_gby5.q.out| 207 +
 .../metastore/utils/TestMetastoreVersionInfo.java  |  28 ++-
 5 files changed, 241 insertions(+), 11 deletions(-)
 create mode 100644 ql/src/test/queries/clientpositive/multi_insert_gby5.q
 create mode 100644 
ql/src/test/results/clientpositive/llap/multi_insert_gby5.q.out
 copy ql/src/test/org/apache/hadoop/hive/ql/plan/TestDropMacroDesc.java => 
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetastoreVersionInfo.java
 (60%)

[hive] 02/02: HIVE-27745: Add unit test to ensure short version is inline with full version (Kirti Ruge reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 089e2727cb9b2855d9c1ad2b7a789b7afb8fc5a5
Author: rkirti 
AuthorDate: Wed Oct 11 13:40:47 2023 +0530

HIVE-27745: Add unit test to ensure short version is inline with full 
version (Kirti Ruge reviewed by Stamatis Zampetakis)

Close apache/hive#4791
---
 .../metastore/utils/TestMetastoreVersionInfo.java  | 45 ++
 1 file changed, 45 insertions(+)

diff --git 
a/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetastoreVersionInfo.java
 
b/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetastoreVersionInfo.java
new file mode 100644
index 000..e256befa62b
--- /dev/null
+++ 
b/standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetastoreVersionInfo.java
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.utils;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestMetastoreVersionInfo {
+
+  private String hiveShortVersion;
+  private String hiveVersion;
+
+  @Before
+  public void setUp() throws Exception {
+hiveShortVersion = MetastoreVersionInfo.getShortVersion();
+hiveVersion = MetastoreVersionInfo.getVersion();
+  }
+
+  @Test
+  public void testValidateHiveShortVersionWithHiveVersion() {
+Assert.assertEquals(hiveVersion.replace("-SNAPSHOT", ""), 
hiveShortVersion);
+  }
+
+  @Test
+  public void testIfHiveVersionHasShortVersionAsPrefix() {
+Assert.assertTrue(hiveVersion.startsWith(hiveShortVersion));
+  }
+}
+

[calcite] branch main updated: [CALCITE-6075] Site: Cloning source code from GitHub using git protocol fails

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new 8821eaf94b [CALCITE-6075] Site: Cloning source code from GitHub using 
git protocol fails
8821eaf94b is described below

commit 8821eaf94b08dd5c55074b900201da7c386c1635
Author: caicancai <77189278+caican...@users.noreply.github.com>
AuthorDate: Thu Oct 26 17:25:40 2023 +0800

[CALCITE-6075] Site: Cloning source code from GitHub using git protocol 
fails

GitHub removed the unecrypted git protocoli a while ago:

https://github.blog/changelog/2022-03-15-removed-unencrypted-git-protocol-and-certain-ssh-keys/

Close apache/calcite#3485
---
 site/develop/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/site/develop/index.md b/site/develop/index.md
index 4ed681157f..180048afa9 100644
--- a/site/develop/index.md
+++ b/site/develop/index.md
@@ -54,7 +54,7 @@ Create a local copy of the Git repository, `cd` to its root 
directory,
 then build using Gradle:
 
 {% highlight bash %}
-$ git clone git://github.com/apache/calcite.git
+$ git clone https://github.com/apache/calcite.git
 $ cd calcite
 $ ./gradlew build
 {% endhighlight %}

[calcite] branch main updated: [CALCITE-5923] SqlOperatorTest using safeParameters are not using overridable fixture

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


The following commit(s) were added to refs/heads/main by this push:
 new f996bc9993 [CALCITE-5923] SqlOperatorTest using safeParameters are not 
using overridable fixture
f996bc9993 is described below

commit f996bc9993546019d0fd475b2daa99ac8b1b9259
Author: Runkang He 
AuthorDate: Sun Aug 13 18:06:37 2023 +0800

[CALCITE-5923] SqlOperatorTest using safeParameters are not using 
overridable fixture

SqlOperatorTest cases using the safeParameters method to generate
parameterized inputs are not using the overridable fixture from the
subclasses. Due to this the customized behavior introduced by
subclasses of SqlOperatorTest is not taken into account by these tests.

This violates the design principle of the SqlOperatorTest class which
explicitly states that subclasses are meant to override the method
and basically decreases code coverage since subclasses will not have
any effect on the affected tests.

1. Remove static modifier from safeParameters to allow the use of
overridable fixture() method.
2. Add TestInstance.Lifecycle.PER_CLASS annotation to allow
@MethodSource to be non-static (see junit-team/junit5#984).
3. Adapt expected results for test cases casting string to timestamps
based on changes from CALCITE-5678 in Avatica.
4. Pickup avatica.version from project properties to run the appropriate
tests while waiting for the Avatica upgrade.

Close apache/calcite#3364
---
 build.gradle.kts   |  1 +
 .../org/apache/calcite/test/SqlOperatorTest.java   | 52 --
 .../java/org/apache/calcite/util/TestUtil.java |  2 +
 3 files changed, 42 insertions(+), 13 deletions(-)

diff --git a/build.gradle.kts b/build.gradle.kts
index 9d4a6e8af9..f026c48583 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -820,6 +820,7 @@ allprojects {
 passProperty("user.language", "TR")
 passProperty("user.country", "tr")
 passProperty("user.timezone", "UTC")
+passProperty("calcite.avatica.version", 
props.string("calcite.avatica.version"))
 val props = System.getProperties()
 for (e in props.propertyNames() as 
`java.util`.Enumeration) {
 if (e.startsWith("calcite.") || e.startsWith("avatica.")) {
diff --git a/testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java 
b/testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java
index 746c330720..fd11a31f05 100644
--- a/testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java
+++ b/testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java
@@ -83,6 +83,7 @@ import org.checkerframework.checker.nullness.qual.Nullable;
 import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Tag;
 import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInstance;
 import org.junit.jupiter.params.ParameterizedTest;
 import org.junit.jupiter.params.provider.Arguments;
 import org.junit.jupiter.params.provider.MethodSource;
@@ -197,6 +198,7 @@ import static java.nio.charset.StandardCharsets.UTF_8;
  * null arguments or null results.
  * 
  */
+@TestInstance(TestInstance.Lifecycle.PER_CLASS)
 @SuppressWarnings("MethodCanBeStatic")
 public class SqlOperatorTest {
   //~ Static fields/initializers -
@@ -441,8 +443,9 @@ public class SqlOperatorTest {
   }
 
   /** Generates parameters to test both regular and safe cast. */
-  static Stream safeParameters() {
-SqlOperatorFixture f = SqlOperatorFixtureImpl.DEFAULT;
+  @SuppressWarnings("unused")
+  private Stream safeParameters() {
+SqlOperatorFixture f = fixture();
 SqlOperatorFixture f2 =
 
SqlOperatorFixtures.safeCastWrapper(f.withLibrary(SqlLibrary.BIG_QUERY), 
"SAFE_CAST");
 SqlOperatorFixture f3 =
@@ -1238,8 +1241,6 @@ public class SqlOperatorTest {
 
 f.checkScalar("cast('1945-02-24 12:42:25' as TIMESTAMP)",
 "1945-02-24 12:42:25", "TIMESTAMP(0) NOT NULL");
-f.checkScalar("cast('1945-2-2 12:2:5' as TIMESTAMP)",
-"1945-02-02 12:02:05", "TIMESTAMP(0) NOT NULL");
 f.checkScalar("cast('  1945-02-24 12:42:25  ' as TIMESTAMP)",
 "1945-02-24 12:42:25", "TIMESTAMP(0) NOT NULL");
 f.checkScalar("cast('1945-02-24 12:42:25.34' as TIMESTAMP)",
@@ -1253,15 +1254,40 @@ public class SqlOperatorTest {
   f.checkScalar("cast('1945-02-24 12:42:25.34' as TIMESTAMP(2))",
   "1945-02-24 12:42:25.34", "TIMES

[hive] branch master updated: HIVE-27772: UNIX_TIMESTAMP should return NULL when date fields are out of bounds (Simhadri Govindappa reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 1c126d94744 HIVE-27772: UNIX_TIMESTAMP should return NULL when date 
fields are out of bounds (Simhadri Govindappa reviewed by Stamatis Zampetakis)
1c126d94744 is described below

commit 1c126d947448ffc9784a1465306e018ba183a014
Author: SimhadriG 
AuthorDate: Thu Oct 5 14:31:12 2023 +0530

HIVE-27772: UNIX_TIMESTAMP should return NULL when date fields are out of 
bounds (Simhadri Govindappa reviewed by Stamatis Zampetakis)

In the case of invalid dates, such as '2001-02-31' (day field exceeds valid 
range for the given month), the UNIX_TIMESTAMP function behaves unexpectedly.
Instead of returning NULL (as it happens in other systems like Spark, 
MySQL, etc.), it provides a value corresponding to another valid date based on 
some resolution rules (e.g., Feb 28th or March 1st).
The resolution rules and results depend on the underlying formatter 
implementation used by UNIX_TIMESTAMP.

By default, the DATETIME formatter uses the SMART resolution style and the 
SIMPLE formatter the LENIENT. Both of these styles are able to resolve 
"invalid" bounds to valid dates.

In order to prevent seemingly "invalid" dates to be parsed correctly we 
have to use the STRICT resolution style.
However, we cannot simply switch the formatters to always use the STRICT 
resolution cause that would break existing applications relying on the existing 
resolution rules.

To address the problem reported here and retain the previous behaviour we 
opted to make the resolution style configurable by adding a new property.
The new property only affects the DATETIME formatter; the SIMPLE formatter 
is almost deprecated so we don't add new features to it.

Close apache/hive#4777
---
 .../java/org/apache/hadoop/hive/conf/HiveConf.java |   4 +
 .../hadoop/hive/conf/TestHiveConfVarsValidate.java |   9 ++
 .../ql/udf/generic/InstantDateTimeFormatter.java   |   6 +-
 .../hive/ql/udf/generic/InstantFormatter.java  |  16 ++-
 ...ericUDFToUnixTimestampEvaluateStringString.java |  30 +++--
 ...nericUDFToUnixTimestampEvaluateStringString.csv | 140 +
 6 files changed, 135 insertions(+), 70 deletions(-)

diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index 5e9c8425ddf..290cd8a4efa 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -3860,6 +3860,10 @@ public class HiveConf extends Configuration {
 "is discouraged. It suffers from known bugs that are unlikely to be 
fixed in subsequent versions of the product." +
 "Furthermore, using SIMPLE formatter may lead to strange behavior, and 
unexpected results when combined " +
 "with SQL functions/operators that are using the new DATETIME 
formatter."),
+HIVE_DATETIME_RESOLVER_STYLE("hive.datetime.formatter.resolver.style", 
"SMART",
+new StringSet("SMART", "STRICT", "LENIENT"),
+"The style used by the hive.datetime.formatter (only applicable to 
DATETIME) to resolve dates amd times." +
+"The possible values are STRICT, SMART, and LENIENT and their behavior 
follows the java.time.format.ResolverStyle API."),
  // HiveServer2 specific configs
 
HIVE_SERVER2_CLEAR_DANGLING_SCRATCH_DIR("hive.server2.clear.dangling.scratchdir",
 false,
 "Clear dangling scratch dir periodically in HS2"),
diff --git 
a/common/src/test/org/apache/hadoop/hive/conf/TestHiveConfVarsValidate.java 
b/common/src/test/org/apache/hadoop/hive/conf/TestHiveConfVarsValidate.java
index 7ac44588c08..42736ddb3d6 100644
--- a/common/src/test/org/apache/hadoop/hive/conf/TestHiveConfVarsValidate.java
+++ b/common/src/test/org/apache/hadoop/hive/conf/TestHiveConfVarsValidate.java
@@ -26,6 +26,7 @@ import java.util.List;
 
 import static org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import static 
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_DATETIME_FORMATTER;
+import static 
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_DATETIME_RESOLVER_STYLE;
 import static 
org.apache.hadoop.hive.conf.HiveConf.ConfVars.HIVE_EXPLAIN_NODE_VISIT_LIMIT;
 import static org.junit.Assert.assertEquals;
 
@@ -62,6 +63,14 @@ public class TestHiveConfVarsValidate {
 list.add(new Object[] { HIVE_DATETIME_FORMATTER, "simple", null });
 list.add(new Object[] { HIVE_DATETIME_FORMATTER, "dateTime", null });
 list.add(new Object[] { HIVE_DATETIME_FORMATTER, "OTHER", "Invalid value.. 
expects one of [datetime, simple]&

[hive] branch master updated: HIVE-27695: HIVE-26828: Intermittent OOM when running TestMiniTezCliDriver (Stamatis Zampetakis reviewed by Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new c126422a91b HIVE-27695: HIVE-26828: Intermittent OOM when running 
TestMiniTezCliDriver (Stamatis Zampetakis reviewed by Ayush Saxena)
c126422a91b is described below

commit c126422a91be695c75ec4a750638a0aa4d1ba6cd
Author: Stamatis Zampetakis 
AuthorDate: Wed Oct 11 12:17:23 2023 +0200

HIVE-27695: HIVE-26828: Intermittent OOM when running TestMiniTezCliDriver 
(Stamatis Zampetakis reviewed by Ayush Saxena)

java.lang.OutOfMemoryError: GC overhead limit exceeded is thrown by the Tez 
Application Master (AM) cause the current heap size (128MB) is not enough to 
accommodate the needs of multiple Tez containers running.
Each running container requires roughly 10MB of memory in the AM. The AM 
accumulates/manipulates multiple configuration objects (some of them retaining 
as much as 1MB of heap) per container.
The heap gradually becomes full and GC is spending a lot of CPU time to 
clean things up without really making much progress since containers are reused 
and heap cannot shrink.

There are multiple solutions to the problem but the easiest and most 
effective is to increase the heap size for the AM. At this point in time, 512MB 
is a good value.

The hybridgrace_hashjoin_2.q test which was failing to due OOM can now be 
re-enabled.

Close apache/hive#4792
---
 data/conf/tez/tez-site.xml  | 2 +-
 ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/data/conf/tez/tez-site.xml b/data/conf/tez/tez-site.xml
index ba4df319a55..88adb6a57e8 100644
--- a/data/conf/tez/tez-site.xml
+++ b/data/conf/tez/tez-site.xml
@@ -1,7 +1,7 @@
 
   
 tez.am.resource.memory.mb
-128
+512
   
   
 tez.task.resource.memory.mb
diff --git a/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q 
b/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
index 93b8c13a49f..6ed771ba685 100644
--- a/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
+++ b/ql/src/test/queries/clientpositive/hybridgrace_hashjoin_2.q
@@ -1,7 +1,6 @@
 --! qt:dataset:srcpart
 --! qt:dataset:src1
 --! qt:dataset:src
---! qt:disabled:HIVE-26820 Disable hybridgrace_hashjoin_2.q flaky test
 set hive.mapred.mode=nonstrict;
 set hive.explain.user=false;
 -- Hybrid Grace Hash Join

[hive] branch master updated (6cd53ed2bbb -> 1843105f27f)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 6cd53ed2bbb HIVE-27747: Generalize TestSchemaToolForMetastore to run 
on every supported DBMS (Stamatis Zampetakis reviewed by Zhihua Deng)
 add 1843105f27f HIVE-27755: Quote identifiers in SQL emitted by SchemaTool 
for MySQL (Stamatis Zampetakis reviewed by Zhihua Deng)

No new revisions were added by this update.

Summary of changes:
 .../hadoop/hive/metastore/MetaStoreSchemaInfo.java   | 16 ++--
 .../metastore/tools/schematool/HiveSchemaHelper.java |  5 +
 .../metastore/tools/schematool/MetastoreSchemaTool.java  |  4 
 3 files changed, 15 insertions(+), 10 deletions(-)

[hive] branch master updated: HIVE-27747: Generalize TestSchemaToolForMetastore to run on every supported DBMS (Stamatis Zampetakis reviewed by Zhihua Deng)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 6cd53ed2bbb HIVE-27747: Generalize TestSchemaToolForMetastore to run 
on every supported DBMS (Stamatis Zampetakis reviewed by Zhihua Deng)
6cd53ed2bbb is described below

commit 6cd53ed2bbbe8ff1f5e52fcfd15e2d55b7ee393f
Author: Stamatis Zampetakis 
AuthorDate: Wed Sep 27 22:04:23 2023 +0200

HIVE-27747: Generalize TestSchemaToolForMetastore to run on every supported 
DBMS (Stamatis Zampetakis reviewed by Zhihua Deng)

The goal is to make the tests run on every supported metastore DBMS (not 
only Derby), to increase code coverage for the SchemaTool. An overview of the 
changes can be seen below.

1. Parameterize the test using all available dockerized DBMS (except 
Mariadb which currently fails due to HIVE-27749)
2. Refactor hardcoded references to derby with the appropriate database type
3. Add missing semicolons in some statements in the test scripts
4. Use appropriate DDL statements for renaming a table based on the 
underlying database
5. Post-process DML statements in test scripts to:
* quote table/column identifiers when necessary (only Postgres at the 
moment)
* pick appropriate literal for indicating false in boolean/bit datatype 
based on the database

Close apache/hive#4754
---
 .../schematool/TestSchemaToolForMetastore.java | 149 +++--
 1 file changed, 110 insertions(+), 39 deletions(-)

diff --git 
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java
 
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java
index f6a029a4470..e82b943915b 100644
--- 
a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java
+++ 
b/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java
@@ -29,10 +29,14 @@ import java.net.URI;
 import java.sql.Connection;
 import java.sql.DatabaseMetaData;
 import java.sql.SQLException;
-import java.util.Random;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
 
 import org.apache.commons.dbcp2.DelegatingConnection;
-import org.apache.commons.io.FileUtils;
 import org.apache.commons.lang3.text.StrTokenizer;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hive.metastore.HiveMetaException;
@@ -40,39 +44,64 @@ import 
org.apache.hadoop.hive.metastore.IMetaStoreSchemaInfo;
 import org.apache.hadoop.hive.metastore.MetaStoreSchemaInfoFactory;
 import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest;
 import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
-import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.Derby;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.Mssql;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.Mysql;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.Oracle;
+import org.apache.hadoop.hive.metastore.dbinstall.rules.Postgres;
+
 import org.junit.After;
 import org.junit.Assert;
 import org.junit.Before;
 import org.junit.Test;
 import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static java.lang.String.format;
 
 @Category(MetastoreCheckinTest.class)
+@RunWith(Parameterized.class)
 public class TestSchemaToolForMetastore {
+  private static final Pattern IDENTIFIER = Pattern.compile("[A-Z_]+");
+
   private MetastoreSchemaTool schemaTool;
   private Connection conn;
   private Configuration conf;
-  private String testMetastoreDB;
+  private final DatabaseRule dbms;
   private PrintStream errStream;
   private PrintStream outStream;
-  private String argsBase;
   private SchemaToolTaskValidate validator;
 
+  public TestSchemaToolForMetastore(DatabaseRule dbms){
+this.dbms = dbms;
+  }
+
+  @Parameterized.Parameters(name = "{0}")
+  public static Collection databases() {
+List dbs = new ArrayList<>();
+dbs.add(new Object[] { new Derby(true) });
+dbs.add(new Object[] { new Mysql() });
+dbs.add(new Object[] { new Oracle() });
+dbs.add(new Object[] { new Postgres() });
+//dbs.add(new Object[] { new Mariadb() }); Disabled due to HIVE-27749
+dbs.add(new Object[] { new Mssql() });
+return dbs;
+  }
+  
   @Before
-  public void setUp() throws HiveMetaException, IOExcept

[hive] branch master updated: HIVE-27760: WHERE condition on DATE type partitioning column leads to wrong results (Dayakar M reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 1e919dd9816 HIVE-27760: WHERE condition on DATE type partitioning 
column leads to wrong results (Dayakar M reviewed by Stamatis Zampetakis)
1e919dd9816 is described below

commit 1e919dd98169212ab10d267de0dc25bbbdf96b1e
Author: mdayakar 
AuthorDate: Fri Sep 29 22:36:01 2023 +0530

HIVE-27760: WHERE condition on DATE type partitioning column leads to wrong 
results (Dayakar M reviewed by Stamatis Zampetakis)

During partitioning pruning a DATE literal follows certain transformations 
that are timezone sensitive.

For timezones ahead of GMT (e.g., Asia/Hong Kong) the literal is shifted by 
one day leading to wrong results.

(2023-01-01) String -> [DateTimeFormatter] -> LocalDate -> java.sql.Date -> 
[SimpleDateFormat] -> String (2022-12-31)

The problem was introduced by HIVE-27373, which brought DateTimeFormatter 
and LocalDate to the mix.
The use of different formatters and Date object representations made 
parsing timezone sensitive and led to the bug.

1. Add new methods in MetaStoreUtils to Encapsulate Date, Timestamp 
conversions from/to string.
2. Replace SimpleDateFormat with DateTimeFormatter to keep the 
parsing/formatting behavior properly aligned.
3. Refactor MetaStoreDirectSql and PartFilterVisitor to use the same 
conversion methods in MetaStoreUtils.
4. Add unit tests for the new methods in MetaStoreUtils and qtests for the 
wrong result problem.

Close apache/hive#4760
---
 .../hadoop/hive/ql/parse/BaseSemanticAnalyzer.java |   7 +-
 .../date_timestamp_partition_filter.q  |  14 ++
 .../llap/date_timestamp_partition_filter.q.out | 214 +
 .../hive/metastore/utils/MetaStoreUtils.java   |  74 +--
 .../hadoop/hive/metastore/MetaStoreDirectSql.java  |  14 +-
 .../hive/metastore/parser/ExpressionTree.java  |   6 +-
 .../hive/metastore/parser/PartFilterVisitor.java   |  20 +-
 .../hive/metastore/utils/TestMetaStoreUtils.java   | 102 ++
 8 files changed, 398 insertions(+), 53 deletions(-)

diff --git 
a/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
b/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
index 0989c412dfa..bc36832b6d1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
@@ -20,7 +20,6 @@ package org.apache.hadoop.hive.ql.parse;
 
 import java.io.IOException;
 import java.io.UnsupportedEncodingException;
-import java.text.ParseException;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashMap;
@@ -1804,9 +1803,9 @@ public abstract class BaseSemanticAnalyzer {
   throw new SemanticException("Unexpected date type " + 
colValue.getClass());
 }
 try {
-  return MetaStoreUtils.PARTITION_DATE_FORMAT.get().format(
-  MetaStoreUtils.PARTITION_DATE_FORMAT.get().parse(value.toString()));
-} catch (ParseException e) {
+  return MetaStoreUtils.convertDateToString(
+  MetaStoreUtils.convertStringToDate(value.toString()));
+} catch (Exception e) {
   throw new SemanticException(e);
 }
   }
diff --git 
a/ql/src/test/queries/clientpositive/date_timestamp_partition_filter.q 
b/ql/src/test/queries/clientpositive/date_timestamp_partition_filter.q
new file mode 100644
index 000..31328aebca8
--- /dev/null
+++ b/ql/src/test/queries/clientpositive/date_timestamp_partition_filter.q
@@ -0,0 +1,14 @@
+--! qt:timezone:Asia/Hong_Kong
+
+CREATE EXTERNAL TABLE testpd(col1 string, col2 String) PARTITIONED 
BY(PartitionDate DATE) STORED AS ORC;
+INSERT into testpd(PartitionDate, col1, col2) 
VALUES('2023-01-01','Value11','Value12');
+INSERT into testpd(PartitionDate, col1, col2) 
VALUES('2023-01-02','Value21','Value22');
+explain extended select * from testpd where PartitionDate = '2023-01-01';
+select * from testpd where PartitionDate = '2023-01-01';
+
+
+CREATE EXTERNAL TABLE testpt(col1 string, col2 String) PARTITIONED 
BY(PartitionTimestamp TIMESTAMP) STORED AS ORC;
+INSERT into testpt(PartitionTimestamp, col1, col2) VALUES('2023-01-01 
10:20:30','Value11','Value12');
+INSERT into testpt(PartitionTimestamp, col1, col2) VALUES('2023-01-02 
20:30:40','Value21','Value22');
+explain extended select * from testpt where PartitionTimestamp = '2023-01-01 
10:20:30';
+select * from testpt where PartitionTimestamp = '2023-01-01 10:20:30';
diff --git 
a/ql/src/test/results/clientpositive/llap/date_timestamp_partition_filter.q.out 
b/ql/src/test/results/clientpositive/llap/date_timestamp_partition_filter.q.out
new file mode 100644
index 000..a676d982e37
--- /dev/null
+++ 
b/ql/src/test/re

[hive] branch master updated (4b8e4e3ca52 -> 49132a32702)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 4b8e4e3ca52 HIVE-27704: Remove PowerMock from jdbc-handler and upgrade 
mockito to 4.11 (#4722). (Kirti Ruge, reviewed by Ayush Saxena, Zsolt Miskolczi)
 add 49132a32702 HIVE-27738: SchemaTool picks incorrect schema script after 
4.0.0-beta-1 release (Kirti Ruge reviewed by Stamatis Zampetakis, Wechar Yu)

No new revisions were added by this update.

Summary of changes:
 pom.xml | 2 +-
 ql/src/test/results/clientpositive/llap/sysdb.q.out | 2 +-
 ...eta-1.postgres.sql => upgrade-3.1.3000-to-4.0.0-beta-2.postgres.sql} | 0
 standalone-metastore/pom.xml| 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename 
standalone-metastore/metastore-server/src/test/resources/sql/postgres/{upgrade-3.1.3000-to-4.0.0-beta-1.postgres.sql
 => upgrade-3.1.3000-to-4.0.0-beta-2.postgres.sql} (100%)

[hive] branch master updated: HIVE-27673: Configurable datetime formatter for date_format (Stamatis Zampetakis reviewed by John Sherman, Aman Sinha, Ayush Saxena)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 16a39b4fe77 HIVE-27673: Configurable datetime formatter for 
date_format (Stamatis Zampetakis reviewed by John Sherman, Aman Sinha, Ayush 
Saxena)
16a39b4fe77 is described below

commit 16a39b4fe77be3f204ddcffac607385f683ed129
Author: Stamatis Zampetakis 
AuthorDate: Fri Sep 8 12:20:37 2023 +0200

HIVE-27673: Configurable datetime formatter for date_format (Stamatis 
Zampetakis reviewed by John Sherman, Aman Sinha, Ayush Saxena)

SimpleDateFormat and DateTimeFormatter present differences in their 
behavior leading to different query results when date_format is used (after 
upgrade).

To avoid sudden changes in query results and allow users to migrate 
gradually their query workloads to use the new patterns the 
`hive.datetime.formatter` property can now be used to select which formatter 
will be used in date_format.

Overview of the change:
1. Generalize the UnixTimeFormatter hierarchy so that it works with 
Instant, which can represent more than epochSeconds, to be used by 
GenericUDFDateFormat class.
2. Adapt slightly other classes using the UnixTimeFormatter to pass or 
retrieve the necessary values from/to Instants.
3. Refactor GenericUDFDateFormat to use the InstantFormatter so that its 
behavior becomes configurable via `hive.datetime.formatter` property.
4. Add configure method in GenericUDFDateFormat to allowing propagation of 
sessions level configurations (formatter/zone) to the runtime (Tez, MR, etc.). 
At runtime there is no SessionState so the only way to configure date_format is 
using the respective method and global configurations in hive-site.xml, etc.
5. Extend the use of `hive.datetime.formatter` to date_format and update 
the description in HiveConf mentioning also the bugs that affect SIMPLE 
formatter.
6. Add unit tests for date_format for both formatters covering:
* Reference dates (1970-01-01) and trivial patterns
* Invalid date inputs (Jul 9 2023) that cannot be parsed
* Timestamp inputs with timezone specification (1970-01-01 00:00:00 
GMT-01:00) that is ignored/dropped silently
* Current date (2023-07-21 09:13:10.123456789) with nano precision and 
pattern variations
* Patterns ('u','S') with different behavior between formatters
* Gregorian dates (1800-01-01 00:00:00) before 1900 and different timezones
* Julian dates (1000-01-01 00:00:00) and different timezones
7. Add end-to-end tests for ensuring that configuration property takes 
effect with (in HS2)/without (in Tez) task conversion.

Essentially the unit tests also demonstrate existing bugs when the SIMPLE 
formatter is used affecting Gregorian dates before 1900 and Julian dates.

Closes #4675
---
 .../hadoop/hive/common/type/TimestampTZ.java   |   3 +
 .../java/org/apache/hadoop/hive/conf/HiveConf.java |   7 +-
 .../expressions/VectorUDFUnixTimeStampString.java  |   8 +-
 .../hive/ql/udf/generic/GenericUDFDateFormat.java  |  57 +--
 .../ql/udf/generic/GenericUDFFromUnixTime.java |  13 ++-
 .../ql/udf/generic/GenericUDFToUnixTimeStamp.java  |  10 +-
 ...ormatter.java => InstantDateTimeFormatter.java} |  20 ++--
 ...nixTimeFormatter.java => InstantFormatter.java} |  49 +-
 ...matterCache.java => InstantFormatterCache.java} |   4 +-
 ...matter.java => InstantSimpleDateFormatter.java} |  20 ++--
 .../generic/TestGenericUDFDateFormatEvaluate.java  | 104 +
 .../test/queries/clientpositive/udf_date_format.q  |   1 +
 .../clientpositive/udf_date_format_nofetchtask.q   |  26 ++
 ...{udf_date_format.q => udf_date_format_simple.q} |   4 +
 .../generic/TestGenericUDFDateFormatEvaluate.csv   |  40 
 .../clientpositive/llap/udf_date_format.q.out  |   2 +-
 .../llap/udf_date_format_nofetchtask.q.out |  83 
 ...e_format.q.out => udf_date_format_simple.q.out} |  18 ++--
 18 files changed, 369 insertions(+), 100 deletions(-)

diff --git 
a/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java 
b/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java
index af181ba124e..3ff06e0eead 100644
--- a/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java
+++ b/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java
@@ -100,4 +100,7 @@ public class TimestampTZ implements Comparable 
{
 return zonedDateTime.toInstant().getNano();
   }
 
+  public Instant toInstant() {
+return zonedDateTime.toInstant();
+  }
 }
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index a76b64f87f5..10b8a34b76e 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.jav

[hive] branch master updated: HIVE-27642: Add missing dependency to StartMiniHS2Cluster with Postgres (Zoltan Ratkai reviewed by Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new 44fdbee484b HIVE-27642: Add missing dependency to StartMiniHS2Cluster 
with Postgres (Zoltan Ratkai reviewed by Stamatis Zampetakis)
44fdbee484b is described below

commit 44fdbee484bc717a1076f2c520f1adafc5190e9f
Author: Zoltan Ratkai 
AuthorDate: Wed Aug 30 14:29:36 2023 +0200

HIVE-27642: Add missing dependency to StartMiniHS2Cluster with Postgres 
(Zoltan Ratkai reviewed by Stamatis Zampetakis)

Closes #4644
---
 itests/hive-unit/pom.xml | 5 +
 1 file changed, 5 insertions(+)

diff --git a/itests/hive-unit/pom.xml b/itests/hive-unit/pom.xml
index af54522bb96..29ece216722 100644
--- a/itests/hive-unit/pom.xml
+++ b/itests/hive-unit/pom.xml
@@ -489,6 +489,11 @@
   ${htmlunit.version}
   test
 
+
+  org.postgresql
+  postgresql
+  test
+
   
   
 

[hive] branch master updated: HIVE-27277: GH actions to build and push docker image (Simhadri Govindappa reviewed by Stamatis Zampetakis, Zhihua Deng, Naveen Gangam, Attila Turoczy, Zoltan Ratkai)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


The following commit(s) were added to refs/heads/master by this push:
 new b7e3e1d6190 HIVE-27277: GH actions to build and push docker image 
(Simhadri Govindappa reviewed by Stamatis Zampetakis, Zhihua Deng, Naveen 
Gangam, Attila Turoczy, Zoltan Ratkai)
b7e3e1d6190 is described below

commit b7e3e1d6190b1a92b9bf5e6e4fb661070d12a398
Author: SimhadriGovindappa 
AuthorDate: Wed Apr 26 13:08:41 2023 +0530

HIVE-27277: GH actions to build and push docker image (Simhadri Govindappa 
reviewed by Stamatis Zampetakis, Zhihua Deng, Naveen Gangam, Attila Turoczy, 
Zoltan Ratkai)

1. Add (manually triggered) action for building images from archives
(for old releases)
2. Add action for building images from sources automatically triggered
on new tag creation.
3. Publish docker images for amd64 and arm64 platforms to dockerhub

Closes #4614
---
 .github/workflows/docker-GA-images.yml | 138 +
 packaging/src/docker/Dockerfile|  42 --
 packaging/src/docker/build.sh  |   1 +
 3 files changed, 174 insertions(+), 7 deletions(-)

diff --git a/.github/workflows/docker-GA-images.yml 
b/.github/workflows/docker-GA-images.yml
new file mode 100644
index 000..cffa8eb89a3
--- /dev/null
+++ b/.github/workflows/docker-GA-images.yml
@@ -0,0 +1,138 @@
+name: Build and Publish docker images for Hive GA
+
+on:
+  create:
+  workflow_dispatch:
+inputs:
+  hiveVersion:
+description: 'Hive version'
+required: true
+default: '4.0.0-beta-1'
+
+
+  hadoopVersion:
+description: 'hadoop version'
+required: true
+default: '3.3.1'
+
+  tezVersion:
+description: 'tez version'
+required: true
+default: '0.10.2'
+
+
+
+jobs:
+  buildFromArchive:
+if: github.event_name == 'workflow_dispatch'
+runs-on: ubuntu-latest
+steps:
+  -
+name: Checkout
+uses: actions/checkout@v3
+
+  -
+name: 'Set up JDK 8'
+uses: actions/setup-java@v1
+with:
+  java-version: 8
+
+  -
+name: Hive version
+run:  echo "HIVE_VERSION=${{ github.event.inputs.hiveVersion }}"  >> 
$GITHUB_ENV
+
+
+  - name: Tag
+run: echo "tag=${{ env.HIVE_VERSION }}" | awk '{print tolower($0)}' >> 
$GITHUB_ENV
+
+  -
+name: Hadoop version
+run: echo "HADOOP_VERSION=${{ github.event.inputs.hadoopVersion }}" >> 
$GITHUB_ENV
+  -
+name: Tez version
+run: echo "TEZ_VERSION=${{ github.event.inputs.tezVersion }}" >> 
$GITHUB_ENV
+
+  - name: Login to Docker Hub
+uses: docker/login-action@v2
+with:
+  username: ${{ secrets.DOCKERHUB_USER }}
+  password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+  -
+name: Set up Docker Buildx
+uses: docker/setup-buildx-action@v2
+
+  - name: Build and push Hive Image to docker hub
+uses: docker/build-push-action@v4
+with:
+  context: ./packaging/src/docker/
+  file: ./packaging/src/docker/Dockerfile
+  platforms: linux/amd64,linux/arm64
+  push: true
+  tags: ${{ secrets.DOCKERHUB_USER }}/hive:${{ env.tag }}
+  build-args:
+|
+HIVE_VERSION=${{ env.HIVE_VERSION }}
+HADOOP_VERSION=${{ env.HADOOP_VERSION }}
+TEZ_VERSION=${{ env.TEZ_VERSION }}
+BUILD_ENV=archive
+
+  buildFromSource:
+if: github.event_name == 'create' && github.event.ref_type == 'tag' && 
startsWith(github.event.ref, 'rel/')
+runs-on: ubuntu-latest
+steps:
+  - name: Checkout
+uses: actions/checkout@v3
+
+  - name: 'Set up JDK 8'
+uses: actions/setup-java@v1
+with:
+  java-version: 8
+
+  - name: Hive version
+run: echo "HIVE_VERSION=$(mvn -f "pom.xml" -q help:evaluate 
-Dexpression=project.version -DforceStdout)" >> $GITHUB_ENV
+
+
+  - name: Tag
+run: echo "tag=${{ env.HIVE_VERSION }}" | awk '{print tolower($0)}' >> 
$GITHUB_ENV
+
+  - name: Hadoop version
+run: echo "HADOOP_VERSION=$(mvn -f "pom.xml" -q help:evaluate 
-Dexpression=hadoop.version -DforceStdout)" >> $GITHUB_ENV
+  - name: Tez version
+run: echo "TEZ_VERSION=$(mvn -f "pom.xml" -q help:evaluate 
-Dexpression=tez.version -DforceStdout)" >> $GITHUB_ENV
+
+  - name: Build project
+run: mvn clean package -DskipTests -Pitests,iceberg -Pdist
+
+  - name: Check for hive tar.gz
+run: ls ./packaging/target/
+
+  - name: Copy to docker directory
+run: cp ./packaging/target/ap

[calcite] 01/02: [CALCITE-5615] Run SQL Logic Test suite using Calcite's HSQLDB JDBC adapter

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit 996692e36a0d1bdff2b3fd1e08fa99fcc755f2ab
Author: Mihai Budiu 
AuthorDate: Tue Apr 11 16:44:36 2023 -0700

[CALCITE-5615] Run SQL Logic Test suite using Calcite's HSQLDB JDBC adapter

Signed-off-by: Mihai Budiu 

Close apache/calcite#3145
---
 bom/build.gradle.kts   |   1 +
 gradle.properties  |   1 +
 plus/build.gradle.kts  |   1 +
 .../java/org/apache/calcite/slt/SqlLogicTests.java | 314 +++
 .../calcite/slt/executors/CalciteExecutor.java |  94 
 .../calcite/slt/executors/package-info.java}   |  22 +-
 .../java/org/apache/calcite/slt/package-info.java  | 105 
 plus/src/test/resources/slttestfailures.txt| 615 +
 8 files changed, 1135 insertions(+), 18 deletions(-)

diff --git a/bom/build.gradle.kts b/bom/build.gradle.kts
index e4b3a82aa2..dd68f3036b 100644
--- a/bom/build.gradle.kts
+++ b/bom/build.gradle.kts
@@ -90,6 +90,7 @@ dependencies {
 apiv("net.hydromatic:foodmart-queries")
 apiv("net.hydromatic:quidem")
 apiv("net.hydromatic:scott-data-hsqldb")
+apiv("net.hydromatic:sql-logic-test")
 apiv("net.hydromatic:tpcds", "hydromatic.tpcds")
 apiv("net.java.dev.jna:jna")
 apiv("net.sf.opencsv:opencsv")
diff --git a/gradle.properties b/gradle.properties
index e99adc707c..e59712d8c8 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -159,6 +159,7 @@ sketches-core.version=0.9.0
 slf4j.version=1.7.25
 spark.version=2.2.2
 sqlline.version=1.12.0
+sql-logic-test.version=0.3
 teradata.tpcds.version=1.2
 testcontainers.version=1.17.3
 tpch.version=1.0
diff --git a/plus/build.gradle.kts b/plus/build.gradle.kts
index 2aef9d28c5..1b58da0029 100644
--- a/plus/build.gradle.kts
+++ b/plus/build.gradle.kts
@@ -29,6 +29,7 @@ dependencies {
 implementation("org.apache.calcite.avatica:avatica-server")
 implementation("org.hsqldb:hsqldb")
 
+testImplementation("net.hydromatic:sql-logic-test")
 testImplementation(project(":testkit"))
 testImplementation("org.incava:java-diff")
 testRuntimeOnly("org.apache.logging.log4j:log4j-slf4j-impl")
diff --git a/plus/src/test/java/org/apache/calcite/slt/SqlLogicTests.java 
b/plus/src/test/java/org/apache/calcite/slt/SqlLogicTests.java
new file mode 100644
index 00..7e6ee0c36e
--- /dev/null
+++ b/plus/src/test/java/org/apache/calcite/slt/SqlLogicTests.java
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.calcite.slt;
+
+import org.apache.calcite.slt.executors.CalciteExecutor;
+import org.apache.calcite.util.trace.CalciteTrace;
+
+import com.google.common.collect.ImmutableSet;
+
+import net.hydromatic.sqllogictest.Main;
+import net.hydromatic.sqllogictest.OptionsParser;
+import net.hydromatic.sqllogictest.TestStatistics;
+
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.TestFactory;
+import org.slf4j.Logger;
+
+import java.io.BufferedReader;
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.PrintStream;
+import java.io.UnsupportedEncodingException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+/**
+ * Tests using sql-logic-test suite.
+ *
+ * For each test file the number of failed tests is saved in a "golden" 
file.
+ 

[calcite] 02/02: Remove mentions of binary distribution from README

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git

commit d802314e69f1eb975558ad3fd6013dc05ddaef7a
Author: Stamatis Zampetakis 
AuthorDate: Tue Jul 18 14:15:01 2023 +0300

Remove mentions of binary distribution from README

The project does not make binary releases at the moment so mentioning
binary distribution in the README is misleading.

Close apache/calcite#3322
---
 README | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/README b/README
index 3e646582f1..8f12ab9c9a 100644
--- a/README
+++ b/README
@@ -1,15 +1,14 @@
 Apache Calcite release 1.35.0
 
-This is a source or binary distribution of Apache Calcite.
+This is a source distribution of Apache Calcite.
 
 Changes since the previous release are described in the
 site/_docs/history.md file.
 
 The LICENSE and NOTICE files contain license information.
 
-If this is a source distribution, you can find instructions how to
-build the release in the "Building from a source distribution" section
-in site/_docs/howto.md.
+Check "Building from a source distribution" section
+in site/_docs/howto.md for instructions on how to build the project.
 
 Further information about Apache Calcite is available at its web site,
 http://calcite.apache.org.

[calcite] branch main updated (eff56f938c -> d802314e69)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/calcite.git


from eff56f938c Update broken link in RelMetadataProvider Javadoc
 new 996692e36a [CALCITE-5615] Run SQL Logic Test suite using Calcite's 
HSQLDB JDBC adapter
 new d802314e69 Remove mentions of binary distribution from README

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 README |   7 +-
 bom/build.gradle.kts   |   1 +
 gradle.properties  |   1 +
 plus/build.gradle.kts  |   1 +
 .../java/org/apache/calcite/slt/SqlLogicTests.java | 314 +++
 .../calcite/slt/executors/CalciteExecutor.java |  94 
 .../calcite/slt/executors}/package-info.java   |   4 +-
 .../java/org/apache/calcite/slt/package-info.java  | 105 
 plus/src/test/resources/slttestfailures.txt| 615 +
 9 files changed, 1136 insertions(+), 6 deletions(-)
 create mode 100644 plus/src/test/java/org/apache/calcite/slt/SqlLogicTests.java
 create mode 100644 
plus/src/test/java/org/apache/calcite/slt/executors/CalciteExecutor.java
 copy {core/src/main/java/org/apache/calcite/rel/type => 
plus/src/test/java/org/apache/calcite/slt/executors}/package-info.java (89%)
 create mode 100644 plus/src/test/java/org/apache/calcite/slt/package-info.java
 create mode 100644 plus/src/test/resources/slttestfailures.txt

[hive] branch master updated (12f7d5ea894 -> 1ebef40aba0)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 12f7d5ea894 HIVE-27431: Clean invalid properties in test module 
(#4408). (zhangbutao,  reviewed by Denys Kuzmenko, Ayush Saxena)
 add 1ebef40aba0 HIVE-27638: Preparing for 4.0.0-beta-2 development 
(Stamatis Zampetakis reviewed by Ayush Saxena, Attila Turoczy)

No new revisions were added by this update.

Summary of changes:
 accumulo-handler/pom.xml   |2 +-
 beeline/pom.xml|2 +-
 classification/pom.xml |2 +-
 cli/pom.xml|2 +-
 common/pom.xml |2 +-
 contrib/pom.xml|2 +-
 druid-handler/pom.xml  |2 +-
 hbase-handler/pom.xml  |2 +-
 hcatalog/core/pom.xml  |2 +-
 hcatalog/hcatalog-pig-adapter/pom.xml  |4 +-
 hcatalog/pom.xml   |4 +-
 hcatalog/server-extensions/pom.xml |2 +-
 hcatalog/webhcat/java-client/pom.xml   |2 +-
 hcatalog/webhcat/svr/pom.xml   |2 +-
 hplsql/pom.xml |2 +-
 iceberg/iceberg-catalog/pom.xml|2 +-
 iceberg/iceberg-handler/pom.xml|2 +-
 iceberg/iceberg-shading/pom.xml|2 +-
 iceberg/patched-iceberg-api/pom.xml|2 +-
 iceberg/patched-iceberg-core/pom.xml   |2 +-
 iceberg/pom.xml|4 +-
 itests/custom-serde/pom.xml|2 +-
 itests/custom-udfs/pom.xml |2 +-
 itests/custom-udfs/udf-classloader-udf1/pom.xml|2 +-
 itests/custom-udfs/udf-classloader-udf2/pom.xml|2 +-
 itests/custom-udfs/udf-classloader-util/pom.xml|2 +-
 .../custom-udfs/udf-vectorized-badexample/pom.xml  |2 +-
 itests/hcatalog-unit/pom.xml   |2 +-
 itests/hive-blobstore/pom.xml  |2 +-
 itests/hive-jmh/pom.xml|2 +-
 itests/hive-minikdc/pom.xml|2 +-
 itests/hive-unit-hadoop2/pom.xml   |2 +-
 itests/hive-unit/pom.xml   |2 +-
 itests/pom.xml |2 +-
 itests/qtest-accumulo/pom.xml  |2 +-
 itests/qtest-druid/pom.xml |2 +-
 itests/qtest-iceberg/pom.xml   |2 +-
 itests/qtest-kudu/pom.xml  |2 +-
 itests/qtest/pom.xml   |2 +-
 itests/test-serde/pom.xml  |2 +-
 itests/util/pom.xml|2 +-
 jdbc-handler/pom.xml   |2 +-
 jdbc/pom.xml   |2 +-
 kafka-handler/pom.xml  |2 +-
 kudu-handler/pom.xml   |2 +-
 llap-client/pom.xml|2 +-
 llap-common/pom.xml|2 +-
 llap-ext-client/pom.xml|2 +-
 llap-server/pom.xml|2 +-
 llap-tez/pom.xml   |2 +-
 metastore/pom.xml  |2 +-
 .../upgrade/hive/hive-schema-4.0.0-beta-2.hive.sql | 2121 
 .../upgrade-4.0.0-beta-1-to-4.0.0-beta-2.hive.sql  |3 +
 metastore/scripts/upgrade/hive/upgrade.order.hive  |3 +-
 packaging/pom.xml  |2 +-
 parser/pom.xml |2 +-
 pom.xml|6 +-
 ql/pom.xml |2 +-
 .../test/results/clientpositive/llap/sysdb.q.out   |8 +-
 serde/pom.xml  |2 +-
 service-rpc/pom.xml|2 +-
 service/pom.xml|2 +-
 shims/0.23/pom.xml |2 +-
 shims/aggregator/pom.xml   |2 +-
 shims/common/pom.xml   |2 +-
 shims/pom.xml  |2 +-
 standalone-metastore/metastore-common/pom.xml  |2 +-
 standalone-metastore/metastore-server/pom.xml  |6 +-
 .../sql/derby/hive-schema-4.0.0-beta-2.derby.sql   |  896 +
 .../upgrade-4.0.0-beta-1-to-4.0.0-beta-2.derby.sql |2 +
 .../src/main/sql/derby/upgrade.order.derby |1 +
 .../sql/mssql/hive-schema-4.0.0-beta-2.mssql.sql   | 1476

[hive] 02/02: HIVE-25576: Configurable datetime formatter for unix_timestamp, from_unixtime (Stamatis Zampetakis reviewed by Aman Sinha, John Sherman, Sai Hemanth Gantasala)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit ed51dfdbdfc109db53b3c631e9f1631e9bb65c34
Author: Stamatis Zampetakis 
AuthorDate: Thu Aug 17 22:02:59 2023 +0300

HIVE-25576: Configurable datetime formatter for unix_timestamp, 
from_unixtime (Stamatis Zampetakis reviewed by Aman Sinha, John Sherman, Sai 
Hemanth Gantasala)

The two Java formatters present diffences in their behavior leading to
different query results. The supported patterns, between the two
formatters, are also different something that makes existing queries
crash at runtime (after upgrade). Adapting to the new behavior of
DateTimeFormatter is a challenging and time-consuming task for end
users especially due to the widespread use of the afforementioned
unixtime functions.

Although DateTimeFormatter is a clear improvement over SimpleDateFormat
some users still want to retain the old behavior for compatibility
reasons thus introducing a property is necessary for facilitating
migration.

Overview of the change:

1. Add hive.datetime.formatter property to control formatter in
unix_timestamp and from_unixtime functions.
2. Add UnixTimeFormatter class hierarchy for encapsulating formatting
and parsing of unixtime based on the configuration.
3. Refactor unix_timestamp (+vectorized) and from_unixtime
implementations to use the new formatter classes.
4. Add parameterized unit tests for the affected UDF implementations.
The test cases are chosen in way to highlight similarities and
differences between the two available formatters and document the
current behavior. Few interesting test cases are discussed in more
detail below but not all of them.

* Dates before 1800 in different timezones

1800-01-01 00:00:00;-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;-5364683608
1800-01-01 00:00:00;-MM-dd HH:mm:ss;Asia/Kolkata;SIMPLE;-5364682200

The DATETIME and SIMPLE formatter use slightly different zone conversion
rules so mapping 1800-01-01 00:00:00 Asia/Kolkata to seconds since
epoch presents differences.

* Invalid pattern and AM/PM timestamps

2023-07-21 09:13PM;-MM-dd HH:mma;Etc/GMT;SIMPLE;1689930780

The SIMPLE formatter returns a wrong result when an invalid pattern is
used. The value 1689930780 actually corrsponds to 2023-07-21 09:13AM
(not PM as it was supposed to ); it seems that 'HH' takes precedence
over 'a'. The combined use of 'H' and 'a' is problematic. When using
AM and PM the 'h' letter is the correct pattern letter.

* Number of pattern letters

Jul 9 2023;MMM dd ;Etc/GMT;DATETIME;null
Jul 9 2023;MMM dd ;Etc/GMT;SIMPLE;1688860800

The SIMPLE formatter does not care how many times a pattern letter is
used when parsing. For this reason although the day appears as a single
digit for the SIMPLE formatter that is completely fine. The same does
not hold for the DATETIME formatter.

Closes #4615
---
 .../java/org/apache/hadoop/hive/conf/HiveConf.java |  13 ++-
 .../hadoop/hive/conf/TestHiveConfVarsValidate.java |   6 ++
 .../expressions/VectorUDFUnixTimeStampString.java  |  27 ++---
 .../ql/udf/generic/GenericUDFFromUnixTime.java |  35 ++
 .../ql/udf/generic/GenericUDFToUnixTimeStamp.java  |  76 -
 .../ql/udf/generic/UnixTimeDateTimeFormatter.java  |  76 +
 .../hive/ql/udf/generic/UnixTimeFormatter.java | 119 +
 .../ql/udf/generic/UnixTimeFormatterCache.java |  50 +
 .../udf/generic/UnixTimeSimpleDateFormatter.java   |  69 
 .../TestVectorUDFUnixTimeStampString.java  | 110 +++
 .../TestGenericUDFFromUnixTimeEvaluate.java| 105 ++
 ...ericUDFToUnixTimestampEvaluateStringString.java |  97 +
 .../expressions/TestVectorUnixTimeStampString.csv  |  36 +++
 .../generic/TestGenericUDFFromUnixTimeEvaluate.csv |  48 +
 ...nericUDFToUnixTimestampEvaluateStringString.csv |  54 ++
 15 files changed, 816 insertions(+), 105 deletions(-)

diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index 108c545a384..14190915020 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -3830,7 +3830,18 @@ public class HiveConf extends Configuration {
 
HIVE_PRIVILEGE_SYNCHRONIZER_INTERVAL("hive.privilege.synchronizer.interval",
 "1800s", new TimeValidator(TimeUnit.SECONDS),
 "Interval to synchronize privileges from external authorizer 
periodically in HS2"),
-
+HIVE_DATETIME_FORMATTER("hive.datetime.formatter", &quo

[hive] branch master updated (4d23badbd9c -> ed51dfdbdfc)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 4d23badbd9c HIVE-27303: Set correct output name to ReduceSink when 
there is a SMB join after Union (Seonggon Namgung, reviewed by Denys Kuzmenko, 
Laszlo Vegh)
 new cd9b42d36d4 HIVE-24771: Enable TransactionalKafkaWriterTest (Kokila N 
reviewed by Akshat Mathur, Attila Turoczy, Stamatis Zampetakis)
 new ed51dfdbdfc HIVE-25576: Configurable datetime formatter for 
unix_timestamp, from_unixtime (Stamatis Zampetakis reviewed by Aman Sinha, John 
Sherman, Sai Hemanth Gantasala)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/hadoop/hive/conf/HiveConf.java |  13 ++-
 .../hadoop/hive/conf/TestHiveConfVarsValidate.java |   6 ++
 .../hive/kafka/TransactionalKafkaWriterTest.java   |   1 -
 .../expressions/VectorUDFUnixTimeStampString.java  |  27 ++---
 .../ql/udf/generic/GenericUDFFromUnixTime.java |  35 ++
 .../ql/udf/generic/GenericUDFToUnixTimeStamp.java  |  76 -
 .../ql/udf/generic/UnixTimeDateTimeFormatter.java  |  76 +
 .../hive/ql/udf/generic/UnixTimeFormatter.java | 119 +
 .../ql/udf/generic/UnixTimeFormatterCache.java |  50 +
 .../udf/generic/UnixTimeSimpleDateFormatter.java   |  69 
 .../TestVectorUDFUnixTimeStampString.java  | 110 +++
 .../TestGenericUDFFromUnixTimeEvaluate.java| 105 ++
 ...ericUDFToUnixTimestampEvaluateStringString.java |  97 +
 .../expressions/TestVectorUnixTimeStampString.csv  |  36 +++
 .../generic/TestGenericUDFFromUnixTimeEvaluate.csv |  48 +
 ...nericUDFToUnixTimestampEvaluateStringString.csv |  54 ++
 16 files changed, 816 insertions(+), 106 deletions(-)
 create mode 100644 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UnixTimeDateTimeFormatter.java
 create mode 100644 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UnixTimeFormatter.java
 create mode 100644 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UnixTimeFormatterCache.java
 create mode 100644 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/UnixTimeSimpleDateFormatter.java
 create mode 100644 
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorUDFUnixTimeStampString.java
 create mode 100644 
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFFromUnixTimeEvaluate.java
 create mode 100644 
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.java
 create mode 100644 
ql/src/test/resources/org/apache/hadoop/hive/ql/exec/vector/expressions/TestVectorUnixTimeStampString.csv
 create mode 100644 
ql/src/test/resources/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFFromUnixTimeEvaluate.csv
 create mode 100644 
ql/src/test/resources/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.csv

[hive] 01/02: HIVE-24771: Enable TransactionalKafkaWriterTest (Kokila N reviewed by Akshat Mathur, Attila Turoczy, Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit cd9b42d36d4f24bc6d37ccb8ffad2cadb745483f
Author: Kokila N 
AuthorDate: Fri Jul 21 18:20:04 2023 +0530

HIVE-24771: Enable TransactionalKafkaWriterTest (Kokila N reviewed by 
Akshat Mathur, Attila Turoczy, Stamatis Zampetakis)

Flaky check passed: http://ci.hive.apache.org/job/hive-flaky-check/728/

Closes #4512
---
 .../test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java  | 1 -
 1 file changed, 1 deletion(-)

diff --git 
a/kafka-handler/src/test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java
 
b/kafka-handler/src/test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java
index 85bf5aac99a..07a3b5a37fe 100644
--- 
a/kafka-handler/src/test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java
+++ 
b/kafka-handler/src/test/org/apache/hadoop/hive/kafka/TransactionalKafkaWriterTest.java
@@ -58,7 +58,6 @@ import java.util.stream.IntStream;
 /**
  * Test Transactional Writer.
  */
-@org.junit.Ignore("HIVE-24771")
 public class TransactionalKafkaWriterTest {
 
   private static final String TOPIC = "TOPIC_TEST";

[hive-site] branch main updated: Apache Hive 4.0.0-beta-1 release

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hive-site.git


The following commit(s) were added to refs/heads/main by this push:
 new 0702866  Apache Hive 4.0.0-beta-1 release
0702866 is described below

commit 07028661978fe25f3d852ff07bf8752281ba7a54
Author: Stamatis Zampetakis 
AuthorDate: Mon Aug 14 19:18:45 2023 +0300

Apache Hive 4.0.0-beta-1 release
---
 content/docs/javadocs.md | 1 +
 content/general/downloads.md | 7 ++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/content/docs/javadocs.md b/content/docs/javadocs.md
index 7e076d0..d89bfbb 100644
--- a/content/docs/javadocs.md
+++ b/content/docs/javadocs.md
@@ -26,6 +26,7 @@ aliases: [/javadoc.html]
 ## Recent versions:
 ---
  javadoc and sources jars for use in an IDE are also available via 
[Nexus](https://repository.apache.org/index.html#nexus-search;gav~org.apache.hive)
+* [Hive 4.0.0-beta-1 
Javadocs](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs//r4.0.0-beta-1/api/index.html)
 * [Hive 4.0.0-alpha-2 
Javadocs](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs//r4.0.0-alpha-2/api/index.html)
 * [Hive 3.1.3 
Javadocs](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r3.1.3/api/index.html)
 * [Hive 4.0.0-alpha-1 
Javadocs](https://svn.apache.org/repos/infra/websites/production/hive/content/javadocs/r4.0.0-alpha-1/api/index.html)
diff --git a/content/general/downloads.md b/content/general/downloads.md
index 2e08cde..8973148 100644
--- a/content/general/downloads.md
+++ b/content/general/downloads.md
@@ -33,7 +33,11 @@ directory.
 
 ## News
 ---
-* ### 16 November 2022: release 4.0.0-alpha-2 available
+*  14 August 2023: release 4.0.0-beta-1 available
+* This release works with Hadoop 3.3.1
+* You can look at the complete [JIRA change log for this 
release][HIVE_4_0_0_B_1_CL].
+
+*  16 November 2022: release 4.0.0-alpha-2 available
 * This release works with Hadoop 3.3.1
 * You can look at the complete [JIRA change log for this 
release][HIVE_4_0_0_A_2_CL].
 
@@ -191,6 +195,7 @@ Hive users for these two versions are encouraged to upgrade.
   * You can look at the complete [JIRA change log for this 
release][HIVE_10_CL].
 
 [HIVE_DL]: http://www.apache.org/dyn/closer.cgi/hive/
+[HIVE_4_0_0_B_1_CL]: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12353351=Text=12310843
 [HIVE_4_0_0_A_2_CL]: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351489=Html=12310843
 [HIVE_3_1_3_CL]: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346277=Html=12310843
 [HIVE_4_0_0_A_1_CL]: 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12351399=Html=12310843

svn commit: r1083919 - in /websites/production/hive/content/javadocs/r4.0.0-beta-1: ./ api/ api/org/ api/org/apache/ api/org/apache/hadoop/ api/org/apache/hadoop/fs/ api/org/apache/hadoop/fs/class-use

[zabetak]

Author: zabetak
Date: Mon Aug 14 18:20:29 2023
New Revision: 1083919

Log:
Hive 4.0.0-beta-1 release


[This commit notification would consist of 7832 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r63446 - /dev/hive/hive-4.0.0-beta-1/ /release/hive/hive-4.0.0-beta-1/

[zabetak]

Author: zabetak
Date: Mon Aug 14 13:16:25 2023
New Revision: 63446

Log:
Move hive-4.0.0-beta-1 release from dev to release

Added:
release/hive/hive-4.0.0-beta-1/
  - copied from r63445, dev/hive/hive-4.0.0-beta-1/
Removed:
dev/hive/hive-4.0.0-beta-1/

svn commit: r63445 - /dev/hive/hive-4.0.0-beta-1/

[zabetak]

Author: zabetak
Date: Mon Aug 14 12:50:51 2023
New Revision: 63445

Log:
Hive 4.0.0-beta-1 release

Added:
dev/hive/hive-4.0.0-beta-1/
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz   (with 
props)
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.asc
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.sha256
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz   (with 
props)
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.asc
dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.sha256

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz
==
Binary file - no diff available.

Propchange: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.asc
==
--- dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.asc (added)
+++ dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.asc Mon Aug 
14 12:50:51 2023
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEEBHSVd/2TRnS5zUXF13wzg/GSdXAFAmTQ17MACgkQ13wzg/GS
+dXDRIg//dCY71SrvDNWHNjYhJPb8AaUfYKwgB0Z7T/Eubbz22KLKzg80ArjPTndE
+xAA13r/mkqT+4wsmvh2RAD+pMfNvmDcg9DytyeXiJw0aaGhx2afwon2UNYNBpRCN
+YYbVbKoyrjuHAoPixgizc5Jq6+QqLrlIHNdQEe9LBWtCZiKEUO37yU/z/1r1Nd4W
+Tjc55aSsXv5ICf0aVFdX4fqhjMho9hsuxM3J9ElNe/LOwvNJHqx7S5R9grUkNP39
+0P0kJvCPdxcra8o5Cpa853OtPQUDihHGXw4Zh9IAoYfpOVF1o1pKAIH6lXvzRcHc
+D07TVNyaGKuTbicoThMiPNpQzvd8wnifaPLafb6d+9Rd+QUByRVCne1IesHSLAS/
+QlBbPRzOm0l2UeQnF3MX3GuOTFXWQIy1QqBfji0D/quw7qpvNJFdqdDFiBidwdIx
+Yaw0I1t3F6P6Rd9VwlPR1UiVM3kYDzTKbtRQvuDP7t/c4I+3HbYvHdfWGH5mqMnx
+BObx/zpjU3Af3fkiaGaM0UxM9i5TtgrdOba73jh8Iio6VX3uFPUgOtNIYBPwMP4l
+H4ZoqDBtxbkVMysR89tzPcQEgI4r0onaojfaVGOnuXwyBntHaCG6ezQcFMvkqbBW
+H2ZgxjCMQt6rpf9zlpxBZGbUmvvInyciuEnzi05WFms2R4QRM78=
+=ixAN
+-END PGP SIGNATURE-

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.sha256
==
--- dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.sha256 
(added)
+++ dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-bin.tar.gz.sha256 Mon 
Aug 14 12:50:51 2023
@@ -0,0 +1 @@
+4114d8e9a523562c77237a8751dec9ed1bcbf6ccbe2e178d72f356ca4e65d466  
apache-hive-4.0.0-beta-1-bin.tar.gz

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz
==
Binary file - no diff available.

Propchange: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.asc
==
--- dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.asc (added)
+++ dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.asc Mon Aug 
14 12:50:51 2023
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEEBHSVd/2TRnS5zUXF13wzg/GSdXAFAmTQ16gACgkQ13wzg/GS
+dXAigxAAtRJZecmPQJcbJ0jfm/Ct+I2Rr+boRCBeMfZlpHvHqCNAUoXT4TPw6jdf
+/i2Hrogy+W7ekwv8i41hhdorTMSLn57ON5eEgdHMJ6eK/hh5sAUOcx3/FIRj1/tg
+C97bvnw2oPu4uyHX0e5iahHON/g/351lOHH4qG7AB0XcPjn1MegVx6414i2VG89y
+MiQhdk9lRdRm4pu5C4nF/fzT3hffmmK/dXiDN+Zio2HY/o9ZPSC6s9lG6b6f4eHu
++5JNJUm9eoLEyuA8FJIqS/xCzkmjKHbbvFO0+HUaJ1Qn/WepkAKLYNFa3mOF3AId
+Vd9ERR+YrpUT5LGti+UTkGB58CcLjmA/eFOeLFd/BmL3mt8D2awmFphxWafav5mj
+zBm8dO28RtCNzNCG3GPpMHb/B3+4cBwARsHDD3bTZgs1sET0//ewI2bqkN/Zps+R
+Q3C+Md2dJ5ihN5yf9ZjUfYnVpO//RpS4FuAt8RCz0WCms464tZgDAOA9Tk8m2pFZ
+arRsPwYAzr3kno2Rvd9ZJ8LgjlgJufbVgaHNMypey95cGGbbLquk62s8H5jsj1LZ
+AkqlGC85bhDEmvR9jzsqjUgzGg0xb9Bwyg2PKn/feDfowBmgM8KaABKaMIrBLl9D
+W1PcISliqSXvV5LZsIyKsescy4iiX9wbyHD94Af+dNAuoGewUlo=
+=sPFM
+-END PGP SIGNATURE-

Added: dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.sha256
==
--- dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.sha256 
(added)
+++ dev/hive/hive-4.0.0-beta-1/apache-hive-4.0.0-beta-1-src.tar.gz.sha256 Mon 
Aug 14 12:50:51 2023
@@ -0,0 +1 @@
+8d157f4dcb9af5e48e51206a4046d1c11414fbc39583c84be31d609606136209  
apache-hive-4.0.0-beta-1-src.tar.gz

[hive] annotated tag release-4.0.0-beta-1-rc0 deleted (was 429d5b0e6dc)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to annotated tag release-4.0.0-beta-1-rc0
in repository https://gitbox.apache.org/repos/asf/hive.git


*** WARNING: tag release-4.0.0-beta-1-rc0 was deleted! ***

   tag was  429d5b0e6dc

The revisions that were on this annotated tag are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[hive] annotated tag rel/release-4.0.0-beta-1 updated (d2310944e41 -> 72977c9c78b)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to annotated tag rel/release-4.0.0-beta-1
in repository https://gitbox.apache.org/repos/asf/hive.git


*** WARNING: tag rel/release-4.0.0-beta-1 was modified! ***

from d2310944e41 (commit)
  to 72977c9c78b (tag)
 tagging d2310944e412b577a39687c7968b2e93eede8433 (commit)
  by Stamatis Zampetakis
  on Mon Aug 14 15:06:16 2023 +0300

- Log -
Hive 4.0.0-beta-1 release
---


No new revisions were added by this update.

Summary of changes:

[hive] annotated tag release-4.0.0-beta-1-rc0 updated (d2310944e41 -> 429d5b0e6dc)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to annotated tag release-4.0.0-beta-1-rc0
in repository https://gitbox.apache.org/repos/asf/hive.git


*** WARNING: tag release-4.0.0-beta-1-rc0 was modified! ***

from d2310944e41 (commit)
  to 429d5b0e6dc (tag)
 tagging d2310944e412b577a39687c7968b2e93eede8433 (commit)
  by Stamatis Zampetakis
  on Mon Aug 7 16:51:29 2023 +0300

- Log -
Apache Hive 4.0.0-beta-1-RC0 release
---


No new revisions were added by this update.

Summary of changes:

[hive] 01/02: Set version to 4.0.0-beta-1

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch branch-4.0.0-beta-1
in repository https://gitbox.apache.org/repos/asf/hive.git

commit a4ce808ee93cc779716cfce530a4d46f431d30b5
Author: Stamatis Zampetakis 
AuthorDate: Mon Aug 7 13:40:35 2023 +0300

Set version to 4.0.0-beta-1
---
 accumulo-handler/pom.xml  | 2 +-
 beeline/pom.xml   | 2 +-
 classification/pom.xml| 2 +-
 cli/pom.xml   | 2 +-
 common/pom.xml| 2 +-
 contrib/pom.xml   | 2 +-
 druid-handler/pom.xml | 2 +-
 hbase-handler/pom.xml | 2 +-
 hcatalog/core/pom.xml | 2 +-
 hcatalog/hcatalog-pig-adapter/pom.xml | 4 ++--
 hcatalog/pom.xml  | 4 ++--
 hcatalog/server-extensions/pom.xml| 2 +-
 hcatalog/webhcat/java-client/pom.xml  | 2 +-
 hcatalog/webhcat/svr/pom.xml  | 2 +-
 hplsql/pom.xml| 2 +-
 iceberg/iceberg-catalog/pom.xml   | 2 +-
 iceberg/iceberg-handler/pom.xml   | 2 +-
 iceberg/iceberg-shading/pom.xml   | 2 +-
 iceberg/patched-iceberg-api/pom.xml   | 2 +-
 iceberg/patched-iceberg-core/pom.xml  | 2 +-
 iceberg/pom.xml   | 4 ++--
 itests/custom-serde/pom.xml   | 2 +-
 itests/custom-udfs/pom.xml| 2 +-
 itests/custom-udfs/udf-classloader-udf1/pom.xml   | 2 +-
 itests/custom-udfs/udf-classloader-udf2/pom.xml   | 2 +-
 itests/custom-udfs/udf-classloader-util/pom.xml   | 2 +-
 itests/custom-udfs/udf-vectorized-badexample/pom.xml  | 2 +-
 itests/hcatalog-unit/pom.xml  | 2 +-
 itests/hive-blobstore/pom.xml | 2 +-
 itests/hive-jmh/pom.xml   | 2 +-
 itests/hive-minikdc/pom.xml   | 2 +-
 itests/hive-unit-hadoop2/pom.xml  | 2 +-
 itests/hive-unit/pom.xml  | 2 +-
 itests/pom.xml| 2 +-
 itests/qtest-accumulo/pom.xml | 2 +-
 itests/qtest-druid/pom.xml| 2 +-
 itests/qtest-iceberg/pom.xml  | 2 +-
 itests/qtest-kudu/pom.xml | 2 +-
 itests/qtest/pom.xml  | 2 +-
 itests/test-serde/pom.xml | 2 +-
 itests/util/pom.xml   | 2 +-
 jdbc-handler/pom.xml  | 2 +-
 jdbc/pom.xml  | 2 +-
 kafka-handler/pom.xml | 2 +-
 kudu-handler/pom.xml  | 2 +-
 llap-client/pom.xml   | 2 +-
 llap-common/pom.xml   | 2 +-
 llap-ext-client/pom.xml   | 2 +-
 llap-server/pom.xml   | 2 +-
 llap-tez/pom.xml  | 2 +-
 metastore/pom.xml | 2 +-
 packaging/pom.xml | 2 +-
 parser/pom.xml| 2 +-
 pom.xml   | 6 +++---
 ql/pom.xml| 2 +-
 serde/pom.xml | 2 +-
 service-rpc/pom.xml   | 2 +-
 service/pom.xml   | 2 +-
 shims/0.23/pom.xml| 2 +-
 shims/aggregator/pom.xml  | 2 +-
 shims/common/pom.xml  | 2 +-
 shims/pom.xml | 2 +-
 standalone-metastore/metastore-common/pom.xml

[hive] 02/02: Add release notes for 4.0.0-beta-1

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch branch-4.0.0-beta-1
in repository https://gitbox.apache.org/repos/asf/hive.git

commit d2310944e412b577a39687c7968b2e93eede8433
Author: Stamatis Zampetakis 
AuthorDate: Mon Aug 7 14:01:08 2023 +0300

Add release notes for 4.0.0-beta-1
---
 RELEASE_NOTES.txt | 332 ++
 1 file changed, 332 insertions(+)

diff --git a/RELEASE_NOTES.txt b/RELEASE_NOTES.txt
index c64f5cf07b3..344e2246da0 100644
--- a/RELEASE_NOTES.txt
+++ b/RELEASE_NOTES.txt
@@ -1,3 +1,335 @@
+Release Notes - Hive - Version 4.0.0-beta-1
+
+** Sub-task
+* [HIVE-23693] - HiveKafkaProducerTest.resumeTransaction is flaky
+* [HIVE-25205] - Reduce overhead of adding write notification log during 
batch loading of partition..
+* [HIVE-26400] - Provide docker images for Hive
+* [HIVE-26484] - Introduce gRPC Proto Definition
+* [HIVE-26585] - Revamp Apache Hive website
+* [HIVE-26586] - Create a Basic Apache Hive website with HUGO
+* [HIVE-26587] - Create a custom hive-theme in HUGO
+* [HIVE-26588] - Move the markdown files from old website and add content 
to the homepage.
+* [HIVE-26589] - Add GitHub Action workflow to build and deploy site to 
gh-pages
+* [HIVE-26590] - Hive website: Fix JS, broken links, add license text and 
section header, and updates images
+* [HIVE-26655] - VectorUDAFBloomFilterMerge should take care of safe batch 
handling when working in parallel
+* [HIVE-26694] - Populate file row position information during vectorized 
Iceberg reads
+* [HIVE-26696] - Implement row iterator for VectorizedRowBatch
+* [HIVE-26697] - Create vectorized Hive DeleteFilter for Iceberg tables
+* [HIVE-26716] - Query based Rebalance compaction on full acid tables
+* [HIVE-26719] - Ability to set number of buckets manually
+* [HIVE-26744] - Add array_max and array_min UDFs 
+* [HIVE-26754] - Implement array_distinct UDF to return an array after 
removing duplicates in it
+* [HIVE-26757] - Add sfs+ofs support
+* [HIVE-26774] - Implement array_slice UDF to get the subset of elements 
from an array (subarray)
+* [HIVE-26848] - Ozone: LOAD data fails in case of move across buckets
+* [HIVE-26889] - Implement array_join udf to concatenate the elements of 
an array with a specified delimiter
+* [HIVE-26968] - Wrong results when shared work optimizer merges TS 
operator with different DPP edges
+* [HIVE-27019] - Split Cleaner into separate manageable modular entities
+* [HIVE-27020] - Implement a separate handler to handle aborted 
transaction cleanup
+* [HIVE-27031] - Iceberg: Implement Copy-On-Write for Delete Queries
+* [HIVE-27081] - Revert HIVE-26717 and HIVE-26718
+* [HIVE-27112] - implement array_except UDF in Hive
+* [HIVE-27118] - implement array_intersect UDF in Hive
+* [HIVE-27197] - Iceberg:  Support Iceberg version travel by reference 
name 
+* [HIVE-27234] - Iceberg:  CREATE BRANCH SQL implementation
+* [HIVE-27235] - Iceberg: CREATE TAG SQL implementation
+* [HIVE-27236] - Iceberg: DROP BRANCH SQL implementation
+* [HIVE-27243] - Iceberg: Implement Load data via temp table
+* [HIVE-27244] - Iceberg: Implement LOAD data for unpartitioned table via 
Append API
+* [HIVE-27245] - Iceberg: Implement LOAD data for partitioned tables via 
Append API
+* [HIVE-27302] - Iceberg: Suport write to iceberg branch
+* [HIVE-27329] - Document usage of the image
+* [HIVE-27332] - Add retry backoff mechanism for abort cleanup
+* [HIVE-27440] - Improve data connector cache
+* [HIVE-27482] - Minor issues in array_intersect udf
+* [HIVE-27503] - Iceberg: Support query iceberg tag
+
+** Bug
+* [HIVE-22173] - Query with multiple lateral views hangs during compilation
+* [HIVE-22383] - `alterPartitions` is invoked twice during dynamic 
partition load causing runtime delay
+* [HIVE-22813] - Hive query fails if table location is in remote EZ and 
it's readonly
+* [HIVE-23935] - Fetching primaryKey through beeline fails with NPE
+* [HIVE-24404] - Hive getUserName close db makes client operations lost 
metaStoreClient connection
+* [HIVE-25707] - SchemaTool may leave the metastore in-between upgrade 
steps
+* [HIVE-26035] - Explore moving to directsql for ObjectStore::addPartitions
+* [HIVE-26065] - Fix incorrect metric active_calls_alter_partitions and 
active_calls_alter_partition
+* [HIVE-26522] - Test for HIVE-22033 and backport to 3.1 and 2.3
+* [HIVE-26575] - Fix missing Copy Serde Description when Alter Table's 
Serde Description
+* [HIVE-26628] - Iceberg table is created when running explain ctas command
+* [HIVE-26632] - Update DelegationTokenSecretManager current key ID to 
prevent erroneous database updates.
+* [HIVE-26658] - INT64 Parquet timestamps cannot be mapped to most Hive 
numeric

[hive] branch branch-4.0.0-beta-1 created (now d2310944e41)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch branch-4.0.0-beta-1
in repository https://gitbox.apache.org/repos/asf/hive.git


  at d2310944e41 Add release notes for 4.0.0-beta-1

This branch includes the following new commits:

 new a4ce808ee93 Set version to 4.0.0-beta-1
 new d2310944e41 Add release notes for 4.0.0-beta-1

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[hive] branch master updated (3bc62cbc2d4 -> 8ea1d78d0dc)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from 3bc62cbc2d4 HIVE-27454: Iceberg: Allow expire snapshots by id. 
(#4441). (Ayush Saxena, reviewed by Denys Kuzmenko)
 new 7139d42e8d8 HIVE-27488: Incorrect escaping of quotes in HPL/SQL 
literals (Dayakar M reviewed by Attila Turoczy, Stamatis Zampetakis)
 new 8ea1d78d0dc HIVE-23693: HIVE-27502: Enable HiveKafkaProducerTest 
(Akshat Mathur reviewed by Attila Turoczy, Stamatis Zampetakis)

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/hive/hplsql/Utils.java| 12 
 .../java/org/apache/hive/hplsql/TestUtils.java | 22 +-
 hplsql/src/test/queries/local/dbms_output.sql  |  6 ++
 hplsql/src/test/results/local/dbms_output.out.txt  |  6 ++
 .../hadoop/hive/kafka/HiveKafkaProducerTest.java   |  1 -
 5 files changed, 33 insertions(+), 14 deletions(-)
 copy ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java => 
hplsql/src/test/java/org/apache/hive/hplsql/TestUtils.java (63%)

[hive] 01/02: HIVE-27488: Incorrect escaping of quotes in HPL/SQL literals (Dayakar M reviewed by Attila Turoczy, Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 7139d42e8d8c75ff0700274b81f4ddf3e90b905c
Author: mdayakar 
AuthorDate: Mon Jul 10 18:26:05 2023 +0530

HIVE-27488: Incorrect escaping of quotes in HPL/SQL literals (Dayakar M 
reviewed by Attila Turoczy, Stamatis Zampetakis)

Closes #4476
---
 .../main/java/org/apache/hive/hplsql/Utils.java| 12 +---
 .../java/org/apache/hive/hplsql/TestUtils.java | 35 ++
 hplsql/src/test/queries/local/dbms_output.sql  |  6 
 hplsql/src/test/results/local/dbms_output.out.txt  |  6 
 4 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/hplsql/src/main/java/org/apache/hive/hplsql/Utils.java 
b/hplsql/src/main/java/org/apache/hive/hplsql/Utils.java
index 2a86f55a3e9..c7d0f202f11 100644
--- a/hplsql/src/main/java/org/apache/hive/hplsql/Utils.java
+++ b/hplsql/src/main/java/org/apache/hive/hplsql/Utils.java
@@ -32,18 +32,22 @@ public class Utils {
  }
  
  int len = s.length();
- StringBuilder s2 = new StringBuilder(len);  
+ StringBuilder s2 = new StringBuilder(len);
+ boolean isEscape = true;
  
  for (int i = 0; i < len; i++) {
char ch = s.charAt(i);
char ch2 = (i < len - 1) ? s.charAt(i+1) : 0;
  
-   if((i == 0 || i == len -1) && (ch == '\'' || ch == '"'))
+   if((i == 0 || i == len - 1) && (ch == '\'' || ch == '"'))
  continue;
else
-   // \' and '' escape sequences
-   if((ch == '\\' && ch2 == '\'') || (ch == '\'' && ch2 == '\''))
+   // \' and '' escape sequences and include ' if last two characters 
are ''
+   if((ch == '\\' && ch2 == '\'') || (ch == '\'' && ch2 == '\'' && 
isEscape && i != len - 2)) {
+ isEscape = false;
  continue;
+   }
+   isEscape = true;

s2.append(ch);  
  }
diff --git a/hplsql/src/test/java/org/apache/hive/hplsql/TestUtils.java 
b/hplsql/src/test/java/org/apache/hive/hplsql/TestUtils.java
new file mode 100644
index 000..1060f6a5e66
--- /dev/null
+++ b/hplsql/src/test/java/org/apache/hive/hplsql/TestUtils.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.hplsql;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestUtils {
+
+  @Test
+  public void testUnquoteStringRemovesOneQuoteWhenTwoConsecutive() {
+Assert.assertEquals("a'a", Utils.unquoteString("'a''a'"));
+Assert.assertEquals("'a", Utils.unquoteString("'''a'"));
+Assert.assertEquals("a'", Utils.unquoteString("'a'''"));
+Assert.assertEquals("''aa", Utils.unquoteString("'aa'"));
+Assert.assertEquals("a''a", Utils.unquoteString("'aa'"));
+Assert.assertEquals("aa''", Utils.unquoteString("'aa'"));
+  }
+}
diff --git a/hplsql/src/test/queries/local/dbms_output.sql 
b/hplsql/src/test/queries/local/dbms_output.sql
index 37d73133ca9..d8855b4336f 100644
--- a/hplsql/src/test/queries/local/dbms_output.sql
+++ b/hplsql/src/test/queries/local/dbms_output.sql
@@ -3,4 +3,10 @@ DECLARE
 BEGIN
   DBMS_OUTPUT.PUT_LINE('Hello, world!');
   DBMS_OUTPUT.PUT_LINE(str);
+  DBMS_OUTPUT.PUT_LINE('a''a');
+  DBMS_OUTPUT.PUT_LINE('''a');
+  DBMS_OUTPUT.PUT_LINE('a''');
+  DBMS_OUTPUT.PUT_LINE('aa');
+  DBMS_OUTPUT.PUT_LINE('aa');
+  DBMS_OUTPUT.PUT_LINE('aa');
 END;
diff --git a/hplsql/src/test/results/local/dbms_output.out.txt 
b/hplsql/src/test/results/local/dbms_output.out.txt
index b6ed0e0eb7f..d68f919a93e 100644
--- a/hplsql/src/test/results/local/dbms_output.out.txt
+++ b/hplsql/src/test/results/local/dbms_output.out.txt
@@ -1,3 +1,9 @@
 Ln:2 DECLARE str VARCHAR = 'Hello, world!'
 Hello, world!
 Hello, world!
+a'a
+'a
+a'
+''aa
+a''a
+aa''

[hive] 02/02: HIVE-23693: HIVE-27502: Enable HiveKafkaProducerTest (Akshat Mathur reviewed by Attila Turoczy, Stamatis Zampetakis)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git

commit 8ea1d78d0dc18fec4305dd0d0a9e341ecdcaef1e
Author: akshat0395 
AuthorDate: Thu Jul 13 16:59:43 2023 +0530

HIVE-23693: HIVE-27502: Enable HiveKafkaProducerTest (Akshat Mathur 
reviewed by Attila Turoczy, Stamatis Zampetakis)

Closes #4483
---
 .../src/test/org/apache/hadoop/hive/kafka/HiveKafkaProducerTest.java | 1 -
 1 file changed, 1 deletion(-)

diff --git 
a/kafka-handler/src/test/org/apache/hadoop/hive/kafka/HiveKafkaProducerTest.java
 
b/kafka-handler/src/test/org/apache/hadoop/hive/kafka/HiveKafkaProducerTest.java
index cf08e2f30ac..8c9ed5f99b1 100644
--- 
a/kafka-handler/src/test/org/apache/hadoop/hive/kafka/HiveKafkaProducerTest.java
+++ 
b/kafka-handler/src/test/org/apache/hadoop/hive/kafka/HiveKafkaProducerTest.java
@@ -53,7 +53,6 @@ import java.util.stream.IntStream;
 /**
  * Test class for Hive Kafka Producer.
  */
-@org.junit.Ignore("flaky HIVE-23693")
 @SuppressWarnings("unchecked") public class HiveKafkaProducerTest {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(HiveKafkaProducerTest.class);

[calcite-avatica] branch main updated: [CALCITE-5804] Upgrade jackson version from 2.14.1 to 2.15.2

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/calcite-avatica.git


The following commit(s) were added to refs/heads/main by this push:
 new c14bcdda6 [CALCITE-5804] Upgrade jackson version from 2.14.1 to 2.15.2
c14bcdda6 is described below

commit c14bcdda68f2abf8ca607a5176f19ec53d92212e
Author: Richard Antal 
AuthorDate: Fri Jun 30 14:30:05 2023 +0200

[CALCITE-5804] Upgrade jackson version from 2.14.1 to 2.15.2

Close apache/calcite-avatica#223
---
 gradle.properties | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gradle.properties b/gradle.properties
index 481275002..e31fd4c3e 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -64,7 +64,7 @@ hsqldb.version=2.4.1
 h2.version=1.4.197
 httpclient5.version=5.1.3
 httpcore5.version=5.1.3
-jackson.version=2.14.1
+jackson.version=2.15.2
 jcip-annotations.version=1.0-1
 jcommander.version=1.72
 jetty.version=9.4.44.v20210927

[hive] branch master updated (f4edd0e4210 -> 5e46e80bc7d)

[zabetak]

This is an automated email from the ASF dual-hosted git repository.

zabetak pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git


from f4edd0e4210 HIVE-27475: Downgrade Kafka version to 2.5.0 to stabilise 
Kafka handler (Akshat Mathur reviewed by Kokila N, Attila Turoczy, Stamatis 
Zmpetakis)
 add 1b5f2aae0a5 HIVE-27477: Upgrade maven plugin versions (Raghav Aggarwal 
reviewed by Attila Turoczy, Stamatis Zampetakis)
 add 5e46e80bc7d HIVE-27468: Restore original license in 
PriorityBlockingDeque (Stamatis Zampetakis reviewed by Attila Turoczy, Ayush 
Saxena)

No new revisions were added by this update.

Summary of changes:
 .../llap/daemon/impl/PriorityBlockingDeque.java| 53 +++---
 pom.xml| 11 +++--
 standalone-metastore/pom.xml   |  4 +-
 storage-api/pom.xml|  4 +-
 upgrade-acid/pom.xml   |  2 +-
 5 files changed, 47 insertions(+), 27 deletions(-)