[hive] branch master updated: HIVE-27353: Show saved snapshot of materialized view source tables (Krisztian Kasa, reviewed by Laszlo Vegh)
This is an automated email from the ASF dual-hosted git repository.
krisztiankasa pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new f5e69dc38d7 HIVE-27353: Show saved snapshot of materialized view
source tables (Krisztian Kasa, reviewed by Laszlo Vegh)
f5e69dc38d7 is described below
commit f5e69dc38d7ff26c70be19adc9d1a3ae90dc4cf2
Author: Krisztian Kasa
AuthorDate: Tue Jun 6 16:53:32 2023 +0200
HIVE-27353: Show saved snapshot of materialized view source tables
(Krisztian Kasa, reviewed by Laszlo Vegh)
---
.../queries/positive/mv_iceberg_partitioned_orc.q | 1 +
.../queries/positive/mv_iceberg_partitioned_orc2.q | 1 +
.../positive/mv_iceberg_partitioned_orc.q.out | 8 +--
.../positive/mv_iceberg_partitioned_orc2.q.out | 8 +--
.../positive/show_iceberg_materialized_views.q.out | 16 ++---
.../desc/formatter/TextDescTableFormatter.java | 69 +++---
.../results/clientpositive/llap/masking_mv.q.out | 8 +--
.../clientpositive/llap/masking_mv_by_text.q.out | 4 +-
.../llap/materialized_view_cluster.q.out | 12 ++--
.../llap/materialized_view_create.q.out| 8 +--
.../llap/materialized_view_create_acid.q.out | 4 +-
.../llap/materialized_view_create_rewrite_4.q.out | 36 +--
.../llap/materialized_view_create_rewrite_5.q.out | 6 +-
...erialized_view_create_rewrite_time_window.q.out | 18 +++---
...ialized_view_create_rewrite_time_window_2.q.out | 18 +++---
.../llap/materialized_view_describe.q.out | 16 ++---
.../llap/materialized_view_distribute_sort.q.out | 14 ++---
.../llap/materialized_view_parquet.q.out | 4 +-
.../llap/materialized_view_partition_cluster.q.out | 12 ++--
.../llap/materialized_view_partitioned.q.out | 4 +-
.../llap/show_materialized_views.q.out | 16 ++---
21 files changed, 157 insertions(+), 126 deletions(-)
diff --git
a/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc.q
b/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc.q
index ad01a992bff..54fa59369ad 100644
---
a/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc.q
+++
b/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc.q
@@ -2,6 +2,7 @@
--! qt:replace:/(\s+uuid\s+)\S+(\s*)/$1#Masked#$2/
-- Mask random snapshot id
--! qt:replace:/(\s+current-snapshot-id\s+)\d+(\s*)/$1#SnapshotId#/
+--! qt:replace:/(.*snapshotId=)\S+(\}.*)/$1#SnapshotId#$2/
-- Mask added file size
--! qt:replace:/(\S\"added-files-size\\\":\\\")(\d+)(\\\")/$1#Masked#$3/
-- Mask total file size
diff --git
a/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc2.q
b/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc2.q
index 7d0a6efaff1..f525b6b97be 100644
---
a/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc2.q
+++
b/iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_partitioned_orc2.q
@@ -1,6 +1,7 @@
-- MV data is stored by partitioned iceberg with partition spec
--! qt:replace:/(\s+uuid\s+)\S+(\s*)/$1#Masked#$2/
--! qt:replace:/(\s+current-snapshot-id\s+)\d+(\s*)/$1#SnapshotId#/
+--! qt:replace:/(.*snapshotId=)\S+(\}.*)/$1#SnapshotId#$2/
-- Mask added file size
--! qt:replace:/(\S\"added-files-size\\\":\\\")(\d+)(\\\")/$1#Masked#$3/
-- Mask total file size
diff --git
a/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_partitioned_orc.q.out
b/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_partitioned_orc.q.out
index f4dbe3b5650..c91810c0971 100644
---
a/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_partitioned_orc.q.out
+++
b/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_partitioned_orc.q.out
@@ -95,8 +95,8 @@ Rewrite Enabled: Yes
Outdated for Rewriting:Unknown
# Materialized View Source table information
-Table name I/U/D since last rebuild
-hive.default.tbl_ice 0/0/0
+Table name Snapshot
+default.tbl_iceSnapshotContext{snapshotId=#SnapshotId#}
PREHOOK: query: select * from mat1
PREHOOK: type: QUERY
PREHOOK: Input: default@mat1
@@ -183,5 +183,5 @@ Rewrite Enabled:Yes
Outdated for Rewriting:Unknown
# Materialized View Source table information
-Table name I/U/D since last rebuild
-hive.default.tbl_ice 0/0/0
+Table name Snapshot
+default.tbl_iceSnapshotContext{snapshotId=#SnapshotId#}
diff --git
a/iceberg/iceberg-handler/src/test/results/positive/mv_iceberg_partitioned_orc2.q.out
[hive] branch master updated (fd80b1502eb -> 03fb1e64ae2)
This is an automated email from the ASF dual-hosted git repository. ngangam pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from fd80b1502eb HIVE-27413: Bump jettison from 1.5.3 to 1.5.4 (#4140). (dependabot, Reviewed by Ayush Saxena, Akshat Mathur) add 03fb1e64ae2 HIVE-27398: SHOW CREATE TABLE doesn't output backticks for CLUSTERED by Col names (#4381) No new revisions were added by this update. Summary of changes: .../apache/hadoop/hive/ql/exec/DDLPlanUtils.java | 2 +- .../queries/clientpositive/show_create_table.q | 13 ++ .../clientpositive/llap/show_create_table.q.out| 53 ++ .../llap/show_create_table_alter.q.out | 10 ++-- .../clientpositive/llap/table_storage.q.out| 6 +-- 5 files changed, 75 insertions(+), 9 deletions(-)
[hive] branch dependabot/maven/org.codehaus.jettison-jettison-1.5.4 deleted (was 8b821681cb4)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.codehaus.jettison-jettison-1.5.4 in repository https://gitbox.apache.org/repos/asf/hive.git was 8b821681cb4 Bump jettison from 1.5.3 to 1.5.4 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[hive] branch master updated (75d14a0cea1 -> fd80b1502eb)
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from 75d14a0cea1 HIVE-27401 : Upgrade arrow to 12.0.0 (#4383). (Aman Raj, reviewed by Ayush Saxena, Attila Turoczy, Akshat Mathur, Zsolt Miskolczi) add fd80b1502eb HIVE-27413: Bump jettison from 1.5.3 to 1.5.4 (#4140). (dependabot, Reviewed by Ayush Saxena, Akshat Mathur) No new revisions were added by this update. Summary of changes:
[hive] branch master updated (c4fb63e92ea -> 75d14a0cea1)
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from c4fb63e92ea HIVE-27393: Upgrade ant to 1.10.13 to fix CVE's (#4376). (Raghav Aggarwal, reviewed by Ayush Saxena, Akshat Mathur) add 75d14a0cea1 HIVE-27401 : Upgrade arrow to 12.0.0 (#4383). (Aman Raj, reviewed by Ayush Saxena, Attila Turoczy, Akshat Mathur, Zsolt Miskolczi) No new revisions were added by this update. Summary of changes: pom.xml| 4 ++-- ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
[hive] branch master updated (838c7d7db57 -> c4fb63e92ea)
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from 838c7d7db57 HIVE-27386: Avoid duplicate audit log in cleanupHandlerContext (#4366). (Wechar Yu, reviewed by Ayush Saxena, Zhihua Deng, Akshat Mathur, Attila Turoczy) add c4fb63e92ea HIVE-27393: Upgrade ant to 1.10.13 to fix CVE's (#4376). (Raghav Aggarwal, reviewed by Ayush Saxena, Akshat Mathur) No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[hive] branch master updated: HIVE-27386: Avoid duplicate audit log in cleanupHandlerContext (#4366). (Wechar Yu, reviewed by Ayush Saxena, Zhihua Deng, Akshat Mathur, Attila Turoczy)
This is an automated email from the ASF dual-hosted git repository.
ayushsaxena pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 838c7d7db57 HIVE-27386: Avoid duplicate audit log in
cleanupHandlerContext (#4366). (Wechar Yu, reviewed by Ayush Saxena, Zhihua
Deng, Akshat Mathur, Attila Turoczy)
838c7d7db57 is described below
commit 838c7d7db57002d1cc408b90b54889bc1364b5b3
Author: Wechar Yu
AuthorDate: Tue Jun 6 15:35:47 2023 +0800
HIVE-27386: Avoid duplicate audit log in cleanupHandlerContext (#4366).
(Wechar Yu, reviewed by Ayush Saxena, Zhihua Deng, Akshat Mathur, Attila
Turoczy)
---
.../main/java/org/apache/hadoop/hive/metastore/HMSHandler.java | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
index 66303f12ef8..206d690b79d 100644
---
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
+++
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java
@@ -173,17 +173,19 @@ public class HMSHandler extends FacebookBase implements
IHMSHandler {
}
static void cleanupHandlerContext() {
+AtomicBoolean cleanedRawStore = new AtomicBoolean(false);
try {
HMSHandlerContext.getRawStore().ifPresent(rs -> {
logAndAudit("Cleaning up thread local RawStore...");
rs.shutdown();
+cleanedRawStore.set(true);
});
} finally {
- HMSHandlerContext.getHMSHandler().ifPresent(handler -> {
-handler.notifyMetaListenersOnShutDown();
- });
+
HMSHandlerContext.getHMSHandler().ifPresent(HMSHandler::notifyMetaListenersOnShutDown);
+ if (cleanedRawStore.get()) {
+logAndAudit("Done cleaning up thread local RawStore");
+ }
HMSHandlerContext.clear();
- logAndAudit("Done cleaning up thread local RawStore");
}
}
[hive] branch master updated: HIVE-27408: Parquet file opened for reading stats is never closed (#4387). (Sourabh Badhya, reviewed by Ayush Saxena, Attila Turoczy, Akshat Mathur)
This is an automated email from the ASF dual-hosted git repository. ayushsaxena pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/master by this push: new e5aebeec8e9 HIVE-27408: Parquet file opened for reading stats is never closed (#4387). (Sourabh Badhya, reviewed by Ayush Saxena, Attila Turoczy, Akshat Mathur) e5aebeec8e9 is described below commit e5aebeec8e9060179c995246cbd40983aaa5f8cf Author: Sourabh Badhya <42867455+sourabhbad...@users.noreply.github.com> AuthorDate: Tue Jun 6 13:00:40 2023 +0530 HIVE-27408: Parquet file opened for reading stats is never closed (#4387). (Sourabh Badhya, reviewed by Ayush Saxena, Attila Turoczy, Akshat Mathur) --- .../hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java b/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java index 5d4131a9615..d64766d62c3 100644 --- a/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java +++ b/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java @@ -140,8 +140,7 @@ public class ParquetRecordWriterWrapper implements RecordWriter
[hive] branch branch-3 updated: HIVE-27378: Backport HIVE-19133 : HS2 WebUI phase-wise performance metrics not showing correctly (Bharathkrishna Guruvayoor Murali reviewed by Zoltan Haindrich, Vihang
This is an automated email from the ASF dual-hosted git repository.
sankarh pushed a commit to branch branch-3
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/branch-3 by this push:
new 63e6941abb2 HIVE-27378: Backport HIVE-19133 : HS2 WebUI phase-wise
performance metrics not showing correctly (Bharathkrishna Guruvayoor Murali
reviewed by Zoltan Haindrich, Vihang Karajgaonkar)
63e6941abb2 is described below
commit 63e6941abb20626dd273d30ba7061a88ae3cfe02
Author: Diksha628 <43694846+diksha...@users.noreply.github.com>
AuthorDate: Tue Jun 6 12:23:14 2023 +0530
HIVE-27378: Backport HIVE-19133 : HS2 WebUI phase-wise performance metrics
not showing correctly (Bharathkrishna Guruvayoor Murali reviewed by Zoltan
Haindrich, Vihang Karajgaonkar)
Signed-off-by: Sankar Hariappan
Closes (#4358)
---
.../org/apache/hadoop/hive/ql/log/PerfLogger.java | 2 --
.../apache/hive/jdbc/miniHS2/TestHs2Metrics.java | 1 -
.../operation/TestOperationLoggingAPIWithMr.java | 1 -
.../operation/TestOperationLoggingAPIWithTez.java | 1 -
.../hive/service/cli/session/TestQueryDisplay.java | 7 +++
ql/src/java/org/apache/hadoop/hive/ql/Driver.java | 22 +-
.../hive/service/cli/operation/SQLOperation.java | 11 +--
7 files changed, 21 insertions(+), 24 deletions(-)
diff --git a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
index 764a832e281..65745f211d0 100644
--- a/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
+++ b/common/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java
@@ -51,14 +51,12 @@ public class PerfLogger {
public static final String SERIALIZE_PLAN = "serializePlan";
public static final String DESERIALIZE_PLAN = "deserializePlan";
public static final String CLONE_PLAN = "clonePlan";
- public static final String TASK = "task.";
public static final String RELEASE_LOCKS = "releaseLocks";
public static final String PRUNE_LISTING = "prune-listing";
public static final String PARTITION_RETRIEVING = "partition-retrieving";
public static final String PRE_HOOK = "PreHook.";
public static final String POST_HOOK = "PostHook.";
public static final String FAILURE_HOOK = "FailureHook.";
- public static final String DRIVER_RUN = "Driver.run";
public static final String TEZ_COMPILER = "TezCompiler";
public static final String TEZ_SUBMIT_TO_RUNNING = "TezSubmitToRunningDag";
public static final String TEZ_BUILD_DAG = "TezBuildDag";
diff --git
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java
b/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java
index 0ec23e1c1e1..9686445f2b2 100644
---
a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java
+++
b/itests/hive-unit/src/test/java/org/apache/hive/jdbc/miniHS2/TestHs2Metrics.java
@@ -109,7 +109,6 @@ public class TestHs2Metrics {
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER,
"api_hs2_sql_operation_PENDING", 1);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER,
"api_hs2_sql_operation_RUNNING", 1);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.COUNTER,
"hs2_completed_sql_operation_FINISHED", 1);
-MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER,
"api_Driver.run", 1);
//but there should be no more active calls.
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.COUNTER,
"active_calls_api_semanticAnalyze", 0);
diff --git
a/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
b/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
index a6aa84629af..c7dade3874a 100644
---
a/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
+++
b/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithMr.java
@@ -59,7 +59,6 @@ public class TestOperationLoggingAPIWithMr extends
OperationLoggingAPITestBase {
expectedLogsPerformance = new String[]{
"",
"",
- "",
""
};
hiveConf = new HiveConf();
diff --git
a/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
b/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
index 388486d9702..28eeda18a1a 100644
---
a/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
+++
b/itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingAPIWithTez.java
@@ -49,7 +49,6 @@ public class TestOperationLoggingAPIWithTez extends
OperationLoggingAPITestBase
expectedLogsPerformance = new String[]{
