(hive) branch branch-4.0 updated (f4302068b6a -> 183f8cb41d3)
This is an automated email from the ASF dual-hosted git repository. dkuzmenko pushed a change to branch branch-4.0 in repository https://gitbox.apache.org/repos/asf/hive.git discard f4302068b6a Updating RELEASE_NOTES, NOTICE, README.md for 4.0.0 new 183f8cb41d3 Updating RELEASE_NOTES, NOTICE, README.md for 4.0.0 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f4302068b6a) \ N -- N -- N refs/heads/branch-4.0 (183f8cb41d3) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(hive) 01/01: Set version to 4.0.0
This is an automated email from the ASF dual-hosted git repository. dkuzmenko pushed a commit to branch branch-4.0 in repository https://gitbox.apache.org/repos/asf/hive.git commit 258d0e98218f60b8e88b8d0f9909c6e17264c828 Author: Denys Kuzmenko AuthorDate: Thu Mar 21 14:47:15 2024 +0100 Set version to 4.0.0 --- accumulo-handler/pom.xml | 2 +- beeline/pom.xml | 2 +- classification/pom.xml| 2 +- cli/pom.xml | 2 +- common/pom.xml| 2 +- contrib/pom.xml | 2 +- druid-handler/pom.xml | 2 +- hbase-handler/pom.xml | 2 +- hcatalog/core/pom.xml | 2 +- hcatalog/hcatalog-pig-adapter/pom.xml | 4 ++-- hcatalog/pom.xml | 4 ++-- hcatalog/server-extensions/pom.xml| 2 +- hcatalog/webhcat/java-client/pom.xml | 2 +- hcatalog/webhcat/svr/pom.xml | 2 +- hplsql/pom.xml| 2 +- iceberg/iceberg-catalog/pom.xml | 2 +- iceberg/iceberg-handler/pom.xml | 2 +- iceberg/iceberg-shading/pom.xml | 2 +- iceberg/patched-iceberg-api/pom.xml | 2 +- iceberg/patched-iceberg-core/pom.xml | 2 +- iceberg/pom.xml | 4 ++-- itests/custom-serde/pom.xml | 2 +- itests/custom-udfs/pom.xml| 2 +- itests/custom-udfs/udf-classloader-udf1/pom.xml | 2 +- itests/custom-udfs/udf-classloader-udf2/pom.xml | 2 +- itests/custom-udfs/udf-classloader-util/pom.xml | 2 +- itests/custom-udfs/udf-vectorized-badexample/pom.xml | 2 +- itests/hcatalog-unit/pom.xml | 2 +- itests/hive-blobstore/pom.xml | 2 +- itests/hive-jmh/pom.xml | 2 +- itests/hive-minikdc/pom.xml | 2 +- itests/hive-unit-hadoop2/pom.xml | 2 +- itests/hive-unit/pom.xml | 2 +- itests/pom.xml| 2 +- itests/qtest-accumulo/pom.xml | 2 +- itests/qtest-druid/pom.xml| 2 +- itests/qtest-iceberg/pom.xml | 2 +- itests/qtest-kudu/pom.xml | 2 +- itests/qtest/pom.xml | 2 +- itests/test-serde/pom.xml | 2 +- itests/util/pom.xml | 2 +- jdbc-handler/pom.xml | 2 +- jdbc/pom.xml | 2 +- kafka-handler/pom.xml | 2 +- kudu-handler/pom.xml | 2 +- llap-client/pom.xml | 2 +- llap-common/pom.xml | 2 +- llap-ext-client/pom.xml | 2 +- llap-server/pom.xml | 2 +- llap-tez/pom.xml | 2 +- metastore/pom.xml | 2 +- packaging/pom.xml | 2 +- parser/pom.xml| 2 +- pom.xml | 6 +++--- ql/pom.xml| 2 +- serde/pom.xml | 2 +- service-rpc/pom.xml | 2 +- service/pom.xml | 2 +- shims/0.23/pom.xml| 2 +- shims/aggregator/pom.xml | 2 +- shims/common/pom.xml | 2 +- shims/pom.xml | 2 +- standalone-metastore/metastore-common/pom.xml | 2 +-
(hive) branch branch-4.0 updated (15ad9048e91 -> 258d0e98218)
This is an automated email from the ASF dual-hosted git repository. dkuzmenko pushed a change to branch branch-4.0 in repository https://gitbox.apache.org/repos/asf/hive.git discard 15ad9048e91 Set version to 4.0.0 new 258d0e98218 Set version to 4.0.0 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (15ad9048e91) \ N -- N -- N refs/heads/branch-4.0 (258d0e98218) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: hcatalog/hcatalog-pig-adapter/pom.xml | 2 +- hcatalog/pom.xml | 2 +- pom.xml | 4 ++-- standalone-metastore/metastore-common/pom.xml | 2 +- standalone-metastore/metastore-server/pom.xml | 4 ++-- standalone-metastore/metastore-tools/metastore-benchmarks/pom.xml | 2 +- standalone-metastore/metastore-tools/pom.xml | 2 +- standalone-metastore/metastore-tools/tools-common/pom.xml | 2 +- standalone-metastore/pom.xml | 6 +++--- storage-api/pom.xml | 2 +- streaming/pom.xml | 2 +- 11 files changed, 15 insertions(+), 15 deletions(-)
(hive) branch branch-4.0 updated: Set version to 4.0.0
This is an automated email from the ASF dual-hosted git repository. dkuzmenko pushed a commit to branch branch-4.0 in repository https://gitbox.apache.org/repos/asf/hive.git The following commit(s) were added to refs/heads/branch-4.0 by this push: new 15ad9048e91 Set version to 4.0.0 15ad9048e91 is described below commit 15ad9048e914b0787a9ae2b9cb963a2b1a7ce65d Author: Denys Kuzmenko AuthorDate: Thu Mar 21 14:47:15 2024 +0100 Set version to 4.0.0 --- accumulo-handler/pom.xml | 2 +- beeline/pom.xml | 2 +- classification/pom.xml | 2 +- cli/pom.xml | 2 +- common/pom.xml | 2 +- contrib/pom.xml | 2 +- druid-handler/pom.xml| 2 +- hbase-handler/pom.xml| 2 +- hcatalog/core/pom.xml| 2 +- hcatalog/hcatalog-pig-adapter/pom.xml| 2 +- hcatalog/pom.xml | 2 +- hcatalog/server-extensions/pom.xml | 2 +- hcatalog/webhcat/java-client/pom.xml | 2 +- hcatalog/webhcat/svr/pom.xml | 2 +- hplsql/pom.xml | 2 +- iceberg/iceberg-catalog/pom.xml | 2 +- iceberg/iceberg-handler/pom.xml | 2 +- iceberg/iceberg-shading/pom.xml | 2 +- iceberg/patched-iceberg-api/pom.xml | 2 +- iceberg/patched-iceberg-core/pom.xml | 2 +- iceberg/pom.xml | 4 ++-- itests/custom-serde/pom.xml | 2 +- itests/custom-udfs/pom.xml | 2 +- itests/custom-udfs/udf-classloader-udf1/pom.xml | 2 +- itests/custom-udfs/udf-classloader-udf2/pom.xml | 2 +- itests/custom-udfs/udf-classloader-util/pom.xml | 2 +- itests/custom-udfs/udf-vectorized-badexample/pom.xml | 2 +- itests/hcatalog-unit/pom.xml | 2 +- itests/hive-blobstore/pom.xml| 2 +- itests/hive-jmh/pom.xml | 2 +- itests/hive-minikdc/pom.xml | 2 +- itests/hive-unit-hadoop2/pom.xml | 2 +- itests/hive-unit/pom.xml | 2 +- itests/pom.xml | 2 +- itests/qtest-accumulo/pom.xml| 2 +- itests/qtest-druid/pom.xml | 2 +- itests/qtest-iceberg/pom.xml | 2 +- itests/qtest-kudu/pom.xml| 2 +- itests/qtest/pom.xml | 2 +- itests/test-serde/pom.xml| 2 +- itests/util/pom.xml | 2 +- jdbc-handler/pom.xml | 2 +- jdbc/pom.xml | 2 +- kafka-handler/pom.xml| 2 +- kudu-handler/pom.xml | 2 +- llap-client/pom.xml | 2 +- llap-common/pom.xml | 2 +- llap-ext-client/pom.xml | 2 +- llap-server/pom.xml | 2 +- llap-tez/pom.xml | 2 +- metastore/pom.xml| 2 +- packaging/pom.xml| 2 +- parser/pom.xml | 2 +- pom.xml | 2 +- ql/pom.xml | 2 +- serde/pom.xml| 2 +- service-rpc/pom.xml | 2 +- service/pom.xml | 2 +- shims/0.23/pom.xml | 2 +- shims/aggregator/pom.xml | 2 +- shims/common/pom.xml | 2 +- shims/pom.xml| 2 +- standalone-metastore/metastore-server/pom.xml| 2 +- streaming/pom.xml| 2 +- testutils/pom.xml| 2 +- udf/pom.xml | 2 +- vector-code-gen/pom.xml | 2 +- 67 files changed, 68 insertions(+), 68 deletions(-) diff --git a/accumulo-handler/pom.xml b/accumulo-handler/pom.xml index b68d2d39597..4e141a81e95 100644 --- a/accumulo-handler/pom.xml +++ b/accumulo-handler/pom.xml @@ -17,7 +17,7 @@ org.apache.hive hive -4.0.0-SNAPSHOT +4.0.0 ../pom.xml hive-accumulo-handler diff --git a/beeline/pom.xml b/beeline/pom.xml index b785a20e2b5..87f1740436b 100644 ---
(hive) branch master updated (c1d592f4f98 -> 9a0ce4e1589)
This is an automated email from the ASF dual-hosted git repository. zabetak pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hive.git from c1d592f4f98 HIVE-28069: Iceberg: Implement Merge task functionality for Iceberg tables (#5076) (Sourabh Badhya reviewed by Denys Kuzmenko, Krisztian Kasa) add 9a0ce4e1589 HIVE-28140: Remove unnecessary shading of test jars in Kafka module to speedup build (Stamatis Zampetakis reviewed by Butao Zhang) No new revisions were added by this update. Summary of changes: kafka-handler/pom.xml | 1 - 1 file changed, 1 deletion(-)
(hive) branch master updated: HIVE-28069: Iceberg: Implement Merge task functionality for Iceberg tables (#5076) (Sourabh Badhya reviewed by Denys Kuzmenko, Krisztian Kasa)
This is an automated email from the ASF dual-hosted git repository.
sbadhya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new c1d592f4f98 HIVE-28069: Iceberg: Implement Merge task functionality
for Iceberg tables (#5076) (Sourabh Badhya reviewed by Denys Kuzmenko,
Krisztian Kasa)
c1d592f4f98 is described below
commit c1d592f4f98e49614ffaed57813983b579b9147f
Author: Sourabh Badhya
AuthorDate: Thu Mar 21 15:48:05 2024 +0530
HIVE-28069: Iceberg: Implement Merge task functionality for Iceberg tables
(#5076) (Sourabh Badhya reviewed by Denys Kuzmenko, Krisztian Kasa)
---
.../iceberg/mr/hive/HiveIcebergInputFormat.java| 2 +-
.../mr/hive/HiveIcebergOutputCommitter.java| 66 +++
.../iceberg/mr/hive/HiveIcebergStorageHandler.java | 27 ++
.../mr/hive/IcebergMergeTaskProperties.java| 55 +++
.../test/queries/positive/iceberg_merge_files.q| 95 +
.../positive/llap/iceberg_merge_files.q.out| 458 +
.../test/resources/testconfiguration.properties| 2 +
.../hadoop/hive/ql/io/CombineHiveInputFormat.java | 13 +-
.../hive/ql/metadata/HiveStorageHandler.java | 17 +
.../hadoop/hive/ql/optimizer/GenMapRedUtils.java | 41 ++
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 13 +-
.../ql/plan/ConditionalResolverMergeFiles.java | 132 --
.../hadoop/hive/ql/plan/MergeTaskProperties.java | 30 ++
13 files changed, 921 insertions(+), 30 deletions(-)
diff --git
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java
index dd329c122aa..1ea78eeba54 100644
---
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java
+++
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java
@@ -191,7 +191,7 @@ public class HiveIcebergInputFormat extends
MapredIcebergInputFormat
@Override
public boolean shouldSkipCombine(Path path, Configuration conf) {
-return true;
+return false;
}
@Override
diff --git
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
index b4d5ce98f59..d9f3116ff84 100644
---
a/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
+++
b/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
@@ -45,7 +45,10 @@ import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
import org.apache.hadoop.hive.ql.Context.Operation;
import org.apache.hadoop.hive.ql.Context.RewritePolicy;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.plan.MapWork;
import org.apache.hadoop.hive.ql.session.SessionStateUtil;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.JobContext;
@@ -167,6 +170,8 @@ public class HiveIcebergOutputCommitter extends
OutputCommitter {
LOG.info("CommitTask found no serialized table in config for
table: {}.", output);
}
}, IOException.class);
+
+ cleanMergeTaskInputFiles(jobConf, tableExecutor, context);
} finally {
if (tableExecutor != null) {
tableExecutor.shutdown();
@@ -741,4 +746,65 @@ public class HiveIcebergOutputCommitter extends
OutputCommitter {
throw new NotFoundException("Can not read or parse committed file: %s",
fileForCommitLocation);
}
}
+
+ public List getOutputFiles(List jobContexts) throws
IOException {
+List outputs = collectOutputs(jobContexts);
+ExecutorService fileExecutor =
fileExecutor(jobContexts.get(0).getJobConf());
+ExecutorService tableExecutor =
tableExecutor(jobContexts.get(0).getJobConf(), outputs.size());
+Collection dataFiles = new ConcurrentLinkedQueue<>();
+try {
+ Tasks.foreach(outputs.stream().flatMap(kv -> kv.jobContexts.stream()
+ .map(jobContext -> new SimpleImmutableEntry<>(kv.table,
jobContext
+ .suppressFailureWhenFinished()
+ .executeWith(tableExecutor)
+ .onFailure((output, exc) -> LOG.warn("Failed to retrieve merge
input file for the table {}", output, exc))
+ .run(output -> {
+JobContext jobContext = output.getValue();
+JobConf jobConf = jobContext.getJobConf();
+LOG.info("Cleaning job for jobID: {}, table: {}",
jobContext.getJobID(), output);
+
+Table table = output.getKey();
+FileSystem fileSystem =
(hive) branch master updated: HIVE-28111: FSCountersSummary is only printed when LLAP IO is enabled (#5133) (Laszlo Bodor reviewed by Ayush Saxena)
This is an automated email from the ASF dual-hosted git repository.
abstractdog pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hive.git
The following commit(s) were added to refs/heads/master by this push:
new 5c05cc76056 HIVE-28111: FSCountersSummary is only printed when LLAP IO
is enabled (#5133) (Laszlo Bodor reviewed by Ayush Saxena)
5c05cc76056 is described below
commit 5c05cc760564e8f2c3ded576d93894bf4612956a
Author: Bodor Laszlo
AuthorDate: Thu Mar 21 10:06:41 2024 +0100
HIVE-28111: FSCountersSummary is only printed when LLAP IO is enabled
(#5133) (Laszlo Bodor reviewed by Ayush Saxena)
---
.../org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
index fbbbf189e48..e5c4f2e09c5 100644
---
a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
+++
b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java
@@ -445,8 +445,8 @@ public class TezJobMonitor {
//llap IO summary
if (HiveConf.getBoolVar(hiveConf, HiveConf.ConfVars.LLAP_IO_ENABLED,
false)) {
new LLAPioSummary(progressMap, dagClient).print(console);
-new FSCountersSummary(progressMap, dagClient).print(console);
}
+ new FSCountersSummary(progressMap, dagClient).print(console);
String wmQueue = HiveConf.getVar(hiveConf,
ConfVars.HIVE_SERVER2_TEZ_INTERACTIVE_QUEUE);
if (wmQueue != null && !wmQueue.isEmpty()) {
new LlapWmSummary(progressMap, dagClient).print(console);
